Rust detection method, system and storage medium based on edge computing

Abstract

The present application provides a method, system and storage medium for rust detection based on edge computing, which relates to the field of computer vision technology. The method includes: using an enhanced shuffle network to perform lightweight improvements on the backbone network of the target detection model, using a bolt data set to train the lightweight target detection model, and then obtaining a lightweight model that adapts to the edge computing hardware platform through format conversion and quantization; using a number of cores that matches the hardware resource specifications in the edge computing hardware platform, and performing collaborative reasoning on the lightweight model based on real-time acquired bolt images to quickly obtain real-time rust detection results. By performing targeted optimization on the architecture and deployment method of the rust detection model, the speed and performance of the model deployed in the edge computing device can be effectively improved, and real-time robust bolt rust detection can be achieved based on edge computing.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a corrosion detection method, system, and storage medium based on edge computing. Background Art

[0002] The corrosion of bolts on large equipment on construction sites seriously affects site safety. To promptly replace corroded bolts, specialized personnel are required for inspections, which is not only costly but also inefficient. Deep learning technology has emerged as a powerful solution to improve the accuracy and efficiency of corrosion detection. Solutions applying deep learning are primarily categorized as single-stage and two-stage object detection algorithms. Single-stage algorithms offer faster computational speeds but suffer a slight loss in accuracy. Typical examples include Single Shot MultiBox Detector (SSD) and You Only Look Once (YOLO). Two-stage object detection algorithms, such as Fast Region-CNN (Fast R-CNN) and CornerNet, offer more comprehensive feature extraction but are relatively slow.

[0003] At the same time, the rise of edge computing has provided new opportunities for edge corrosion detection. Edge computing can process and compute data closer to the source of data generation, reducing data transmission and latency. Currently, the deployment of the YOLO series of algorithms on edge nodes is a hot topic of research. Some scholars have deployed the YOLOV5 algorithm for target detection on edge computing chips from manufacturers such as Rockchip, Huawei HiSilicon, and NVIDIA. Others have improved the backbone networks of algorithms such as YOLOV5, such as using the MobileNetV1 network to replace the backbone of the original YOLOV5 network. This can effectively increase the speed of edge computing, but the overall algorithm accuracy will be reduced. Furthermore, some scholars have combined the backbone network of YOLOV7 with the weighted bidirectional feature pyramid structure (BiFPN) to address the problem of information loss during sampling.

[0004] Most of the research on these object detection algorithms remains limited to theoretical simulation. Furthermore, even after improvements to the existing models for rust detection applications in edge computing, they still suffer from issues such as large size and high hardware resource consumption. Due to limited computing resources in edge computing devices, performance and speed cannot be guaranteed when deployed on these devices, making them difficult to implement in practice.

[0005] Therefore, how to optimize the corrosion detection model, improve the speed and performance of the model deployed in edge computing devices, and achieve real-time and robust corrosion detection is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a corrosion detection method, system and storage medium based on edge computing, which optimizes the architecture and deployment method of the corrosion detection model, effectively improves the speed and performance of the model deployment in edge computing devices, and realizes real-time and robust bolt corrosion detection based on edge computing.

[0007] To achieve the above objectives, in a first aspect, the present application provides a corrosion detection method based on edge computing, the method comprising:

[0008] S1. Construct a bolt dataset based on multiple training bolt images;

[0009] S2. Adding an enhanced shuffle network to the backbone network of the target detection model to obtain a lightweight target detection model. The enhanced shuffle network includes: a phantom module for simplifying the convolution process and a compression-excitation module based on the attention mechanism;

[0010] S3. Based on the bolt dataset, training the lightweight object detection model to obtain a training model that meets a preset training goal;

[0011] S4. Convert and quantize the weight parameters of the training model to obtain a lightweight model that matches the format of the edge computing hardware platform;

[0012] S5. Transmit the lightweight model to the edge computing hardware platform, use a core number that matches the hardware resource specifications of the edge computing hardware platform, perform collaborative reasoning on the lightweight model based on the bolt image acquired in real time, and output the corrosion detection result of the bolt image, wherein the corrosion detection result includes: the location of the corroded bolt and the corrosion probability.

[0013] In one possible implementation, the target detection model is a YOLOv7-tiny model for edge computing, and step S2 includes:

[0014] S21. Build a YOLOv7-tiny network training project based on the PyTorch training framework;

[0015] S22. Based on the YOLOv7-tiny network training project, the phantom module and the compressed excitation module are added to the backbone network of the YOLO v7-tiny model. A network structure whose performance meets the preset performance target is searched through neural network search technology, and the lightweight target detection model is generated based on the network structure.

[0016] In one possible implementation, step S4 includes:

[0017] S41, format conversion: converting the weight parameters of the training network model into a weight file in an open neural network exchange format;

[0018] S42, quantization: calling the configuration interface to obtain the preprocessing parameters of the neural network software development kit that matches the edge computing hardware platform, creating an RKNN object to initialize the neural network software development kit;

[0019] Importing the weight file into the neural network model development tool, calling the creation interface, and creating an initial RKNN model based on the RKNN object;

[0020] Calling the application program interface of the neural network model development tool to evaluate the memory usage of the initial RKNN model and perform simulated reasoning until the initial RKNN model achieves the preset expected effect;

[0021] The initial RKNN model that achieves the expected effect is quantized through the 8-bit asymmetric quantization function of the neural network software development kit, and the lightweight model in RNKK format is exported.

[0022] In one possible implementation, step S5 includes:

[0023] Acquire a real-time video stream to be detected, wherein the real-time video stream includes multiple frames of the bolt image;

[0024] Normalizing, 8-bit asymmetric quantizing, and format converting the real-time video stream in the original video format to obtain a video stream in a target format, where the target format matches the hardware resource specifications of the edge computing platform;

[0025] Inputting the target format video stream into the edge computing platform, using a core number that matches the hardware resource specifications of the edge computing hardware platform, performing corrosion detection based on the real-time acquired bolt image, and performing collaborative reasoning on the lightweight model to obtain the corrosion detection result;

[0026] The rust detection result is superimposed on the corresponding bolt image in the video stream, the superimposed video stream is converted into the original video format, and output to an image display device.

[0027] In one possible implementation, step S1 includes:

[0028] S11. Extract multiple training bolt images from the video captured by the UAV in the target site;

[0029] S12. Enhance the training bolt image by one or more of the following methods:

[0030] Increasing the number of bolts in the training bolt image by copying and pasting the bolts multiple times;

[0031] performing one or more of random rotation, scaling, cropping, brightness adjustment, noise addition, and solar flare addition on the training bolt image;

[0032] S13. Use an image annotation tool to annotate the enhanced training bolt image, convert the training bolt image into a YOLO dataset format containing target categories and target position coordinates, and obtain the bolt dataset; divide the bolt dataset into a training set and a test set for neural network training and testing according to a preset ratio.

[0033] In one possible implementation, the training bolt images are acquired at the target site in multiple urban areas under various weather conditions.

[0034] In a second aspect, a corrosion detection system based on edge computing is provided, comprising: an image acquisition device, an edge computing hardware platform, and an image output device;

[0035] The image acquisition device is used to capture a real-time video stream containing bolt images and transmit the real-time video stream to the edge computing hardware platform;

[0036] The edge computing hardware platform is configured to perform corrosion detection on the bolt image contained in the real-time video stream based on the lightweight model described in any one of the embodiments of the first aspect, and output a corrosion detection result, wherein the corrosion detection result includes: the location of the corroded bolt and the corrosion probability;

[0037] The image output device is used to superimpose the corrosion detection result onto the real-time video stream and output the real-time detection video stream, wherein the real-time detection video stream includes the position of the corroded bolt and the corrosion probability.

[0038] In one possible implementation, the edge computing hardware platform implements multi-threaded model reasoning based on a multi-core neural network processing unit;

[0039] The image acquisition device and the image output device use a central processing unit and / or an image processing unit to perform corresponding image processing.

[0040] In a third aspect, a computing device is provided, which includes a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the edge computing-based corrosion detection method provided in the first aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is executed by a processor to implement the edge computing-based corrosion detection method provided in the first aspect.

[0042] The technical solution provided by this application includes at least the following technical effects:

[0043] The enhanced shuffle network is used to lightweight the backbone network of the target detection model. The deployed model is then collaboratively reasoned with a number of cores that match the hardware resource specifications of the edge computing hardware platform, quickly obtaining real-time detection results. Therefore, the technical solution of this application effectively improves the speed and performance of model deployment in edge computing devices, and realizes real-time and robust bolt corrosion detection based on edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a corrosion detection method based on edge computing provided by the present application according to an exemplary embodiment;

[0045] Figure 2 This is a structural diagram of a lightweight YOLOv7-tiny model provided by the present application according to an exemplary embodiment;

[0046] Figure 3 1 is a schematic diagram of a network module in a lightweight YOLOv7-tiny model provided by the present application according to an exemplary embodiment;

[0047] Figure 4 This is a corrosion detection result diagram obtained using YOLOv7-tiny-PicoDet according to an exemplary embodiment of the present application;

[0048] Figure 5 This is a structural diagram of a multi-threaded edge computing corrosion detection system provided by the present application according to an exemplary embodiment;

[0049] Figure 6 is a schematic diagram of a corrosion detection method based on edge computing provided by the present application according to an exemplary embodiment;

[0050] Figure 7 It is a hardware structure diagram of a computing device provided by the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] To further illustrate various embodiments, this application includes accompanying drawings. These drawings form part of the disclosure and are primarily used to illustrate the embodiments and, in conjunction with the relevant description in the specification, to explain the operating principles of the embodiments. By referring to these drawings, those skilled in the art will understand other possible implementations and the advantages of this application. The components in the figures are not drawn to scale, and similar reference numerals are generally used to represent similar components.

[0052] The present application will now be further described with reference to the accompanying drawings and specific implementation methods.

[0053] Example 1

[0054] The edge computing-based corrosion detection method provided in this application includes several key processes: dataset preparation, model building, and model deployment. The PC device is responsible for lightweight model building and model conversion. Once the lightweight model for corrosion detection is obtained, it is ported to the edge computing hardware platform for deployment, which then performs real-time corrosion detection. Figure 1 A flow chart of a corrosion detection method based on edge computing provided in this application is shown below. Figure 1 This paper introduces the corrosion detection method based on edge computing provided by this application. Figure 1 The method includes the following steps S1 to S6.

[0055] S1. The PC device builds a bolt dataset based on multiple training bolt images.

[0056] In the embodiment of the present application, firstly, a data set is collected and processed to construct a bolt data set for training the model. This step S1 includes the following steps S11 to S13.

[0057] S11. Extract multiple training bolt images from the video captured by the UAV in the target site.

[0058] In some embodiments, training bolt images are collected from target sites in multiple urban areas under various weather conditions to enhance the environmental background richness of the bolt images. For example, a drone inspects large equipment on a construction site in different urban areas and in various weather conditions, captures video, and captures multiple training bolt images from the video. Optionally, a target number of images, for example, 1,000, are selected from the captured training bolt images. Optionally, the images are in RGB format. S12: Enhance the training bolt images using one or more of the following methods.

[0059] Method 1: Oversampling the training bolt image. For example, the number of bolts in the training bolt image is increased by copying and pasting the bolts multiple times.

[0060] Method 2: Perform one or more of the following random rotations, scaling, cropping, brightness adjustment, noise addition, and solar flare addition on the training bolt images.

[0061] Because bolts are numerous and small in construction site equipment, they are challenging to detect. Therefore, we oversampled images containing bolts by copying and pasting them multiple times to increase the number of bolts in each image and improve the matching rate of corresponding anchors. Furthermore, we augmented the dataset with a series of operations, including random rotation, scaling, cropping, brightness adjustment, noise addition, and solar flare, to increase data diversity.

[0062] In some embodiments, through the above enhancement process, a larger number of training bolt images can be obtained. For example, 1,000 images are selected during collection and 8,000 images are obtained through enhancement.

[0063] S13. Use an image annotation tool to annotate the enhanced training bolt image, convert the training bolt image into a YOLO dataset format containing target categories and target position coordinates, and obtain a bolt dataset; divide the bolt dataset into a training set and a test set for neural network training and testing according to a preset ratio.

[0064] In some embodiments, the enhanced training bolt images are manually labeled using the LabelImg tool, and the image data is converted into a YOLO dataset format containing object categories and target location coordinates, which serves as a bolt dataset for neural network training. Optionally, the bolt dataset is divided into a training set and a test set according to a preset ratio (e.g., 8:2).

[0065] S2. The PC device adds an enhanced shuffle network (ESNet) to the backbone network of the target detection model to obtain a lightweight target detection model. The enhanced shuffle network includes: a ghost module (Ghost Block) for simplifying the convolution process and a squeeze and excitation block (SE Block) based on the attention mechanism.

[0066] The target detection model used in the embodiments of this application is the YOLOv7-tiny model, and the generated lightweight target detection model is also a lightweight YOLOv7-tiny model. YOLOv7-tiny is a basic model optimized for edge image processing units. The suffix "tiny" in the computer vision model means that it is optimized for edge artificial intelligence and deep learning workloads, and is more lightweight and can run on mobile computing devices or distributed edge servers and devices.

[0067] In the embodiments of the present application, considering the limited hardware resources of the device and the fact that the YOLO v7-tiny model has room for optimization, the YOLO v7-tiny model is improved by combining the lightweight characteristics of the lightweight object detection algorithm PicoDet to reduce the number of model parameters and the amount of computation, making it suitable for resource-constrained environments. Optionally, the improved lightweight YOLO v7-tiny model can also be referred to as YOLOv7-tiny-PicoDet.

[0068] In some embodiments, this step S2 includes the following steps S21 and S22.

[0069] S21. Build a YOLOv7-tiny network training project based on the PyTorch training framework.

[0070] S22. Based on the YOLOv7-tiny network training project, Ghost Block and SE Block are added to the backbone network of the YOLO v7-tiny model. A network structure whose performance meets the preset performance target is searched through neural network search technology, and a lightweight YOLO v7-tiny model (lightweight target detection model) is generated based on the network structure.

[0071] Figure 2 This is a structural diagram of a lightweight YOLOv7-tiny model provided by this application according to an exemplary embodiment. Figure 3 This is a schematic diagram of a network module in a lightweight YOLOv7-tiny model provided by this application according to an exemplary embodiment. Figure 2 and Figure 3 This article introduces the lightweight YOLOv7-tiny model provided by this application.

[0072] refer to Figure 2 The lightweight YOLOv7-tiny model (YOLOv7-tiny-PicoDet) provided in this application includes: input (Input), backbone network (Backbone) and feature fusion prediction network (Neck+Head).

[0073] The input is the input of the training bolt images (bolt dataset). Adaptive anchor box calculation, adaptive image scaling, and data augmentation are used to preprocess the input images before they are fed into the Backbone layer. This enriches the diversity of the input images, significantly improves the detection of small objects, and enhances the network model's ability to detect bolts.

[0074] Backbone introduces ESNet, a lightweight network that is mobile-friendly and improved based on the ShuffleNetV2 network. ESNNet is composed of two basic modules, ESBlock1 and ESBlock2. ESBlock1 contains SE Block and Ghost Block from GhostNet. ESBlock2 contains SE Block. The embodiment of this application uses a one-shot neural network search technology (NAS) to automatically find the optimal architecture for target detection.

[0075] See also Figure 3 , CBL unit is composed of convolution layer (Conv) – bottleneck layer (BN) – activation layer (LeakyReLU). YOLOv7-tiny uses LeakyReLU as the activation function, while other models use SiLU as the activation function. UP refers to upsampling, and MP refers to max pooling.

[0076] Ghost Block consists of a depthwise separable convolution (pw conv, dw conv) unit and a connection unit (Contact). SE Block consists of a global pooling unit, a fully connected unit (FC), an activation function (ReLU, Sigmoid), and a connection unit (Contact). ESBlock1 includes a channel split unit, GhostBlock, SE Block, a depthwise separable convolution (pw conv, dw conv) unit, a connection unit (Contact), and a channel shuffle unit. ESBlock2 includes a depthwise separable convolution (pw conv, dw conv) unit, SEBlock, and a connection unit (Contact). Among them, pw conv refers to pointwise convolution, and dwconv refers to depthwise convolution.

[0077] See also Figure 2The Neck+Head layer is used to enhance the network's feature extraction capabilities and further improve the network's ability to detect bolt images. Specifically, it integrates the features provided by the Backbone layer to generate bounding boxes and predict categories. The Neck+Head layer mainly includes a Spatial Pyramid Pooling Cross Stage Partial (SPPCSP) layer, an Efficient Layer Aggregation Network (ELAN) layer, several Convolutional (Conv) layers, and a Maximum Pooling (MPConv) layer. Among them, the Head layer outputs three feature maps of different sizes at the 75th, 88th, and 101st layers, and outputs three prediction results of different sizes after the Reparameterized Structure (REP) layer, which facilitates the subsequent deployment of the algorithm. See [1]. Figure 3 , SPPCSP includes: CBL unit, maximum pooling (MP) unit and connection unit (Contact); ELAN includes: CBL unit and connection unit (Contact).

[0078] S3 and PC devices train a lightweight target detection model based on bolt data to obtain a training model that meets the preset training objectives.

[0079] In this embodiment of the present application, the bolt dataset from step S1 is fed into the lightweight YOLOv7-tiny model (YOLOv7-tiny-PicoDet) obtained in step S2 for training to obtain a training model. Specifically, the lightweight YOLOv7-tiny model is used for training, the number of training rounds is set to 300, the input image size is 640*640, and the model that meets the preset training target during the training process is saved, where the preset training target can be the number of training rounds reaching the set number of training rounds, or the recognition error of the model reaching a preset threshold.

[0080] Through steps S1 to S3, we complete dataset preparation, model lightweighting, and model training, resulting in a lightweight model. Next, through steps S4 and S5, we will describe how to convert the lightweight model format and quickly port it to edge computing hardware.

[0081] S4. The PC device converts and quantizes the weight parameters of the training model to obtain a lightweight model that matches the format of the edge computing hardware platform.

[0082] This application example uses Rockchip's RK3588S platform with a built-in NPU (neural network processing unit) as the edge computing hardware platform for deploying the model. The RK3588S supports AI acceleration and has a higher-performance CPU architecture and stronger GPU performance.

[0083] Among them, the neural network model development tool is RKNN-Toolkit2 provided by Rockchip, which is used to achieve rapid transplantation and application development of trained models.

[0084] In the embodiment of the present application, step S4 includes:

[0085] S41. Format conversion: Convert the weight parameters of the training model into a weight file in the Open Neural Network Exchange (ONNX) format.

[0086] In some embodiments, before exporting the model in ONNX format, the exported operator version is pre-set. ONNX supports multiple operator versions, each version corresponding to a different set of operations. Specifically, the exported operator version is pre-set according to the operator version supported by the edge computing hardware platform to be deployed. For example, in the embodiment of the present application, the highest operator version supported by the RK3588S platform is 12, so the operation set version (opset_version) variable is set to 12.

[0087] S42. Quantization: Based on the quantization function of the neural network model development tool (Software Development Kit, SDK), the weight file is converted into a lightweight model in the RKNN (Rockchip Neural Network) format.

[0088] Specifically, first call the configuration interface to obtain the preprocessing parameters of the neural network model development tool (RKNN-Toolkit2) that matches the edge computing hardware platform, create an RKNN object to initialize the software development environment of RKNN-Toolkit2; import the weight file into RKNN-Toolkit2, call the build interface, and create an initial RKNN model based on the RKNN object; call the application program interface (API) of RKNN-Toolkit2 to evaluate the memory usage of the initial RKNN model, and perform simulated inference until the initial RKNN model achieves the preset expected effect; by turning on the 8-bit asymmetric quantization function of the neural network software development kit, the initial RKNN model that achieves the expected effect is quantized, and a lightweight model in RNKK format is exported.

[0089] In this embodiment of the present application, the PC device exports the weight parameters of the trained lightweight model and uses RKNN Toolkit2 to quantize the model and convert the format. The model is quantized using an asymmetric 8-bit quantization method. After the model is quantized, the model weight information is saved with lower precision. After deployment, the model's memory space usage can be reduced and the model reasoning speed can be accelerated. Finally, the model is converted to RKNN format for rapid porting to the edge computing hardware platform.

[0090] In some embodiments, the edge computing hardware platform needs to be configured with software and hardware resources related to the model application and the PC. For example, the Debian system and a tailored ARM-Linux kernel are pre-ported to the RK3588S platform, and the firmware version supported by the NPU, along with the development environment and library files required for the deep learning network, are installed. For application software, a cross-compilation environment must be established with the PC, Opencv and Cmake-gui must be installed, and model files must be transferred using MobaXterm.

[0091] S5. The PC device transmits the lightweight model to the edge computing hardware platform. The edge computing hardware platform uses a core number that matches the hardware resource specifications of the edge computing hardware platform. Based on the real-time bolt image, it performs collaborative inference on the lightweight model and outputs the corrosion detection results of the bolt image. The corrosion detection results include the location of the corroded bolts and the corrosion probability.

[0092] For example, a lightweight model in RKNN format received from a PC device is loaded and run on the RK3588S platform, and the model is inferred by calling the RKNN API interface. Specifically, based on the hardware resource specifications of the RK3588S itself, the model reasoning is optimized, and a multi-threaded collaborative computing solution is enabled according to the number of cores that match the hardware resource specifications. This can improve the performance and computing speed of the model in the hardware platform, enabling subsequent edge computing hardware platforms to perform rust identification in real time at a higher frame rate.

[0093] Specifically, taking the edge computing hardware platform including a multi-core NPU as an example, a multi-core NPU can implement parallel computing based on multiple threads. Based on the hardware resource specifications, the channel width used by the NPU hardware DMA to transfer image data can be determined. Based on the channel width, the image data processed by the model is split into multiple parts. For each part, data is transferred according to the channel width. Then, the NPU can perform model inference on the image data in parallel based on multiple threads.

[0094] In some embodiments, this step S5 includes:

[0095] 1. Obtain a real-time video stream to be inspected, which includes multiple frames of bolt images. Specifically, the real-time video stream can be obtained from an image acquisition device.

[0096] 2. Standardize, 8-bit asymmetric quantize, and convert the real-time video stream in the original video format to obtain the video stream in the target format, which matches the hardware resource specifications of the edge computing platform.

[0097] In some embodiments, a third-party video image processing program can be used to perform image pre-processing. This pre-processing includes the aforementioned standardization of the input video stream, asymmetric 8-bit quantization, and format conversion. Specifically, NHWC represents the dimensional order of the image, where: N represents the batch size (the number of images); H represents the image height; W represents the image width; and C represents the number of image channels. The data formats before and after pre-processing are: NHWC and NC1HWC2, respectively.

[0098] C1 and C2 divide the image channel into two parts (or more or fewer parts depending on different hardware resource specifications). The image data corresponding to the two parts can be calculated in parallel in the NPU (edge ​​computing hardware platform). Based on this, the efficiency and performance of model inference of the edge computing hardware platform can be improved.

[0099] 3. Input the target format video stream into the edge computing platform, use the number of cores that matches the hardware resource specifications of the edge computing hardware platform, perform rust detection based on the real-time bolt images, and perform collaborative reasoning on the lightweight model to obtain the rust detection results. The rust detection results include: the location of the rusted bolts and the rust probability.

[0100] 4. Superimpose the rust detection results on the corresponding bolt image in the video stream, convert the superimposed video stream into the original video format, and output it to the image display device.

[0101] Among them, the above steps 2 and 4 can be implemented using a CPU or GPU, and step 3 can be implemented using a multi-core NPU.

[0102] Figure 4 This is a corrosion detection result diagram obtained by using YOLOv7-tiny-PicoDet according to an exemplary embodiment of the present application, see Figure 4 , the position of the rusted bolts is marked with a rectangular box in the figure, and the rust value is used above the rectangular box to indicate the probability of rusting of the bolts, for example, Figure 4 The probability of rusting of the bolt marked in the upper right sub-graph is 0.91.

[0103] This application provides a corrosion detection method based on edge computing. It uses an enhanced shuffle network to lightweight the backbone network of the YOLOv7-tiny model. In the edge computing hardware platform, a number of cores matching the hardware resource specifications of the edge computing hardware platform are used to perform collaborative inference on the deployed model, quickly obtaining a lightweight model. The edge computing hardware platform can use the lightweight model to efficiently perform real-time corrosion detection. Therefore, the technical solution of this application effectively improves the speed and performance of model deployment in edge computing devices.

[0104] Furthermore, the technical solution of this application uses multiple dataset expansion techniques to expand the bolt corrosion dataset, and then leverages the multi-core capabilities of the NPU in the edge computing hardware platform to accelerate model reasoning. In some embodiments, the lightweight model deployed in the edge computing hardware platform can achieve a real-time corrosion detection speed of 60 frames per second in real-time detection, meeting the requirements of real-time and robust bolt corrosion detection.

[0105] Example 2

[0106] Based on the above-mentioned edge computing-based rust detection method, this application provides an edge computing-based rust detection system, which includes an image acquisition device, an edge computing hardware platform and an image output device.

[0107] The image acquisition device is used to capture a real-time video stream containing bolt images and transmit the real-time video stream to the edge computing hardware platform. The real-time video stream can be a local video input or a construction site video captured in real time by an image acquisition device such as a camera. In some embodiments, the acquired real-time video stream is also pre-processed (see step 2 above) to facilitate input into the neural network model for calculation.

[0108] The edge computing hardware platform is used to perform corrosion detection on bolt images contained in a real-time video stream based on the lightweight model provided in Example 1, and output corrosion detection results, including the location of the corroded bolts and the corrosion probability. This model inference is performed by the NPU on the RK3588S platform, and the program design uses a multi-threaded approach to utilize the NPU's multi-core model to improve computing speed.

[0109] This image output device is used to superimpose the corrosion detection results onto a real-time video stream, outputting a real-time detection video stream that includes the location of the corroded bolts and the corrosion probability. The output real-time detection video stream can be displayed on an image display device, such as a PC screen, to show the location of the corroded bolts and the detected corrosion probability in real time.

[0110] In some embodiments, the edge computing hardware platform implements multi-threaded model reasoning based on a multi-core neural network processing unit; the image acquisition device and the image output device use a central processing unit and / or an image processing unit to perform corresponding image processing.

[0111] The system provided in this application utilizes a trained lightweight model, is deployed on the RK3588S platform, and optimizes the algorithm operation scheme based on hardware resources to improve computing speed. The processing flow of the system includes image data pre-processing, model inference, and output post-processing.

[0112] Figure 5 This is a structural diagram of a multi-threaded edge computing rust detection system provided by this application according to an exemplary embodiment. During the rust identification process, the image data (real-time video stream) is first pre-processed by an image acquisition device integrated with a CPU / GPU. The pre-processing includes standardization, quantization, and format conversion of the image data. Then, an edge computing hardware platform integrated with a multi-core NPU (for example, including core0-2) performs multi-threaded model inference. During the inference process, RKNNAPI can be called for model inference. Finally, the rust identification results and image data are post-processed (including format conversion and dequantization) by an image output device integrated with a CPU / GPU, and sent to an image display device for display. For the introduction of format conversion, refer to step 2 above.

[0113] In combination with the above-mentioned embodiment 1 and embodiment 2, the present application provides a schematic diagram of a corrosion detection method based on edge computing according to an exemplary embodiment. Figure 6 First, corrosion data (bolt images) is collected using an image acquisition device such as a camera. Dataset enhancement is then performed on the collected data to complete dataset preparation. During the model building phase, the YOLOv7-tiny model is lightweighted. During the model deployment phase, the lightweight YOLOv7-tiny model is trained and ported to the RK3588S platform. Based on this, the image data collected by the camera undergoes pre-processing before being transmitted to the RK3588S platform for corrosion detection. The corrosion detection results undergo post-processing and are then transmitted to an image display device for real-time detection results.

[0114] In order to better illustrate the performance of the lightweight improved YOLOv7-tiny model in the technical solution of this application, this application also provides experimental analysis process and experimental data.

[0115] Specifically, the experimental analysis was conducted in Python under the Windows 11 operating system. The network model was trained using the Pytorch 1.13.1 framework, the CPU was an AMD R5 3600, and the GPU was an NVIDIA GeForce RTX 3060. Before model training, parameters such as the number of categories, category names, and training path were initialized. The image input size was set to 640×640, the maximum number of iterations was set to 300, the number of images per training cycle was set to 8, the momentum factor was set to 0.9, the weight decay coefficient was set to 0.0005, and the initial learning rate was set to 0.001.

[0116] Table 1 compares the parameter count and computational speed of different models. YOLOv5s, YOLOv7-tiny, and YOLOv8s are the original, unmodified models. YOLOv7-tiny-GhostNet replaces the YOLOv7-tiny backbone with GhostNet. YOLOv7-tiny-PicoDet is the improved model from this application. The fourth column shows the number of floating-point operations per second (FLOPs).

[0117] Table 1 Comparison of parameters and computing speed of different models

[0118]

[0119]

[0120] To verify the impact of the improvements on model performance, the training platform configuration remained unchanged. YOLOv7-tiny and its two improved models were trained on the same training set. The optimal weights for each network model were selected for comparative analysis on the test set. Table 2 shows the performance comparison of the different models. In Table 2, F1 is a metric that takes both precision and recall into account. A higher F1 indicates a higher overall performance of the model. mAP@0.5 represents the average precision of the model for object detection; a higher percentage indicates greater accuracy.

[0121] Table 2 Performance comparison of different models

[0122]

[0123] It should be noted that the model size in Table 2 refers to the number of bytes of storage space occupied by the model. As shown in Table 2, after replacing the backbone network ELAN with GhostNet, the model memory usage was reduced by 7.5% and the accuracy increased by 0.17%. The improved YOLOv7-tiny-PicoDet model in this application also reduced the memory usage by 25.8% and increased the accuracy by 2.9%, significantly reducing device resource consumption and improving model performance.

[0124] Figure 7 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. Figure 7 As shown, the computing device can be implemented as any of the above-mentioned image acquisition devices, image output devices, PC devices or edge computing hardware platforms.

[0125] The computing device includes a processor 701, a memory 702, a bus 703, and a computer program stored in the memory 702 and executable on the processor 701. The processor 701 includes one or more processing cores. The memory 702 is connected to the processor 701 via the bus 703. The memory 702 is used to store program instructions. When the processor executes the computer program, all or part of the steps in the above-mentioned method embodiments provided in this application are implemented.

[0126] Furthermore, as an executable solution, the computing device may be a computer unit, which may be a computing device such as a desktop computer, a laptop, a PDA, or a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the composition of the computer unit described above is merely an example of a computer unit and does not constitute a limitation of the computer unit. The computer unit may include more or fewer components than described above, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, a bus, etc., which are not limited in the present embodiment.

[0127] Furthermore, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit and connects various parts of the entire computer unit using various interfaces and lines.

[0128] The memory can be used to store the computer programs and / or modules, and the processor implements the various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0129] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method in the embodiment of the present application are implemented.

[0130] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0131] Although the present application has been specifically shown and described in conjunction with preferred embodiments, it should be understood by those skilled in the art that various changes in form and details may be made to the present application without departing from the spirit and scope of the present application as defined by the appended claims, and all such changes are within the scope of protection of the present application.

Claims

1. A corrosion detection method based on edge computing, characterized in that: The method comprises: S1. Construct a bolt dataset based on multiple training bolt images; S2. Adding an enhanced shuffle network to the backbone network of the target detection model to obtain a lightweight target detection model. The enhanced shuffle network includes: a phantom module for simplifying the convolution process and a compression-excitation module based on the attention mechanism; S3. Based on the bolt dataset, training the lightweight object detection model to obtain a training model that meets a preset training goal; S4. Convert and quantize the weight parameters of the training model to obtain a lightweight model that matches the format of the edge computing hardware platform; S5. Transmit the lightweight model to the edge computing hardware platform, perform collaborative reasoning on the lightweight model based on the bolt image acquired in real time, using a number of cores that matches the hardware resource specifications of the edge computing hardware platform, and output a corrosion detection result for the bolt image, the corrosion detection result including: the location of the corroded bolt and the corrosion probability; The step S4 comprises: S41, format conversion: converting the weight parameters of the training model into a weight file in an open neural network exchange format; S42, quantization: calling the configuration interface to obtain the preprocessing parameters of the neural network software development kit that matches the edge computing hardware platform, creating an RKNN object to initialize the neural network software development kit; Importing the weight file into the neural network model development tool, calling the creation interface, and creating an initial RKNN model based on the RKNN object; Calling the application program interface of the neural network model development tool to evaluate the memory usage of the initial RKNN model and perform simulated reasoning until the initial RKNN model achieves the preset expected effect; Quantize the initial RKNN model that achieves the expected effect through the 8-bit asymmetric quantization function of the neural network software development kit, and export the lightweight model in RNKK format; The step S5 comprises: Acquire a real-time video stream to be detected, wherein the real-time video stream includes multiple frames of the bolt image; Normalizing, 8-bit asymmetric quantizing, and format converting the real-time video stream in the original video format to obtain a video stream in a target format, where the target format matches the hardware resource specifications of the edge computing hardware platform; Inputting the target format video stream into the edge computing hardware platform, using a core number that matches the hardware resource specifications of the edge computing hardware platform, performing corrosion detection based on the real-time acquired bolt image, and performing collaborative reasoning on the lightweight model to obtain the corrosion detection result; The rust detection result is superimposed on the corresponding bolt image in the video stream, the superimposed video stream is converted into the original video format, and output to an image display device.

2. The corrosion detection method according to claim 1, characterized in that: The target detection model is a YOLOv7-tiny model for edge computing, and step S2 includes: S21. Build a YOLOv7-tiny network training project based on the PyTorch training framework; S22. Based on the YOLOv7-tiny network training project, the phantom module and the compressed excitation module are added to the backbone network of the YOLO v7-tiny model. A network structure whose performance meets the preset performance target is searched through neural network search technology, and the lightweight target detection model is generated based on the network structure.

3. The corrosion detection method according to claim 1, characterized in that: The step S1 comprises: S11. Extract multiple training bolt images from the video captured by the UAV in the target site; S12. Enhance the training bolt image by one or more of the following methods: Increasing the number of bolts in the training bolt image by copying and pasting the bolts multiple times; performing one or more of random rotation, scaling, cropping, brightness adjustment, noise addition, and solar flare addition on the training bolt image; S13. Use an image annotation tool to annotate the enhanced training bolt image, convert the training bolt image into a YOLO dataset format containing target categories and target position coordinates, and obtain the bolt dataset; divide the bolt dataset into a training set and a test set for neural network training and testing according to a preset ratio.

4. The method according to claim 3, characterized in that The training bolt images are collected from the target sites in multiple urban areas under various weather conditions.

5. A corrosion detection system based on edge computing, characterized in that: include: Image acquisition equipment, edge computing hardware platform, and image output equipment; The image acquisition device is used to capture a real-time video stream containing bolt images and transmit the real-time video stream to the edge computing hardware platform; The edge computing hardware platform is configured to perform corrosion detection on the bolt image contained in the real-time video stream based on the lightweight model in the edge computing-based corrosion detection method according to any one of claims 1 to 4, and output a corrosion detection result, wherein the corrosion detection result includes: the location of the corroded bolt and the corrosion probability; The image output device is used to superimpose the corrosion detection result onto the real-time video stream and output the real-time detection video stream, wherein the real-time detection video stream includes the position of the corroded bolt and the corrosion probability.

6. The corrosion detection system according to claim 5, characterized in that: The edge computing hardware platform implements multi-threaded model reasoning based on a multi-core neural network processing unit; The image acquisition device and the image output device use a central processing unit and / or an image processing unit to perform corresponding image processing.

7. A computer-readable storage medium, characterized in that At least one program is stored in the storage medium, and the at least one program is executed by the processor to implement the edge computing-based corrosion detection method as described in any one of claims 1 to 4.

Images