Edge end equipment crack detection method and device and storage medium

Abstract

The invention belongs to the technical field of visual identification. The invention discloses an edge end equipment crack detection method and device and a storage medium, and the method comprises the following steps: S1, collecting a crack detection data set, and dividing the data set into a training set and a test set; s2, a YOLOv3-Tiny model is constructed, and the model is trained and optimized; s3, processing the image to be detected; and S4, inputting the processed image data into the optimized YOLOv3-Tiny model so as to detect surface cracks of the outdoor engineering building, comparing an output result of the YOLOv3-Tiny model with a set threshold value, outputting a detection result greater than the threshold value, analyzing the detection result, constructing a visual target frame, and displaying the visual target frame. And marking the detected crack on the image in the form of a target frame. According to the method, the occupation of detection computing resources can be reduced, the computing time is shortened, and accurate and efficient identification and detection of cracks are realized.

Description

Technical Field

[0001] The present invention belongs to the field of visual recognition technology, and specifically relates to a method, device and storage medium for detecting cracks in edge devices. Background Art

[0002] At present, crack detection methods for engineering infrastructure such as bridges, roads, tunnels, dams, and towers are mainly divided into two categories: traditional methods and deep learning methods. Traditional methods mainly include methods combining integral channel features with random forests. Deep learning mainly includes semantic segmentation methods and deep full convolutional encoding and decoding network algorithms.

[0003] Traditional crack detection methods use classic template matching methods, and detection mainly relies on a series of traditional two-dimensional image processing technologies, which can be divided into several categories according to different principles, such as edge detection, threshold segmentation, texture analysis, wavelet analysis, and minimum path search. Although these traditional methods can achieve certain results when processing some simple scenes, they often have large computational complexity, low detection efficiency, are easily affected by lighting, and have poor detection generalization, making it difficult to cover the huge number of engineering infrastructure crack detection conditions today.

[0004] In order to pursue accuracy, the crack detection methods based on deep learning in the existing technology need to increase the weight ratio of enhanced machine vision and deep learning systems, which will increase memory and reasoning time, and these systems must use cloud servers to complete their detection tasks. For field environments with poor network quality or no network, it is difficult to adapt to their detection tasks. In order to solve the problem of crack detection system dependence on servers, the current conventional practice is to deploy its enhanced machine vision and deep learning algorithms on edge GPUs, which can solve the problem of server dependence, but GPUs are expensive, power-hungry, heavy, short battery life, inconvenient to carry, and difficult to adapt to crack detection in field environments. There are also many crack detection methods that are algorithmic improvements that can only be tested on the computer side and are not very practical. In addition, the convolutional neural network used in the existing crack detection methods is too deep, resulting in a large network model, large amount of calculation, and slow detection speed; and the model needs to target specific types of images when extracting features, and the generalization is low. There are still some challenges in data processing, mainly including difficulty in feature fusion and lack of global dependency capture capabilities.

[0005] After searching, a Chinese patent document with application number CN202410476364.0 discloses a road crack detection method based on adversarial and adaptive cross-consistency training, which includes the following steps: obtaining road crack image data; constructing a segmentation network model and a discriminator, wherein the segmentation network model includes a main segmentation network and an auxiliary segmentation network, and the main segmentation network includes a main encoder and a main decoder; using an adversarial training strategy to train the main segmentation network and the discriminator; adding the calculation weight of the high confidence prediction area at the output end of the discriminator, adaptively guiding the cross-consistency training, training the main segmentation network to learn the mapping relationship from the input road crack image data to the crack segmentation result, and saving the main segmentation network model parameters with the best performance on the validation set; in the test phase, inputting the road crack image to the trained main segmentation network to obtain pixel-level road crack detection results. However, the existing crack detection method belongs to the segmentation category, which requires huge computational complexity and is almost impossible to deploy on terminal devices. Although this method can segment the shape of the crack, it requires people or other methods to tell it that there is a crack somewhere, and then use this method to segment that place and obtain the shape of the crack. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems raised in the above-mentioned background technology, and provides a method, device and storage medium for edge device crack detection. By optimizing the deep learning model, the model is made smaller and the reasoning steps are reduced. The deep learning model is deployed in the terminal device, and crack detection tasks can be performed outdoors. It can also reduce the detection computing resource usage and shorten the calculation time, thereby realizing accurate and efficient identification and detection of cracks.

[0007] In a first aspect, the present invention provides a method for detecting cracks in an edge device, comprising the following steps:

[0008] S1. Collect crack detection data set and divide the data set into training set and test set;

[0009] S2. Build a YOLOv3-Tiny model, train and optimize the model, use the training set to train the YOLOv3-Tiny model, use the test set to test the model performance, and finally obtain the weight information and quantization information of the optimal model;

[0010] S3. Process the image to be detected, and the processing method includes: first, preprocessing the image to be detected to obtain preprocessed image data; second, extracting image features of the preprocessed image data by convolution calculation; then using the INT8 quantization algorithm to quantize the image features; then performing data nonlinear transformation on the quantized feature data, and using the LeakyReLU function to activate; then compressing the feature data after nonlinear transformation based on the maximum pooling method; finally, using the upsampling method to perform multi-layer feature fusion operations on feature maps of different levels to obtain fused feature data;

[0011] S4. Further feature extraction is performed on the fused feature data to finally obtain the output result, compare the output result with the set threshold, output the detection result greater than the threshold, and parse the detection result to obtain the target frame information containing the crack coordinate information and size information; convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks on the image in the form of a target frame.

[0012] Furthermore, in S2, training and optimizing the model includes the following steps:

[0013] S2.1. Build the YOLOv3-Tiny model and create a crack detection dataset based on the crack shape and the characteristics of the deployed equipment.

[0014] S2.2, put the training set into the YOLOv3-Tiny model for training;

[0015] S2.3. Save the optimal model during model training;

[0016] S2.4. After training, load the saved optimal model and merge the convolutional layer with the batch normalization layer;

[0017] S2.5. Perform INT8 quantization on the optimized model weights to reduce the amount of floating-point calculations and the size of the model weights;

[0018] S2.6, put the test set into the quantized model for adaptive training, and save the quantized model after the training is completed;

[0019] S2.7, load the quantized model and obtain its weight information and quantization information;

[0020] Furthermore, in step S3, extracting image features of the preprocessed image data by convolution calculation includes the following steps:

[0021] S3.1.1, construct a multi-channel convolution calculation matrix;

[0022] S3.1.2, design a DSP multiplexing method, first shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculation data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations;

[0023] S3.1.3, sort out the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = Wix i +B i , where Z is the result of convolution calculation, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term;

[0024] S3.1.4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

[0025] Furthermore, in step S3, the specific steps of quantization processing are as follows:

[0026] S3.2.1. Arrange the quantitative formula of dot product calculation and get:

[0027] Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number;

[0028] S3.2.2. Optimize the quantization algorithm. The optimized formula is:

[0029] in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and It is the result of convolution calculation;

[0030] S3.2.3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

[0031] Furthermore, in S3, the feature data after nonlinear transformation is compressed based on the maximum pooling method, including a method for completing data comparison of a 2X2 matrix through time series characteristics, and the specific steps are: the first step is to construct a 2X2 data matrix, compare the two adjacent columns of data in the first row, and save the maximum value to the memory; the second step is to compare the two adjacent columns of data in the second row and retain the maximum value; the third step is to read the memory data in the first step and compare it with the maximum value retained in the second step, and the final maximum value obtained is the maximum pooling result.

[0032] Furthermore, the method of multi-layer feature fusion operation is: first, read the data in the storage area of ​​the i-th layer, and expand the size of the data of this layer by upsampling to meet the size requirement of fusion with the i-th layer data, then read the data in the storage area of ​​the n-th layer, and finally complete the fusion operation of the i-th layer data and the n-th layer data, and store them in the new area, where i and n represent the number of convolutional layers respectively.

[0033] Furthermore, analyzing the test results includes the following specific steps:

[0034] S4.1. Score calculation: Score calculation is performed on output layer A and output layer B respectively;

[0035] S4.2, score comparison, compare the data retained by output layer A with the data retained by output layer B, and retain four sets of maximum score data, then complete maximum value suppression and restore target box information;

[0036] Step S4.1 includes the following specific steps:

[0037] S4.1.1, read the data in the memory;

[0038] S4.1.2. Calculate the target box score value of the output layer A;

[0039] S4.1.3, determine whether the current target frame score is greater than the minimum score in the array, if yes, execute step S4.1.4, otherwise execute step S4.1.5;

[0040] S4.1.4, replace the minimum score value and the corresponding target box information in the array;

[0041] S4.1.5. Determine whether all target frame scores of the output layer A have been calculated. If yes, end the process. Otherwise, return to step S4.1.2 to continue calculating and comparing the target frames.

[0042] The parsing steps of output layer B are the same as those of output layer A;

[0043] Step S4.2 includes the following specific steps:

[0044] S4.2.1. Read the data results saved in layer A and layer B;

[0045] S4.2.2, comparing the score of the target frame at layer A with the score of the target frame at layer B;

[0046] S4.2.3, determine whether the score of the target box in layer A is large, if so, execute step S4.2.4, otherwise execute step S4.2.6;

[0047] S4.2.4, determine whether the score of the target frame in layer A is greater than the threshold, if yes, execute step S4.2.5, otherwise execute step S4.2.8;

[0048] S4.2.5. Save the target frame score and target frame information of layer A;

[0049] S4.2.6, determine whether the score of the target box in layer B is greater than the threshold, if yes, execute step S4.2.7, otherwise execute step S4.2.8;

[0050] S4.2.7, save the target frame score and target frame information of layer B;

[0051] S4.2.8. Determine whether the comparison of the scores of the two target frames is completed. If yes, execute step S4.2.9. Otherwise, return to step S4.2.2 to continue the comparison of the scores of the two target frames.

[0052] S4.2.9, calculate the coordinate information and size information of the target frame;

[0053] S4.2.10. Transmit the coordinate information and size information of the target frame to the PL end.

[0054] In a second aspect, the present invention provides an edge device crack detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the memory, wherein when the processor executes the program, the edge device crack detection method as described in any one of the first aspects is implemented.

[0055] In a third aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the edge device crack detection method as described in any one of the first aspects.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The modular lightweight convolutional neural network designed by the present invention has a smaller model size and a faster detection speed. By using a multi-channel deep convolutional neural network, it is possible to extract more levels of image features, upsample and fuse the compressed feature data. The fused features can better capture the complexity of the data, improve model performance, enhance generalization ability, obtain richer and more comprehensive feature representation, and improve the crack detection effect in complex environments.

[0058] 2. A method for detecting cracks in edge devices of the present invention performs collaborative improvement processing on the image to be detected in many aspects: first, the image to be detected is preprocessed to obtain preprocessed image data; secondly, the image features of the preprocessed image data are extracted by convolution calculation; the image features are extracted by convolution calculation, which has stronger adaptability and better detection effect; then the image features are quantized using the INT8 quantization algorithm; then the quantized feature data is subjected to data nonlinear transformation, and activated using the LeakyReLU function, which greatly improves the nonlinear expression ability of the system and improves the generalization of the system; then the feature data after nonlinear transformation is compressed based on the maximum pooling method, and the most important feature data is extracted while reducing the amount of data; finally, the feature maps at different levels are fused by a multi-channel upsampling method, and the fused feature data is further extracted, so as to obtain higher detection accuracy and efficiency, and the model size is smaller and the detection speed is faster.

[0059] 3. The present invention can reduce the number of inference steps and the number of network parameters by combining the convolution layer and the batch normalization layer, thereby achieving the goal of lightweighting the network. Quantizing the parameters makes the model weights and biases smaller, and the model is lightweight. The convolution results are quantized, thereby discarding floating-point operations. This effectively solves the problems of large model weights, large floating-point calculations, and complex inference steps, thereby reducing the system inference time.

[0060] 4. The present invention preprocesses the image to be detected so that cracks in the entire image can be detected.

[0061] The present invention adopts a multi-channel parallel computing solution, which can greatly improve the computing throughput; and adopts a DSP multiplexing method, which greatly improves the utilization rate of DSP.

[0062] 5. The edge device crack detection method provided by the present invention can deploy the deep learning model to the edge device (for example, to GPU, embedded system, FPGA, ZYNQ, etc.). The method mainly emphasizes model optimization and deployment of the model on the device side, and can perform crack detection tasks outdoors. For example, the method can deploy the model to a ZYNQ processor, which has a PL end and a PS end, wherein the PL end is responsible for completing high-density computing tasks, and the PS end completes tasks such as control, data analysis, and data scheduling. It can complete crack detection without computer control and network, and is suitable for outdoor engineering construction surface crack detection tasks. The model hierarchy information is constructed through the PS end, and different models can be replaced by modifying the hierarchy information, which improves the scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a method for detecting cracks in edge devices of the present invention;

[0064] Figure 2 It is the main flow chart of image processing and crack detection;

[0065] Figure 3 is a flow chart of model training and optimization of the present invention;

[0066] Figure 4 This is a schematic diagram of image downsampling;

[0067] Figure 5 Schematic diagram for expanding data channels;

[0068] Figure 6 Schematic diagram of multi-channel convolution calculation;

[0069] Figure 7 This is a schematic diagram of multi-channel parallel computing of the system;

[0070] Figure 8 It is the flow chart of DSP multiplexing data processing;

[0071] Fig. 9 It is the flowchart of nonlinear transformation of data;

[0072] Fig.10 It is a schematic diagram of feature data compression;

[0073] Fig.11 It is the flow chart of data analysis of output layer 16;

[0074] Fig.12 It is the data parsing flow chart of output layer 22;

[0075] Fig.13 It is the maximum suppression flow chart;

[0076] Fig.14This is a crack detection effect diagram of the present invention. DETAILED DESCRIPTION

[0077] The present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the specific embodiments of the present invention are only for the purpose of more clearly describing the technical solution and cannot be used as a limitation on the protection scope of the present invention.

[0078] See also Figure 1-Figure 13 In a first aspect, the present invention provides a method for detecting cracks in an edge device, comprising the following steps:

[0079] S1. Collect crack detection data set and divide the data set into training set and test set;

[0080] S2. Build a YOLOv3-Tiny model, train and optimize the model, use the training set to train the YOLOv3-Tiny model, use the test set to test the model performance, and finally obtain the weight information and quantization information of the optimal model;

[0081] S3, processing the image to be detected collected by the camera, the processing method includes: first preprocessing the image to be detected to obtain preprocessed image data; secondly, extracting image features of the preprocessed image data by convolution calculation; then using the INT8 quantization algorithm to quantize the image features; then performing data nonlinear transformation on the quantized feature data, and using the LeakyReLU function to activate; then compressing the feature data after nonlinear transformation based on the maximum pooling method; finally, using the multi-channel upsampling method to fuse feature maps of different levels to obtain fused feature data;

[0082] S4. Further feature extraction is performed on the fused feature data to finally obtain the output result, compare the output result with the set threshold, output the detection result greater than the threshold, and parse the detection result to obtain the target frame information containing the crack coordinate information and size information; convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of a target frame on the image and display it on the screen.

[0083] See also Figure 3 ,, in S2, training and optimizing the model includes the following steps:

[0084] S2.1. Build the YOLOv3-Tiny model and create a crack detection dataset based on the crack shape and the characteristics of the deployed equipment.

[0085] S2.2, put the training set into the YOLOv3-Tiny model for training;

[0086] S2.3. Save the optimal model during model training;

[0087] S2.4. After the training is completed, the saved optimal model is loaded and the convolution layer is merged with the batch normalization layer. Specifically, the method of merging the convolution layer with the batch normalization layer is used for the training model for the first time, and the second time it is deployed on the ZYNQ processor. S2 is the training stage, which is also the weight data preparation stage. S3 is the deployment stage, and the parameters obtained by S2 training are put into the ZYNQ processor to build the YOLOv3-tiny model, thereby realizing crack detection on the device side. The method of merging the convolution layer with the batch normalization layer is described in detail below.

[0088] S2.5. Perform INT8 quantization on the optimized model weights to reduce the amount of floating-point calculations and the size of the model weights;

[0089] S2.6, put the test set into the quantized model for adaptive training, and save the quantized model after the training is completed;

[0090] S2.7, load the quantized model, obtain its weight information and quantization information, and prepare for subsequent detection;

[0091] Furthermore, preprocessing the image to be detected means using the camera to transform the size of the image frame to be detected, using the downsampling method to reduce the image size, and finally obtaining image data that meets the model input size. The downsampling method and effect are as follows: Figure 4 As shown, the number of data channels of the image frame to be detected is expanded to achieve the multiplexing condition. The effect of the channel number expansion is as follows Figure 5 shown.

[0092] Furthermore, in step S3, extracting image features of the preprocessed image data by convolution calculation includes the following steps:

[0093] S3.1.1. Construct a multi-channel convolution calculation matrix. In the convolution calculation, the corresponding data point needs to be multiplied by the weight corresponding to the corresponding convolution kernel. The convolution calculation is as follows:

[0094]

[0095] In multi-channel calculation, each input channel needs to be calculated with its corresponding convolution kernel channel. The calculation process is as follows: Figure 6 In order to improve the inference speed, the 3x3 convolution calculation of 8 input channels and 8 convolution kernel channels will be completed in a single clock cycle. The calculation process diagram is shown in Figure 7 shown.

[0096] S3.1.2, design a DSP multiplexing method, first shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculated data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations; specifically, taking a, b (the same multiplication factor for the two multiplication operations), and c as input data as an example, the specific implementation steps are as follows: Figure 8 As shown:

[0097] 1) Shift a;

[0098] 2) Expand the sign bits of a, b, and c;

[0099] 3) Add a and c;

[0100] 4) Multiply the sum of a and c by b;

[0101] 5) Get data with a bit width of n, the upper n / 2 bits are the result of a*b, and the lower n / 2 bits are the result of c*b.

[0102] S3.1.3, sort out the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = Wix i +B i , where Z is the result of convolution calculation, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term;

[0103] S3.1.3 uses the convolutional layer and the batch normalization layer to combine calculations to reduce the steps of feature extraction. The specific arrangement process is as follows:

[0104] Convolution calculation formula:

[0105] y i =w i x i +b i ;y i is the convolution result, w i is the weight, x i is the input data, b i For bias.

[0106] Batch normalization formula:

[0107] Sample mean (μ): m: the current batch data size, y i : Batch input data.

[0108] Sample variance (σ 2 ):

[0109] Standardizing Data ε is a small non-zero value to prevent the denominator from being zero.

[0110] Translation and scaling processing: r: trainable scale parameter; β: trainable bias parameter.

[0111] The YOLOv3-Tiny model has no bias (b i ) item, therefore, the calculation formula for combining the convolutional layer and the BN layer is as follows:

[0112]

[0113] After sorting:

[0114]

[0115] Final formula:

[0116] Z=Wix i +B i ;in:

[0117] Through the above reasoning and the preprocessing of the training results in the first step, after the convolution layer and the batch normalization layer are combined, only one multiplication and one addition are required to obtain the result of feature extraction.

[0118] S3.1.4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

[0119] Furthermore, in step S3, the specific steps of quantization processing are as follows:

[0120] S3.2.1. Arrange the quantitative formula of dot product calculation and get:

[0121] Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number;

[0122] S3.2.2, Data quantization By judging the level of the input data, loading the quantization information of the corresponding level, and using the INT8 quantization algorithm to quantize the feature extraction results, we can get a fixed-point number that meets the system reasoning requirements. The quantization formula is as follows:

[0123] r = S*(qZ);

[0124]

[0125] r represents the original floating-point number, S represents the conversion coefficient, q represents the quantized integer data, Z represents the value of the floating-point number 0 mapped to the fixed-point number, and the round() function has the function of rounding.

[0126] The reasoning process of the original model is a floating-point multiplication process. The formula for converting floating-point multiplication calculation to fixed-point multiplication calculation is as follows:

[0127] 3x3 convolution point multiplication calculation:

[0128] 1) r3 = r1 * r2 + bias;

[0129] Substituting r=S*(qZ) into 1) yields:

[0130] 2)S3*(q3-Z3)=S1*(q1-Z1)*S2(q2-Z2)+bias;

[0131] Further transformation yields:

[0132] 3)

[0133] r3 represents the floating-point multiplication result, r1 and r2 represent the input floating-point number and weight floating-point number respectively, bias represents the bias in the convolution calculation, S i Indicates the conversion factor corresponding to the floating point number, q i Indicates the corresponding floating point quantized integer data, Z i Indicates the value of 0 corresponding to the floating-point number mapped to the fixed-point value.

[0134] In the Zynq processor, the speed of shifting is faster than that of multiplication and division, and it can reduce resource consumption and solve the problem of tight timing. It is divided into two parts. The first part is 2 to the power of negative n, which is convenient for the subsequent processor to complete the shift operation. The second part is M0, which has been sorted in S2.

[0135] The final optimized quantization formula is:

[0136] in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and It is the result of convolution calculation;

[0137] S3.2.3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

[0138] 1. In the step of performing nonlinear transformation on the quantized feature data, the formula of the LeakyReLU activation function in the nonlinear transformation is as follows:

[0139] LeakyReLU(x)=x, if x>=0;

[0140] LeakyReLU(x)=negative_slope*x,if x<0;negative_slope default value is 0.01,x is the input value.

[0141] If the conventional calculation method is used to complete the nonlinear transformation of data, more PL resources will be consumed and the calculation complexity will be increased. Therefore, the present invention adopts the method of table search to realize the activation operation to solve the above problem. The specific steps are as follows: Fig. 9 As shown:

[0142] The first step is to sort out the activation function lookup table for each layer;

[0143] The second step is to store the lookup table in the SD card for future use;

[0144] The third step is to determine the current layer and read the current layer lookup table data;

[0145] Step 4: Store the current layer lookup table data into RAM;

[0146] The fifth step is to use the input data as the address to read the data in the RAM, and the data output by the RAM is the activation result.

[0147] Further, in S3, the feature data after nonlinear transformation is compressed based on the maximum pooling method, including a method for completing the data comparison of the 2X2 matrix through the time series characteristics, please refer to Fig.10 The specific steps are as follows: first, the pooling operation needs to be divided into three small steps. The first step is to build a 2X2 data matrix, compare the data of two adjacent columns in the first row, and save the maximum value to the memory; the second step is to compare the data of two adjacent columns in the second row and retain the maximum value; the third step is to read the memory data in the first step and compare it with the maximum value retained in the second step. The final maximum value is the maximum pooling result.

[0148] Furthermore, the method of multi-layer feature fusion operation is: first, read the data of the i-th layer storage area, and expand the size of the layer data by upsampling to meet the size requirements of fusion with the i-th layer data, then read the n-th layer storage area data, and finally complete the fusion operation of the i-th layer data and the n-th layer data, and store them in the new area, where i and n represent the number of convolution layers respectively. In this embodiment, i is selected as 13 and n is selected as 8.

[0149] Furthermore, analyzing the test results includes the following specific steps:

[0150] S4.1. Score calculation: Score calculation is performed on output layer A and output layer B respectively;

[0151] S4.2, score value comparison, compare the data retained by output layer A with the data retained by output layer B, and retain four sets of maximum score value data, then complete maximum value suppression and restore target frame information; in this embodiment, A is selected as 16 and B is selected as 23.

[0152] Please refer to Fig.11 and Fig.12 , step S4.1 includes the following specific steps:

[0153] S4.1.1, read the data in the memory;

[0154] S4.1.2. Calculate the target box score value of the output layer A;

[0155] S4.1.3, determine whether the current target frame score is greater than the minimum score in the array, if yes, execute step S4.1.4, otherwise execute step S4.1.5;

[0156] S4.1.4, replace the minimum score value and the corresponding target box information in the array;

[0157] S4.1.5. Determine whether all target frame scores of the output layer A have been calculated. If yes, end the process. Otherwise, return to step S4.1.2 to continue calculating and comparing the target frames.

[0158] The parsing steps of output layer B are the same as those of output layer A;

[0159] Please refer to Fig.13 , step S4.2 includes the following specific steps:

[0160] S4.2.1. Read the data results saved in layer A and layer B;

[0161] S4.2.2, comparing the score of the target frame at layer A with the score of the target frame at layer B;

[0162] S4.2.3, determine whether the score of the target box in layer A is large, if so, execute step S4.2.4, otherwise execute step S4.2.6;

[0163] S4.2.4, determine whether the score of the target frame in layer A is greater than the threshold, if yes, execute step S4.2.5, otherwise execute step S4.2.8;

[0164] S4.2.5. Save the target frame score and target frame information of layer A;

[0165] S4.2.6, determine whether the score of the target box in layer B is greater than the threshold, if yes, execute step S4.2.7, otherwise execute step S4.2.8;

[0166] S4.2.7, save the target frame score and target frame information of layer B;

[0167] S4.2.8. Determine whether the comparison of the scores of the two target frames is completed. If yes, execute step S4.2.9. Otherwise, return to step S4.2.2 to continue the comparison of the scores of the two target frames.

[0168] S4.2.9, calculate the coordinate information and size information of the target frame;

[0169] S4.2.10. Transmit the coordinate information and size information of the target frame to the PL end.

[0170] The final crack detection effect diagram of the implementation method of this system is as follows: Fig.14 As shown in the figure, it can be seen that the crack detection method of the present invention can accurately complete the crack detection.

[0171] In a second aspect, the present invention provides an edge device crack detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the memory, wherein when the processor executes the program, the edge device crack detection method as described in any one of the first aspects is implemented.

[0172] In a third aspect, the present invention provides a computer-readable storage medium storing computer executable instructions, characterized in that the computer executable instructions are used to execute the edge device crack detection method as described in any one of the first aspects. It can be understood by those of ordinary skill in the art that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0173] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for detecting cracks in edge devices, characterized in that: The steps include: S1. Collect crack detection data set and divide the data set into training set and test set; S2. Build a YOLOv3-Tiny model, train and optimize the model, use the training set to train the YOLOv3-Tiny model, use the test set to test the model performance, and finally obtain the weight information and quantization information of the optimal model; S3. Process the image to be detected, and the processing method includes: first, preprocessing the image to be detected to obtain preprocessed image data; second, extracting image features of the preprocessed image data by convolution calculation; then using the INT8 quantization algorithm to quantize the image features; then performing data nonlinear transformation on the quantized feature data, and using the LeakyReLU function to activate; then compressing the feature data after nonlinear transformation based on the maximum pooling method; finally, using the upsampling method to perform multi-layer feature fusion operations on feature maps of different levels to obtain fused feature data; S4. Further feature extraction is performed on the fused feature data to finally obtain the output result, compare the output result with the set threshold, output the detection result greater than the threshold, and parse the detection result to obtain the target frame information containing the crack coordinate information and size information; convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks on the image in the form of a target frame.

2. The edge device crack detection method according to claim 1, characterized in that: In S2, training and optimizing the model includes the following steps: S2.

1. Build the YOLOv3-Tiny model and create a crack detection dataset based on the crack shape and the characteristics of the deployed equipment. S2.2, put the training set into the YOLOv3-Tiny model for training; S2.

3. Save the optimal model during model training; S2.

4. After training, load the saved optimal model and merge the convolutional layer with the batch normalization layer; S2.

5. Perform INT8 quantization on the optimized model weights to reduce the amount of floating-point calculations and the size of the model weights; S2.6, put the test set into the quantized model for adaptive training, and save the quantized model after the training is completed; S2.

7. Load the quantized model and obtain its weight information and quantization information.

3. The edge device crack detection method according to claim 1, characterized in that: In step S3, extracting image features of the preprocessed image data by convolution calculation includes the following steps: S3.1.1, construct a multi-channel convolution calculation matrix; S3.1.2, design a DSP multiplexing method, first shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculation data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations; S3.1.3, sort out the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = Wix i +B i , where Z is the result of convolution calculation, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term; S3.1.

4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

4. The edge device crack detection method according to claim 1, characterized in that: In step S3, the specific steps of quantization processing are as follows: S3.2.

1. Arrange the quantitative formula of dot product calculation and get: Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number; S3.2.

2. Optimize the quantization algorithm. The optimized formula is: in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and It is the result of convolution calculation; S3.2.

3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

5. The edge device crack detection method according to claim 1, characterized in that: In the S3, the feature data after nonlinear transformation is compressed based on the maximum pooling method, including a method for completing data comparison of a 2X2 matrix through time series characteristics, and the specific steps are: the first step is to construct a 2X2 data matrix, compare the two adjacent columns of data in the first row, and save the maximum value to the memory; the second step is to compare the two adjacent columns of data in the second row and retain the maximum value; the third step is to read the memory data in the first step and compare it with the maximum value retained in the second step, and the final maximum value obtained is the maximum pooling result.

6. The edge device crack detection method according to claim 1, characterized in that: The method of multi-layer feature fusion operation is: first, read the data in the storage area of ​​the i-th layer, and expand the size of the data of this layer by upsampling to meet the size requirement of fusion with the i-th layer data, then read the data in the storage area of ​​the n-th layer, and finally complete the fusion operation of the i-th layer data and the n-th layer data, and store them in the new area, where i and n represent the number of convolutional layers respectively.

7. The edge device crack detection method according to claim 1, characterized in that: The analysis of the test results includes the following specific steps: S4.

1. Score calculation: Score calculation is performed on output layer A and output layer B respectively; S4.2, score comparison, compare the data retained by output layer A with the data retained by output layer B, and retain four sets of maximum score data, then complete maximum value suppression and restore target box information; Step S4.1 includes the following specific steps: S4.1.1, read the data in the memory; S4.1.

2. Calculate the target box score value of the output layer A; S4.1.

3. Determine whether the current target frame score is greater than the minimum score in the array. If so, execute step S4.1.4; otherwise, execute step S4.1.

5. S4.1.4, replace the minimum score value and the corresponding target box information in the array; S4.1.5, determine whether all target frame scores of the output layer A have been calculated, if yes, end the process, otherwise return to step S4.1.2 to continue to complete the target frame calculation and comparison; The parsing steps of output layer B are the same as those of output layer A; Step S4.2 includes the following specific steps: S4.2.

1. Read the data results saved in layer A and layer B; S4.2.2, comparing the score of the target frame at layer A with the score of the target frame at layer B; S4.2.3, determine whether the score of the target box in layer A is large, if so, execute step S4.2.4, otherwise execute step S4.2.6; S4.2.4, determine whether the score of the target frame in layer A is greater than the threshold, if yes, execute step S4.2.5, otherwise execute step S4.2.8; S4.2.

5. Save the target frame score and target frame information of layer A; S4.2.6, determine whether the score of the target box in layer B is greater than the threshold, if yes, execute step S4.2.7, otherwise execute step S4.2.8; S4.2.7, save the target frame score and target frame information of layer B; S4.2.

8. Determine whether the comparison of the scores of the two target frames is completed. If yes, execute step S4.2.

9. Otherwise, return to step S4.2.2 to continue the comparison of the scores of the two target frames. S4.2.9, calculate the coordinate information and size information of the target frame; S4.2.

10. Transmit the coordinate information and size information of the target frame to the PL end.

8. An edge device crack detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the memory, characterized in that: When the processor executes the program, the edge device crack detection method as described in any one of claims 1-8 is implemented.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the edge device crack detection method as described in any one of claims 1 to 7.

Images