Crack detection method and system based on edge end equipment

Abstract

The invention discloses a crack detection method and system based on edge end equipment, and the method comprises the following steps: collecting an image frame of a crack condition to be detected, and carrying out the preprocessing of the image frame; performing matrix construction on the preprocessed data, and performing image feature extraction on the preprocessed data by using convolution calculation; performing quantization processing on a feature extraction result by using an INT8 quantization algorithm; inputting the quantized data into a Leaky ReLU function to carry out data nonlinear transformation; constructing a 2X2 data matrix, and processing the data after nonlinear transformation by using a maximum pooling method; carrying out fusion operation on the feature maps of different levels; performing further feature extraction on the fused feature data to obtain output layer data; analyzing the output layer data; and converting the target frame information into RGB data for visual display. The target detection effect can be improved, and the problems of large model size, large equipment weight, high system power consumption, dependence on a network and the like in the existing field environment crack detection technology are solved.

Description

Technical Field

[0001] The present invention belongs to the field of visual recognition technology, and specifically relates to a crack detection method and system based on edge devices. Background Art

[0002] Today, there are a large number of bridges, roads, tunnels, dams, towers, etc. in my country. These engineering infrastructures are exposed to the outdoor environment and will be eroded by wind and rain. Therefore, it is essential to conduct safety inspections on these engineering infrastructures, and crack detection is one of the first indicators that can be detected.

[0003] The traditional crack detection system uses the classic template matching method, which is difficult to cover the current large number of engineering infrastructure crack detection conditions, and its adaptability and accuracy are reduced accordingly. In order to overcome the limitations of traditional methods, thanks to the advancement of artificial intelligence technology, researchers have begun to use various machine learning algorithms to detect cracks in engineering infrastructure. Prior art, the patent document with application number CN20210881065.1 discloses a road damage detection method, including: obtaining a road surface image to be detected; performing an expansion process based on brightness and contrast adjustment on the image to be detected; inputting a feature extraction network, obtaining the output of the feature extraction network as a feature image; inputting the feature image into a multi-layer feature pyramid network for feature fusion, and finally outputting a feature vector obtained after feature fusion at four different scales; fusing the confidence score output by the multi-layer feature pyramid network to obtain the fused confidence; comparing the fused confidence with the set threshold to obtain the prediction box position; determining the four types of damage of the road surface, namely, transverse cracks, longitudinal cracks, cracks, and block cracks according to the prediction box position. The existing patents for crack detection methods that enhance machine vision and deep learning have the following problems: (1) The network model is very large, the amount of calculation is large, and the detection speed is slow; the processing time of this method is long, the real-time performance is poor, the detection effect is not accurate enough, and it cannot provide fast detection calculation and decision support for real-time analysis and decision-making of engineering infrastructure; (2) The target detection effect is poor for complex scenes, such as road areas; (3) In order to pursue accuracy, it is necessary to increase the weight ratio of enhanced machine vision and deep learning systems, the amount of calculation is large and complex, which will increase memory and reasoning time, and these systems must use cloud servers to complete their detection tasks. For field environments with poor network quality or no network, it is difficult to adapt to their detection tasks. In addition, for those with network, the cost of cloud servers is also high, which is limited to scattered use and difficult to promote quickly. In order to solve the problem of crack detection system's dependence on servers, the current conventional practice is to deploy its enhanced machine vision and deep learning algorithms on edge GPUs, which can solve the problem of server dependence, but GPUs are costly, power-hungry, heavy, and have short battery life. They are not easy to carry and are difficult to adapt to crack detection in field environments. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems raised in the above-mentioned background technology, and provides a crack detection method and system based on edge devices. Through the combined improvement of software and hardware, it can solve the problems of large model volume, heavy equipment weight, high system power consumption, and network dependence in the existing field environment crack detection technology, improve the target detection effect, reduce the detection computing resource occupation, shorten the calculation time, so that it can provide real-time detection analysis calculation and decision support for cracks on engineering infrastructure.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a crack detection method based on an edge device, characterized in that it comprises the following steps:

[0007] S1. Establish a YOLOv3-Tiny model, collect image frames of cracks to be detected, and preprocess the image frames to obtain preprocessed data;

[0008] S2, constructing a matrix for the preprocessed data, and extracting image features from the preprocessed data using convolution calculation to obtain corresponding feature data;

[0009] S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number;

[0010] S4, input the quantized data into the LeakyReLU function for data nonlinear transformation;

[0011] S5, construct a 2X2 data matrix, use the maximum pooling method to process the data after nonlinear transformation, and obtain compressed feature data;

[0012] S6, traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is in a multi-layer feature fusion layer, if so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the feature extraction of the image; otherwise, execute S7;

[0013] S7, determine whether the current layer is an output layer, if yes, execute step S8, otherwise return to step S2 to continue to complete the feature extraction of the image;

[0014] S8, parsing the feature data of the output layer to obtain target frame information including crack position information and size information;

[0015] S9. Convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera for visual display.

[0016] Preferably, the specific method for preprocessing the image frame is:

[0017] Firstly, the image frames to be detected captured by the camera are resized and the image size is reduced using the downsampling method, finally obtaining image data that meets the input size of the YOLOv3-Tiny model; then, the number of data channels of the image frames to be detected is expanded to meet the conditions for module reuse.

[0018] Preferably, in step S2, using convolution calculation to extract image features from the preprocessed data includes the following steps:

[0019] S2.1, construct a multi-channel convolution calculation matrix;

[0020] S2.2, design a DSP multiplexing method, first shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculation data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations;

[0021] S2.3, sort out the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = Wix i +B i , and use the code to represent it, where Z is the convolution calculation result, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term;

[0022] S2.4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

[0023] Preferably, in step S3, the specific steps of quantization processing are as follows:

[0024] S3.1.1. Arrange the quantitative formula of dot multiplication calculation, and finally get:

[0025] Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number;

[0026] S3.1.2. Optimize the quantization algorithm. The optimized formula is:

[0027] in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and For the convolution calculation results, the optimized algorithm can greatly reduce resource consumption and reduce the time required for calculation;

[0028] S3.1.3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

[0029] Preferably, the nonlinear transformation of the data in step S4 includes arranging the lookup table and implementing the activation function, and the specific steps are as follows:

[0030] S4.1, take 0 to 255 as input data, and Z in each layer of quantization information i For comparison, if the input data is greater than Z i , then execute LeakyReLU(x)=0.1*x, otherwise execute LeakyReLU(x)=x, where x is the input data;

[0031] S4.2, quantify the above execution results and store them for future use;

[0032] S4.3. When activation processing is required, the corresponding lookup table is written into the memory, and the data to be activated is sent to the memory as an address. The value output by the memory is the activation result.

[0033] In a second aspect, the present invention provides a crack detection system based on an edge device, comprising:

[0034] An image acquisition module, used for acquiring image frames of crack conditions to be detected;

[0035] The image preprocessing module is used to transform the size of the collected image frame to be detected, reduce the image size by using the downsampling method, and expand the number of data channels of the image frame to be detected;

[0036] The feature extraction module is used to construct a matrix for the preprocessed data and extract image features from the preprocessed data using convolution calculation to obtain corresponding feature data;

[0037] The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point numbers.

[0038] The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for data nonlinear transformation.

[0039] The feature data compression module is used to construct a 2X2 data matrix and use the maximum pooling method to process the data after nonlinear transformation to obtain compressed feature data.

[0040] The multi-layer feature fusion module is used to fuse feature maps at different levels through upsampling to obtain fused feature data.

[0041] The data analysis module is used to analyze the feature data of the output layer to obtain target frame information including crack location information and size information.

[0042] The target frame construction module is used to convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera for visual display.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Combining the convolution layer and batch normalization layer can reduce the number of inference steps. Quantizing the parameters makes the model weights and biases smaller, and the model is lightweight. Quantizing the convolution results discards floating-point operations. This effectively solves the problems of large model weights, large floating-point calculations, and complex inference steps, thereby reducing the system inference time.

[0045] 2. By preprocessing the data collected by the camera, cracks in the entire image can be detected.

[0046] 3. The system adopts a multi-channel parallel computing solution to greatly improve the computing throughput, solve the problem of insufficient computing power of some devices, and reduce the delay of the system's forward reasoning.

[0047] 4. The system adopts DSP multiplexing method to greatly improve the utilization of DSP and provide different solutions for processors with different DSP resources, making the system more adaptable.

[0048] 5. The system uses ZYNQ as a development platform, which has a PL side and a PS side. The PL side is responsible for completing high-density computing tasks, and the PS side completes tasks such as control, data analysis, and data scheduling. The system can complete crack detection without computer control and network.

[0049] 6. The system builds model hierarchy information through the PS side. Different models can be replaced by modifying the hierarchy information, which improves the scalability of the system.

[0050] 7. The ZYNQ processor has the advantages of low power consumption, small size and light weight, which can improve the system's endurance and is easy to carry, making it more suitable for outdoor crack detection tasks.

[0051] 8. The system does not need to rent a server, and the ZYNQ processor is low-priced, which can reduce the manufacturing cost and use cost of the system equipment.

[0052] 9. The present invention uses the ZYNQ processor as the edge processing unit to construct its edge device, which not only makes the hardware and power supply battery very light, but also is suitable for field crack detection and easy to carry. In terms of crack detection, the convolution layer and the normalization layer of the YOLOv3-Tiny algorithm are combined, and the convolution kernel and the convolution result are quantized. The optimized YOLOv3-Tiny algorithm has a smaller algorithm calculation amount, a faster running speed, and a better target detection effect. It can be better deployed on the edge device of the ZYNQ processor, realizing the real lightweight of the crack detection system and the device, which is better suitable for outdoor engineering crack detection applications and carrying. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a crack detection method based on edge devices of the present invention;

[0054] Figure 2 A block diagram of a crack detection system based on edge devices of the present invention;

[0055] Figure 3 This is a schematic diagram for expanding the data channel;

[0056] Figure 4 Schematic diagram of multi-channel convolution calculation;

[0057] Figure 5 This is a schematic diagram of multi-channel parallel computing of the system;

[0058] Figure 6 This is a crack detection effect diagram in one embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of system data flow;

[0060] Figure 8 This is a schematic diagram of ping-pong operation;

[0061] Fig. 9 The power consumption graph is calculated in parallel for 8 input channels and 8 convolution kernel channels;

[0062] Fig.10 The power consumption graph is calculated for 16 input channels and 16 convolution kernel channels in parallel. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention. These embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. In addition, the embodiments in this application and the features in the embodiments may be combined with each other without conflict.

[0064] See also Figure 1-Figure 10 , first of all, such as Figure 1 As shown, the present invention provides a crack detection method based on edge devices, comprising the following steps:

[0065] S1. Establish a YOLOv3-Tiny model, use a camera to collect image frames of cracks to be detected, and preprocess the image frames to obtain preprocessed data;

[0066] S2. Matrix construction is performed on the preprocessed data, and image features are extracted from the preprocessed data using convolution calculation to obtain the corresponding feature data; the multi-channel input data matrix and convolution kernel matrix are constructed to meet the requirements of multi-channel parallel calculation. In terms of calculation method, by using the method of combining convolution layer and batch normalization layer to reduce the number of inference steps and extract important feature data. In convolution calculation, the system also uses DSP multiplexing method to improve resource utilization.

[0067] S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number;

[0068] S4, input the quantized fixed-point data into the LeakyReLU function to perform nonlinear transformation of the data, so that it completes the activation processing to improve the nonlinear expression ability of the model;

[0069] S5, construct a 2X2 data matrix, divide the data after nonlinear transformation into blocks, use the maximum pooling method to compress the data after nonlinear transformation, finally eliminate the remaining data, retain the largest feature data in the matrix, and obtain compressed feature data;

[0070] S6, traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is in a multi-layer feature fusion layer, if so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the feature extraction of the image; otherwise, execute S7;

[0071] S7, determine whether the current layer is an output layer, if yes, execute step S8, otherwise return to step S2 to continue to complete the feature extraction of the image;

[0072] S8, parsing the feature data of the output layer to obtain target frame information containing crack location information and size information; specifically, parsing the results obtained by the system forward reasoning, calculating the score of each target frame, comparing the score of each target frame, retaining the 4 highest score values ​​and their corresponding target frame information, and completing non-maximum suppression, thereby obtaining useful target frame location information, and finally transmitting the restored target frame information to the PL end;

[0073] S9. Convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera for visual display.

[0074] Preferably, the specific method for preprocessing the image frame is:

[0075] First, the image frame to be detected captured by the camera is resized, and the image size is reduced using the downsampling method, and finally the image data that meets the input size of the YOLOv3-Tiny model is obtained; then, the number of data channels of the image frame to be detected is expanded from the original 3-channel data to 8-channel data to achieve the condition of module reuse. The effect of channel number expansion is as follows: Figure 3 shown.

[0076] Preferably, in step S2, using convolution calculation to extract image features from the preprocessed data includes the following steps:

[0077] S2.1. Construct a multi-channel convolution calculation matrix. By constructing a multi-channel input data matrix and a convolution kernel matrix, multi-channel parallel calculation is completed, so that important features can be extracted more quickly. In the convolution calculation, the corresponding data point needs to be multiplied by the weight corresponding to the corresponding convolution kernel. The convolution calculation is as follows:

[0078]

[0079] In multi-channel calculation, each input channel needs to be calculated with its corresponding convolution kernel channel. The calculation process is as follows: Figure 4 In order to improve the inference speed, this system will complete the 3x3 convolution calculation of 8 input channels and 8 convolution kernel channels in a single clock cycle. The calculation process diagram is shown in Figure 5 As shown;

[0080] S2.2, design a DSP multiplexing method to improve resource utilization; first, shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculation data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations;

[0081] S2.3, sort out the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = Wix i +B i , and use the code to represent it, where Z is the convolution calculation result, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term;

[0082] The convolutional layer and the batch normalization layer are combined to reduce the steps of feature extraction. The specific arrangement process is as follows:

[0083] Convolution calculation formula:

[0084] y i =w i x i +b i ;y i is the convolution result, w i is the weight, x i is the input data, b i For bias.

[0085] Batch normalization formula:

[0086] Sample mean (μ): m: the current batch data size, y i : Batch input data.

[0087] Sample variance (σ 2 ):

[0088] Standardizing Data ε is a small non-zero value to prevent the denominator from being zero.

[0089] Translation and scaling processing: r: trainable scale parameter; β: trainable bias parameter.

[0090] The YOLOv3-Tiny model has no bias (b i ) item, therefore, the calculation formula for combining the convolutional layer and the BN layer is as follows:

[0091]

[0092] After sorting:

[0093]

[0094] Final formula:

[0095] Z=Wix i +B i ;in:

[0096] S2.4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

[0097] Preferably, in step S3, the specific steps of quantization processing are as follows:

[0098] S3.1.1. Arrange the quantitative formula of dot multiplication calculation, and finally get:

[0099] Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number;

[0100] S3.1.2, Data quantization By judging the level of the input data, loading the quantization information of the corresponding level, and using the INT8 quantization algorithm to quantize the feature extraction results, a fixed-point number that meets the system reasoning requirements is obtained. The quantization formula is as follows:

[0101] r = S*(qZ);

[0102]

[0103] r represents the original floating-point number, S represents the conversion coefficient, q represents the quantized integer data, Z represents the value of the floating-point number 0 mapped to the fixed-point number, and the round() function has the function of rounding.

[0104] The reasoning process of the original model is a floating-point multiplication process. The formula for converting floating-point multiplication calculation to fixed-point multiplication calculation is as follows:

[0105] 3x3 convolution point multiplication calculation:

[0106] 1) r3 = r1 * r2 + bias;

[0107] Substituting r=S*(qZ) into 1) yields:

[0108] 2)S3*(q3-Z3)=S1*(q1-Z1)*S2(q2-Z2)+bias;

[0109] Further transformation yields:

[0110] 3)

[0111] r3 represents the floating-point multiplication result, r1 and r2 represent the input floating-point number and weight floating-point number respectively, bias represents the bias in the convolution calculation, S i Indicates the conversion factor corresponding to the floating point number, q i Indicates the corresponding floating point quantized integer data, Z i Indicates the value of 0 corresponding to the floating-point number mapped to the fixed-point value.

[0112] In the Zynq processor, the speed of shifting is faster than that of multiplication and division, and it can reduce resource consumption and solve the problem of tight timing. It is divided into two parts. The first part is 2 to the power of negative n, which is convenient for the subsequent processor to complete the shift operation. The second part is M0, which has been sorted in S1.

[0113] The final optimized quantization formula is:

[0114] in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and For the convolution calculation results, the optimized algorithm can greatly reduce resource consumption and reduce the time required for calculation;

[0115] S3.1.3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

[0116] Preferably, the nonlinear transformation of the data in step S4 includes arranging the lookup table and implementing the activation function, and the specific steps are as follows:

[0117] S4.1, take 0 to 255 as input data, and Z in each layer of quantization information i For comparison, if the input data is greater than Z i , then execute LeakyReLU(x)=0.1*x, otherwise execute LeakyReLU(x)=x, where x is the input data;

[0118] S4.2, quantify the above execution results and store them for future use;

[0119] S4.3. When activation processing is required, the corresponding lookup table is written into the memory, and the data to be activated is sent to the memory as an address. The value output by the memory is the activation result.

[0120] The specific steps of feature data compression in step S5 are as follows:

[0121] S5.1. Complete the comparison of the data of two adjacent columns in the first row by controlling the timing, and retain the maximum value;

[0122] S5.2, wait for the comparison of the two adjacent columns of the second row to be completed, compare the maximum value of the two adjacent columns of the second row with the maximum value obtained in the first step, and finally obtain the maximum value of the 2x2 data matrix. In the process of completing the feature data compression, it is necessary to pay attention to the step size of the maximum pooling.

[0123] The multi-layer feature fusion in the above step S6 includes upsampling and data fusion, and the specific steps are as follows:

[0124] S6.1, read the data in the storage area of ​​the 13th layer, and use upsampling to expand the data size of this layer;

[0125] S6.2, read the data in the storage area of ​​the 8th layer and fuse it with the up-sampled data in the first step;

[0126] S6.3. Store the fused data in the new area.

[0127] In the above step S8, the analysis operation includes data analysis, score value comparison, maximum value suppression and target frame recovery. The specific steps are as follows:

[0128] S8.1, read the data stored in the 15th layer storage area, and calculate the score value of each target box;

[0129] S8.2, compare the score value of each target frame in the 15th layer, and retain the 4 maximum score values ​​and their corresponding target frame information;

[0130] S8.3, read the 23rd layer data, and calculate the score value of each target box;

[0131] S8.4, compare the score value of each target frame in the 23rd layer, and retain the 4 maximum score values ​​and the corresponding target frame position information;

[0132] S8.5, comparing the score values ​​retained at the 15th layer and the 23rd layer, and saving the four maximum score values ​​and their corresponding target box information;

[0133] S8.6, performing maximum value suppression on the four maximum score values ​​saved in step 5;

[0134] S8.7. Proportionally restore the target boxes whose scores are greater than the threshold.

[0135] Second, as Figure 2 As shown, the present invention provides a crack detection system based on edge devices, comprising:

[0136] An image acquisition module is used to acquire image frames of cracks to be detected; the image acquisition module is composed of an OV5640 camera to obtain image information to be detected with a resolution of 720P or above;

[0137] The data processing module of the crack detection system is completed on the processor. In this embodiment, the processor is a ZYNQ processor. The data processing module includes: image preprocessing module, feature extraction module, data quantization module, data nonlinear transformation module, feature data compression module, multi-layer feature fusion module, data analysis module, target frame construction module, wherein:

[0138] The image preprocessing module is used to transform the size of the collected image frames to be detected, reduce the image size by downsampling to meet the input requirements of the model, and expand the number of data channels of the image frames to be detected to meet the multiplexing conditions of the module;

[0139] The feature extraction module is used to construct a matrix for the preprocessed data, and use convolution calculation to extract image features from the preprocessed data to obtain the corresponding feature data; by constructing the convolution calculation matrix and using the convolution layer and batch normalization layer combined calculation method, the corresponding feature data can be obtained while reducing the inference steps;

[0140] The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point numbers that meet the system's forward reasoning.

[0141] The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for data nonlinear transformation and complete the activation processing to improve the nonlinear expression ability of the model.

[0142] The feature data compression module is used to construct a 2X2 data matrix, use the maximum pooling method to process the data after nonlinear transformation, and finally retain the largest feature data in the matrix, that is, to obtain the compressed feature data.

[0143] The multi-layer feature fusion module is used to fuse feature maps of different layers through upsampling methods to obtain fused feature data; for example, the size of the 13th layer data is enlarged by upsampling methods to meet the size requirements for fusion with the 8th layer data.

[0144] The data parsing module is used to parse the feature data of the output layer to obtain the target frame information containing the crack location information and size information. After the model inference is completed, the PS side reads the data in DDR3, parses the location information and size information of the target frame, and sends it to the PL side.

[0145] The target frame construction module is used on the PL side to convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera, and visualize them through the LCD display.

[0146] The final crack detection effect diagram of the implementation method of this system is as follows: Figure 6 As shown in the figure, it can be seen that the crack detection method based on edge devices of the present invention can accurately complete the crack detection. The advantages of the system are further described below:

[0147] In order to improve the scalability of the system, this system stores the hierarchical information of the model on the PS side, and controls the PL side to complete the calculation of different layers by calling and parsing the hierarchical information. If other models need to be replaced later, only the hierarchical structure of the PS side needs to be modified.

[0148] Figure 7 It is the data transmission direction during model inference. First, the camera data is stored in DDR3 as a frame buffer, and then the PS side sends the stored camera data to the PL side for calculation. The calculation result of the PL side is transmitted back to DDR3 for storage. The operation is repeated according to the scheduling information of the PS side, and the output layer data is finally parsed on the PS side. In the data storage scheme, the present invention adopts the ping-pong operation method to complete data storage and reading. The ping-pong operation diagram is shown in the figure. Figure 8 As shown in the figure. The input data required for the calculation of the nth layer is read from area 2, and the calculation results are written to area 1. When the system reaches the n+1th layer, the required input data is read from area 1, and the calculation results are written to area 2. The above ping-pong operation can improve the utilization of storage space while ensuring that data is read correctly.

[0149] A large number of multiplication calculations are required during the model inference process, so DSP resources are particularly important. In order to adapt to the medium and low performance ZNYQ chip, this system adopts a parallel calculation solution of 8 input channels and 8 convolution kernel channels. The system inference time of this solution is 194ms, and the total on-chip power consumption of the system is 2.651W. The power consumption results are shown in Fig. 9As shown. When the DSP resources of the processor are greater than 310, the system can be used directly for crack detection. When the DSP resources of the processor are insufficient, it is necessary to reduce the number of parallel computing channels or use logic resources instead of DSP resources. If you want to increase the inference speed, you can increase the number of parallel computing channels. When the system uses 16 input channels and 16 convolution kernel channels for parallel computing, the time required for inference will be reduced to 86ms, and the total on-chip power consumption of the system is 3.331W. The power consumption results are shown in Fig.10 shown.

[0150] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A crack detection method based on edge devices, characterized in that: The steps include: S1. Establish a YOLOv3-Tiny model, collect image frames of cracks to be detected, and preprocess the image frames to obtain preprocessed data; S2, constructing a matrix for the preprocessed data, and extracting image features from the preprocessed data using convolution calculation to obtain corresponding feature data; S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number; S4, input the quantized data into the LeakyReLU function for data nonlinear transformation; S5, construct a 2X2 data matrix, use the maximum pooling method to process the data after nonlinear transformation, and obtain compressed feature data; S6, traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is in a multi-layer feature fusion layer, if so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the feature extraction of the image; otherwise, execute S7; S7, determine whether the current layer is an output layer, if yes, execute step S8, otherwise return to step S2 to continue to complete the feature extraction of the image; S8, parsing the data of the output layer to obtain target frame information including crack position information and size information; S9. Convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera for visual display.

2. A crack detection method based on edge devices according to claim 1, characterized in that: The specific method for preprocessing image frames is: Firstly, the image frames to be detected captured by the camera are resized and the image size is reduced using the downsampling method, finally obtaining image data that meets the input size of the YOLOv3-Tiny model; then, the number of data channels of the image frames to be detected is expanded to meet the conditions for module reuse.

3. The crack detection method based on edge device according to claim 1, characterized in that: In step S2, using convolution calculation to extract image features from the preprocessed data includes the following steps: S2.1, construct a multi-channel convolution calculation matrix; S2.2, design a DSP multiplexing method, first shift and fill the three input multiplication factors (a, b, c), then use the pre-adder in the DSP to process the calculation data, and perform multiplication calculation on the processed data, and finally further process the output result of the DSP to obtain the results of a*b and c*b, where b is the multiplication factor shared by the two multiplication calculations; S2.

3. Arrange the calculation formula after the convolution layer and the batch normalization layer are combined, and finally get Z = W i x i +B i , where Z is the result of convolution calculation, W i is the weight term after sorting, x i is the input data, B i is the adjusted bias term; S2.

4. Complete multi-channel parallel convolution calculation to obtain corresponding feature data.

4. The crack detection method based on edge device according to claim 1, characterized in that: In step S3, the specific steps of quantization processing are as follows: S3.1.

1. Arrange the quantitative formula of dot multiplication calculation, and finally get: Among them, bias represents the convolution calculation bias, S i Indicates the conversion coefficient, q3 indicates the quantized integer data of the multiplication result, q1 and q2 indicate the quantized integer data of the multiplication factor, Z i Indicates the value of 0 of the floating point number mapped to the fixed point number; S3.1.

2. Optimize the quantization algorithm. The optimized formula is: in, 2 of the above -n The processing can be completed by shifting, M0 can be extracted from the quantization information, and It is the result of convolution calculation; S3.1.

3. Implement the above algorithm and round off the calculation results to reduce the error of the results.

5. The crack detection method based on edge device according to claim 4, characterized in that: The nonlinear transformation of data in step S4 includes arranging the lookup table and implementing the activation function. The specific steps are as follows: S4.1, take 0 to 255 as input data, and Z in each layer of quantization information i For comparison, if the input data is greater than Z i , then execute LeakyReLU(x)=0.1*x, otherwise execute LeakyReLU(x)=x, where x is the input data; S4.2, quantify the above execution results and store them for future use; S4.

3. When activation processing is required, the corresponding lookup table is written into the memory, and the data to be activated is sent to the memory as an address. The value output by the memory is the activation result.

6. A crack detection system based on edge devices, characterized in that: include: An image acquisition module, used for acquiring image frames of crack conditions to be detected; The image preprocessing module is used to transform the size of the collected image frame to be detected, reduce the image size by using the downsampling method, and expand the number of data channels of the image frame to be detected; The feature extraction module is used to construct a matrix for the preprocessed data and extract image features from the preprocessed data using convolution calculation to obtain corresponding feature data; The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point numbers. The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for data nonlinear transformation. The feature data compression module is used to construct a 2X2 data matrix and use the maximum pooling method to process the data after nonlinear transformation to obtain compressed feature data. The multi-layer feature fusion module is used to fuse feature maps at different levels through upsampling to obtain fused feature data. The data analysis module is used to analyze the feature data of the output layer to obtain target frame information including crack location information and size information. The target frame construction module is used to convert the target frame information into RGB data, construct a visual target frame, and mark the detected cracks in the form of target frames on the image frames of the cracks to be detected collected by the camera for visual display.

Images