A Crack Detection Method and System Based on Edge Devices

Abstract

A crack detection method and system based on edge devices is disclosed. The method includes the following steps: acquiring image frames of the crack to be detected and preprocessing the image frames; constructing a matrix from the preprocessed data and extracting image features from the preprocessed data using convolution calculation; quantizing the feature extraction results using the INT8 quantization algorithm; inputting the quantized data into the LeakyReLU function for nonlinear data transformation; constructing a 2x2 data matrix and processing the nonlinearly transformed data using max pooling; fusing feature maps at different levels; further extracting features from the fused feature data to obtain output layer data; parsing the output layer data; and converting the target bounding box information into RGB data for visualization. This invention can improve the target detection effect and solve the problems of large model size, heavy equipment, high system power consumption, and network dependence in existing field environment crack detection technologies.

Description

Technical Field

[0001] This invention belongs to the field of visual recognition technology, specifically relating to a crack detection method and system based on edge devices. Background Technology

[0002] Today, my country has a large number of bridges, highways, tunnels, dams, iron towers, and other engineering infrastructure. These infrastructure projects are exposed to the outdoor environment and are subject to wind and rain erosion. Therefore, it is essential to conduct safety inspections on these infrastructure projects, and crack detection is one of the first indicators that can be detected.

[0003] Traditional crack detection systems employ classic template matching methods, which struggle to cover the vast number of crack detection scenarios in modern engineering infrastructure, resulting in reduced adaptability and accuracy. To overcome the limitations of traditional methods, researchers have begun using various machine learning algorithms to detect cracks in engineering infrastructure, thanks to advancements in artificial intelligence. Existing technology, patent application number CN20210881065.1, discloses a pavement damage detection method, comprising: acquiring an image of the pavement to be detected; performing augmentation processing on the image based on brightness and contrast adjustments; inputting the image into a feature extraction network and obtaining the output of the feature extraction network as a feature image; inputting the feature image into a multi-layer feature pyramid network for feature fusion, ultimately outputting feature vectors obtained after feature fusion at four different scales; fusing the confidence scores output by the multi-layer feature pyramid network to obtain a fused confidence score; comparing the fused confidence score with a set threshold to obtain the predicted bounding box position; and determining four types of pavement damage based on the predicted bounding box position: transverse cracks, longitudinal cracks, alligator cracks, and block cracks. The existing patents for crack detection methods that enhance machine vision and deep learning have the following problems: (1) The network model is very large, the computation is large, and the detection speed is slow; the processing time of this method is long, the real-time performance is poor, the detection effect is not accurate enough, and it cannot provide fast detection calculation and decision support for real-time analysis and decision-making of engineering infrastructure; (2) The target detection effect is poor in complex scenes, such as road areas; (3) In order to pursue accuracy, it is necessary to increase the weight ratio of the enhanced machine vision and deep learning system, which is computationally large and complex, which will lead to an increase in memory and inference time. Moreover, these systems must use cloud servers to complete their detection tasks. For field environments with poor network quality or no network, it is difficult to adapt to their detection tasks. In addition, for those with network, the cost of cloud servers is also large, which limits their use to scattered use and makes it difficult to promote them quickly. In order to solve the problem of crack detection system dependence on servers, the current conventional approach is to deploy its enhanced machine vision and deep learning algorithms on edge GPUs, which can solve the problem of server dependence. However, GPUs are expensive, consume a lot of power, are heavy, have short battery life, are not easy to carry, and are difficult to adapt to crack detection in field environments. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems mentioned in the background art above, and provides a crack detection method and system based on edge devices. Through the combined improvement of software and hardware, it can solve the problems of large model size, large equipment weight, high system power consumption and network dependence in existing field environment crack detection technology, improve the target detection effect, reduce the detection computing resource occupation, shorten the computing time, and enable it to provide real-time detection, analysis and calculation and decision support for cracks on engineering infrastructure.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a crack detection method based on an edge-end device, characterized by comprising the following steps:

[0007] S1. Establish a YOLOv3-Tiny model, collect image frames of the crack to be detected, and preprocess the image frames to obtain preprocessed data;

[0008] S2. Construct a matrix from the preprocessed data and use convolution to extract image features from the preprocessed data to obtain the corresponding feature data.

[0009] S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number.

[0010] S4. Input the quantized data into the LeakyReLU function for nonlinear data transformation;

[0011] S5. Construct a 2x2 data matrix and use max pooling to process the nonlinearly transformed data to obtain compressed feature data.

[0012] S6. Traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is a multi-layer feature fusion layer. If so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the image feature extraction; otherwise, execute S7.

[0013] S7. Determine whether the current layer is the output layer. If yes, proceed to step S8; otherwise, return to step S2 to continue extracting features from the image.

[0014] S8. Parse the feature data of the output layer to obtain the target bounding box information containing crack location and size information;

[0015] S9. Convert the target box information into RGB data, construct a visual target box, and mark the detected cracks as target boxes on the image frame of the crack to be detected captured by the camera for visualization.

[0016] The preferred method for preprocessing image frames is as follows:

[0017] First, the image frames to be detected captured by the camera are resized by downsampling to reduce the image size, ultimately obtaining image data that conforms to the input size of the YOLOv3-Tiny model. Then, the number of data channels of the image frames to be detected is increased to achieve the condition of module reuse.

[0018] Preferably, in step S2, the image feature extraction of the preprocessed data using convolution calculation includes the following steps:

[0019] S2.1 Construct a multi-channel convolution calculation matrix;

[0020] S2.2 Design a DSP multiplexing method. First, shift and fill the three input multiplication factors (a, b, c). Second, use the pre-adder in the DSP to process the calculation data and perform multiplication calculation on the processed data. Finally, further process the output result of the DSP to obtain the results of a*b and c*b. The above b is a multiplication factor common to the two multiplication calculations.

[0021] S2.3. After refining the calculation formula for the combined convolutional layer and batch normalized layer, we finally obtain Z = W. i x i +B i And represent it using code, where Z is the result of the convolution calculation, W i For the adjusted weights, x i For input data, B i The adjusted bias term;

[0022] S2.4 Complete the multi-channel parallel convolution calculation to obtain the corresponding feature data.

[0023] Preferably, the specific steps of the quantization process in step S3 are as follows:

[0024] S3.1.1. Reorganize the quantitative formula for dot product calculation, and finally obtain:

[0025] Where bias represents the bias used in convolution calculation, S i Z represents the conversion factor, q3 represents the quantized integer data of the multiplication result, q1 and q2 represent the quantized integer data of the multiplication factors, and Z represents the multiplication factor. i This represents the value of 0 in a floating-point number mapped to a fixed-point number.

[0026] S3.1.2 Optimize the quantization algorithm. The optimized formula is as follows:

[0027] in, 2 above -n Processing can be accomplished using bit shifting; M0 can extract information from quantized data, while... For the convolution calculation results, the optimized algorithm can greatly reduce resource consumption and reduce the computation time required;

[0028] S3.1.3 Implement the above algorithm and round the calculation results to reduce the error.

[0029] Preferably, the nonlinear data transformation in step S4 includes tidying up the lookup table and implementing the activation function, with the specific steps as follows:

[0030] S4.1. Using 0 to 255 as input data, and comparing it with the Z-value in each layer of quantization information. i For comparison, if the input data is greater than Z i If the result is positive, then LeakyReLU(x) = 0.1 * x is executed; otherwise, LeakyReLU(x) = x is executed, where x is the input data.

[0031] S4.2 Quantify the above execution results and store them for later use;

[0032] S4.3 When activation is required, the corresponding lookup table is written into the memory, the data to be activated is sent to the memory as the address, and the value output by the memory is the activation result.

[0033] Secondly, the present invention provides a crack detection system based on an edge-end device, comprising:

[0034] The image acquisition module is used to acquire image frames of the crack to be detected;

[0035] The image preprocessing module is used to resize the acquired image frames to be detected, reduce the image size using downsampling, and increase the number of data channels of the image frames to be detected.

[0036] The feature extraction module is used to construct a matrix from the preprocessed data and use convolution calculation to extract image features from the preprocessed data to obtain the corresponding feature data.

[0037] The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point numbers.

[0038] The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for data nonlinear transformation.

[0039] The feature data compression module is used to construct a 2x2 data matrix and process the nonlinearly transformed data using the max pooling method to obtain compressed feature data.

[0040] The multi-layer feature fusion module is used to fuse feature maps at different levels through upsampling to obtain fused feature data.

[0041] The data parsing module is used to parse the feature data of the output layer to obtain target bounding box information containing crack location and size information.

[0042] The target bounding box construction module is used to convert target bounding box information into RGB data, construct visual target bounding boxes, and mark the detected cracks as target bounding boxes on the image frames of the cracks to be detected captured by the camera for visualization.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Combining convolutional layers and batch normalized layers reduces inference steps. Quantization of parameters reduces the size of model weights and biases, resulting in a lighter model. Quantization of convolution results eliminates floating-point operations. This effectively solves the problems of large model weight size, high floating-point computation cost, and complex inference steps, thereby reducing system inference time.

[0045] 2. By preprocessing the data collected by the camera, cracks in the entire image can be detected.

[0046] 3. The system adopts a multi-channel parallel computing scheme, which greatly improves the computing throughput, solves the problem of insufficient computing power of some devices, and reduces the latency of forward inference of the system.

[0047] 4. The system adopts a DSP multiplexing method, which greatly improves the utilization of DSP and provides different solutions for processors with different DSP resources, making the system more adaptable.

[0048] 5. The system uses ZYNQ as the development platform, which has a PL (Programmer) terminal and a PS (Power Supply) terminal. The PL terminal is responsible for high-density computing tasks, while the PS terminal handles control, data parsing, and data scheduling. The system can perform crack detection without computer control or network access.

[0049] 6. The system constructs model hierarchy information through the PS terminal. By modifying the hierarchy information, different models can be replaced, thus improving the system's scalability.

[0050] 7. The ZYNQ processor has the advantages of low power consumption, small size and light weight, which can improve the system's battery life, make it easy to carry, and make it more suitable for outdoor crack detection tasks.

[0051] 8. The system does not require renting servers, and the low price of ZYNQ processors can reduce the manufacturing and operating costs of system equipment.

[0052] 9. This invention constructs an edge device using a ZYNQ processor as the edge processing unit, achieving lightweight hardware and power supply batteries, making it suitable for field crack detection and easy to carry. In crack detection, the convolutional and normalization layers of the YOLOv3-Tiny algorithm are merged, and the convolution kernel and results are quantized. This optimized YOLOv3-Tiny algorithm has lower computational load, faster running speed, and better target detection performance. It can be better deployed on edge devices using ZYNQ processors, achieving true lightweighting of the crack detection system and device, making it better suited for outdoor engineering crack detection applications and portability. Attached Figure Description

[0053] Figure 1 This is a flowchart of a crack detection method based on an edge-end device according to the present invention;

[0054] Figure 2 This is a block diagram of a crack detection system based on an edge-end device according to the present invention;

[0055] Figure 3 Diagram illustrating the expansion of data channels;

[0056] Figure 4 This is a schematic diagram of multi-channel convolution calculation;

[0057] Figure 5 This is a schematic diagram of the system's multi-channel parallel computing.

[0058] Figure 6 This is a crack detection effect diagram in one embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the system data flow;

[0060] Figure 8 This is a diagram illustrating the operation of a ping-pong game.

[0061] Figure 9 Power consumption diagram for parallel computation of 8 input channels and 8 convolutional kernel channels;

[0062] Figure 10 The power consumption diagram for parallel computation of 16 input channels and 16 convolutional kernel channels. Detailed Implementation

[0063] The preferred embodiments of the present invention will now be described with reference to the accompanying drawings and examples. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. These embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the disclosure to those skilled in the art. Furthermore, the embodiments and features described herein can be combined with each other without conflict.

[0064] Please see Figures 1-10 Firstly, such as Figure 1 As shown, the present invention provides a crack detection method based on an edge-end device, comprising the following steps:

[0065] S1. Establish a YOLOv3-Tiny model, use a camera to collect image frames of the crack to be detected, and preprocess the image frames to obtain preprocessed data.

[0066] S2. The preprocessed data is matrix-constructed, and convolutional computation is used to extract image features, yielding the corresponding feature data. A multi-channel input data matrix and convolutional kernel matrix are constructed to meet the requirements of multi-channel parallel computation. In terms of computation method, the convolutional layer and batch normalization layer are combined for computation, reducing the inference steps while extracting important feature data. In the convolutional computation, the system also employs DSP reuse to improve resource utilization.

[0067] S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number.

[0068] S4. Input the quantized fixed-point data into the LeakyReLU function for nonlinear data transformation to complete the activation process and improve the nonlinear expressive power of the model.

[0069] S5. Construct a 2x2 data matrix, divide the nonlinearly transformed data into blocks, use the max pooling method to compress the nonlinearly transformed data, finally remove the remaining data, retain the largest feature data in the matrix, and obtain the compressed feature data.

[0070] S6. Traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is a multi-layer feature fusion layer. If so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the image feature extraction; otherwise, execute S7.

[0071] S7. Determine whether the current layer is the output layer. If yes, proceed to step S8; otherwise, return to step S2 to continue extracting features from the image.

[0072] S8. The feature data of the output layer is parsed to obtain the target box information containing crack location and size information. Specifically, the results obtained by the forward inference of the system are parsed, the score value of each target box is calculated, the score values ​​of each target box are compared, the four highest score values ​​and their corresponding target box information are retained, and non-maximum suppression is performed to obtain useful target box location information. Finally, the restored target box information is transmitted to the PL end.

[0073] S9. Convert the target box information into RGB data, construct a visual target box, and mark the detected cracks as target boxes on the image frame of the crack to be detected captured by the camera for visualization.

[0074] The preferred method for preprocessing image frames is as follows:

[0075] First, the image frames to be detected captured by the camera are resized using downsampling to reduce the image size, ultimately obtaining image data that conforms to the input size of the YOLOv3-Tiny model. Then, the number of data channels in the image frames to be detected is expanded from the original 3 channels to 8 channels to achieve module reuse. The channel expansion effect is as follows: Figure 3 As shown.

[0076] Preferably, in step S2, the image feature extraction of the preprocessed data using convolution calculation includes the following steps:

[0077] S2.1 Constructing a multi-channel convolution calculation matrix; By constructing a multi-channel input data matrix and a convolution kernel matrix, multi-channel parallel computation is achieved, thereby extracting important features more quickly. In convolution calculation, the corresponding data points need to be multiplied by the weights corresponding to their respective convolution kernels. The convolution calculation is as follows:

[0078]

[0079] In multi-channel computation, each input channel needs to be processed with its corresponding convolutional kernel channel. The computation process is as follows: Figure 4 As shown. To improve inference speed, this system will complete 3x3 convolution calculations for 8 input channels and 8 convolution kernel channels within a single clock cycle. A schematic diagram of the calculation process is shown below. Figure 5 As shown;

[0080] S2.2 Design a DSP multiplexing method to improve resource utilization. First, shift and fill the three input multiplication factors (a, b, c). Second, use the pre-adder in the DSP to process the calculation data and perform multiplication calculation on the processed data. Finally, further process the output of the DSP to obtain the results of a*b and c*b. b is a multiplication factor common to the two multiplication calculations.

[0081] S2.3. After refining the calculation formula for the combined convolutional layer and batch normalized layer, we finally obtain Z = W. i x i +B i And represent it using code, where Z is the result of the convolution calculation, W i For the adjusted weights, x i For input data, B i The adjusted bias term;

[0082] The method of combining convolutional layers and batch normalization layers to reduce the feature extraction steps is summarized as follows:

[0083] Convolution calculation formula:

[0084] y i =w i x i +b i ;y i For the convolution result, w i As the weight, x i For input data, b i For bias.

[0085] Batch standardization formula:

[0086] Sample mean (μ): m: Size of the current batch of data, y i Batch input data.

[0087] Sample variance (σ) 2 ):

[0088] Standardized data ε is a small non-zero value used to prevent the denominator from being 0.

[0089] Translation and scaling: r: Trainable proportional parameter; β: Trainable bias parameter.

[0090] The YOLOv3-Tiny model has no bias (b i Therefore, the combined calculation formula for convolutional layers and BN layers is as follows:

[0091]

[0092] After sorting:

[0093]

[0094] Final formula:

[0095] Z = W i x i +B i ;in:

[0096] S2.4 Complete the multi-channel parallel convolution calculation to obtain the corresponding feature data.

[0097] Preferably, the specific steps of the quantization process in step S3 are as follows:

[0098] S3.1.1. Reorganize the quantitative formula for dot product calculation, and finally obtain:

[0099] Where bias represents the bias used in convolution calculation, S i Z represents the conversion factor, q3 represents the quantized integer data of the multiplication result, q1 and q2 represent the quantized integer data of the multiplication factors, and Z represents the multiplication factor. i This represents the value of 0 in a floating-point number mapped to a fixed-point number.

[0100] S3.1.2 Data Quantization: By determining the level of the input data, the corresponding level's quantization information is loaded. The INT8 quantization algorithm is used to quantize the feature extraction results, obtaining a fixed-point number that meets the system's inference requirements. The quantization formula is as follows:

[0101] r = S*(qZ);

[0102]

[0103] r represents the original floating-point number, S represents the conversion factor, q represents the quantized integer data, Z represents the value of the floating-point number 0 mapped to the fixed-point number, and the round() function has the function of rounding.

[0104] The original model's inference process involves floating-point multiplication. The formula for converting floating-point multiplication to fixed-point multiplication is as follows:

[0105] Calculating the dot product of a 3x3 convolution:

[0106] 1) r3 = r1 * r2 + bias;

[0107] Substituting r = S*(qZ) into 1), we get:

[0108] 2)S3*(q3-Z3)=S1*(q1-Z1)*S2(q2-Z2)+bias;

[0109] Further transformation yields:

[0110] 3)

[0111] r3 represents the result of the floating-point multiplication, r1 and r2 represent the input floating-point number and the weight floating-point number, respectively, bias represents the bias in the convolution calculation, and S i q represents the conversion factor for the corresponding floating-point number. i Z represents the integer data after quantization of the corresponding floating-point number. i This represents the value of a fixed-point number that corresponds to the value of 0 in the floating-point number.

[0112] In the ZYNQ processor, shift operations are faster than multiplication and division, reducing resource consumption and alleviating timing constraints. Therefore, this design will... It is divided into two parts. The first part is 2 to the power of negative n, which facilitates the subsequent processor to complete the shift operation. The second part is M0, which has already been sorted in S1.

[0113] The final optimized quantization formula is as follows:

[0114] in, 2 above -n Processing can be accomplished using bit shifting; M0 can extract information from quantized data, while... For the convolution calculation results, the optimized algorithm can greatly reduce resource consumption and reduce the computation time required;

[0115] S3.1.3 Implement the above algorithm and round the calculation results to reduce the error.

[0116] Preferably, the nonlinear data transformation in step S4 includes tidying up the lookup table and implementing the activation function, with the specific steps as follows:

[0117] S4.1. Using 0 to 255 as input data, and comparing it with the Z-value in each layer of quantization information. i For comparison, if the input data is greater than Z i If the result is positive, then LeakyReLU(x) = 0.1 * x is executed; otherwise, LeakyReLU(x) = x is executed, where x is the input data.

[0118] S4.2 Quantify the above execution results and store them for later use;

[0119] S4.3 When activation is required, the corresponding lookup table is written into the memory, the data to be activated is sent to the memory as the address, and the value output by the memory is the activation result.

[0120] The specific steps for feature data compression in step S5 above are as follows:

[0121] S5.1. By controlling the timing, the data of two adjacent columns in the first row is compared, and the maximum value is retained;

[0122] S5.2. Wait for the comparison of the data in the two adjacent columns of the second row to complete. Compare the maximum value of the two adjacent columns of the second row with the maximum value obtained in the first step to obtain the maximum value of the 2x2 data matrix. During the feature data compression process, pay attention to the step size of max pooling.

[0123] The multi-layer feature fusion in step S6 above includes upsampling and data fusion, and the specific steps are as follows:

[0124] S6.1 Read the data from the 13th layer storage area and use upsampling to expand the data size of this layer;

[0125] S6.2 Read the data from the 8th layer storage area and fuse it with the upsampled data from the first step;

[0126] S6.3 Store the merged data in the new area.

[0127] In step S8 above, the parsing operation includes data parsing, score comparison, maximum value suppression, and target box recovery. The specific steps are as follows:

[0128] S8.1 Read the data from the 15th layer of storage area and calculate the score for each target box;

[0129] S8.2 Compare the score values ​​of each target box in the 15th layer, and retain the 4 maximum score values ​​and their corresponding target box information;

[0130] S8.3 Read the data from layer 23 and calculate the score for each target box;

[0131] S8.4 Compare the score values ​​of each target box in the 23rd layer, and retain the 4 maximum score values ​​and the corresponding target box position information;

[0132] S8.5 Compare the scores retained in the 15th and 23rd layers, and save the four maximum scores and their corresponding bounding boxes.

[0133] S8.6. Perform maximum value suppression on the four maximum score values ​​saved in step five;

[0134] S8.7. Restore the bounding boxes whose scores are greater than the threshold proportionally.

[0135] Secondly, such as Figure 2 As shown, the present invention provides a crack detection system based on an edge-end device, comprising:

[0136] The image acquisition module is used to acquire image frames of the crack to be detected; the image acquisition module consists of an OV5640 camera, which can acquire image information of the crack to be detected at a resolution of 720P or higher.

[0137] The data processing module of the crack detection system is implemented on a processor; in this embodiment, the processor is a ZYNQ processor. The data processing module includes: an image preprocessing module, a feature extraction module, a data quantization module, a data nonlinear transformation module, a feature data compression module, a multi-layer feature fusion module, a data parsing module, and a target bounding box construction module, wherein:

[0138] The image preprocessing module is used to transform the size of the acquired image frames to be detected, reduce the image size using downsampling to meet the input requirements of the model, and expand the number of data channels of the image frames to be detected to meet the module's reuse conditions.

[0139] The feature extraction module is used to construct a matrix from the preprocessed data and use convolution calculation to extract image features from the preprocessed data to obtain the corresponding feature data. By constructing a convolution calculation matrix and using the method of merging convolutional layers and batch normalization layers for calculation, the corresponding feature data can be obtained while reducing the inference steps.

[0140] The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the fixed-point number that satisfies the forward inference of the system.

[0141] The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for nonlinear transformation and activation processing, thereby improving the nonlinear expressive power of the model.

[0142] The feature data compression module is used to construct a 2x2 data matrix, process the nonlinearly transformed data using the max pooling method, and finally retain the largest feature data in the matrix, which is the compressed feature data.

[0143] The multi-layer feature fusion module is used to fuse feature maps from different layers using an upsampling method to obtain fused feature data. For example, the upsampling method can be used to expand the size of the 13th layer data to meet the size requirements for fusion with the 8th layer data.

[0144] The data parsing module is used to parse the feature data of the output layer to obtain target bounding box information containing crack location and size information. After the model inference is completed, the PS terminal reads the data in DDR3, parses the target bounding box location and size information, and sends it to the PL terminal.

[0145] The target bounding box construction module is used by the PL end to convert the target bounding box information into RGB data, construct a visual target bounding box, and mark the detected cracks as target bounding boxes on the image frame of the crack to be detected captured by the camera, and display it visually on the LCD screen.

[0146] The final crack detection effect diagram of this system implementation is shown below. Figure 6 As shown in the figure, the crack detection method based on edge-end devices of this invention can accurately detect cracks. The advantages of the system are further elaborated below:

[0147] To improve system scalability, this system stores the model's hierarchical information on the PS (Power Supply) terminal. By calling and parsing this hierarchical information, the PL (Power Logic Terminal) terminal is controlled to complete calculations for different layers. If a different model needs to be used later, only the hierarchical structure on the PS terminal needs to be modified.

[0148] Figure 7 This describes the data transmission direction during model inference. First, camera data is stored in DDR3 as a frame buffer. Then, the PS (PS terminal) sends the stored camera data to the PL (PListor) for computation. The PL's computation result is then transmitted back to DDR3 for storage. This process is repeated according to the PS's scheduling information, and finally, the output layer data is parsed at the PS. In the data storage scheme, this invention uses a ping-pong operation method to complete data storage and retrieval. A schematic diagram of the ping-pong operation is shown below. Figure 8 As shown in the diagram, the input data required for the nth layer calculation is read from region 2, and the calculation result is written to region 1. When the system reaches the (n+1)th layer, it reads the required input data from region 1, and the calculation result is written to region 2. This ping-pong operation improves the utilization of storage space while ensuring correct data reading.

[0149] Model inference requires a large number of multiplication calculations, making DSP resources particularly important. To adapt to the low-to-medium performance ZNYQ chip, this system employs a parallel computing scheme with 8 input channels and 8 convolutional kernel channels. This scheme achieves an inference time of 194ms and a total on-chip power consumption of 2.651W. The power consumption results are as follows... Figure 9As shown. When the processor's DSP resources are greater than 310, the system can be directly used for crack detection. When the processor's DSP resources are insufficient, it is necessary to reduce the number of parallel computing channels or use logic resources instead of DSP resources. To improve inference speed, the number of parallel computing channels can be increased. When the system uses a scheme of 16 input channels and 16 convolution kernel channels for parallel computing, the inference time will be reduced to 86ms, and the total on-chip power consumption of the system is 3.331W. The power consumption results are shown in the figure. Figure 10 As shown.

[0150] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit of the present invention should fall within the patent scope covered by the present invention.

Claims

1. A crack detection method based on edge-end devices, characterized in that, Includes the following steps: S1. Establish a YOLOv3-Tiny model, collect image frames of the crack to be detected, and preprocess the image frames to obtain preprocessed data; S2. Construct a matrix from the preprocessed data and use convolution to extract image features from the preprocessed data to obtain the corresponding feature data. S3. Determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number. S4. Input the quantized data into the LeakyReLU function for nonlinear data transformation; S5. Construct a 2x2 data matrix and use max pooling to process the nonlinearly transformed data to obtain compressed feature data. S6. Traverse each network layer of the YOLOv3-Tiny model, determine whether the current layer is a multi-layer feature fusion layer. If so, perform upsampling and fusion operations on the compressed feature data, and then return to S2 to continue to complete the image feature extraction; otherwise, execute S7. S7. Determine whether the current layer is the output layer. If yes, proceed to step S8; otherwise, return to step S2 to continue extracting features from the image. S8. Parse the data of the output layer to obtain the target bounding box information containing crack location and size information; S9. Convert the target box information into RGB data, construct a visual target box, and mark the detected cracks as target boxes on the image frame of the crack to be detected captured by the camera for visualization.

2. The crack detection method based on edge-end devices according to claim 1, characterized in that, The specific method for preprocessing image frames is as follows: First, the image frames to be detected captured by the camera are resized by downsampling to reduce the image size, ultimately obtaining image data that conforms to the input size of the YOLOv3-Tiny model. Then, the number of data channels of the image frames to be detected is increased to achieve the condition of module reuse.

3. The crack detection method based on edge-end devices according to claim 1, characterized in that, In step S2, the image feature extraction of the preprocessed data using convolution calculation includes the following steps: S2.1 Construct a multi-channel convolution calculation matrix; S2.2 Design a DSP multiplexing method, which shifts and fills the three input multiplication factors a, b, and c, then uses the pre-adder in the DSP to process the calculation data, and performs multiplication calculation on the processed data. Finally, further processing is performed on the DSP output to obtain the results a*b and c*b, where b is a common multiplication factor in the two multiplication calculations. S2.

3. After refining the calculation formula for the combined convolutional layer and batch normalized layer, we finally obtain Z = W. i x i +B i Where Z is the result of the convolution calculation, and W i For the adjusted weights, x i For input data, B i The adjusted bias term; S2.4 Complete the multi-channel parallel convolution calculation to obtain the corresponding feature data.

4. The crack detection method based on edge-end devices according to claim 1, characterized in that, The specific steps of the quantization process in step S3 are as follows: S3.1.

1. Reorganize the quantitative formula for dot product calculation, and finally obtain: Where bias represents the convolution calculation bias, q3 represents the quantized integer data of the multiplication calculation result, and q1 and q2 represent the quantized integer data of the multiplication factors. S3.1.2 Optimize the quantization algorithm. The optimized formula is as follows: in, 2 above -n Processing can be accomplished using bit shifting; M0 can extract information from quantized data, while... This is the result of the convolution calculation; S3.1.3 Implement the above algorithm and round the calculation results to reduce the error.

5. The crack detection method based on edge-end devices according to claim 4, characterized in that, The nonlinear data transformation in step S4 includes tidying up the lookup table and implementing the activation function. The specific steps are as follows: S4.

1. Take 0 to 255 as input data and compare it with Z3 in each layer of quantization information. If the input data is greater than Z3, then execute LeakyReLU(x) = 0.1 * x; otherwise, execute LeakyReLU(x) = x, where x is the input data. S4.2 Quantify the above execution results and store them for later use; S4.3 When activation is required, the corresponding lookup table is written into the memory, the data to be activated is sent to the memory as the address, and the value output by the memory is the activation result.

6. A crack detection system based on edge-end devices, characterized in that, include: The image acquisition module is used to acquire image frames of the crack to be detected; The image preprocessing module is used to resize the acquired image frames to be detected, reduce the image size using downsampling, and increase the number of data channels of the image frames to be detected. The feature extraction module is used to construct a matrix from the preprocessed data and use convolution calculation to extract image features from the preprocessed data to obtain the corresponding feature data. The data quantization module is used to determine the level of the input feature data, load the quantization information of the corresponding level, and use the INT8 quantization algorithm to quantize the feature extraction results to obtain the corresponding fixed-point number. The data nonlinear transformation module is used to input the quantized data into the LeakyReLU function for data nonlinear transformation; The feature data compression module is used to construct a 2x2 data matrix and process the nonlinearly transformed data using the max pooling method to obtain compressed feature data. The multi-layer feature fusion module is used to fuse feature maps at different levels through upsampling to obtain fused feature data. The data parsing module is used to parse the feature data of the output layer to obtain target bounding box information containing crack location and size information; The target bounding box construction module is used to convert target bounding box information into RGB data, construct visual target bounding boxes, and mark the detected cracks as target bounding boxes on the image frames of the cracks to be detected captured by the camera for visualization.

Images