Intelligent recording and analyzing system and method for whole process of structure test

Abstract

The invention relates to a structure test whole process intelligent recording and analyzing system and method, and belongs to the technical field of structure test monitoring. The system is based on computer vision and comprises a camera, and a video acquisition module, a crack positioning module, a crack segmentation module, a parameter measurement module and a crack tracking module which are in communication connection with the camera, in addition, a corresponding structure test whole process intelligent recording analysis method is provided, and the recording analysis method sequentially comprises a crack positioning method, a crack segmentation method and a crack tracking method. According to the method, the whole damage development process of the test component can be intuitively mastered, including same-frequency display and real-time recording, and the method has a wide application prospect.

Description

Technical Field

[0001] The invention relates to a structural test recording method, in particular to a computer vision-based structural test full-process intelligent recording and analysis system and method, belonging to the technical field of structural test monitoring. Background Art

[0002] The measurement and analysis of cracks in structural components in civil engineering structural tests is an important part of structural damage identification and assessment. During the test, recording the cracking and damage of components can serve as an important basis for studying structural performance and behavior. For example, for reinforced concrete structures, the appearance and development of cracks can be tracked throughout the test to explore their expansion patterns and rates, and thus predict the durability and remaining life of the structure.

[0003] Although structural engineering tests are widely carried out, the test process and test result records still have the following deficiencies:

[0004] (1) Crack inspection of typical reinforced concrete components is still marked by hand-drawing. As the size and number of cracks continue to develop and evolve during the test, the manual marking method consumes a lot of working time of the test personnel and has low work efficiency.

[0005] (2) The results of traditional artificial crack inspection are not objective enough. The inspection accuracy of fine cracks is low, the crack recording error is large, and it is impossible to accurately indicate the size and direction of the cracks.

[0006] (3) During the crack inspection process, the components are still under load. Especially when the components are close to failure, they may experience drastic changes in deflection and deformation. Manual inspection and recording have high risks.

[0007] In addition, the test output of traditional reinforced concrete structural components is usually a load-displacement curve or a load-strain curve. Although this method can reflect the mechanical properties of the components, it cannot achieve the same-frequency display and real-time recording of the actual changes in the properties of the components during loading, and lacks intuitive grasp of the entire process of overall damage development of the test components. Summary of the invention

[0008] The purpose of the present invention is to propose a computer vision-based intelligent recording and analysis system and method for the entire process of structural testing, which is used to solve or overcome the above-mentioned technical problems existing in the existing methods, namely, to intuitively grasp the entire process of overall destruction development of the test component, including same-frequency display and real-time recording.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] The whole process intelligent recording and analysis system of the structural test is based on computer vision, including a camera and a video acquisition module, a crack location module, a crack segmentation module, a parameter measurement module and a crack tracking module connected to the camera;

[0011] The camera records the test process based on the following processing method: in order to meet the crack recognition capability of the component, multiple cameras are connected in parallel, and the number of cameras is configured as needed for the entire test component and key parts, the shooting parameters are adjusted, and the supporting fill-in lighting conditions are set;

[0012] The video acquisition module is used to obtain video images of the entire test process based on the following processing: time code synchronization of multiple cameras is performed to accurately trace the test process from multiple angles; after unifying the clock source, the same instruction is used to trigger different cameras to image and collect data simultaneously on the same timeline; finally, video fusion is performed to seamlessly splice video streams from different cameras to provide more comprehensive test scene coverage and richer structural state change information;

[0013] The crack location module is used to identify component cracks based on the following processing: the test monitoring video acquired by the video acquisition module is processed frame by frame, and the monitoring image is analyzed using an improved YOLOv8 crack detection model to quickly locate the crack position of the test component;

[0014] The crack segmentation module is used to extract crack information in a refined manner based on the following processing: for cracks at key positions of the located test components, the improved DeeplabV3+ crack segmentation model is used to perform pixel-level crack segmentation to eliminate the interference of background noise and improve the precision and accuracy of crack identification;

[0015] The parameter measurement module is used to quantitatively analyze crack parameters based on the following processing: first, a scaling factor is defined to convert the pixel scale of the crack into a real physical scale; the physical length, width and area of ​​the crack are calculated based on the length, width, area and scaling factor of the crack at the pixel level, and the crack distribution density is calculated based on the total surface area of ​​the crack and the total area of ​​the surface image of the test component;

[0016] The crack tracking module is used to dynamically analyze the crack evolution process based on the following processing: after the crack is identified and the feature is extracted by the model, a motion model is established based on the DeepSORT method to synchronize it with the test loading process. During the tracking process, a matching matrix between cracks is constructed and the Hungarian algorithm is used to ensure the accuracy of cross-frame crack tracking. Finally, the crack extension path is presented in a visual manner, and time series data of crack features changing over time is generated to provide support for the analysis and prediction of crack extension laws. Among them, the identification of cracks refers to the detection by the crack positioning module and the segmentation by the crack segmentation module, and the feature extraction refers to the length and width features of the cracks obtained by the parameter measurement module.

[0017] Furthermore, the camera parallel connection method and data transmission method use WiFi technology, Zigbee technology or Bluetooth technology; for data storage, the video data of multiple cameras can be stored in the same storage device in chronological order, or a distributed storage method can be used to store the data of some cameras in the local device and the rest in the cloud server to improve data security and availability.

[0018] Furthermore, the number of cameras is determined according to the stress characteristics of different test components, so as to fully obtain the changes of the components throughout the process. The installation method can use a tripod or fix it on the surrounding walls. The shooting parameters are determined according to actual needs. The resolution, frame rate, focal length, viewing angle, etc. are based on the ability to accurately capture the cracks of the components; the supporting fill light condition can be a white light fill light. The angle of the fill light needs to match the shooting angle of the camera to ensure that the fill light area is the area photographed by the camera to avoid uneven lighting or no light in some areas.

[0019] Furthermore, in the video acquisition module, the time code synchronization method adopts the NTP network time protocol. After each camera is connected to the network, the accurate time is obtained by configuring the NTP server, and then the camera sends a request to the server regularly to obtain the latest time and calibrate it. On this basis, the trigger script is written in Python language, the IP addresses or device identifiers of all cameras are looped, and the network communication library and the camera API are used to send instructions to start recording to multiple cameras at the same time;

[0020] Based on the above-mentioned time synchronization method, video acquisition is performed. After the acquisition is completed, the videos of each camera are spliced ​​and fused to represent the changes of components in the same time. First, the video is decomposed into frames, and then a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm is used to extract features for each frame, and matching points between different frames are found. The frames are spliced ​​together by calculating the transformation matrix, and the spliced ​​frames are recombined into a video.

[0021] Furthermore, in the parameter measurement module, the proportional factor α is defined as follows:

[0022] α= d real

[0023] d pixel

[0024] Among them, d real is the actual physical size of the test component, d pixel is the pixel size of the corresponding component in the image, then the physical size of the crack is calculated as:

[0025] D real =αD pixel

[0026] Among them, D can represent length, width and area respectively;

[0027] The crack image after model recognition is converted into a binary image, with white set as the crack and black set as the background. The area of ​​the crack is calculated by counting the total number of white pixels in the crack area in the binary image; the crack length uses the crack skeletonization algorithm, which is an iterative refinement algorithm or a distance transformation method. The crack area is extracted as a single-pixel wide line and the total pixel length of the skeleton line is calculated; the crack width is calculated by the closest distance between the crack area and the skeleton; the crack distribution density is obtained by dividing the total area of ​​the crack by the total area of ​​the apparent image of the test component.

[0028] A method for intelligently recording and analyzing the entire process of a structural test adopts the above-mentioned intelligent recording and analyzing system for the entire process of a structural test. The method for intelligently recording and analyzing the entire process of a structural test sequentially comprises a crack locating method, a crack segmenting method and a crack tracking method.

[0029] Furthermore, the crack locating method comprises the following steps:

[0030] S11. Construct a crack dataset based on the previously collected open source crack images and test monitoring images; perform geometric transformation, color transformation and noise addition on the crack images to expand the number of datasets and increase the diversity of data;

[0031] S12. Annotate the data set with a bounding box, clearly identify the location of the cracks, and indicate the crack category and bounding box coordinates to establish a crack detection data set;

[0032] S13. Improve the YOLOv8 crack detection model algorithm and build a fused CFE module. By adding the lightweight and efficient convolutional neural network FasterNet, the resource consumption of the model is reduced and the running time required for detection is reduced; at the same time, the efficient multi-scale attention module of EMA cross-space learning is introduced to batch reconstruct some channels, making the YOLOv8 model more targeted when extracting crack features;

[0033] S14. Divide the data set and train the model, use distribution focus loss and complete intersection-over-union loss as loss functions, and use precision, recall and average precision to evaluate the model performance;

[0034] S15. According to actual needs, the trained improved YOLOv8 crack detection model is deployed to the camera. Considering the computing power and storage resources of the device, the improved YOLOv8 crack detection model is appropriately compressed and optimized to ensure that the model can run efficiently.

[0035] Furthermore, the crack segmentation method comprises the following steps:

[0036] S21. Based on the above crack dataset, data enhancement method is used to ensure that the images have different crack shapes, sizes, directions and different background environments;

[0037] S22. Annotate the crack image at pixel level, where the crack part is the foreground and the rest is the background, and establish a crack segmentation dataset;

[0038] S23. Improve the DeepLabV3+ model algorithm and use MobilenetV2 to replace the original backbone feature extraction network to reduce the number of model parameters and improve the operation speed; in addition, add the ECANet cross-channel interactive attention module to make the model more focused on crack information and enhance the performance of crack segmentation;

[0039] S24. Divide the data set and train the model, use cross entropy loss and Dice loss as loss functions, and use category average pixel accuracy and average intersection-over-union ratio to evaluate model performance;

[0040] S25. Deploy the improved DeepLabV3+ crack segmentation model trained in S24 into the camera for real-time crack segmentation. At the same time, collect data feedback from the actual test process and continuously optimize the model based on these feedbacks to adapt to different crack segmentation environments and requirements.

[0041] Furthermore, the crack tracking method comprises the following steps:

[0042] S31. Combine the motion information and appearance features of the cracks, and achieve accurate association across time frames based on the DeepSORT method; after identifying and extracting the features of the cracks, use the Kalman filter to establish the motion model of the cracks, and predict the position and size of the cracks in the current frame based on the state of the previous frame to provide a preliminary association reference; at the same time, use the appearance features of the cracks extracted by the above module for embedding; during the tracking process, combine the motion model distance predicted by the Kalman filter and the similarity of the appearance features to build a matching matrix between the cracks; use the Hungarian algorithm to optimally allocate the detection frame and the tracking trajectory to ensure that the same crack remains consistent in consecutive frames;

[0043] S32. After the matching is completed, the state of the crack is dynamically updated, the unmatched cracks are initialized as new trajectories, and finally the tracking number of the crack and its evolution trajectory are output. The crack extension path is presented in a visual way, and the time series data of the crack characteristics changing over time is generated.

[0044] Furthermore, in S11, the geometric transformation method is rotation, flipping, cropping, and scaling; the color transformation method is color adjustment, brightness adjustment, and contrast adjustment; and the noise addition method is Gaussian noise and salt and pepper noise;

[0045] In S12, the Make Sense annotation tool is used to perform bounding box annotation and category annotation on the crack information in the data set, clarify the location of the crack and indicate the crack category and bounding box coordinate content, so as to directly generate the label format required by the YOLO series algorithm and establish a crack detection data set;

[0046] In S13, the FasterNet network is integrated with the EMA attention mechanism to construct a CFE module and apply it to the backbone network and the neck network of the model; wherein FasterNet, as a lightweight and efficient convolutional neural network, is used to reduce the resource consumption of the model and reduce the running time required for detection; EMA, as an efficient multi-scale attention module for cross-space learning, can batch reconstruct some channels, making the model more targeted when extracting crack features;

[0047] In S14, the crack detection data set is divided into a training set and a test set according to a certain ratio, wherein the training set is used to train the model, and the test set is used to evaluate the performance of the model;

[0048] Use distribution focal loss and complete intersection-over-union loss as loss functions;

[0049] The distribution focus loss is described as follows: for a continuous value y∈R, R is a set of real numbers; it is discretized into the integer interval [l,u], where l is the maximum integer not greater than y rounded down; u is the minimum integer not less than y rounded up; assuming that the predicted distribution probability p = {p0, p1, ..., p n}, corresponding to the discrete position {0,1,…,n}, the calculation formula of the distribution focus loss is:

[0050] L DFL =-((uy)log(p l )+(yl)log(p u ))

[0051] Among them, y is the target value, p l and p u is the predicted probability corresponding to the interval [l,u]; uy and yl are weights, reflecting the degree of deviation of the target value within the interval;

[0052] Complete intersection-over-union loss assumes that the predicted box is B p , the target box is B g, the overlapping area is Area, and the Euclidean distance between the center point of the prediction box and the target box is ρ(λ p ,λ g ), the diagonal length of the minimum enclosing box surrounding the prediction box and the target box is d, and the height and width of the prediction box and the target box are h respectively p , w p and h g , w g , the weight factor is α, which is used to balance the influence of the aspect ratio term and other terms. The calculation formula is:

[0053]

[0054] In S14, the model performance is evaluated using precision, recall and average precision.

[0055] TP represents the number of samples that are actually cracks and are correctly detected as cracks, FN represents the number of samples that are actually cracks but are mistakenly detected as background, FP represents the number of samples that are actually background but are mistakenly detected as cracks, and TN represents the number of samples that are actually background and are correctly detected as background.

[0056] Precision refers to the ratio of the number of samples TP detected by the model as actually cracks to the total number of samples detected as cracks TP+FP, expressed as:

[0057]

[0058] The recall rate refers to the ratio of the number of samples TP detected by the model as actually cracks to the number of samples TP+FN as actually cracks, expressed as:

[0059]

[0060] The average precision is the area below the precision-recall curve. The higher the value, the easier it is for the model to maintain high precision at a high recall rate. The calculation formula is:

[0061]

[0062] In the S15, after the model is iteratively trained and its performance meets the actual requirements, it is packaged using TorchScript or TensorFlow Lite and deployed in the camera for execution;

[0063] In the S23, firstly, a lightweight neural network MobileNetV2 is used to replace the backbone feature extraction network Xception in the original DeepLabV3+ semantic segmentation model, so as to effectively reduce the number of parameters and balance the calculation accuracy and speed, so as to make the model more suitable for embedded devices such as cameras; secondly, an attention module ECANet is added after the ASPP void space pyramid pooling module extracts features and before the low-order features are spliced, so that the model can better focus on the crack information of the test component, suppress invalid and complex background features, and improve the accuracy of the model;

[0064] In S24, the cross entropy loss L CE Measures the difference between the predicted probability distribution and the true distribution, calculated as:

[0065]

[0066] Where N is the total number of pixels in the image, C is the number of categories, and y i,c is the true label of the i-th pixel belonging to category c, is the predicted probability that the i-th pixel belongs to category c;

[0067] Dice loss is based on the Dice similarity coefficient L Dice , which is used to measure the overlap between the prediction and the true segmentation. The formula is:

[0068]

[0069] Among them, p i is the predicted probability that the i-th pixel belongs to a crack, g i is the true label of the i-th pixel belonging to the crack;

[0070] In S24, the category average pixel accuracy MPA is used to measure the proportion of pixels in each category that are correctly classified, and then the average is calculated for all categories:

[0071]

[0072] Among them, TP c Indicates the number of pixels whose true label is category c and whose prediction is also category c, FN c Indicates the number of pixels whose true label is category c but is predicted to be other categories;

[0073] The mean intersection over union (MIoU) measures the ratio of the intersection and union of the predicted result to the true segmentation, averaging all categories:

[0074]

[0075] Among them, FP cIndicates the number of pixels whose predicted label is category c but whose true label is not c;

[0076] After the model’s performance reaches the expected performance on the crack segmentation dataset, the model size is reduced and the inference speed is improved through pruning and quantization methods. Then, the model is converted into ONNX format or TensorRT format suitable for deployment, packaged and deployed in the camera for execution;

[0077] In S31, the DeepSORT crack tracking method mainly refers to combining a simple and efficient Kalman filter motion model and deep feature measurement to perform real-time crack extension state tracking;

[0078] Kalman filtering assumes that the crack state is a vector x k , the observed value is z k , then the change of crack state is described by the state transfer equation:

[0079] x k =F k x k-1 +B k u k +w k

[0080] Among them, x k represents the state vector of the crack at time k, F k is the state transfer matrix, u k Control input vector, B k Control input matrix, w k is the process noise;

[0081] The motion model of the crack uses Kalman filtering to model the geometric characteristics and position state of the crack: the state vector is defined as the center point position (x, y) of the crack, the size of the bounding box and its change speed; the Kalman filter predicts the possible position and size of the crack in the next frame based on the state of the current frame, providing a reference for the association;

[0082] The crack matching and association method is that in each frame of the image, the new crack detected needs to be matched with the crack tracked in the previous frame, which is achieved by combining distance measurement;

[0083] The distance measurement uses motion model distance and appearance feature distance; the motion model distance calculates the Euclidean distance between the state predicted by Kalman filter and the actual detection result, and the appearance feature distance uses the extracted feature embedding to calculate the cosine distance or Euclidean distance between the crack features of the current frame and the previous frame;

[0084] The above two distances are weighted and fused to construct a matching matrix. The optimal allocation of crack recognition results and tracking trajectories is achieved through the Hungarian algorithm, that is, a set of minimum matching pairs is found on the matching matrix, so that each trajectory matches at most one detection frame, and each detection frame matches at most one trajectory; unmatched detection frames will be initialized as new crack trajectories;

[0085] In S32, the dynamic update method updates the position, size and other states and appearance features of the successfully matched crack trajectory to keep it consistent with the actual state of the crack. For the unmatched crack trajectory, it continues to track according to the prediction of the Kalman filter. If no new detection results are matched for a period of time, it is considered that the crack has stopped expanding;

[0086] After tracking is completed, the dynamic trajectory of the crack is visualized and output in the following ways: displaying the crack number and trajectory line on the original image, drawing curves of crack width and length changing with time, and generating an animation of crack expansion to demonstrate its dynamic evolution process.

[0087] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0088] The present invention can effectively realize the same-frequency display and real-time recording of the actual property changes of the component during the loading process, and can intuitively grasp the whole process of the overall destruction development of the test component. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a schematic diagram of the structure of an embodiment of the present invention;

[0090] Figure 2 It is a schematic diagram of the arrangement of the test component and the camera of the present invention;

[0091] Figure 3 This is a schematic diagram of the improved YOLOv8 crack detection network of the present invention;

[0092] Figure 4 This is a schematic diagram of the improved DeepLabV3+ crack segmentation network of the present invention;

[0093] Figure 5 This is the core flow chart of the crack dynamic tracking method of the present invention. DETAILED DESCRIPTION

[0094] The following is combined with Figure 1-5 The present invention is further described in detail with specific implementations to facilitate a clear understanding of the present invention, but they do not constitute a limitation on the present invention.

[0095] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0096] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0097] As attached Figure 1-2 As shown, the intelligent recording and analysis system for the whole process of structural test of this embodiment is based on computer vision and includes a camera 1 and a video acquisition module, a crack positioning module, a crack segmentation module, a parameter measurement module and a crack tracking module connected to the camera 1;

[0098] The camera 2 records the test process based on the following processing method: in order to meet the crack recognition capability of the component, multiple cameras 1 are connected in parallel, and the number of cameras is configured as needed for the entire test component 2 and key parts, the shooting parameters are adjusted, and the matching fill light conditions are set; Figure 2 As shown, the parallel connection mode of the camera 1 and the data transmission method use WiFi technology, Zigbee technology or Bluetooth technology; for data storage, the video data of multiple cameras 1 can be stored in the same storage device, and the storage method is in chronological order, or a distributed storage method is adopted to store the data of some cameras in the local device, and the rest are stored in the cloud server to improve the security and availability of the data. In this embodiment, the number of cameras 1 is determined according to the stress characteristics of different test components 2, so as to fully obtain the changes of the components throughout the process. The installation method can use a tripod or fix it on the surrounding walls. The shooting parameters are determined according to actual needs, and the resolution, frame rate, focal length, viewing angle, etc. are based on the ability to accurately capture the cracks of the components; the supporting fill light condition can be a white light fill light, and the angle of the fill light needs to match the shooting angle of the camera to ensure that the fill light area is the area shot by the camera to avoid uneven lighting or no light in some areas.

[0099] The video acquisition module is used to obtain video images of the entire test process based on the following processing: time code synchronization of multiple cameras 1 is performed to accurately trace the test process from multiple angles; after unifying the clock source, the same instruction is used to trigger different cameras 1 to form images, and data is collected simultaneously on the same timeline; finally, video fusion is performed to seamlessly splice video streams from different cameras 1 to provide more comprehensive test scene coverage and richer structural state change information; wherein, in the video acquisition module, the time code synchronization method adopts the NTP network time protocol, after each camera 1 is connected to the network, the accurate time is obtained by configuring the NTP server, and then the camera 1 sends a request to the server at regular intervals to obtain the latest time and calibrate; on this basis, a trigger script is written in Python language, the IP addresses or device identifiers of all cameras 1 are looped through, and the network communication library and the API of the camera 1 are used to send instructions to start recording to multiple cameras 1 at the same time;

[0100] Based on the above-mentioned time synchronization method, video acquisition is performed. After the acquisition is completed, the videos of each camera 1 are spliced ​​and fused to represent the changes of the components in the same time; first, the video is decomposed into frames, and then a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm is used to extract features for each frame, and matching points between different frames are found. The frames are spliced ​​together by calculating the transformation matrix, and the spliced ​​frames are recombined into a video.

[0101] The crack locating module is used to identify component cracks based on the following processing: processing the test monitoring video acquired by the video acquisition module frame by frame, and analyzing the monitoring image using the improved YOLOv8 crack detection model to quickly locate the crack position of the test component 2;

[0102] The crack segmentation module is used to extract crack information in a refined manner based on the following processing: for cracks at key positions of the located test component 2, pixel-level crack segmentation is performed using an improved DeeplabV3+ crack segmentation model to eliminate interference from background noise and improve the precision and accuracy of crack identification;

[0103] The parameter measurement module is used to quantitatively analyze crack parameters based on the following processing: first, a scaling factor is defined to convert the pixel scale of the crack into a real physical scale; the physical length, width and area of ​​the crack are calculated based on the length, width, area and scaling factor of the crack at the pixel level, and the crack distribution density is calculated based on the total surface area of ​​the crack and the total area of ​​the surface image of the test component;

[0104] The definition method of the proportional factor α is as follows:

[0105]

[0106] Among them, d real is the actual physical size of the test component, d pixel is the pixel size of the corresponding component in the image, then the physical size of the crack is calculated as:

[0107] D real =αD pixel

[0108] Among them, D can represent length, width and area respectively;

[0109] The crack image after model recognition is converted into a binary image, with white set as the crack and black set as the background. The area of ​​the crack is calculated by counting the total number of white pixels in the crack area in the binary image; the crack length uses the crack skeletonization algorithm, which is an iterative refinement algorithm or a distance transformation method. The crack area is extracted as a single-pixel wide line and the total pixel length of the skeleton line is calculated; the crack width is calculated by the closest distance between the crack area and the skeleton; the crack distribution density is obtained by dividing the total area of ​​the crack by the total area of ​​the apparent image of the test component.

[0110] The crack tracking module is used to dynamically analyze the crack evolution process based on the following processing: after the crack is identified and the feature is extracted by the model, a motion model is established based on the DeepSORT method to synchronize it with the test loading process. During the tracking process, a matching matrix between cracks is constructed and the Hungarian algorithm is used to ensure the accuracy of cross-frame crack tracking. Finally, the crack extension path is presented in a visual way, and time series data of crack features changing over time is generated to provide support for the analysis and prediction of crack extension laws. Among them, the identification of cracks refers to the detection by the crack positioning module and the segmentation by the crack segmentation module, and the feature extraction refers to the length and width features of the cracks obtained by the parameter measurement module.

[0111] Example 2

[0112] In the present embodiment, a method for intelligently recording and analyzing the entire process of a structural test is provided, which adopts various modules of the system for intelligently recording and analyzing the entire process of a structural test in Embodiment 1. The method for intelligently recording and analyzing the entire process of a structural test includes, in sequence, a crack locating method, a crack segmenting method, and a crack tracking method.

[0113] Wherein, the crack locating method comprises the following steps:

[0114] S11. Construct a crack dataset based on the previously collected open source crack images, such as CFD datasets or SDNET datasets, and test monitoring images; perform geometric transformation, color transformation, and noise addition on the crack images to expand the number of datasets and increase the diversity of data; the geometric transformation methods include rotation, flipping, cropping, and scaling; the color transformation methods include color adjustment, brightness adjustment, and contrast adjustment; and the noise addition methods include Gaussian noise and salt and pepper noise;

[0115] S12. Annotate the dataset with bounding boxes, clarify the locations of cracks, indicate the crack categories and bounding box coordinates, and establish a crack detection dataset; specifically, use the Make Sense annotation tool to annotate the crack information in the dataset with bounding boxes and categories, clarify the locations of cracks, indicate the crack categories and bounding box coordinates, so as to directly generate the label format required by the YOLO series algorithm and establish a crack detection dataset;

[0116] S13. Improve the YOLOv8 crack detection model algorithm and build a fused CFE module. By adding the FasterNet lightweight and efficient convolutional neural network, the resource consumption of the model is reduced and the running time required for detection is reduced; at the same time, the EMA cross-space learning efficient multi-scale attention module is introduced to batch reconstruct some channels, making the YOLOv8 model more targeted when extracting crack features; in S13, Figure 3 As shown in the figure, the FasterNet network is integrated with the EMA attention mechanism to construct the CFE module and apply it to the backbone network and neck network of the model. Among them, FasterNet, as a lightweight and efficient convolutional neural network, is used to reduce the resource consumption of the model and reduce the running time required for detection. EMA, as an efficient multi-scale attention module for cross-space learning, can batch reconstruct some channels, making the model more targeted when extracting crack features.

[0117] S14. Divide the data set and train the model, use distribution focus loss and complete intersection-over-union loss as loss functions, and use precision, recall and average precision to evaluate the model performance;

[0118] In S14, the crack detection data set is divided into a training set and a test set according to a certain ratio (9:1 / 8:2 / 7:3), wherein the training set is used to train the model, and the test set is used to evaluate the performance of the model;

[0119] Distribution Focal Loss and Complete Intersection over Union Loss are used as loss functions;

[0120] The distribution focus loss (DFL) is described as follows: for a continuous value y∈R, R is a set of real numbers; discretize it into the integer interval [l,u], where Assume that the predicted distribution probability p = {p0,p1,…,p n}, corresponding to the discrete position {0,1,…,n}, the calculation formula of the distribution focus loss is:

[0121] L DFL =-((uy)log(p l )+(yl)log(p u ))

[0122] Among them, y is the target value, p l and p u is the predicted probability corresponding to the interval [l,u]; uy and yl are weights, reflecting the degree of deviation of the target value within the interval;

[0123] Complete Intersection over Union Loss (CIoUL) Assume that the predicted box is B p , the target box is B g , the overlapping area is Area, and the Euclidean distance between the center point of the prediction box and the target box is ρ(λ p ,λ g ), the diagonal length of the minimum enclosing box surrounding the prediction box and the target box is d, and the height and width of the prediction box and the target box are h respectively p , w p and h g , w g , the weight factor is α, which is used to balance the influence of the aspect ratio term and other terms. The calculation formula is:

[0124]

[0125] In S14, the model performance is evaluated using precision, recall, and average precision (AP).

[0126] TP represents the number of samples that are actually cracks and are correctly detected as cracks, FN represents the number of samples that are actually cracks but are mistakenly detected as background, FP represents the number of samples that are actually background but are mistakenly detected as cracks, and TN represents the number of samples that are actually background and are correctly detected as background.

[0127] Precision refers to the ratio of the number of samples TP detected by the model as actually cracks to the total number of samples detected as cracks TP+FP, expressed as:

[0128]

[0129] The recall rate refers to the ratio of the number of samples TP detected by the model as actually cracks to the number of samples TP+FN as actually cracks, expressed as:

[0130]

[0131] The average precision is the area below the precision-recall curve. The higher the value, the easier it is for the model to maintain high precision at a high recall rate. The calculation formula is:

[0132]

[0133] S15. According to actual needs, deploy the trained YOLOv8 crack detection model to the camera, consider the computing power and storage resources of the device, and appropriately compress and optimize the model to ensure that the model can run efficiently. That is, after the model is iteratively trained and its performance meets actual needs, use TorchScript or TensorFlow Lite to encapsulate it and deploy it in the camera (1) for execution.

[0134] In this embodiment, the crack segmentation method comprises the following steps:

[0135] S21. Based on the above crack dataset, data enhancement method is used to ensure that the images have different crack shapes, sizes, directions and different background environments;

[0136] S22. Annotate the crack image at pixel level, where the crack part is the foreground and the rest is the background, and establish a crack segmentation dataset;

[0137] S23. Improve the DeepLabV3+ model algorithm and use MobilenetV2 to replace the original backbone feature extraction network to reduce the number of model parameters and improve the operation speed; in addition, add the ECANet cross-channel interactive attention module to make the model more focused on crack information and enhance the performance of crack segmentation; Figure 4 As shown, in S23, firstly, a lightweight neural network MobileNetV2 is used to replace the backbone feature extraction network Xception in the original DeepLabV3+ semantic segmentation model, so as to effectively reduce the number of parameters and balance the calculation accuracy and speed, so as to make the model more suitable for embedded devices such as cameras; secondly, an attention module ECANet is added after the ASPP void space pyramid pooling module extracts features and before the low-order features are spliced, so that the model can better focus on the crack information of the test component (2), and suppress invalid and complex background features, so as to improve the accuracy of the model;

[0138] S24. Divide the data set and train the model, use cross entropy loss and Dice loss as loss functions, and use category average pixel accuracy (MPA) and mean intersection over union (MIoU) to evaluate model performance; specifically, preferably, use cross entropy loss (Cross-Entropy Loss) and Dice loss (Dice Loss) as loss functions.

[0139] In S24, the cross entropy loss L CE Measures the difference between the predicted probability distribution and the true distribution, calculated as:

[0140]

[0141] Where N is the total number of pixels in the image, C is the number of categories, and y i,c is the true label of the i-th pixel belonging to category c, is the predicted probability that the i-th pixel belongs to category c;

[0142] Dice loss is based on the Dice similarity coefficient L Dice , which is used to measure the overlap between the prediction and the true segmentation. The formula is:

[0143]

[0144] Among them, p i is the predicted probability that the i-th pixel belongs to a crack, g i is the true label of the i-th pixel belonging to the crack;

[0145] In S24, the category average pixel accuracy MPA is used to measure the proportion of pixels in each category that are correctly classified, and then the average is calculated for all categories:

[0146]

[0147] Among them, TP c Indicates the number of pixels whose true label is category c and whose prediction is also category c, FN c Indicates the number of pixels whose true label is category c but is predicted to be other categories;

[0148] The mean intersection over union (MIoU) measures the ratio of the intersection and union of the predicted result to the true segmentation, averaging all categories:

[0149]

[0150] Among them, FP c Indicates the number of pixels whose predicted label is category c but whose true label is not c;

[0151] After the model's performance reaches the expected performance on the crack segmentation dataset, the model size is reduced and the inference speed is improved through pruning and quantization methods, and then converted into ONNX format or TensorRT format suitable for deployment, which is packaged and deployed in the camera (1) for execution;

[0152] S25. Deploy the improved DeepLabV3+ crack segmentation model trained in S24 into the camera for real-time crack segmentation. At the same time, collect data feedback during the actual test process and continuously optimize the model based on these feedbacks to adapt to different crack segmentation environments and requirements.

[0153] Figure 5 A core flow chart of a crack dynamic tracking method is given. The crack tracking method comprises the following steps:

[0154] S31. Combine the motion information and appearance features of the cracks, and achieve accurate association across time frames based on the DeepSORT method; after identifying and extracting the features of the cracks, use the Kalman filter to establish the motion model of the cracks, and predict the position and size of the cracks in the current frame based on the state of the previous frame to provide a preliminary association reference; at the same time, use the appearance features of the cracks extracted by the above module for embedding; during the tracking process, combine the motion model distance predicted by the Kalman filter and the similarity of the appearance features to build a matching matrix between the cracks; use the Hungarian algorithm to optimally allocate the detection frame and the tracking trajectory to ensure that the same crack remains consistent in consecutive frames;

[0155] In S31, the DeepSORT crack tracking method mainly refers to combining a simple and efficient Kalman filter motion model and deep feature measurement to perform real-time crack extension state tracking;

[0156] Kalman filtering assumes that the crack state is a vector x k , the observed value is z k , then the change of crack state is described by the state transfer equation:

[0157] x k =F k x k-1 +B k u k +w k

[0158] Among them, x k represents the state vector of the crack at time k, F k is the state transfer matrix, u k Control input vector, B k Control input matrix, w k is the process noise;

[0159] The motion model of the crack uses Kalman filtering to model the geometric characteristics and position state of the crack: the state vector is defined as the center point position (x, y) of the crack, the size of the bounding box and its change speed; the Kalman filter predicts the possible position and size of the crack in the next frame based on the state of the current frame, providing a reference for the association;

[0160] The crack matching and association method is that in each frame of the image, the new crack detected needs to be matched with the crack tracked in the previous frame, which is achieved by combining distance measurement;

[0161] The distance measurement uses motion model distance and appearance feature distance; the motion model distance calculates the Euclidean distance between the state predicted by Kalman filter and the actual detection result, and the appearance feature distance uses the extracted feature embedding to calculate the cosine distance or Euclidean distance between the crack features of the current frame and the previous frame;

[0162] The above two distances are weighted and fused to construct a matching matrix. The optimal allocation of crack recognition results and tracking trajectories is achieved through the Hungarian algorithm, that is, a set of minimum matching pairs is found on the matching matrix, so that each trajectory matches at most one detection frame, and each detection frame matches at most one trajectory; unmatched detection frames will be initialized as new crack trajectories;

[0163] S32. After the matching is completed, the state of the crack is dynamically updated, and the unmatched cracks are initialized as new trajectories. Finally, the tracking number of the crack and its evolution trajectory are output, and the crack expansion path is presented in a visual way. At the same time, the time series data of the crack characteristics changing over time is generated. Specifically, in S32, the dynamic update method updates the position, size and other states and appearance features of the successfully matched crack trajectory to keep it consistent with the actual state of the crack. For the unmatched crack trajectory, continue to track it according to the prediction of the Kalman filter. If no new detection results are matched for a period of time, it is considered that the crack has stopped expanding;

[0164] After tracking is completed, the dynamic trajectory of the crack is visualized and output in the following ways: displaying the crack number and trajectory line on the original image, drawing curves of crack width and length changing with time, and generating an animation of crack expansion to demonstrate its dynamic evolution process.

[0165] The above is only a preferred embodiment of the present invention, and does not impose any formal limitation on the structure of the present invention. The layout type and the number of uses of the present invention are not limited to this example, and can be optimized and selected according to the actual project. Any modification, equivalent change and decoration of the above embodiment based on the technical principle of the present invention that does not deviate from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. Intelligent recording and analysis system for the whole process of structural test, based on computer vision, characterized by: It comprises a camera (1) and a video acquisition module, a crack positioning module, a crack segmentation module, a parameter measurement module and a crack tracking module which are connected to the camera for communication; The camera (1) records the test process based on the following processing method: in order to meet the crack recognition capability of the component, multiple cameras (1) are connected in parallel, and the number of cameras is configured as needed for the entire test component (2) and key parts, and the shooting parameters are adjusted and the matching fill-in lighting conditions are set; The video acquisition module is used to obtain video images of the entire test process based on the following processing: time code synchronization of multiple cameras (1) arranged to accurately trace the test process from multiple angles; After unifying the clock source, the same command is used to trigger different cameras (1) to image and collect data simultaneously on the same timeline; finally, video fusion is performed to seamlessly splice the video streams from different cameras (1) to provide more comprehensive test scene coverage and richer structural state change information; The crack locating module is used to identify component cracks based on the following processing: processing the test monitoring video acquired by the video acquisition module frame by frame, analyzing the monitoring image using an improved YOLOv8 crack detection model to quickly locate the crack position of the test component (2); The crack segmentation module is used to extract crack information in a refined manner based on the following processing: for cracks at key positions of the located test component (2), pixel-level crack segmentation is performed using an improved DeeplabV3+ crack segmentation model to eliminate interference from background noise and improve the precision and accuracy of crack identification; The parameter measurement module is used to quantitatively analyze crack parameters based on the following processing: first, a scaling factor is defined to convert the pixel scale of the crack into a real physical scale; the physical length, width and area of ​​the crack are calculated based on the length, width, area and scaling factor of the crack at the pixel level, and the crack distribution density is calculated based on the total surface area of ​​the crack and the total area of ​​the surface image of the test component; The crack tracking module is used to dynamically analyze the crack evolution process based on the following processing: after the crack is identified and the feature is extracted by the model, a motion model is established based on the DeepSORT method to synchronize it with the test loading process. During the tracking process, a matching matrix between cracks is constructed and the Hungarian algorithm is used to ensure the accuracy of cross-frame crack tracking. Finally, the crack extension path is presented in a visual manner, and time series data of crack features changing over time is generated to provide support for the analysis and prediction of crack extension laws. Among them, the identification of cracks refers to the detection by the crack positioning module and the segmentation by the crack segmentation module, and the feature extraction refers to the length and width features of the cracks obtained by the parameter measurement module.

2. The intelligent recording and analysis system for the whole process of structural testing according to claim 1 is characterized by: The camera (1) parallel connection mode and data transmission method use WiFi technology, Zigbee technology or Bluetooth technology; for data storage, the video data of multiple cameras (1) can be stored in the same storage device in chronological order, or a distributed storage method is used to store the data of some cameras in a local device and the rest in a cloud server, so as to improve the security and availability of the data.

3. The whole process intelligent recording and analysis system for structural test according to claim 2 is characterized by: The number of cameras (1) is determined according to the stress characteristics of different test components (2), so as to be able to fully obtain the changes of the components throughout the process. The installation method can use a tripod or be fixed on the surrounding walls. The shooting parameters are determined according to actual needs. The resolution, frame rate, focal length, viewing angle, etc. are based on being able to accurately capture the cracks of the components; the matching fill light condition can be a white light fill light. The angle of the fill light needs to match the shooting angle of the camera to ensure that the fill light area is the area shot by the camera to avoid uneven lighting or a lack of light in some areas.

4. The whole process intelligent recording and analysis system for structural test according to claim 3 is characterized by: In the video acquisition module, the time code synchronization method adopts the NTP network time protocol. After each camera (1) is connected to the network, the accurate time is obtained by configuring the NTP server, and then the camera (1) sends a request to the server at a regular interval to obtain the latest time and calibrate it. On this basis, a trigger script is written in Python language to loop through the IP addresses or device identifiers of all cameras (1), and a network communication library and the API of the camera (1) are used to send a command to start recording to multiple cameras (1) at the same time. Based on the above time synchronization method, video acquisition is performed, and after acquisition is completed, the videos of each camera (1) are spliced ​​and merged to represent the changes of the components in the same time; first, the video is decomposed into frames, and then a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm is used to extract features for each frame, and matching points between different frames are found. The frames are spliced ​​together by calculating the transformation matrix, and the spliced ​​frames are reassembled into a video.

5. The whole process intelligent recording and analysis system for structural test according to claim 4 is characterized by: In the parameter measurement module, the proportional factor α is defined as follows: Among them, d real is the actual physical size of the test component, d pixel is the pixel size of the corresponding component in the image, then the physical size of the crack is calculated as: D real =αD pixel Among them, D can represent length, width and area respectively; The crack image after model recognition is converted into a binary image, with white set as the crack and black set as the background. The area of ​​the crack is calculated by counting the total number of white pixels in the crack area in the binary image; the crack length uses the crack skeletonization algorithm, which is an iterative refinement algorithm or a distance transformation method. The crack area is extracted as a single-pixel wide line and the total pixel length of the skeleton line is calculated; the crack width is calculated by the closest distance between the crack area and the skeleton; the crack distribution density is obtained by dividing the total area of ​​the crack by the total area of ​​the apparent image of the test component.

6. A method for intelligently recording and analyzing the entire process of a structural test, which adopts the intelligent recording and analyzing system for the entire process of a structural test according to claim 5, characterized in that: The intelligent recording and analysis method for the whole process of structural testing includes a crack positioning method, a crack segmentation method and a crack tracking method in sequence.

7. The method for intelligent recording and analyzing the entire process of structural testing according to claim 6 is characterized in that: The crack locating method comprises the following steps: S11. Construct a crack dataset based on the previously collected open source crack images and test monitoring images; perform geometric transformation, color transformation and noise addition on the crack images to expand the number of datasets and increase the diversity of data; S12. Annotate the data set with a bounding box, clearly identify the location of the cracks, and indicate the crack category and bounding box coordinates to establish a crack detection data set; S13. Improve the YOLOv8 crack detection model algorithm and build a fused CFE module. By adding the lightweight and efficient convolutional neural network FasterNet, the resource consumption of the model is reduced and the running time required for detection is reduced. At the same time, the efficient multi-scale attention module of EMA cross-space learning is introduced to batch reconstruct some channels, making the YOLOv8 model more targeted when extracting crack features. S14. Divide the data set and train the model, use distribution focus loss and complete intersection-over-union loss as loss functions, and use precision, recall and average precision to evaluate the model performance; S15. According to actual needs, the trained improved YOLOv8 crack detection model is deployed to the camera. Considering the computing power and storage resources of the device, the improved YOLOv8 crack detection model is appropriately compressed and optimized to ensure that the model can run efficiently.

8. The method for intelligent recording and analyzing the entire process of structural testing according to claim 7 is characterized in that: The crack segmentation method comprises the following steps: S21. Based on the above crack dataset, data enhancement method is used to ensure that the images have different crack shapes, sizes, directions and different background environments; S22. Annotate the crack image at pixel level, where the crack part is the foreground and the rest is the background, and establish a crack segmentation dataset; S23. Improve the DeepLabV3+ model algorithm and use MobilenetV2 to replace the original backbone feature extraction network to reduce the number of model parameters and improve the operation speed; in addition, add the ECANet cross-channel interactive attention module to make the model more focused on crack information and enhance the performance of crack segmentation; S24. Divide the data set and train the model, use cross entropy loss and Dice loss as loss functions, and use category average pixel accuracy and average intersection-over-union ratio to evaluate model performance; S25. Deploy the improved DeepLabV3+ crack segmentation model trained in S24 into the camera for real-time crack segmentation. At the same time, collect data feedback from the actual test process and continuously optimize the model based on these feedbacks to adapt to different crack segmentation environments and requirements.

9. The method for intelligent recording and analyzing the entire process of structural testing according to claim 8 is characterized in that: The crack tracking method comprises the following steps: S31. Combine the motion information and appearance features of the cracks, and achieve accurate association across time frames based on the DeepSORT method; after identifying and extracting the features of the cracks, use the Kalman filter to establish the motion model of the cracks, and predict the position and size of the cracks in the current frame based on the state of the previous frame to provide a preliminary association reference; at the same time, use the appearance features of the cracks extracted by the above module for embedding; during the tracking process, combine the motion model distance predicted by the Kalman filter and the similarity of the appearance features to build a matching matrix between the cracks; use the Hungarian algorithm to optimally allocate the detection frame and the tracking trajectory to ensure that the same crack remains consistent in consecutive frames; S32. After the matching is completed, the state of the crack is dynamically updated, the unmatched cracks are initialized as new trajectories, and finally the tracking number of the crack and its evolution trajectory are output. The crack extension path is presented in a visual way, and the time series data of the crack characteristics changing over time is generated.

10. The method for intelligent recording and analyzing the entire process of structural testing according to claim 9 is characterized in that: In S11, the geometric transformation method is rotation, flipping, cropping, and scaling; the color transformation method is color adjustment, brightness adjustment, and contrast adjustment; and the noise addition method is Gaussian noise and salt and pepper noise; In S12, the Make Sense annotation tool is used to perform bounding box annotation and category annotation on the crack information in the data set, clarify the location of the crack and indicate the crack category and bounding box coordinate content, so as to directly generate the label format required by the YOLO series algorithm and establish a crack detection data set; In S13, the FasterNet network is integrated with the EMA attention mechanism to construct a CFE module and apply it to the backbone network and the neck network of the model; wherein FasterNet, as a lightweight and efficient convolutional neural network, is used to reduce the resource consumption of the model and reduce the running time required for detection; EMA, as an efficient multi-scale attention module for cross-space learning, can batch reconstruct some channels, making the model more targeted when extracting crack features; In S14, the crack detection data set is divided into a training set and a test set according to a certain ratio, wherein the training set is used to train the model, and the test set is used to evaluate the performance of the model; Use distribution focal loss and complete intersection-over-union loss as loss functions; The distribution focus loss is described as follows: for a continuous value y∈R, R is a set of real numbers; discretize it into an integer interval [l,u], where l is the maximum integer not greater than y rounded down; u is the minimum integer not less than y rounded up; assuming that the predicted distribution probability p = {p0, p1, ..., p n }, corresponding to the discrete position {0,1,…,n}, the calculation formula of the distribution focus loss is: L DFL =-((u-y)log(p l )+(y-l)log(p u )) Among them, y is the target value, p l and p u is the predicted probability corresponding to the interval [l,u]; uy and yl are weights, reflecting the degree of deviation of the target value within the interval; Complete intersection-over-union loss assumes that the predicted box is B p , the target box is B g , the overlapping area is Area, and the Euclidean distance between the center point of the prediction box and the target box is ρ(λ p ,λ g ), the diagonal length of the minimum enclosing box surrounding the prediction box and the target box is d, and the height and width of the prediction box and the target box are h respectively p , w p and h g , w g , the weight factor is α, which is used to balance the influence of the aspect ratio term and other terms. The calculation formula is: In S14, the model performance is evaluated using precision, recall and average precision. Precision refers to the ratio of the number of samples TP detected by the model as actually cracks to the total number of samples detected as cracks TP+FP, expressed as: The recall rate refers to the ratio of the number of samples TP detected by the model as actually cracks to the number of samples TP+FN as actually cracks, expressed as: The average precision is the area below the precision-recall curve. The higher the value, the easier it is for the model to maintain high precision at a high recall rate. The calculation formula is: TP represents the number of samples that are actually cracks and are correctly detected as cracks, FN represents the number of samples that are actually cracks but are incorrectly detected as background, and FP represents the number of samples that are actually background but are incorrectly detected as cracks; In S15, after the model is iteratively trained and its performance meets the actual requirements, it is packaged using TorchScript or TensorFlowLite and deployed in the camera (1) for execution; In the S23, firstly, a lightweight neural network MobileNetV2 is used to replace the backbone feature extraction network Xception in the original DeepLabV3+ semantic segmentation model, so as to effectively reduce the number of parameters and balance the calculation accuracy and speed, so as to make the model more suitable for embedded devices such as cameras; secondly, an attention module ECANet is added after the ASPP void space pyramid pooling module extracts features and before the low-order features are spliced, so that the model can better focus on the crack information of the test component (2) and suppress invalid and complex background features, thereby improving the accuracy of the model; In S24, the cross entropy loss L CE Measures the difference between the predicted probability distribution and the true distribution, calculated as: Where N is the total number of pixels in the image, C is the number of categories, and y i,c is the true label of the i-th pixel belonging to category c, is the predicted probability that the i-th pixel belongs to category c; Dice loss is based on the Dice similarity coefficient L Dice , which is used to measure the overlap between the prediction and the true segmentation. The formula is: Among them, p i is the predicted probability that the i-th pixel belongs to a crack, g i is the true label of the i-th pixel belonging to the crack; In S24, the category average pixel accuracy MPA is used to measure the proportion of pixels in each category that are correctly classified, and then the average is calculated for all categories: Among them, TP c Indicates the number of pixels whose true label is category c and whose prediction is also category c, FN c Indicates the number of pixels whose true label is category c but is predicted to be other categories; The mean intersection over union (MIoU) measures the ratio of the intersection and union of the predicted result to the true segmentation, averaging all categories: Among them, FP c Indicates the number of pixels whose predicted label is category c but whose true label is not c; After the model's performance reaches the expected performance on the crack segmentation dataset, the model size is reduced and the inference speed is improved through pruning and quantization methods, and then converted into ONNX format or TensorRT format suitable for deployment, which is packaged and deployed in the camera (1) for execution; In S31, the DeepSORT crack tracking method mainly refers to combining a simple and efficient Kalman filter motion model and deep feature measurement to perform real-time crack extension state tracking; Kalman filtering assumes that the crack state is a vector x k , the observed value is z k , then the change of crack state is described by the state transfer equation: x k =F k x k-1 +B k u k +w k Among them, x k represents the state vector of the crack at time k, F k is the state transfer matrix, u k Control input vector, B k Control input matrix, w k is the process noise; The motion model of the crack uses Kalman filtering to model the geometric characteristics and position state of the crack: the state vector is defined as the center point position (x, y) of the crack, the size of the bounding box and its change speed; the Kalman filter predicts the possible position and size of the crack in the next frame based on the state of the current frame, providing a reference for the association; The crack matching and association method is that in each frame of the image, the new crack detected needs to be matched with the crack tracked in the previous frame, which is achieved by combining distance measurement; The distance measurement uses motion model distance and appearance feature distance; the motion model distance calculates the Euclidean distance between the state predicted by Kalman filter and the actual detection result, and the appearance feature distance uses the extracted feature embedding to calculate the cosine distance or Euclidean distance between the crack features of the current frame and the previous frame; The above two distances are weighted and fused to construct a matching matrix. The optimal allocation of crack recognition results and tracking trajectories is achieved through the Hungarian algorithm, that is, a set of minimum matching pairs is found on the matching matrix, so that each trajectory matches at most one detection frame, and each detection frame matches at most one trajectory; unmatched detection frames will be initialized as new crack trajectories; In S32, the dynamic update method updates the position, size and other states and appearance features of the successfully matched crack trajectory to keep it consistent with the actual state of the crack. For the unmatched crack trajectory, it continues to track according to the prediction of the Kalman filter. If no new detection results are matched for a period of time, it is considered that the crack has stopped expanding; After tracking is completed, the dynamic trajectory of the crack is visualized and output in the following ways: displaying the crack number and trajectory line on the original image, drawing curves of crack width and length changing with time, and generating an animation of crack expansion to demonstrate its dynamic evolution process.

Images