Visual vibration measurement method with high generalization capability

Abstract

The application provides a visual vibration measurement method with high generalization capability, S1. selecting a label with high generalization capability, including: collecting historical labels and making a data set; selecting a classic deep learning algorithm to train and make a decision on the best label and rule; S2. efficient target detection, including: design of an efficient detector; a simple target tracker; and conversion of vibration displacement. The application significantly improves the practicability, reliability and accuracy of visual measurement of vibration, and provides a solid technical foundation for application in related fields.

Description

Technical Field

[0001] This invention relates to the field of vibration measurement based on computer vision, specifically to a visual vibration measurement method with high generalization capability, mainly aimed at high-speed visual vibration measurement that can adapt to multiple scenarios. Background Technology

[0002] In recent years, due to the drawbacks of traditional contact vibration measurement methods, such as load effects, complex installation, and expensive supporting instruments, coupled with the rapid rise of computer vision, non-contact vibration measurement methods based on computer vision have developed rapidly and become an important tool in engineering vibration measurement. Currently, visual vibration measurement methods are mainly divided into traditional measurement methods and deep learning-based measurement methods, based on their different principles. Traditional measurement methods mainly include point tracking methods based on edge detection and feature point matching, and digital image correlation methods. However, these methods have poor generalization ability, are difficult to use under complex lighting conditions, and are even more difficult to generalize.

[0003] With the development of deep learning, methods combining deep learning with traditional vision are increasingly common. This method, based on edge detection and feature point detection, incorporates deep learning to transform detection computation into logistic regression, thereby improving generalization ability. In recent years, Convolutional Neural Networks (CNNs) have been widely used in structural engineering problems such as image classification, object detection, and segmentation. Compared with traditional computer vision algorithms, one advantage of these data-driven methods is their robustness to background noise and changes in external environmental conditions. Deep learning methods can automatically learn feature representations with tens of thousands of parameters, and deep models make the feature learning process more efficient. While all of the above methods have achieved high accuracy on specific structural objects, few studies have focused on their generalization. Summary of the Invention

[0004] This invention mainly considers the following five aspects:

[0005] 1. Labels with high generalization ability: In reality, the accuracy of natural feature methods is relatively low, making them difficult to apply in practical engineering. Manual labeling methods primarily use high-contrast patterns, so choosing the right pattern to improve overall generalization ability is a crucial issue.

[0006] 2. Efficient Target Detection Algorithm: Target detection is the first step in locating key points of a target structure. Current deep learning-based target detection algorithms suffer from complex network structures, leading to increased computational load and making them unsuitable for practical vibration measurements. Therefore, it is necessary to design a target detection algorithm that combines high speed with high accuracy.

[0007] 3. Efficient target tracking method: The target tracking method is the key method for establishing the frame between image frames, and can perform appropriate frame replacement in the case of frame loss or missing frames.

[0008] 4. Vibration displacement conversion: By visually measuring the vibration of the target object, the in-plane displacement of the camera can be directly obtained. However, it is still necessary to convert the pixel displacement into actual displacement.

[0009] 5. Auxiliary accuracy improvement method: In the actual measurement process, there are problems such as the difficulty in measuring minute vibrations and the low measurement accuracy caused by low camera pixels.

[0010] This invention provides a visual vibration measurement method with high generalization ability. First, the optimal artificial visual label is determined, and the target detection points are standardized. Then, an efficient target detection algorithm is designed, and the vibration displacement measurement algorithm is completed by combining target tracking and monocular and binocular vision principles.

[0011] The specific technical solution is as follows:

[0012] A visual vibration measurement method with high generalization ability consists of two parts: S1. Selecting labels with high generalization ability; S2. Efficient target detection.

[0013] S1. Select tags with high generalization ability, which includes two parts:

[0014] S1.1. Collect historical tags and create a dataset.

[0015] We selected commonly used visual measurement labels from previous years, fixed them in a uniform frame, and took photos using a mobile phone at different angles, with different numbers, and against different backgrounds on both cloudy and sunny days. The images were 4032×3204 pixels in JPG format. Some images were used as the training set, and some as the test set. Labels were used for manual image labeling.

[0016] S1.2. Select classic deep learning algorithms for training to determine the best labels and patterns.

[0017] Object detection methods include one-stage and two-stage approaches. Among the object detection methods used in deep learning, the You-Only-Look-Once (YOLO) algorithm segments the image into multiple grids, predicts the bounding box within each grid and the object category it contains, and uses the Non-Maximum Suppression (NMS) algorithm to eliminate overlapping bounding boxes. We selected the classic object detection methods YOLO_V5 and YOLO_V7 and performed a reverse quantization analysis on their label generalization ability. The YOLO series uses the same detection method, transforming the detection problem into a regression problem. It then divides the image into an S*S grid to predict the presence of objects within the grids and uses a regression method to predict location information and confidence scores. The confidence score calculation formula is as follows.

[0018]

[0019] P r (Class i |Object) represents the conditional probability that an object belongs to a certain category given that a grid contains the object; It is the ratio of the intersection area to the union area of ​​the predicted bounding box and the actual bounding box. P r The (Object) value is 1 if the object is present in the grid, and 0 if it is not. Predicted boxes are generated and filtered using non-maximum suppression. This score reflects the probability of the class appearing within the predicted box and the degree of overlap between the predicted and ground truth boxes.

[0020] For object detection algorithms, performance evaluation metrics are used to evaluate the algorithm model. YOLO-based algorithms use precision, recall, and average precision (AP) as evaluation metrics. The formulas for calculating precision, recall, and F1 score are as follows. In these formulas, TP represents the number of true positives; FN represents the number of false negatives; TN represents the number of true negatives; and FP represents the number of false positives.

[0021]

[0022] AP determines the value between precision and recall by integration, ranging from 0 to 1. 50 This refers to the average precision (AP) at different recall rates when IoU = 0.5. 75 This is the average precision for different recall rates when IoU = 0.75.

[0023] AP=∫0 1 Precision*Recal (3)

[0024] Training was performed on a server equipped with an NVIDIA GeForce RTX 3080Ti GPU and an Intel Core i9-10700 K CPU. The initial learning rate was set to 0.01, the batch size to 8, and the training epochs to 300. AP was also used. 50 and AP 75 As evaluation criteria, tests were conducted using YOLOv5 and YOLOv7 respectively.

[0025] S2. Highly efficient target detection, including:

[0026] S2.1 Design of an efficient object detector based on deep learning for outlining marked points.

[0027] S2.2 In order to improve the accuracy of target detection in vibration target displacement measurement, this invention optimizes the spatiotemporal correlation between frames by introducing a target tracking algorithm, thus overcoming the independence of adjacent frame detection algorithms.

[0028] S2.3, Vibration Displacement Conversion: To obtain physical displacement from the captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. This invention uses both monocular and binocular methods to convert vibration displacement for measurement. When using monocular vision, with the camera optical axis perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane. In this case, based on the principle of similar triangles, the true vibration displacement can be obtained.

[0029] Step S2.1 specifically includes:

[0030] The detector is based on the FastestDet algorithm. By replacing the backbone network and redesigning the connection pattern of the fusion layer, feature maps at different resolutions are input into the head module to obtain the target's location information and corresponding confidence scores. The network framework of the target detection algorithm includes a backbone network, a feature fusion module, and a prediction framework structure.

[0031] The Backbone network uses a four-stage hierarchical structure, preceded by either an embedding layer (a standard Conv4×4 with a stride of 4) or a merging layer (a standard Conv2×2 with a stride of 2) for spatial downsampling and channel expansion. Each stage contains a set of FasterNet blocks. Batch normalization (BN) is used, merging the normalized data into adjacent Conv layers. ReLU is used as the activation layer to consider runtime and efficiency.

[0032] A bidirectional fusion strategy relying on top-down and bottom-up approaches is adopted. The outputs are obtained from the Merging operations of stage 2 and stage 3 in the backbone, and then Avgpool2d operation is performed. At the same time, the UmSample operation is performed after the Merging operation of stage 4. Finally, the target feature maps of the three different channels are concatenated.

[0033] SimSppf is used as the backbone network module to fuse the outputs.

[0034] The fused information from SimSppf is subjected to a 1×1 convolution, the number of channels is adjusted, and a feature map is obtained. Then, convolution operations are performed on the head layer. The head structure consists of two convolutional layers, one 5×5 and the other 1×1. Batch normalization (BN) is performed after each convolution operation, and the ReLU activation function is used in the 5×5 convolutional layer.

[0035] The labels used in this invention are squares, which have more significant angular features. Therefore, SIoU is used as the target bounding box loss function in this study. SIoU further considers the vector angle between the ground truth bounding box and the predicted bounding box, and redefines the relevant loss function, which specifically includes four parts: angle cost, distance cost, shape cost, and IoU cost.

[0036] Preferably, in step S2.2:

[0037] A target tracking algorithm is added to avoid the independence of detection algorithms between adjacent frames. Kalman filtering is directly used as the tracker. Monocular and binocular video sequences are input into the detector to obtain the target bounding boxes for the left and right cameras, which are then used as input to the tracking part. Kalman filtering is used to predict the position of the tracking trajectory in the next frame, and the IoU between the predicted box and the actual detected box is used as the similarity between the two matches. After successful tracking, the center points of the monocular and binocular target bounding boxes are obtained and output. Since the center coordinates are pixel coordinates, the actual vibration displacement value is finally obtained through monocular measurement and the principle of binocular parallax.

[0038] Preferably, in step S2.3:

[0039] Establish the relationship between pixel coordinates and physical coordinates. Vibration displacement is measured using both monocular and binocular methods. When using monocular vision, with the camera optical axis perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane. Based on the principle of similar triangles, the true vibration displacement is obtained.

[0040]

[0041] In the formula, D is the size of the known object selected in the target plane; D is the corresponding physical length in the imaging plane; are the known physical dimensions and the corresponding image dimensions, respectively; f is the lens focal length; z is the distance from the camera optical center to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor.

[0042] When using binocular vision combined with the parallax principle for measurement, the three-dimensional coordinates of a spatial point are calculated by inversely using the pixel difference between corresponding points on the imaging surfaces of the left and right cameras. The left and right cameras are parallel to each other. The origin of the coordinate system is O. L and O R , which is also the optical center of the left and right cameras. The horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f is the camera focal length. Let a point P(x w y w , z w The projection points on the imaging planes of the two cameras are P and P', respectively. l (x l y l ) and P r (x r y r Equation (5) describes the relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel point:

[0043]

[0044] By finding the projected position of the target point in the left and right cameras, and using the camera focal length f and the baselines b of the two cameras, the depth information z can be determined, thus accurately recovering the three-dimensional information of the target point in the camera coordinates. However, in practical applications, due to lens distortion, the imaging position of the target point will change, so it is necessary to perform stereo matching after calibration to obtain a distortion-free pixel plane.

[0045] The technical effects of this invention are as follows:

[0046] (1) Enhanced generalization ability: By selecting labels with high generalization ability and efficient target detection algorithms, this method can still maintain high measurement accuracy and reliability under complex lighting conditions, thereby broadening the application range of visual measurement in different environments.

[0047] (2) Improved detection speed: The optimized target detection algorithm not only improves the generalization ability under complex light sources, but also improves the target detection speed, enhances the real-time performance of the system, and is suitable for vibration measurement in dynamic scenes.

[0048] (3) Effective spatiotemporal characteristic correlation: Through efficient target tracking methods, the spatiotemporal characteristics between consecutive image frames can be accurately correlated, ensuring the continuity and stability of the target during the measurement process, thereby improving the accuracy of the measurement results.

[0049] (4) Precise displacement conversion: For different application scenarios, it can effectively convert pixel displacement to real displacement, so that the measurement results have practical application value and meet specific needs.

[0050] (5) Improved measurement accuracy: The proposed accuracy improvement strategy can achieve high-precision measurement at a lower pixel resolution, reducing the dependence on equipment requirements and expanding the application potential of the system.

[0051] (6) Wide application adaptability: Ultimately, the design of this method enables it to be used under most lighting conditions, enhancing the practicality of visual measurement technology and meeting the needs of different fields and scenarios.

[0052] In summary, through innovative research concepts and multifaceted technical optimizations, this invention significantly improves the practicality, reliability, and accuracy of visual vibration measurement, providing a solid technical foundation for applications in related fields. Attached Figure Description

[0053] Figure 1 This is a flowchart of the visual vibration measurement process of the present invention.

[0054] Figure 2 This is for the integration of historical tag datasets for the present invention.

[0055] Figure 3 This is a diagram of the target detection algorithm of the present invention.

[0056] Figure 4 This is the network fusion layer in the algorithm of this invention.

[0057] Figure 5 This is the detection head prediction module in this embodiment;

[0058] Figure 6 This is the angle loss calculation scheme for this embodiment;

[0059] Figure 7 This is a schematic diagram of the distance loss calculation scheme in this embodiment;

[0060] Figure 8 This is a flowchart illustrating the vibration displacement conversion process of the present invention.

[0061] Figure 9 AP of the present invention 50 The YOLOv5 test results serve as an evaluation standard.

[0062] Figure 10AP of the present invention 75 The YOLOv5 test results serve as an evaluation standard.

[0063] Figure 11 AP of the present invention 50 The results of the YOLOv7 test serve as an evaluation standard.

[0064] Figure 12 AP of the present invention 75 The results of the YOLOv7 test were used as an evaluation standard. Detailed Implementation

[0065] The specific technical solutions of the present invention will be described with reference to the embodiments.

[0066] The visual vibration measurement method with high generalization ability provided by this invention includes two parts: S1. Selecting labels with high generalization ability; S2. Efficient target detection. The process is as follows: Figure 1 As shown.

[0067] S1. Select labels with high generalization ability;

[0068] In visual vibration measurement systems, not only are high-level target detection algorithms required, but the high generalization ability of target points is also crucial. Choosing appropriate labels can effectively reduce the impact of strong changes in ambient light in practical applications, improving the overall anti-interference capability of the visual vibration measurement system. This section mainly includes two parts:

[0069] S1.1. Collect historical tags and create a dataset.

[0070] In order to determine the best label, this invention searched for commonly used visual measurement labels over the years, and after selection, retained 20 types of labels, such as... Figure 2 As shown, they all use simple patterns, mostly circles and squares, as the shapes for object detection. This invention fixes the following 20 labels in a uniform frame, taken with a mobile phone from different angles, numbers, and backgrounds on both cloudy and sunny days. The image size is 4,032 × 3,204 pixels, in JPG format. A total of 1200 images were collected. Of these, 1000 images were selected as the training set, and 200 images were selected as the test set. Labels were used for manual image labeling.

[0071] S1.2. Select classic deep learning algorithms for training to determine the best labels and patterns.

[0072] Object detection methods mainly include one-stage and two-stage approaches. Among the object detection methods used in deep learning, the You-Only-Look-Once (YOLO) algorithm segments the image into multiple grids, predicts the bounding box and the object category within each grid, and uses the Non-Maximum Suppression (NMS) algorithm to eliminate overlapping bounding boxes. Due to its speed, the YOLO series algorithms are widely used in vibration measurement. Therefore, this invention selects the classic object detection methods YOLO_V5 and YOLO_V7 to perform a reverse quantization analysis of the label generalization ability. The YOLO series uses the same detection method, transforming the detection problem into a regression problem. It then divides the image into an S*S grid to predict whether a target exists within the grid, and uses a regression method to predict location information and confidence levels. The confidence level calculation formula is as follows.

[0073]

[0074] P r (Class i |Object) represents the conditional probability that an object belongs to a certain category given that a grid contains the object; It is the ratio of the intersection area to the union area of ​​the predicted bounding box and the actual bounding box. P r The (Object) value is 1 if the object is present in the grid, and 0 if it is not. Predicted boxes are generated and filtered using non-maximum suppression. This score reflects the probability of the class appearing within the predicted box and the degree of overlap between the predicted and ground truth boxes.

[0075] For object detection algorithms, performance evaluation metrics are needed to evaluate the algorithm model. In this invention, based on YOLO, precision, recall, and average precision (AP) are used as evaluation metrics. The formulas for calculating precision, recall, and F1 score are as follows. In these formulas, TP represents the number of true positives; FN represents the number of false negatives; TN represents the number of true negatives; and FP represents the number of false positives.

[0076]

[0077] The formula for calculating AP is shown in formula (3). The value between precision and recall is determined by integration, and the range is between 0 and 1. AP 50 This refers to the average precision (AP) at different recall rates when IoU = 0.5. 75 This is the average precision for different recall rates when IoU = 0.75.

[0078] AP=∫0 1 Precision*Recal (3)

[0079] In terms of hardware, this invention uses a server with an NVIDIA GeForce RTX 3080Ti GPU and an Intel Core i9-10700 K CPU for training. The initial learning rate was set to 0.01, the batch size to 8, and the training epochs to 300. AP was also employed. 50 and AP 75 As evaluation criteria, tests were conducted using YOLOv5 and YOLOv7 respectively, and the experimental results are shown in the table below. Figure 9 and Figure 10 As shown, in the YOLOv5 test, the black and white blocks and the white-background double rings achieved the fastest convergence speed while maintaining maximum accuracy. Figure 11 and Figure 12 As shown, in the YOLOv7 test, only black and white blocks achieve maximum accuracy while converging at the fastest speed. Therefore, this invention uses this pattern as a label for object detection.

[0080] S2. High-efficiency target detection:

[0081] This section mainly includes three parts: the design of a high-efficiency detector; a simple target tracker; and the conversion of vibration displacement.

[0082] Step S2.1: Design of an efficient object detector based on deep learning for outlining marked points.

[0083] Step S2.2: In order to improve the accuracy of target detection in vibration target displacement measurement, this invention optimizes the spatiotemporal correlation between frames by introducing a target tracking algorithm, thus overcoming the independence of adjacent frame detection algorithms.

[0084] Step S2.3: Vibration Displacement Conversion. To obtain the physical displacement from the captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. This invention uses both monocular and binocular methods to convert vibration displacement for measurement. When using monocular vision, with the camera optical axis perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane. In this case, based on the principle of similar triangles, the true vibration displacement can be obtained.

[0085] Preferably, step S2.1 specifically includes:

[0086] The detector designed in this invention is mainly based on the FastestDet algorithm. By replacing the backbone network and redesigning the connection mode of the fusion layer, feature maps of different resolutions are input into the head module to obtain the target's location information and corresponding confidence scores. This method achieves significant improvements in both speed and accuracy. The flowchart is as follows: Figure 3As shown, the basic network framework of the object detection algorithm includes a backbone network, a feature fusion module, and a prediction framework structure.

[0087] Backbone uses the following number of network layers: Figure 4 As shown, it has four layered stages, each preceded by an embedding layer (a regular Conv4×4, stride 4) or a merging layer (a regular Conv2×2, stride 2) for spatial downsampling and channel expansion. Each stage contains a set of FasterNet blocks. Batch normalization (BN) is used, which can be merged into adjacent Conv layers for faster inference while maintaining similar efficiency. Considering runtime and effectiveness, ReLU was empirically used as the activation layer. The overall structure is shown in Table 1. Layer represents the input network layer, Output represents the output size and number of channels for each network layer, and Operator represents the convolution operation of that module. In this invention, the acquired image is resized to a size of 352×352 with 3 channels before being input into the backbone network. The output size and number of channels change during Embedding and Merging; FasterNet-Block only performs feature extraction, keeping the input and output size and number of channels unchanged. Different FasterNet-Blocks... "Conv_k_c_s" represents a convolutional layer with a kernel size of k, output channels of c, and stride of s. "PConv_k_c_s_r" represents a partial convolution with an additional parameter r, i.e., a partial ratio, otherwise it is the same as "Conv_k_c_s".

[0088] Table 1 Backbone Network Structure

[0089]

[0090] This invention employs a bidirectional fusion strategy relying on both top-down and bottom-up approaches. It outputs the features after the Merging operations of stages 2 and 3 in the backbone, and performs an Avgpool2d operation. Simultaneously, it performs an UmSample operation after the Merging operation of stage 4, and finally concatenates the target feature maps from the three different channels. This invention uses SimSppf as the backbone network module to fuse the outputs. The entire feature fusion process is shown in Table 2. SimSppf, proposed in YOLOv6, is an advanced version of the feature pyramid network. The spatial feature pyramid network can fully utilize the dimensionality to distinguish details in large images and extract macroscopic features in small images, better utilizing original information and achieving a variable receptive field effect, showing good performance for feature images of different sizes. In the feature extraction process, compared with SPPF, the SiLU activation function is replaced with ReLU, effectively avoiding image distortion caused by cropping and scaling of image regions while maintaining accuracy. It solves the problem of repetitive feature extraction of graph-related features in convolutional neural networks, greatly improving the speed of generating candidate boxes. The specific structure of the module is as follows... Figure 4 As shown.

[0091] Table 2. Network Structure of the Fusion Layer

[0092]

[0093] The head structure is an important component of target localization, mainly used to predict the type and location of the target. Since classification focuses more on the texture content of the target, while localization focuses more on the edge information of the target, this invention does not adopt the traditional coupled detection but continues to use the original decoupled detection head design, which can effectively prevent conflicts between classification and regression tasks. This invention is slightly different from the currently popular decoupled detection heads (such as the EfficientdecopledHead, EDHead of Yolov6

[78] and Yolox

[79] , and DecopledHead). Figure 5 As shown, the fused information from SimSppf is mainly subjected to a 1×1 convolution, the number of channels is adjusted, and a feature map is obtained. Then, convolution operations are performed through the Head layer. The Head structure of this invention consists of two convolutional layers, one 5×5 and the other 1×1. Batch normalization (BN) is performed after each convolution operation, and the ReLU activation function is used in the 5×5 convolutional layer. This method enables it to extract image features and predict the category of each pixel more quickly, while also achieving faster convergence.

[0094] like Figure 6 and Figure 7 The labels used are squares, which have more significant angular features. Therefore, SIoU is used as the loss function for the target bounding box in this study. SIoU further considers the vector angle between the ground truth bounding box and the predicted bounding box, and redefines the relevant loss function, which specifically includes four parts: angle cost, distance cost, shape cost, and IoU cost.

[0095] Angular cost:

[0096]

[0097] in:

[0098]

[0099]

[0100] Distance cost: The angle cost defined above was taken into account when redefining distance cost.

[0101] Δ=∑ t＝x，y (1-e -γρt (11)

[0102] in:

[0103]

[0104] Shape cost:

[0105]

[0106] in:

[0107]

[0108] Finally, define the loss function:

[0109]

[0110] in:

[0111]

[0112] Preferably, in step S2.2:

[0113] The purpose of target detection is to predict the position and category of a target object, ignoring the spatiotemporal correlation between adjacent frames. To optimize the correlation between detected target frames and more accurately measure the displacement of vibrating targets, this invention adds a target tracking algorithm, avoiding the independence of detection algorithms between adjacent frames. This invention directly uses Kalman filtering as the tracker. This invention can input monocular and binocular video sequences into the detector to obtain the target bounding boxes of the left and right cameras, which are also used as input to the tracking part. Kalman filtering is used to predict the position of the tracking trajectory in the next frame, and the IoU between the predicted box and the actual detected box is used as the similarity between the two matches. After successful tracking, the center points of the monocular and binocular target bounding boxes can be obtained as outputs. Since the center coordinates are pixel coordinates, the actual vibration displacement value needs to be obtained finally through monocular measurement and the principle of binocular parallax. Overall, as follows... Figure 8 As shown.

[0114] Preferably, in step S2.3:

[0115] To obtain physical displacement from a captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. This invention uses both monocular and binocular methods to measure vibration displacement. When using monocular vision, with the camera optical axis perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane. Figure 8 (As shown in the third figure). In this case, based on the principle of triangle similarity, the actual vibration displacement can be obtained.

[0116]

[0117] In the formula, D is the size of the known object selected in the target plane; D' is the corresponding physical length in the imaging plane; are the known physical dimensions and corresponding image dimensions, respectively; f is the lens focal length; z is the distance from the camera optical center to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor. When using binocular vision combined with the parallax principle for measurement, the three-dimensional coordinates of the spatial point can be obtained by using the pixel difference between corresponding points on the imaging surfaces of the left and right cameras of the spatial point. In such cases... Figure 8 The image shown in the middle is a parallel binocular vision model. The left and right cameras are parallel to each other. The origin of the coordinate system is O. L and O R This is also the optical center of the left and right cameras. The horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f in the diagram is the camera focal length. Let a point P(x) w y w , z w The projection points on the imaging planes of the two cameras are P and P', respectively. l (x l y l ) and Pr (x r y r Equation (5) can be used to describe the relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel.

[0118]

[0119] As can be seen from the formula, if the projected positions of the target point in the left and right cameras can be found, the depth information z can be determined using the camera focal length f and the baselines b of the two cameras, thus accurately recovering the three-dimensional information of the target point in the camera coordinates. However, in practical applications, due to lens distortion, the imaging position of the target point will change. Therefore, most cameras still need to be calibrated and stereo matched to become a distortion-free pixel plane in practical applications. The pseudocode for the entire process is shown in Table 3.

[0120] Table 3. Pseudocode for vibration displacement conversion

[0121]

[0122]

[0123] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A visual vibration measurement method with high generalization ability, characterized in that, It consists of two parts: S1. Selecting labels with high generalization ability; S2. High-efficiency target detection and its vibration displacement conversion; S1. Select tags with high generalization ability, including: S1.

1. Collect historical tags and create a dataset; S1.

2. Select classic deep learning algorithms for training to determine the best labels and patterns; S1.

2. Select classic deep learning algorithms for training to determine the best labels and patterns. The specific method is as follows: In object detection methods used in deep learning, the YOLO algorithm divides the image into multiple grids, predicts the bounding box and the object category contained in each grid, and uses the Non-Maximum Suppression (NMS) algorithm to eliminate overlapping bounding boxes. For object detection methods, the generalization ability of the labels is analyzed using inverse quantization. The detection problem is transformed into a regression problem. Then, the image is divided into an S*S grid to predict whether there is an object in the grid, and a regression method is used to predict the location information and confidence score. The confidence score calculation formula is as follows: P r (Classi|Object) represents the conditional probability that an object belongs to a certain class, given that a grid contains the object. It is the ratio of the intersection area to the union area of ​​the predicted bounding box and the actual bounding box; P r (Object) is 1 when the object is contained in the grid and 0 when it is not; the predicted box is generated and filtered through nonmaximum suppression; this score reflects the probability of the class appearing in the predicted box and the degree of overlap between the predicted box and the ground truth box; Performance evaluation metrics are used to evaluate the algorithm model; based on YOLO, Precision, Recall, and Average Precision (AP) are used as evaluation metrics; in the formula, TP represents the number of true positives; FN represents the number of false negatives; TN represents the number of true negatives; and FP represents the number of false positives. AP determines the value between precision and recall by integration; S2. Highly efficient target detection, including: S2.1, Design of a high-efficiency target detector based on deep learning, which improves detection efficiency while outlining marked points; S2.2, A simplified target tracker introduces a target tracking algorithm to optimize the spatiotemporal correlation between frames and overcome the independence of adjacent frame detection algorithms; To address the independence of detection algorithms between adjacent frames, a target tracking algorithm is added. Kalman filtering is directly used as the tracker. Monocular and binocular video sequences are input into the detector to obtain the target bounding boxes for the left and right cameras, which are then used as input for the tracking part. Kalman filtering is used to predict the position of the tracking trajectory in the next frame, and the IoU between the predicted box and the actual detected box is used as the similarity between the two matches. After successful tracking, the center points of the monocular and binocular target bounding boxes are obtained and output. Since the center coordinates are pixel coordinates, the actual vibration displacement value is finally obtained through monocular measurement and the principle of binocular parallax. S2.3, Transformation of vibration displacement; Establish the relationship between pixel coordinates and physical coordinates; use both monocular and binocular methods to measure vibration displacement; when using monocular, when the camera optical axis is perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane, and the true vibration displacement can be obtained based on the principle of similar triangles. In the formula, D is the size of the known object selected in the target plane; D is the corresponding physical length in the imaging plane; are the known physical dimensions and the corresponding image dimensions, respectively; f is the lens focal length; z is the distance from the camera optical center to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor. When using binocular vision combined with the parallax principle for measurement, the three-dimensional coordinates of the spatial point are calculated by using the pixel difference between corresponding points on the imaging surfaces of the left and right cameras; the left and right cameras are parallel to each other; the origin of the coordinate system is O. L and O R , which is also the optical center of the left and right cameras; the horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f is the camera focal length; let a point P(x w y w , z w The projection points on the imaging planes of the two cameras are P and P', respectively. l (x l y l ) and P r (x r y r Equation (5) describes the relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel point: By finding the projected position of the target point in the left and right cameras, and using the camera focal length f and the baseline b of the two cameras, the depth information z can be determined, thereby accurately recovering the three-dimensional information of the target point in the camera coordinates.

2. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that, S1.

1. Collect historical tags and create a dataset. The specific method is as follows: Visual measurement labels commonly used over the years were selected, all employing simple patterns, with circles and squares as the shapes for object detection. The labels were fixed in a uniform scene, and photos were taken with a mobile phone on cloudy and sunny days from different angles, with different numbers and backgrounds. The image size was 4,032×3,204 pixels, in JPG format. Images were collected; some images were used as the training set, and some as the test set. Labels were used to manually label the images.

3. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that, Step S2.1 specifically includes: The detector is based on the FastestDet algorithm. By changing the backbone network and redesigning the connection mode of the fusion layer, feature maps of different resolutions are input into the head module to obtain the target's location information and corresponding confidence. The network framework of the FastestDet algorithm includes a backbone network, a feature fusion module, and a prediction framework structure. The backbone network uses a four-layer hierarchical stage, with an embedding layer or a merging layer preceding each stage for spatial downsampling and channel expansion; each stage contains a set of FasterNet blocks. Normalization is performed using batch normalization, which is merged into adjacent Conv layers; ReLU is used as the activation layer. A bidirectional fusion strategy relying on top-down and bottom-up approaches is adopted. The outputs are obtained from the Merging operations of stage 2 and stage 3 in the backbone, and Avgpool2d operation is performed. At the same time, the UmSample operation is performed after the Merging operation of stage 4. Finally, the target feature maps of the three different channels are stitched together. SimSppf is used as the backbone network module to fuse the outputs; The fused information from SimSppf is subjected to a 1×1 convolution, the number of channels is adjusted, and a feature map is obtained. Then, convolution operations are performed through the Head layer. The Head structure consists of two convolutional layers, one 5×5 and the other 1×1. Batch normalization (BN) is performed after each convolution operation, and the ReLU activation function is used in the 5×5 convolutional layer.

Images