Visual vibration measurement method with high generalization ability

Abstract

The invention provides a visual vibration measurement method with high generalization ability, which comprises the following steps: S1, selecting labels with high generalization ability: collecting historical labels and making the historical labels into a data set; a classic deep learning algorithm is selected for training to decide an optimal label and rule; s2, high-efficiency target detection: designing a high-efficiency detector; a simple target tracker is provided; and converting vibration displacement. According to the method, the practicability, the reliability and the accuracy of vision measurement vibration are remarkably improved, and a solid technical foundation is provided for application in related fields.

Description

Technical Field

[0001] The present invention relates to the field of vibration measurement based on computer vision, and in particular to a visual vibration measurement method with high generalization capability, which is mainly aimed at high-speed visual vibration measurement that is adaptable to multiple scenarios. Background Art

[0002] In recent years, due to the shortcomings of traditional contact vibration measurement methods, such as load effect, complex installation, and expensive supporting instruments, as well as the rapid rise of computer vision, non-contact vibration measurement methods based on computer vision have developed rapidly and become an important tool in engineering vibration measurement. At present, visual vibration measurement methods are mainly divided into traditional measurement methods and deep learning-based measurement methods according to different principles. Traditional measurement methods mainly include point tracking methods based on edge detection and feature point matching, and digital image correlation methods. However, this method has poor generalization ability, is difficult to use under complex light sources, and is even more difficult to generalize.

[0003] With the development of deep learning, there are more and more methods that combine deep learning with traditional vision. This method is based on edge detection and feature point detection, and adds deep learning methods to turn the detection calculation into a logistic regression method, thereby improving the generalization ability. In recent years, convolutional neural networks (CNN) have been widely used in structural engineering problems such as image classification, target detection, and segmentation. Compared with traditional computer vision algorithms, one advantage of these data-driven methods is that they are robust to background noise and changes in external environmental conditions. Deep learning methods can automatically learn feature representations with tens of thousands of parameters, and deep models make the feature learning process more efficient. The above methods have achieved high accuracy on specific structural objects, but few people have done research on generalization. Summary of the invention

[0004] The present invention mainly considers the following five aspects:

[0005] 1. Labels with high generalization ability: The accuracy of the natural feature method in reality is low and it is difficult to apply it to actual projects. The labels of the manual labeling method are mainly high-contrast patterns, so choosing what kind of pattern can improve the overall generalization ability is a key issue.

[0006] 2. Efficient target detection algorithm: Target detection is the first step to locate the key point information of the target structure. The complex network structure of the current deep learning-based target detection algorithm increases the amount of calculation and is difficult to use in actual vibration measurement. Therefore, it is necessary to design a target detection algorithm that has both high speed and high accuracy.

[0007] 3. Efficient target tracking method: The target tracking method is the key method to establish the connection between image frames, and can perform appropriate frame filling in the case of frame loss or leakage.

[0008] 4. Conversion of vibration displacement: By visually measuring the vibration of the target object, the in-plane displacement of the camera can be directly obtained, and the pixel displacement needs to be converted into actual displacement.

[0009] 5. Auxiliary accuracy improvement method: In the actual measurement process, it is difficult to measure tiny vibrations, and the camera pixels are too low, resulting in low measurement accuracy.

[0010] The present invention provides a visual vibration measurement method with high generalization ability. First, the optimal artificial visual label is determined to unify the target detection points. Then, an efficient target detection algorithm is designed, and the vibration displacement measurement algorithm is completed by combining target tracking and monocular and binocular vision principles.

[0011] The specific technical solution is:

[0012] A visual vibration measurement method with high generalization ability includes two parts: S1. Selecting labels with high generalization ability; S2. Efficient target detection.

[0013] S1. Selecting labels with high generalization ability includes two parts:

[0014] S1.1. Collect historical labels and make them into datasets.

[0015] We selected visual measurement labels that have been commonly used over the years, fixed the labels in a unified picture, and took pictures with a mobile phone at different angles, different numbers, and different backgrounds on cloudy and sunny days. The image size is 4,032×3,204 pixels, and the format is JPG. We collected images. Some of the images were used as training sets, and some were used as test sets. We also used Labelme to manually label the images.

[0016] S1.2. Select classic deep learning algorithms for training to determine the best labels and rules.

[0017] Methods based on target detection include one-stage and two-stage. Among the target detection methods used in deep learning, the You-Only-Look-Once (YOLO) algorithm divides the image into multiple grids, predicts the bounding box and the object category contained in each grid, and uses the non-maximum suppression (NMS) algorithm to eliminate overlapping bounding boxes. The classic target detection methods YOLO_V5 and YOLO_V7 are selected to perform reverse quantitative analysis on the generalization ability of the label. The YOLO series uses the same detection method to convert the detection problem into a regression problem, and then divides the image into S*S grids to predict whether there is a target in the grid, and uses the regression method to predict the position information and confidence. The confidence calculation formula is as follows.

[0018]

[0019] P r (Class i |Object) represents the conditional probability that a target belongs to a certain category under the premise that a grid contains the target; It is the ratio of the intersection area of ​​the predicted bounding box to the union area of ​​the actual bounding box. r (Object) is 1 if the grid contains the object and 0 if it does not. The prediction box is generated and filtered by non-maximum suppression. This score reflects the probability of the class appearing in the prediction box and the degree of overlap between the prediction box and the true box.

[0020] For the target detection algorithm, performance evaluation indicators are used to evaluate the algorithm model. Based on YOLO, precision (Precision), recall (Recall) and average precision (Average Precision, AP) are used as evaluation indicators. The calculation formulas for precision, recall and F1 score are as follows. In these formulas, TP represents the number of true positive examples; FN represents the number of false negative examples; TN represents the number of true negative examples; FP represents the number of false positive examples.

[0021]

[0022] AP is determined by integrating the precision and recall, ranging from 0 to 1. 50 is the average precision of different recall rates when IoU = 0.5, AP 75 It is the average precision of different recall rates when IoU=0.75.

[0023] AP = ∫0 1 Precision*Recal (3)

[0024] We used a server with an NVIDIA GeForce RTX 3080Ti GPU and an Intel Core i9-10700 K CPU for training. The initial learning rate was set to 0.01, the batch size was set to 8, and the training epoch was set to 300. We also used AP 50 and AP 75 As evaluation criteria, yolov5 and yolov7 were used for testing respectively.

[0025] S2. Efficient target detection, including:

[0026] S2.1. Design of efficient object detector based on deep learning for outlining landmark points.

[0027] S2.2. In order to improve the accuracy of target detection in vibration target displacement measurement, the present invention optimizes the temporal and spatial correlation between frames by introducing a target tracking algorithm, thereby overcoming the independence of adjacent frame detection algorithms.

[0028] S2.3, Conversion of vibration displacement, in order to obtain physical displacement from the captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. The present invention uses two methods of converting vibration displacement for measurement: monocular and binocular. When using a monocular camera, when the camera optical axis is perpendicular to the surface of the object, all points on the object plane can be equally reduced to the image plane. In this case, according to the principle of triangle similarity, the real vibration displacement can be obtained.

[0029] The step S2.1 is specifically as follows:

[0030] The detector is based on the FastestDet algorithm. After replacing the backbone network, the connection mode of the fusion layer is redesigned. Finally, the feature maps of different resolutions are input into the head module to obtain the location information and corresponding confidence of the target. The network framework of the target detection algorithm includes the backbone network, feature fusion module and prediction framework structure.

[0031] The network layer used by Backbone has four hierarchical stages, each of which is preceded by an embedding layer (a regular Conv4×4, stride 4) or a merging layer (a regular Conv2×2, stride 2) for spatial downsampling and channel number expansion. Each stage has a stack of FasterNet blocks. Normalization uses batch normalization (Batchnormalization, BN) and is merged into adjacent Conv layers. Considering the running time and effectiveness, ReLU is used as the activation layer.

[0032] A bidirectional fusion strategy based on top-down and bottom-up is adopted to output the Merging operations of stage2 and stage3 in the backbone, and perform Avgpool2d operation. At the same time, the UmSample operation is performed after the Merging operation of stage4, and finally the target feature maps of three different channels are spliced.

[0033] Use SimSppf as the backbone network module to fuse the output.

[0034] The fusion information of SimSppf is convolved with 1×1, the number of channels is adjusted, and the feature map is obtained. Then the convolution operation is performed through the Head layer. The Head structure includes two convolution layers of 5×5 and 1×1 respectively. Batch normalization (BN) operation is used after each convolution operation, and the ReLU activation function is used in the 5×5 convolution layer.

[0035] The label used in this invention is a square, and the angle feature is more significant, so SIoU is used in this target box loss function. SIoU further considers the vector angle between the real box and the predicted box, and redefines the relevant loss function, which specifically includes four parts: angle loss (Angle cost), distance loss (Distance cost), shape loss (Shapecost), and IoU loss (IoU cost).

[0036] Preferably, in step S2.2:

[0037] The target tracking algorithm is added to avoid the mutual independence of the detection algorithms between adjacent frames. The Kalman filter is directly used as the tracker. The monocular and binocular video sequences are input into the detector to obtain the target frames of the left and right cameras, which are also used as the input of the tracking part. The Kalman filter is used to predict the position of the tracking trajectory of the current frame in the next frame, and the IoU between the predicted frame and the actual detection frame is used as the similarity of the two matches. After successful tracking, the center point of the monocular and binocular target frame is obtained and used as the output. Since the center coordinates are pixel coordinates, the real vibration displacement value is finally obtained through monocular measurement and binocular parallax principle.

[0038] Preferably, in step S2.3:

[0039] Establish the relationship between pixel coordinates and physical coordinates. Use monocular and binocular methods to convert vibration displacement for measurement. When using a monocular camera, when the camera optical axis is perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane, and the real vibration displacement can be obtained based on the principle of triangle similarity.

[0040]

[0041] Where D is the size of the known object selected in the target plane; D is the corresponding physical length in the imaging plane; are the known physical size and the corresponding image size, respectively; f is the focal length of the lens; z is the distance from the optical center of the camera to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor.

[0042] When using binocular vision combined with the parallax principle for measurement, the pixel difference of the corresponding points on the imaging surface of the left and right cameras of the space point is used to reverse the three-dimensional coordinates of the space point. The left and right cameras are parallel to each other. The origin of the coordinate system is O L and O R , which is also the optical center of the left and right cameras. The horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f is the focal length of the camera. Suppose a point P(x w ,y w , z w ) on the two camera imaging planes are P l (x l ,y l ) and P r (x r ,y r ). The relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel point is described by equation (5):

[0043]

[0044] Find the projection position of the target point in the left and right cameras, and use the camera focal length f and the baseline b of the two cameras to determine the depth information z, thereby accurately restoring the three-dimensional information of the target point in the camera coordinates. However, in actual applications, due to lens distortion, the imaging position of the target point will change, so it is necessary to calibrate it and perform stereo matching to become a distortion-free pixel plane.

[0045] The technical effects of the present invention are as follows:

[0046] (1) Enhanced generalization ability: By selecting labels with high generalization ability and efficient target detection algorithms, this method can still maintain high measurement accuracy and reliability under complex lighting conditions, thereby broadening the application scope of visual measurement in different environments.

[0047] (2) Improved detection speed: The optimized target detection algorithm not only improves the generalization ability under complex light sources, but also improves the speed of target detection and enhances the real-time performance of the system, making it suitable for vibration measurement in dynamic scenes.

[0048] (3) Effective association of spatiotemporal characteristics: Through efficient target tracking methods, the spatiotemporal characteristics between consecutive image frames can be accurately associated, ensuring the continuity and stability of the target during the measurement process, thereby improving the accuracy of the measurement results.

[0049] (4) Accurate displacement conversion: For different application scenarios, it can achieve effective conversion from pixel displacement to real displacement, making the measurement results have practical application value and meeting specific needs.

[0050] (5) Improved measurement accuracy: The proposed accuracy improvement strategy can achieve high-precision measurement at lower pixel resolution, reducing the dependence on equipment requirements and expanding the application potential of the system.

[0051] (6) Wide application adaptability: Ultimately, the method is designed so that it can be used under most lighting conditions, enhancing the practicality of visual measurement technology and being able to meet the needs of different fields and scenarios.

[0052] In summary, the present invention significantly improves the practicability, reliability and accuracy of visual vibration measurement through innovative research concepts and multi-faceted technical optimization, and provides a solid technical foundation for applications in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of visual vibration measurement of the present invention.

[0054] Figure 2 The historical label dataset is integrated for the present invention.

[0055] Figure 3 This is a target detection algorithm diagram of the present invention.

[0056] Figure 4 It is the network fusion layer in the algorithm of the present invention.

[0057] Figure 5 is the detection head prediction module of this embodiment;

[0058] Figure 6 This is the angle loss calculation scheme of this embodiment;

[0059] Figure 7 This is a schematic diagram of the distance loss calculation scheme of this embodiment;

[0060] Figure 8 It is a flow chart of vibration displacement conversion of the present invention;

[0061] Fig. 9 The AP of the present invention 50 As the evaluation criteria yolov5 test results;

[0062] Fig.10The AP of the present invention 75 As the evaluation criteria yolov5 test results;

[0063] Fig.11 The AP of the present invention 50 As the evaluation criteria yolov7 test results;

[0064] Fig.12 The AP of the present invention 75 As evaluation criteria yolov7 test results. DETAILED DESCRIPTION

[0065] The specific technical solution of the present invention is explained in conjunction with embodiments.

[0066] The visual vibration measurement method with high generalization ability provided by the present invention includes two parts: S1. Selecting labels with high generalization ability; S2. Efficient target detection. The process is as follows Figure 1 shown.

[0067] S1. Select labels with high generalization ability;

[0068] In the visual vibration measurement system, not only the target detection algorithm is required to be high, but also the high generalization ability of the target point is also a key part. Selecting appropriate labels can effectively reduce the impact of strong changes in ambient light in practical applications and improve the anti-interference ability of the overall visual vibration measurement system. This part mainly includes two parts:

[0069] S1.1. Collect historical labels and make them into datasets.

[0070] In order to determine the best label, the present invention has searched for visual measurement labels that have been widely used over the years, and after selection, a total of 20 labels are left, such as Figure 2 As shown, they all use simple patterns, mostly circles and squares, as the shapes of target detection. The present invention fixes the following 20 labels in a unified picture, and takes pictures with a mobile phone at different angles, different numbers, and different backgrounds on cloudy and sunny days. The picture size is 4,032×3,204 pixels, and the format is JPG. A total of 1,200 images were collected. Among them, 1,000 images were selected as the training set and 200 images were selected as the test set. Labelme was used to manually label the images.

[0071] S1.2. Select classic deep learning algorithms for training to determine the best labels and rules.

[0072] The method based on target detection mainly includes one stage and two stages. Among the target detection methods used in deep learning, the You-Only-Look-Once (YOLO) algorithm divides the image into multiple grids, predicts the bounding box in each grid and the object category it contains, and uses the non-maximum suppression (NMS) algorithm to eliminate overlapping bounding boxes. Because the Yolo series algorithm has the advantage of being fast, it is widely used in vibration measurement. Therefore, the present invention selects the classic target detection methods of YOLO_V5 and YOLO_V7 to perform reverse quantitative analysis on the generalization ability of the label. The YOLO series adopts the same detection method, converts the detection problem into a regression problem, and then divides the image into S*S grids to predict whether there is a target in the grid, and uses the regression method to predict the position information and confidence. The confidence calculation formula is as follows.

[0073]

[0074] P r (Class i |Object) represents the conditional probability that a target belongs to a certain category under the premise that a grid contains the target; It is the ratio of the intersection area of ​​the predicted bounding box to the union area of ​​the actual bounding box. r (Object) is 1 if the grid contains the object and 0 if it does not. The prediction box is generated and filtered by non-maximum suppression. This score reflects the probability of the class appearing in the prediction box and the degree of overlap between the prediction box and the true box.

[0075] For target detection algorithms, performance evaluation indicators are needed to evaluate the algorithm model. In the present invention, precision, recall and average precision (AP) are used as evaluation indicators based on YOLO. The calculation formulas for precision, recall and F1 score are as follows. In these formulas, TP represents the number of true positive examples; FN represents the number of false negative examples; TN represents the number of true negative examples; FP represents the number of false positive examples.

[0076]

[0077] The calculation formula of AP is shown in formula (3). The value between precision and recall is determined by integration, ranging from 0 to 1. 50 is the average precision of different recall rates when IoU = 0.5, AP 75 It is the average precision of different recall rates when IoU=0.75.

[0078] AP = ∫0 1 Precision*Recal (3)

[0079] In terms of hardware, the present invention uses a server with NVIDIA GeForce RTX 3080Ti GPU and Intel Core i9-10700 K CPU for training. The initial learning rate is set to 0.01, the batch size is set to 8, and the training epoch is set to 300. AP 50 and AP 75 As the evaluation criteria, yolov5 and yolov7 were used for testing, and the experimental results are shown in the table. Fig. 9 and Fig.10 As shown, in the yolov5 test, the black and white blocks and the double circles with white background achieved the fastest convergence speed while achieving the maximum accuracy. Fig.11 and Fig.12 As shown, in the yolov7 test, only the black and white blocks achieve the maximum accuracy and converge at the fastest speed. Therefore, the present invention also uses this pattern as a label for target detection.

[0080] S2. Efficient target detection:

[0081] This section mainly includes three parts: the design of efficient detector; simple target tracker; and the conversion of vibration displacement.

[0082] Step S2.1: Design an efficient target detector based on deep learning to frame the marker points.

[0083] Step S2.2: In order to improve the accuracy of target detection in vibration target displacement measurement, the present invention optimizes the temporal and spatial correlation between frames by introducing a target tracking algorithm, thereby overcoming the independence of adjacent frame detection algorithms.

[0084] Step S2.3, conversion of vibration displacement, in order to obtain physical displacement from the captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. The present invention uses two methods of converting vibration displacement, monocular and binocular, for measurement. When using a monocular camera, when the camera optical axis is perpendicular to the surface of the object, all points on the object plane can be equally reduced to the image plane. In this case, according to the principle of triangle similarity, the real vibration displacement can be obtained.

[0085] Preferably, the step S2.1 is specifically:

[0086] The detector designed by the present invention is mainly based on the FastestDet algorithm. After replacing the backbone network, the connection mode of the fusion layer is redesigned. Finally, the feature maps of different resolutions are input into the head module to obtain the location information and corresponding confidence of the target. This method has been greatly improved in speed and accuracy. The flow chart is as follows Figure 3As shown in Figure 2, the network framework of the target detection algorithm basically includes the backbone network, feature fusion module and prediction framework structure.

[0087] The number of network layers used by Backbone is as follows Figure 4 As shown in Figure 1, it has four hierarchical stages, each of which is preceded by an embedding layer (a regular Conv4×4, stride 4) or a merging layer (a regular Conv2×2, stride 2) for spatial downsampling and channel number expansion. Each stage has a stack of FasterNet blocks. The normalization uses batch normalization (BN), which can be merged into adjacent Conv layers to make reasoning faster while being as effective as others. Considering the running time and effectiveness, ReLU is used as the activation layer based on experience. As shown in Table 1, Layer represents the input network layer, Output represents the output size and number of channels of each network layer; Operator represents the convolution operation of the module. The present invention resizes the acquired image to a size of 352×352 and keeps the number of channels as 3 before inputting it into the backbone network. The output size and number of channels will be changed after Embedding and Merging. FasterNet-Block only performs feature extraction, and the input and output sizes and number of channels remain unchanged. Among them, different FasterNet-Blocks. "Conv_k_c_s" represents a convolutional layer with kernel size k, output channels c, and stride s. "PConv_k_c_s_r" represents a partial convolution with an additional parameter r, i.e., partial ratio, and the rest is the same as "Conv_k_c_s".

[0088] Table 1 Backbone network structure

[0089]

[0090] The present invention adopts a bidirectional fusion strategy based on top-down and bottom-up, and outputs from the Merging operations of stage2 and stage3 in the backbone respectively, and performs Avgpool2d operation. At the same time, the UmSample operation is performed after the Merging operation of stage4, and finally the target feature maps of three different channels are spliced. The present invention uses SimSppf as the backbone network module to fuse the output, and the entire feature fusion process is shown in Table 2. SimSppf is proposed in YOLOv6 and is an advanced version of the feature pyramid network. The dimension of the spatial feature pyramid network can make full use of the details on large pictures and extract macro-dimensional features on small pictures. It can better utilize the original information and achieve the effect of variable receptive field, and has good effects on feature images of different sizes. In the feature extraction process, compared with SPPF, the SiLU activation function is replaced by ReLU. In addition to ensuring accuracy, it effectively avoids problems such as image distortion caused by cropping and scaling operations on the image area. It solves the problem of repeated feature extraction of graph-related features by convolutional neural networks and greatly improves the speed of generating candidate frames. The specific structure of the module is as follows Figure 4 shown.

[0091] Table 2 Fusion layer network structure

[0092]

[0093] The head structure is an important component of target positioning, and is mainly used to predict the type and location of the target. Since classification focuses more on the texture content of the target, and positioning focuses more on the edge information of the target, the present invention does not adopt the traditional coupled detection but continues to use the original decoupled detection head design, which can effectively prevent conflicts between classification and regression tasks. The present invention is slightly different from the currently popular decoupled detection heads (such as the Efficient decoupled Head (EDHead) of Yolov6

[78] and Yolox

[79] , and DecopledHead,). Figure 5 As shown, the fusion information of SimSppf is mainly convolved with 1×1, the number of channels is adjusted, and a feature map is obtained. Then, the convolution operation is performed through the Head layer. The Head structure of the present invention includes two convolution layers of 5×5 and 1×1, respectively. Batch normalization (BN) operation is used after each convolution operation, and ReLU activation function is used in the 5×5 convolution layer. This method enables it to better and faster extract image features and predict the category of each pixel, while it can converge faster.

[0094] like Figure 6 and Figure 7 , the label used is a square, and the angle feature is more significant, so SIoU is used in this target box loss function. SIoU further considers the vector angle between the real box and the predicted box, and redefines the relevant loss function, which specifically includes four parts: angle loss (Anglecost), distance loss (Distance cost), shape loss (Shape cost), and IoU loss (IoUcost).

[0095] Angular cost:

[0096]

[0097] in:

[0098]

[0099]

[0100] Distance Cost: The distance cost is redefined taking into account the angle cost defined above:

[0101] Δ=∑ t＝x，y (1-e -γρt ) (11)

[0102] in:

[0103]

[0104] Shape cost:

[0105]

[0106] in:

[0107]

[0108] Finally, define the loss function:

[0109]

[0110] in:

[0111]

[0112] Preferably, in step S2.2:

[0113] The purpose of target detection is to predict the position and category of the target object while ignoring the spatiotemporal correlation between adjacent frames. In order to optimize the correlation between the detection target frames and more accurately measure the displacement offset of the vibrating target, the present invention adds a target tracking algorithm to avoid the mutual independence of the detection algorithms between adjacent frames. The present invention directly uses Kalman filtering as the tracker of the present invention. The present invention can input the monocular and binocular video sequences into the detector and obtain the target frames of the left and right cameras. At the same time, as the input of the tracking part, the Kalman filter is used to predict the position of the tracking trajectory of the current frame in the next frame, and the IoU between the predicted frame and the actual detection frame is used as the similarity of the two matches. After successful tracking, the center point of the monocular and binocular target frame can be obtained as output. Since the center coordinates are pixel coordinates, it is necessary to finally obtain the real vibration displacement value through monocular measurement and binocular parallax principle. As a whole Figure 8 shown.

[0114] Preferably, in step S2.3:

[0115] In order to obtain physical displacement from the captured image, it is necessary to establish the relationship between pixel coordinates and physical coordinates. The present invention uses two methods of converting vibration displacement, monocular and binocular, to measure. When using a monocular camera, when the camera optical axis is perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane ( Figure 8 In this case, the true vibration displacement can be obtained based on the principle of triangle similarity.

[0116]

[0117] Where D is the size of the known object selected in the target plane; D is the corresponding physical length in the imaging plane; are the known physical size and the corresponding image size, respectively; f is the focal length of the lens; z is the distance from the optical center of the camera to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor. When using binocular vision combined with the parallax principle for measurement, the three-dimensional coordinates of the spatial point can be inversely calculated using the pixel difference between the corresponding points on the imaging surface of the left and right cameras of the spatial point. Figure 8 The parallel binocular vision model shown in the middle. The left and right cameras are parallel to each other. The origin of the coordinate system is O L and O R , which is also the optical center of the left and right cameras. The horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f in the figure is the focal length of the camera. Suppose a point P(x w ,y w , z w ) on the two camera imaging planes are P l (x l ,y l ) and Pr (x r ,y r ). The relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel point can be described by equation (5).

[0118]

[0119] It can be seen from the formula that if the projection position of the target point in the left and right cameras can be found, the depth information z can be determined using the camera focal length f and the baseline b of the two cameras, thereby accurately restoring the three-dimensional information of the target point in the camera coordinates. However, in practical applications, due to lens distortion, the imaging position of the target point will change, so most cameras need to be calibrated and then stereo matched to become a distortion-free pixel plane in practical applications. The pseudo code of the whole process is shown in Table 3.

[0120] Table 3 Pseudocode for vibration displacement conversion

[0121]

[0122]

[0123] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A visual vibration measurement method with high generalization ability, characterized in that: It includes two parts: S1. Select labels with high generalization ability; S2. Efficient target detection and vibration displacement conversion; S1. Select labels with high generalization ability, including: S1.

1. Collect historical labels and make them into datasets; S1.

2. Select classic deep learning algorithms for training and determine the best labels and rules; S2. Efficient target detection, including: S2.

1. Design of efficient target detector based on deep learning, which is used to frame the marker points and improve the detection efficiency; S2.2, Simple Target Tracker,The target tracking algorithm introduces the optimization of the temporal and spatial correlation between frames,,overcoming the independence of the detection algorithms of adjacent frames; S2.

3. Conversion of vibration displacement, establishing the relationship between pixel coordinates and physical coordinates; using single-eye and binocular methods to convert vibration displacement for measurement.

2. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that: S1.

1. Collect historical labels and make them into a data set. The specific method is as follows: We selected visual measurement labels that have been commonly used over the years, fixed the labels in a unified picture, and took pictures with a mobile phone at different angles, different numbers, and different backgrounds on cloudy and sunny days. The picture size was 4,032×3,204 pixels and the format was JPG. We collected images; some of the images were used as training sets and some as test sets; and Labelme was used to manually label the images.

3. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that: S1.

2. Select the classic deep learning algorithm for training to determine the best labels and rules. The specific method is: Among the object detection methods used in deep learning, the YOLO algorithm divides the image into multiple grids, predicts the bounding box within each grid and the object category it contains, and uses the non-maximum suppression (NMS) algorithm to eliminate overlapping bounding boxes; The target detection method is used to reversely quantify the generalization ability of the label; the detection problem is converted into a regression problem, and then the image is divided into an S*S grid to predict whether there is a target in the grid, and the regression method is used to predict the position information and confidence. The confidence calculation formula is as follows: P r (Class i |Object) represents the conditional probability that a target belongs to a certain category under the premise that a grid contains the target; is the ratio of the intersection area to the union area of ​​the predicted bounding box and the actual bounding box; P r (Object) is 1 when the object is included in the grid and 0 when it is not. The prediction box is generated and filtered by non-maximum suppression. This score reflects the probability of the class appearing in the prediction box and the degree of overlap between the prediction box and the true box. Use performance evaluation indicators to evaluate the algorithm model; based on YOLO, precision, recall and average precision AP are used as evaluation indicators; in the formula, TP represents the number of true positive examples; FN represents the number of false negative examples; TN represents the number of true negative examples; FP represents the number of false positive examples; AP determines the value between precision and recall by integration; 4. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that: The step S2.1 is specifically as follows: The detector is based on the FastestDet algorithm. After replacing the backbone network, the connection mode of the fusion layer is redesigned. Finally, the feature maps of different resolutions are input into the head module to obtain the location information and corresponding confidence of the target. The network framework of the target detection algorithm includes the backbone network, feature fusion module and prediction framework structure. The network layers used by Backbone have four hierarchical stages, each of which is preceded by an embedding layer or a merging layer for spatial downsampling and channel number expansion; each stage has a stack of FasterNet blocks; Normalization uses batch normalization and is merged into the adjacent Conv layer; ReLU is used as the activation layer; A bidirectional fusion strategy based on top-down and bottom-up is adopted. The Merging operations of stage2 and stage3 in the backbone are output respectively, and the Avgpool2d operation is performed. At the same time, the UmSample operation is performed after the Merging operation of stage4, and finally the target feature maps of three different channels are spliced. Use SimSppf as the backbone network module to fuse the output; The fusion information of SimSppf is subjected to 1×1 convolution, the number of channels is adjusted, and the feature map is obtained; then the convolution operation is performed through the Head layer. The Head structure includes two convolution layers of 5×5 and 1×1 respectively. Batch normalization (BN) operation is used after each convolution operation, and the ReLU activation function is used in the 5×5 convolution layer.

5. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that: In step S2.2: Add a target tracking algorithm to avoid the mutual independence of detection algorithms between adjacent frames; directly use Kalman filter as the tracker; input the monocular and binocular video sequences into the detector and obtain the target frames of the left and right cameras, which are also used as the input of the tracking part. Use Kalman filter to predict the position of the tracking trajectory of the current frame in the next frame, and use the IoU between the predicted frame and the actual detection frame as the similarity of the two matches; after successful tracking, obtain the center point of the monocular and binocular target frame as the output; since the center coordinates are pixel coordinates, the real vibration displacement value is finally obtained through monocular measurement and binocular parallax principle.

6. The visual vibration measurement method with high generalization ability according to claim 1, characterized in that: In step S2.3: Establish the relationship between pixel coordinates and physical coordinates; use monocular and binocular methods to convert vibration displacement for measurement; when using a monocular camera, when the camera optical axis is perpendicular to the object surface, all points on the object plane can be equally reduced to the image plane, and the real vibration displacement can be obtained based on the principle of triangle similarity; Where D is the size of the known object selected in the target plane; D is the corresponding physical length in the imaging plane; are the known physical size and the corresponding image size, respectively; F is the focal length of the lens; z is the distance from the optical center of the camera to the object plane; after obtaining the conversion factor, the physical dynamic displacement can be calculated by multiplying the pixel displacement by the conversion factor; When binocular vision combined with the parallax principle is used for measurement, the pixel difference of the corresponding points on the imaging surface of the left and right cameras of the space point is used to reverse the three-dimensional coordinates of the space point; the left and right cameras are parallel to each other; the origin of the coordinate system is O L and O R , which is also the optical center of the left and right cameras; the horizontal distance b between the origins of the left and right camera coordinate systems is the baseline, and f is the focal length of the camera; let a point P(x w ,y w , z w ) on the two camera imaging planes are P l (x l ,y l ) and P r (x r ,y r ); The relationship between the coordinates of a spatial point and the coordinates of its corresponding pixel point is described by equation (5): Find the projection position of the target point in the left and right cameras, and use the camera focal length f and the baseline b of the two cameras to determine the depth information z, thereby accurately restoring the three-dimensional information of the target point in the camera coordinates.

Images