Flexible suspension bracket key point detection method based on lightweight YOLOPose

Through the flexible suspension key point detection method based on lightweight YOLOPose, the deformation problem of flexible contact network suspension in outdoor environment is solved, and the detection of key point without heat map is realized, and the coordinates and three-dimensional coordinate calculations are directly predicted to meet the real-time rail transit Inspection requirements improve detection accuracy and speed.

CN118887291APending Publication Date: 2024-11-01SHANGHAI TIANLIAN RAIL TRANSIT INSPECTION TECH CO LTD

Patent Information

Application Number
CN202411008649.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-01

Smart Images

  • Figure CN118887291A_ABST
    Figure CN118887291A_ABST
Patent Text Reader

Abstract

The invention discloses a lightweight YOLOPose-based flexible suspension bracket key point detection method, which comprises the following steps of: shooting a picture of a suspension bracket by using a binocular camera, and marking key points of the suspension bracket to obtain a training set; an improved YOLOPose network is constructed, and the improved YOLOPose network is trained by using the training set; and detecting the key points of the suspension bracket by using the trained improved YOLOPose network. The flexible suspension bracket key point detection method based on lightweight YOLOPose is a heat-map-free key point detection mode, coordinates of key points can be directly predicted, x and y coordinates of each key point are directly output in a regression mode, the process of generating and analyzing a heat map is omitted, the precision and the regression reasoning speed are considered, and the method is suitable for large-scale popularization and application. And the requirement of real-time detection application can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a flexible suspension frame key point detection method based on lightweight YOLOPose, and belongs to the technical field of weld tracking. Background Art

[0002] By the end of 2023, the national railway operating mileage will reach 159,000 kilometers, including 45,000 kilometers of high-speed railways. Both subways and high-speed railways require a contact network suspension system to power the trains and drive them.

[0003] See also Figure 1 and Figure 2 As shown in the figure, from the physical form, the overhead contact network suspension system is divided into rigid and flexible. The rigid overhead contact network is mainly in the tunnel, and the contact is fastened to the top of the tunnel through components. The flexible overhead contact network system is mainly outdoors, and consists of columns fixed beside the railway, suspension frames and contact wires.

[0004] The rigid contact network is mainly distributed in the relatively stable environment of the tunnel, and the probability of failure is small. However, the flexible suspension frame is in the outdoor environment for a long time. Under the influence of external forces such as high temperature and freezing, strong wind, and rain, snow and dust erosion, the fasteners are prone to fall off, causing the suspension frame to deform. At the least, it will lead to unstable power supply of the running train, resulting in various electrical failures on the train, and at the worst, it will lead to major failures such as power outage and loss of connection of the train.

[0005] Therefore, a flexible suspension key point detection method based on lightweight YOLOPose is needed to solve the above problems. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a key point detection method for a flexible suspension frame based on lightweight YOLOPose.

[0007] A flexible suspension frame key point detection method based on lightweight YOLOPose includes the following steps: Step 1: Use a camera to take a picture of the suspension frame, and mark the key points of the suspension frame to obtain a training set; Step 2: construct a TinyYOLOPose network, and train the TinyYOLOPose network using the training set obtained in step 1; The TinyYOLOPose network includes a backbone network, a PANet multi-scale feature fusion module, a detection head and a multi-scale prediction module; The backbone network includes a first convolution module, a second convolution module, a first 3*C3 module, a third convolution module, a first 6*C3 module, a fourth convolution module, a first 9*C3 module, a fifth convolution module, a second 3*C3 module and an SPPF module connected in sequence; The PANet multi-scale feature fusion module includes a sixth convolution module, a first upsampling module, a first connection module, a third 3*C3 module, a seventh convolution module, a second connection module, a fourth 3*C3 module, an eighth convolution module, a third connection module and a fifth 3*C3 module connected in sequence; The first connection module is connected to the first 9*C3 module, the sixth convolution module is connected to the SPFF module, and the second connection module is connected to the sixth convolution module; The detection head includes a first detection head head3 and a second detection head head4, and the first detection head head3 and the second detection head head4 are connected to the fourth 3*C3 module and the fifth 3*C3 module respectively; The multi-scale prediction module includes a first prediction module and a second prediction module, the first prediction module and the second prediction module are respectively connected to the first detection head head3 and the second detection head head4, and the first prediction module and the second prediction module both include a prediction box prediction module and a key point prediction module; Step 3: Use the trained improved YOLOPose network obtained in step 2 to detect the key points of the suspension frame.

[0008] Furthermore, the first 3*C3 module, the second 3*C3 module, the third 3*C3 module, the fourth 3*C3 module and the fifth 3*C3 module are all superpositions of 3 C3 modules, the first 6*C3 module is a superposition of 6 C3 modules, and the first 9*C3 module is a superposition of 9 C3 modules.

[0009] Furthermore, the C3 module includes a first CBS module, a CSPBottleneck module, a third connection module and a second CBS module connected in sequence, and also includes a third CBS module, the first CBS module is connected to the third connection module through the third CBS module, the output of the second CBS module is the output of the C3 module, the CSPBottleneck module includes two CBS modules, the first CBS module, the second CBS module and the third CBS module are all CBS modules, and the CBS module includes a convolution module, a normalization layer and an adaptive activation function.

[0010] Furthermore, the SPFF module includes a fourth CBS module, a fourth connection module and a fifth CBS module connected in sequence, and also includes a first Maxpool module, a second Maxpool module and a third Maxpool module. The fourth CBS module, the first Maxpool module, the second Maxpool module and the third Maxpool module are connected in sequence, the first Maxpool module, the second Maxpool module and the third Maxpool module are all connected to the fourth connection module, the output of the fifth CBS module is the output of the SPFF module, the first Maxpool module, the second Maxpool module and the third Maxpool module are all maximum pooling layers, the fourth CBS module and the fifth CBS module are both CBS modules, and the CBS module includes a convolution module, a normalization layer and an adaptive activation function.

[0011] Furthermore, the loss function of the TinyYOLOPose network is: , , , , , in, represents the comprehensive loss of the improved YOLOPose network, represents the classification loss, represents the anchor box loss, represents the key point loss, represents the confidence of the key point, OKS represents the similarity of the object key point, BCE represents the binary cross entropy loss function, CIoU represents the complete intersection-over-union function, is the Euclidean distance between the predicted keypoint and the true keypoint, is a fixed constant used to adjust the importance and visibility of different key points. Is a sign of whether the key point is visible. represents the confidence of the nth key point prediction, represents the total number of all key points, Represents the prediction box, and the anchor box scaling rate is , Represents a preset frame, Represents the coordinates of the four vertices of the box.

[0012] Furthermore, , , , .

[0013] Invention principle: Aiming at the demand of rail transit suspension frame deformation detection, a method for distinguishing deformation is defined, which can accurately measure the deformation of the suspension frame. At the same time, an improved YOLOPose model is proposed. By making the YOLOPose model lightweight, the special diagnosis extraction layer of large target resolution is eliminated, so that the modified model can achieve the accuracy of the original model while greatly improving the model's reasoning speed, thereby meeting the real-time detection needs of rail transit.

[0014] Beneficial effect: The key point detection method of the flexible suspension frame based on lightweight YOLOPose of the present invention is a key point detection method without heat map, which can directly predict the coordinates of the key points, and directly output the x and y coordinates of each key point by regression, omitting the process of generating and parsing the heat map, and it takes into account both accuracy and regression reasoning speed, and can meet the needs of real-time detection applications of rail transit suspension frame deformation detection.

[0015] A method for detecting deformation of key points of a rail transit flexible suspension frame, characterized in that it comprises the following steps: Step 11, take pictures of the same suspension rack by using the left and right cameras to obtain two pictures of the suspension rack respectively; Step 12: Find the position of the same key point in the two images by using the flexible suspension frame key point detection method based on lightweight YOLOPose as described in any one of claims 1 to 6, and calculate the three-dimensional coordinates of the key point; Step 13: Repeat steps 11 and 12 to obtain the three-dimensional coordinates of all key points on the suspension frame, calculate the distances between different key points, and thus evaluate the deformation of the suspension frame.

[0016] Beneficial effect: The deformation detection method of the rail transit flexible suspension frame of the present invention is to take pictures of the same suspension frame through two left and right cameras, and find the position of the same key point in the two pictures through the flexible suspension frame key point detection method based on lightweight YOLOPose, so as to solve the three-dimensional coordinates of the key points of the suspension frame, obtain the distance between the key points, and evaluate the deformation of the suspension frame. Among them, the flexible suspension frame key point detection method based on lightweight YOLOPose has no heat map key point detection method, can directly predict the coordinates of the key points, and directly output the x and y coordinates of each key point through regression, omitting the process of generating and parsing the heat map, and it takes into account both accuracy and regression reasoning speed, and can meet the needs of real-time detection applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flexible contact network; Figure 2 It is a rigid contact network; Figure 3 The fasteners of the flexible suspension frame are sent to cause deformation; Figure 4 Define graphs for the key points of the hanger; Figure 5 Define the 2D concept diagram for the suspension bracket; Figure 6 This is an example diagram of binocular key point distance detection; Figure 7 It is the original YOLOPose network topology; Figure 8 The original image (left) and the hanger image found for target recognition (right); Fig. 9 This is a diagram of the lightweight YOLOPose network architecture; Fig.10 Decomposition diagram for submodules;

[0018] Fig.11 The following is a comparison of the detection effects of the three models. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings to more clearly and completely illustrate the technical solutions of the present invention.

[0020] 1. Passive binocular detection of suspension frame deformation:

[0021] At present, deformation detection of flexible suspension frames is a difficult problem in the field of rail transit. The solution we adopt is to determine whether the suspension frame has deformed based on the distance between the fasteners on the suspension frame through passive binocular detection. The principle of binocular detection is to take pictures of the same suspension frame with two cameras on the left and right, and use the algorithm to find the position of the same key point in the two pictures, so as to solve the three-dimensional coordinates of the key points of the suspension frame, obtain the distance between the key points, and thus evaluate the deformation of the suspension frame.

[0022] pass Figure 4 The suspension bracket is positioned and its two-dimensional model is extracted, such as Figure 5 shown.

[0023] Once the three-dimensional coordinates of each key point are obtained, the distance between the key points can be calculated. The calculation formula is as follows: (1) (2) (3) (4) (5) in, Indicates each key point, Represents the distance between key points, such as Figure 6 shown.

[0024] 2. Based on non-heat map key point detection:

[0025] The accuracy of 3D detection by passive binocular cameras is related to the image quality, binocular baseline position, camera focal length, and key point positioning accuracy. In the case of rail transit detection, how to correctly find the position of key points in two images is a key factor affecting binocular positioning accuracy.

[0026] Currently, there are two main technical directions for detecting key points based on deep learning, one is the heat map-based method, and the other is the heat map-free method.

[0027] In the heatmap-based method, the model usually outputs a set of heatmaps, each heatmap corresponds to a key point. Each pixel value in the heatmap represents the probability that the position is a key point. Through these heatmaps, the pixel position with the highest probability can be found to determine the position of the key point. The advantage of this method is that the key point prediction is relatively accurate. The disadvantage is that a large number of heatmaps need to be generated, which cannot meet the real-time processing requirements, and the loss optimization of generating heatmaps and regressing key point coordinates is separate, making it difficult to find the global optimum. In contrast, the heatmap-free method directly predicts the coordinates of the key points. This method usually directly outputs the x and y coordinates of each key point by regression, omitting the process of generating and parsing heatmaps, and can meet the needs of real-time detection applications. However, the disadvantage is that regression problems are usually more difficult to train than classification problems, especially when key points need to be accurately located. In addition, methods that directly predict coordinates may be more sensitive to the distribution and annotation quality of the data.

[0028] YOLOPose is a non-heatmap key point detection method, which was first proposed to solve the problem of human posture estimation. Here, it is improved and applied to the application scenario of suspension frame key point detection.

[0029] 3. Lightweight YOLOPose key point detection: The YOLOPose network evolved from the YOLO target detection framework and is an end-to-end training model optimized for OKS.

[0030] YOLOPose is a deep learning model proposed for human posture estimation. Before this, there were two technical solutions, the top-down solution and the bottom-up solution.

[0031] The top-down approach is to first identify each person in the entire image (usually using a bounding box), and then estimate the pose for each detected person separately. The advantage of this approach is high accuracy because it can focus on the pose estimation of a single person, but the disadvantage is that the amount of computation is large, especially when there are many people. And the timeliness of detection is positively correlated with the number of people in a picture.

[0032] The bottom-up method first identifies all possible key points in the entire image, and then combines these key points into poses of different people. The advantage of this method is that it is fast and suitable for processing large-scale data, but sometimes it may sacrifice some accuracy. In the bottom-up method of human pose estimation, the model first identifies all the key points in the image, and then a post-processing step is required to group these scattered key points according to the individuals they belong to to form a complete skeleton structure. This grouping process is a necessary step in the bottom-up method because the model initially only identifies single key points without considering the association between them.

[0033] YOLOPose effectively combines the high accuracy of top-down methods and the high efficiency of bottom-up methods by performing these two tasks simultaneously in a single forward pass. This combination can improve the overall performance and application flexibility, especially in scenarios where a large number of people and real-time applications need to be processed simultaneously. It avoids the complexity of requiring additional post-processing steps. Each detected bounding box not only indicates the location of the individual, but is also directly associated with a complete posture. This means that the key points have been intrinsically grouped according to their respective skeletons during detection.

[0034] This inherent grouping of keypoints simplifies the overall processing pipeline as it reduces the reliance on complex post-processing algorithms. The associated pose of each bounding box directly provides the correct grouping of keypoints, thus speeding up the processing and potentially improving overall accuracy and efficiency.

[0035] YOLOPose detects the suspension system based on YOLO, while the original YOLO focuses on 80 types of COCO object detection. There is only one type of target detection for the suspension, with only 6 elements. There is one element for the category, 4 elements for the bounding box, and one element for the confidence of the class. Corresponding to each grid position, there are 3 anchor boxes of different shapes.

[0036] According to the modeling of the key point positions of the suspension frame, there are a total of 9 eigenvalues, and each feature point corresponds to three values, namely: (6) Therefore, for each box, a keypoint detection output vector contains 27 elements: (7) YOLO-Pose uses CSP-darknet53 as the backbone network for feature extraction, and uses PANet to fuse features of different scales in the backbone network. Next, it detects different feature sizes and obtains prediction boxes and key points, such as Figure 7 shown.

[0037] Although YOLOPose has excellent detection efficiency compared to other key point detection models, the network model is too complex and inefficient for the specific scenario of rail transit suspension frame deformation detection.

[0038] Because first, the system of the present invention will initially screen out the hanging rack pictures that meet the format size, such as Figure 8 shown.

[0039] from Figure 7 It can be seen that the pixel size occupied by the key points is relatively fixed, which means that the large-scale feature layer is meaningless in this detection scenario. At the same time, the PANet feature fusion layer does not require the fusion of large-scale features. The network is adjusted to obtain the lightweight YOLOPose network structure of the present invention, such as Fig. 9 shown.

[0040] exist Figure 7 and Fig. 9 In the figure, Conv is a convolution module. N*C3 means N C3 modules are superimposed (6*C3 means 6 C3 modules are superimposed), Upsample is upsampling, and Concat is accumulating feature layers of different sizes. The detection head (head1, head2, head3, head4) performs target detection on feature maps of different scales (resolutions), and finally obtains the detection vector of key points, which contains Box and Key points, representing the target recognition information and key point coordinate information respectively.

[0041] The C3 and SPPF networks are decomposed as follows: Fig.10 shown. Fig.10 In the figure, BN is the normalization layer, SiLU is the adaptive activation function, and Maxpool is the maximum pooling layer.

[0042] The present invention uses CIoU loss to converge the bounding box. For one of the boxes, the position is , the scaling factor is , the loss function is defined as follows: (8)

[0043] in is the prediction box, and the anchor box is scaled by , suffix Represents the coordinates of the four vertices of the box.

[0044] OKS is used to evaluate the similarity between the key point position and the true key point position. It is an important indicator for evaluating the performance of the algorithm in the key point detection task. Unlike the ordinary IoU loss, the ordinary IoU loss will have the problem of gradient vanishing when there is no overlap, while the OKS loss will not have this problem.

[0045] For each bounding box, we store the entire keypoint information. So if a The real bounding box is matched with the anchor point at that location. We predict the key point with the center of the anchor box. OKS calculates the similarity of each key point respectively, and then finally obtains the key point OKS loss, as shown in the following formula: (9) (10) is the Euclidean distance between the predicted keypoint and the true keypoint, It is a fixed constant that is related to the type of key point and is used to adjust the importance and visibility of different key points. Is a flag that indicates whether the key point is visible.

[0046] Corresponding to each keypoint, a confidence parameter is learned that shows whether the keypoint is a point inside this suspension: (11) represents the confidence of the prediction of the nth key point, and formula 11 represents the scaling ratio of , BCE represents the binary cross entropy loss function, Is a sign of whether the key point is visible. Represents the total number of all key points.

[0047] All loss functions are shown below: (12) , , , ,These 4 parameters are hyperparameters and need to be adjusted dynamically during the training process.

[0048] 5. Experimental Data

[0049] We use OpenPose, YOLOPose and improved YOLOPose models for experimental comparison. OpenPose is a key point detection model based on heatmaps, YOLOPose is a native model, and improved YOLOPose is a lightweight YOLOPose model. The size of the input image is 512*512.

[0050] First of all, from the overall detection effect, the three models have little difference in detection, such as Fig.11 shown.

[0051] The AP value detection summary is shown in Table 1: By annotating key points of 1,000 samples and training with different network models, we obtain the following parameters:

[0052] Table 1 Comparison of accuracy of different models

[0053] It can be seen from Table 1 that although the parameters of the present invention are reduced and the multi-size fusion network is removed, its accuracy performance is slightly better than OpenPose and YOLPPose.

[0054] At the same time, for the trained model, convert it into onnx format, and perform key point inference on a single image on the same hardware platform. The image format is 2100*1200. The final time for inferring a single image is as follows:

[0055] It can be seen that the present invention has an obvious advantage in reasoning speed.

Claims

1. A flexible suspension frame key point detection method based on lightweight YOLOPose, characterized in that: The steps include: Step 1: Use a camera to take a picture of the suspension frame, and mark the key points of the suspension frame to obtain a training set; Step 2: construct an improved YOLOPose network, and train the improved YOLOPose network using the training set obtained in step 1; The improved YOLOPose network includes a backbone network, a PANet multi-scale feature fusion module, a detection head and a multi-scale prediction module; The backbone network includes a first convolution module, a second convolution module, a first 3*C3 module, a third convolution module, a first 6*C3 module, a fourth convolution module, a first 9*C3 module, a fifth convolution module, a second 3*C3 module and an SPPF module connected in sequence; The PANet multi-scale feature fusion module includes a sixth convolution module, a first upsampling module, a first connection module, a third 3*C3 module, a seventh convolution module, a second connection module, a fourth 3*C3 module, an eighth convolution module, a third connection module and a fifth 3*C3 module connected in sequence; The first connection module is connected to the first 9*C3 module, the sixth convolution module is connected to the SPFF module, and the second connection module is connected to the sixth convolution module; The detection head includes a first detection head head3 and a second detection head head4, and the first detection head head3 and the second detection head head4 are connected to the fourth 3*C3 module and the fifth 3*C3 module respectively; The multi-scale prediction module includes a first prediction module and a second prediction module, the first prediction module and the second prediction module are respectively connected to the first detection head head3 and the second detection head head4, and the first prediction module and the second prediction module both include a prediction box prediction module and a key point prediction module; Step 3: Use the trained improved YOLOPose network obtained in step 2 to detect the key points of the suspension frame.

2. The method for detecting key points of a flexible suspension frame based on lightweight YOLOPose according to claim 1, characterized in that: The first 3*C3 module, the second 3*C3 module, the third 3*C3 module, the fourth 3*C3 module and the fifth 3*C3 module are all superpositions of 3 C3 modules, the first 6*C3 module is a superposition of 6 C3 modules, and the first 9*C3 module is a superposition of 9 C3 modules.

3. The method for detecting key points of a flexible suspension frame based on lightweight YOLOPose as claimed in claim 2, characterized in that: The C3 module includes a first CBS module, a CSP Botleneck module, a third connection module and a second CBS module connected in sequence, and also includes a third CBS module, the first CBS module is connected to the third connection module through the third CBS module, the output of the second CBS module is the output of the C3 module, the CSP Botleneck module includes two CBS modules, the first CBS module, the second CBS module and the third CBS module are all CBS modules, and the CBS module includes a convolution module, a normalization layer and an adaptive activation function.

4. The method for detecting key points of a flexible suspension frame based on lightweight YOLOPose according to claim 1, characterized in that: The SPFF module includes a fourth CBS module, a fourth connection module and a fifth CBS module connected in sequence, and also includes a first Maxpool module, a second Maxpool module and a third Maxpool module. The fourth CBS module, the first Maxpool module, the second Maxpool module and the third Maxpool module are connected in sequence, the first Maxpool module, the second Maxpool module and the third Maxpool module are all connected to the fourth connection module, the output of the fifth CBS module is the output of the SPFF module, the first Maxpool module, the second Maxpool module and the third Maxpool module are all maximum pooling layers, the fourth CBS module and the fifth CBS module are both CBS modules, and the CBS module includes a convolution module, a normalization layer and an adaptive activation function.

5. The method for detecting key points of a flexible suspension frame based on lightweight YOLOPose according to claim 1, characterized in that: The loss function of the improved YOLOPose network is: , , , , , in, represents the comprehensive loss of the improved YOLOPose network, represents the classification loss, represents the anchor box loss, represents the key point loss, represents the confidence of the key point, OKS represents the similarity of the object key point, BCE represents the binary cross entropy loss function, CIoU represents the complete intersection-over-union function, is the Euclidean distance between the predicted keypoint and the true keypoint, is a fixed constant used to adjust the importance and visibility of different key points. Is a sign of whether the key point is visible. represents the confidence of the nth key point prediction, represents the total number of all key points, Represents the prediction box, and the anchor box scaling rate is , Represents a preset frame, Represents the coordinates of the four vertices of the box.

6. The method for detecting key points of a flexible suspension frame based on lightweight YOLOPose according to claim 5, characterized in that: , , , 。 7. A method for detecting deformation of a rail transit flexible suspension frame, characterized in that: The following steps are involved: Step 11, take pictures of the same suspension rack by using the left and right cameras to obtain two pictures of the suspension rack respectively; Step 12: Find the position of the same key point in the two images by using the flexible suspension frame key point detection method based on lightweight YOLOPose as described in any one of claims 1 to 6, and calculate the three-dimensional coordinates of the key point; Step 13: Repeat steps 11 and 12 to obtain the three-dimensional coordinates of all key points on the suspension frame, calculate the distances between different key points, and thus evaluate the deformation of the suspension frame.

Citation Information

Patent Citations

  • Real-time key point detection method based on lightweight neural network

    CN110378253A

  • Human body acupuncture point prediction method based on key point derivation and massage robot

    CN116721150A

  • Student classroom action detection method, system and device and storage medium

    CN116978128A

  • Target detection method and apparatus, electronic device and storage medium

    WO2022170742A1

Cited By

  • Unmanned aerial vehicle target intelligent identification and perception system and method based on deep learning

    CN119649254A

  • Target detection segmentation method, system and device for engineering drawing steel bar bulk sample annotation area and storage medium

    CN120894381A