A method and system for recognizing a normal position state of a hand brake engine hand wheel

Abstract

The application discloses a human brake engine hand wheel normal position state recognition method and system, and relates to a train component image recognition method and system. The method aims to overcome the problem that the original feature extraction network of the existing Faster R-CNN deep learning algorithm lacks the use of three-dimensional space information, thereby causing insufficient feature extraction capability and relatively low accuracy of fault recognition results. The method comprises the following steps: step 1, inputting a to-be-detected image into a hand wheel normal position detection model; the hand wheel normal position detection model is an improved Faster R-CNN, and the feature extraction network in the improved Faster R-CNN is a multi-modal feature extraction network; the multi-modal feature extraction network is used for extracting the features of the to-be-detected image and the features of a pseudo point cloud image and fusing the features to obtain a fused feature map; and step 2, the hand wheel normal position detection model detects whether the hand wheel in the to-be-detected image is in a normal position state, and the recognition of the hand wheel normal position state is completed.

Description

Technical Field

[0001] This invention relates to a method and system for image recognition of train components. Background Technology

[0002] The manual brake handwheel is an important component of a train, primarily used to replace the thrust of compressed air acting on the brake cylinder piston. It is operated manually to achieve the braking and releasing functions of the vehicle. However, after a period of operation, the manual brake handwheel may become misaligned, causing the brake to malfunction and affecting the safe operation of railway freight cars. Therefore, it is necessary to detect the alignment of the manual brake handwheel near the coupler during operation. Existing methods for detecting the alignment of the manual brake handwheel combine image processing and deep learning technologies, such as Faster R-CNN, to achieve automatic fault identification and alarm, effectively improving the quality and efficiency of train inspection operations.

[0003] However, the existing feature extraction network of the Faster R-CNN deep learning algorithm mainly relies on feature extraction from two-dimensional images and lacks the utilization of three-dimensional spatial information, resulting in insufficient feature extraction capability and relatively low accuracy of fault identification results. Summary of the Invention

[0004] The purpose of this invention is to overcome the problem that the existing feature extraction network of the Faster R-CNN deep learning algorithm lacks the utilization of three-dimensional spatial information, resulting in insufficient feature extraction capability and relatively low accuracy of fault identification results. This invention provides a method and system for identifying the correct position of the handwheel of a manual brake.

[0005] This invention provides a method for identifying the correct position of a manual brake handwheel, comprising:

[0006] Step 1: Input the image to be detected into the handwheel alignment detection model;

[0007] The image to be detected includes a handwheel;

[0008] The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map.

[0009] The pseudo point cloud image is generated from the image to be detected;

[0010] Step 2: The handwheel alignment detection model detects whether the handwheel in the image to be detected is in the correct position, thus completing the identification of the handwheel's correct position.

[0011] Furthermore, in step 1, the process of obtaining the fused feature map is as follows:

[0012] Step 11: Use the handwheel mask of the image to be detected to crop the multi-view depth map, and keep only the part corresponding to the handwheel in the multi-view depth map to obtain the handwheel depth map;

[0013] Multi-view depth maps are obtained by interpolating the depth map of the image to be detected;

[0014] The depth map of the image to be detected is obtained by calculating the pseudo point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera.

[0015] Step 12: Map the handwheel depth map to voxel space based on camera parameters to obtain the three-dimensional virtual points of the handwheel;

[0016] Step 13: Use the handwheel depth map as a new channel of the image to be detected, and stitch it with the image to be detected to obtain an updated image to be detected;

[0017] Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image;

[0018] Step 14: Extract the updated features of the image to be detected to obtain a pixel feature map;

[0019] Extract features from the updated pseudo-point cloud image to obtain a voxel feature map;

[0020] Step 15: Obtain the updated pixel features and updated voxel features through the pixel feature map and voxel feature map;

[0021] Step 16: Perform cross-attention calculation on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

[0022] Furthermore, in step 15, the specific process for obtaining the updated pixel features and the updated voxel features is as follows:

[0023] Step 151: Calculate the feature vector of the voxel keypoint corresponding to each pixel in the pixel feature map in the voxel space. :

[0024]

[0025]

[0026] Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space.

[0027] Furthermore, the feature vector of the pixel keypoint corresponding to each voxel in the voxel feature map is calculated. ;

[0028]

[0029]

[0030] Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space.

[0031] Step 152: Extract the feature vector of each voxel keypoint. Feature vectors of each pixel keypoint The updated pixel features are obtained by fusing them using the following formula. and updated voxel features :

[0032]

[0033]

[0034] in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

[0035] Furthermore, the specific process of step 16 is as follows:

[0036] Step 161: Set the features Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query :

[0037] in, , ;

[0038] Step 162, put the key ,value and query The feature value out is obtained by performing cross-attention mechanism calculation:

[0039]

[0040] Where d is the cross-attention ratio factor;

[0041] Step 163: Pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

[0042] Further, in step 1, the image to be detected is converted into a pseudo-point cloud image using the following formula:

[0043]

[0044] Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

[0045] The present invention also provides a system for identifying the correct position of a manual brake handwheel, comprising:

[0046] The input module is used to input the image to be detected into the handwheel alignment detection model;

[0047] The image to be detected includes a handwheel;

[0048] The handwheel alignment detection model is used to detect whether the handwheel in the image to be detected is in the correct position, thus completing the identification of the handwheel's alignment status.

[0049] The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map.

[0050] The pseudo point cloud image is generated from the image to be detected.

[0051] Furthermore, the multimodal feature extraction network includes:

[0052] The feature extraction preprocessing module is used to crop the multi-view depth map using the handwheel mask of the image to be detected, and only retain the part of the multi-view depth map corresponding to the handwheel to obtain the handwheel depth map;

[0053] Multi-view depth maps are obtained by interpolating the depth map of the image to be detected;

[0054] The depth map of the image to be detected is obtained by calculating the pseudo point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera.

[0055] And used to map the handwheel depth map to voxel space based on camera parameters to obtain three-dimensional virtual points of the handwheel;

[0056] And used to use the handwheel depth map as a new channel of the image to be detected, and to stitch it with the image to be detected to obtain an updated image to be detected;

[0057] Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image;

[0058] And used to extract updated features from the image to be detected to obtain a pixel feature map;

[0059] Extract features from the updated pseudo-point cloud image to obtain a voxel feature map;

[0060] A multimodal mutual guidance and complementation network is used to obtain updated pixel features and updated voxel features through pixel feature maps and voxel feature maps;

[0061] The fusion network is used to perform cross-attention calculations on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

[0062] Furthermore, the multimodal mutual guidance and complementarity network includes:

[0063] The feature vector of each pixel in the pixel feature map corresponding to the voxel keypoint in the voxel space is calculated. :

[0064]

[0065]

[0066] Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space.

[0067] And used to calculate the feature vector of the pixel key point corresponding to each voxel in the voxel feature map in the pixel space. ;

[0068]

[0069]

[0070] Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space.

[0071] The supplementary module is used to extract the feature vectors of each voxel keypoint. Feature vectors of each pixel keypoint The updated pixel features are obtained by fusing them using the following formula. and updated voxel features :

[0072]

[0073]

[0074] in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

[0075] Furthermore, the converged network includes:

[0076] The projection module is used to project the feature set. Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query :

[0077] in, , ;

[0078] The cross-attention calculation module is used to calculate the key. ,value and query The feature value out is obtained by performing cross-attention mechanism calculation:

[0079]

[0080] Where d is the cross-attention ratio factor;

[0081] The feature map generation module is used to pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

[0082] Furthermore, the multimodal feature extraction network includes:

[0083] The detection preprocessing module is used to convert the image to be detected into a pseudo-point cloud image using the following formula:

[0084]

[0085] Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

[0086] The beneficial effects of this invention are:

[0087] This invention replaces the feature extraction network of the existing Faster R-CNN algorithm, introduces the concept of multimodality, and fuses the original image to be detected with image heterogeneous modal data, namely pseudo point cloud images. It fully combines the spatial information and texture details of the image to achieve complementary advantages, which can enhance the spatial understanding of the target, construct an improved feature extraction network, improve the network's detection capability in three-dimensional scenes, and improve the accuracy of fault detection. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the structure of a multimodal feature extraction network in a method for identifying the positive position of the handwheel of a manual brake mechanism according to the present invention.

[0089] Figure 2 This is a flowchart of the training process of an improved Faster R-CNN in a method for recognizing the correct position of a manual brake handwheel according to the present invention. Detailed Implementation

[0090] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0092] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention. Specific Implementation Method 1

[0094] The present embodiment provides a method for identifying the positive position of the handwheel of a manual brake, comprising the following steps:

[0095] Step 1: Input the image to be detected into the handwheel alignment detection model;

[0096] The image to be detected includes a handwheel;

[0097] The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map.

[0098] The pseudo point cloud image is generated from the image to be detected;

[0099] Step 2: The handwheel alignment detection model detects whether the handwheel in the image to be detected is in the correct position, thus completing the identification of the handwheel's correct position. Specific Implementation Method Two

[0101] This embodiment is a further explanation of embodiment one. In this embodiment, the process of obtaining the fused feature map in step 1 is as follows:

[0102] Step 11: Use the handwheel mask of the image to be detected to crop the multi-view depth map, and keep only the part corresponding to the handwheel in the multi-view depth map to obtain the handwheel depth map;

[0103] Multi-view depth maps are obtained by interpolating the depth map of the image to be detected;

[0104] The depth map of the image to be detected is obtained by calculating the pseudo point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera.

[0105] Step 12: Map the handwheel depth map to voxel space based on camera parameters to obtain the three-dimensional virtual points of the handwheel;

[0106] Step 13: Use the handwheel depth map as a new channel of the image to be detected, and stitch it with the image to be detected to obtain an updated image to be detected;

[0107] Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image;

[0108] Step 14: Extract the updated features of the image to be detected to obtain a pixel feature map;

[0109] Extract features from the updated pseudo-point cloud image to obtain a voxel feature map;

[0110] Step 15: Obtain the updated pixel features and updated voxel features through the pixel feature map and voxel feature map;

[0111] Step 16: Perform cross-attention calculation on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

[0112] The other technical features in this embodiment are exactly the same as those in Embodiment 1. Specific Implementation Method 3

[0114] This embodiment is a further explanation of embodiment two. In this embodiment, the specific process of obtaining the updated pixel features and updated voxel features in step 15 is as follows:

[0115] Step 151: Calculate the feature vector of the voxel keypoint corresponding to each pixel in the pixel feature map in the voxel space. :

[0116]

[0117]

[0118] Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space.

[0119] Furthermore, the feature vector of the pixel keypoint corresponding to each voxel in the voxel feature map is calculated. ;

[0120]

[0121]

[0122] Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space.

[0123] Step 152: Extract the feature vector of each voxel keypoint. Feature vectors of each pixel keypoint The updated pixel features are obtained by fusing them using the following formula. and updated voxel features :

[0124]

[0125]

[0126] in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

[0127] The other technical features in this embodiment are exactly the same as those in Embodiment 2. Specific Implementation Method Four

[0129] This embodiment is a further explanation of embodiment three. In this embodiment, the specific process of step 16 is as follows:

[0130] Step 161: Set the features Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query :

[0131] in, , ;

[0132] Step 162, put the key ,value and query The feature value out is obtained by performing cross-attention mechanism calculation:

[0133]

[0134] Where d is the cross-attention ratio factor;

[0135] Step 163: Pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

[0136] The other technical features in this embodiment are exactly the same as those in embodiment three. Detailed Implementation Method Five

[0138] This embodiment is a further explanation of embodiment four. In this embodiment, in step 1, the image to be detected is converted into a pseudo-point cloud image using the following formula:

[0139]

[0140] Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

[0141] The other technical features in this embodiment are exactly the same as those in embodiment four. Specific Implementation Method Six

[0143] This embodiment of a manual brake handwheel positioning identification system includes:

[0144] The input module is used to input the image to be detected into the handwheel alignment detection model;

[0145] The image to be detected includes a handwheel;

[0146] The handwheel alignment detection model is used to detect whether the handwheel in the image to be detected is in the correct position, thus completing the identification of the handwheel's alignment status.

[0147] The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map.

[0148] The pseudo point cloud image is generated from the image to be detected. Detailed Implementation Method Seven

[0150] This embodiment is a further explanation of embodiment six. In this embodiment, the multimodal feature extraction network includes:

[0151] The feature extraction preprocessing module is used to crop the multi-view depth map using the handwheel mask of the image to be detected, and only retain the part of the multi-view depth map corresponding to the handwheel to obtain the handwheel depth map;

[0152] Multi-view depth maps are obtained by interpolating the depth map of the image to be detected;

[0153] The depth map of the image to be detected is obtained by calculating the pseudo point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera.

[0154] And used to map the handwheel depth map to voxel space based on camera parameters to obtain three-dimensional virtual points of the handwheel;

[0155] And used to use the handwheel depth map as a new channel of the image to be detected, and to stitch it with the image to be detected to obtain an updated image to be detected;

[0156] Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image;

[0157] And used to extract updated features from the image to be detected to obtain a pixel feature map;

[0158] Extract features from the updated pseudo-point cloud image to obtain a voxel feature map;

[0159] A multimodal mutual guidance and complementation network is used to obtain updated pixel features and updated voxel features through pixel feature maps and voxel feature maps;

[0160] The fusion network is used to perform cross-attention calculations on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

[0161] The other technical features in this embodiment are exactly the same as those in embodiment six. Detailed Implementation Method Eight

[0163] This embodiment is a further explanation of embodiment seven. In this embodiment, the multimodal mutual guidance and complementarity network includes:

[0164] The feature vector of each pixel in the pixel feature map corresponding to the voxel keypoint in the voxel space is calculated. :

[0165]

[0166]

[0167] Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space.

[0168] And used to calculate the feature vector of the pixel key point corresponding to each voxel in the voxel feature map in the pixel space. ;

[0169]

[0170]

[0171] Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space.

[0172] The supplementary module is used to extract the feature vectors of each voxel keypoint. Feature vectors of each pixel keypoint The updated pixel features are obtained by fusing them using the following formula. and updated voxel features :

[0173]

[0174]

[0175] in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

[0176] The other technical features in this embodiment are exactly the same as those in embodiment seven.

[0177] Furthermore, the converged network includes:

[0178] The projection module is used to project the feature set. Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query :

[0179] in, , ;

[0180] The cross-attention calculation module is used to calculate the key. ,value and query The feature value out is obtained by performing cross-attention mechanism calculation:

[0181]

[0182] Where d is the cross-attention ratio factor;

[0183] The feature map generation module is used to pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

[0184] The other technical features in this embodiment are exactly the same as those in embodiment eight. Detailed Implementation Method Ten

[0186] This embodiment is a further explanation of embodiment nine. In this embodiment, the multimodal feature extraction network includes:

[0187] The detection preprocessing module is used to convert the image to be detected into a pseudo-point cloud image using the following formula:

[0188]

[0189] Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

[0190] The other technical features in this embodiment are exactly the same as those in embodiment nine.

[0191] Example

[0192] This invention improves the feature extraction network. Addressing the issue of insufficient feature extraction capabilities, it employs a multimodal approach. To enhance the learning ability of the feature network and reduce the influence of irrelevant information on feature fusion, an attention mechanism is used to improve the information supplementation network based on multimodal mutual guidance, generating a new feature extraction network. This new network serves as the feature extraction network for Faster R-CNN. The specific implementation process is as follows:

[0193] I. Acquisition of Linear Scan Images

[0194] High-definition equipment is installed at the bottom of the freight train tracks to capture images of the underside of the high-speed passing freight trains. Line scanning is used to achieve seamless image stitching, generating a large-field-of-view, high-precision two-dimensional image.

[0195] II. Coarse Positioning of Components

[0196] Based on the basic information of the truck, the position of the manual brake handwheel is roughly located. By extracting a local area image containing the component from the large bottom image, the time required for fault identification can be effectively reduced and the identification accuracy improved.

[0197] III. Establishing the Original Sample Dataset

[0198] Truck components can be affected by natural or human conditions such as rain, mud, oil stains, and black paint. Furthermore, images taken at different sites may vary. Therefore, the collection of the training image dataset must ensure diversity, collecting images from various conditions and different sites as much as possible.

[0199] Each sample image dataset consists of two parts: the original image set and the labeled information set. The original image set comprises images captured by the device, including the bottom manual brake handwheel. The labeled information set contains images of the bottom manual brake handwheel in an off-center configuration, obtained through manual labeling. There is a one-to-one correspondence between the original image set and the labeled information set; both parts are grayscale images, meaning each image corresponds to one labeled data point.

[0200] IV. Dataset Expansion

[0201] Although the sample dataset includes images under various conditions, data augmentation is still necessary to improve the robustness of the algorithm. Augmentation includes operations such as image translation, scaling, and brightness adjustment, each performed under random conditions. Horizontal and vertical flipping are used to enhance image diversity; image brightness and contrast are adjusted to reduce the impact of uneven lighting and highlight edge features of defects; histogram equalization is used to enhance image contrast; Gaussian blur and random noise are added. Furthermore, due to different camera angles, the shape of the manual brake handwheel seen in the acquired images varies, thus adding multiple shapes of the manual brake handwheel to improve the model's generalization ability.

[0202] V. Model Training

[0203] After the training dataset was established, a deep learning object detection algorithm was used to detect the bottom manual brake handwheel. The model training process is as follows: Figure 1 As shown.

[0204] When detecting handwheels on manual brake machines, the distinction between their normal and misaligned forms is often unclear, leading to frequent detection errors. Even the most basic network-generated bounding boxes are inaccurate. To address this issue, the network was further improved. To fully utilize feature information, a multimodal approach was introduced, fusing the original image with heterogeneous image modal data. This approach effectively combines spatial information and texture details from the image. Furthermore, to reduce the impact of irrelevant spatial information on feature fusion, a multimodal mutual guidance information supplementation network was adopted, combined with an attention mechanism. A novel feature extraction network was designed and embedded into the Faster R-CNN feature extraction network. Experiments show that the improvement effectively enhances the accuracy of manual brake handwheel detection.

[0205] 1. Replacement of the feature extraction network in the Faster R-CNN deep learning algorithm

[0206] In object detection, the ability to extract features significantly impacts detection performance. While Faster R-CNN, as one of the most accurate object detection algorithms currently available, outperforms human performance in detecting general objects, the detection results of existing Faster R-CNN networks using VGG16 as the feature extraction network are still not ideal. This is because the VGG16 network cannot effectively extract high-level features. Therefore, to improve feature extraction capabilities, based on the concept of multimodal approaches, image-to-pseudo-point-cloud data is transformed through projection transformation, achieving a unified representation of image and point-cloud data and narrowing the gap between heterogeneous modal data. Simultaneously, to reduce the impact of irrelevant information on feature fusion, an information supplementation network based on multimodal mutual guidance is used, combined with an attention mechanism. This network can adaptively select useful information from another modality for the supplementary modality, further enhancing feature extraction capabilities.

[0207] 2. Preprocessing of input data

[0208] First, in the given input image, after brightness adjustment, all pixels are back-projected into 3D coordinates using a projection matrix, resulting in a dense frame of pseudo-point cloud data. The spatial correspondence between the image and the pseudo-point cloud is as follows:

[0209] (1)

[0210] In the formula, a and b represent the coordinates of pixels on the two-dimensional image, X, Y and Z represent points in 3D space, w is an adjustment factor used to convert homogeneous coordinates to non-homogeneous coordinates, K is the intrinsic parameter matrix of the camera, and lidar→cam is the projection matrix between the point cloud and the image.

[0211] 3. Modify the information supplementation network based on multimodal mutual guidance.

[0212] The original image is used to obtain an instance-level mask through labeling information, and the original pseudo-point cloud is used to calculate a depth map through intrinsic and extrinsic parameter calculations. Then, the instance-level mask and depth map are used in a multimodal guided complementary network at the data level to obtain an instance-level depth map and instance-level 3D virtual points, which are then used to supplement the original image and pseudo-point cloud inputs, respectively. Next, high-dimensional image features are extracted based on the supplemented inputs, and these features are further supplemented by a multimodal feature-level guided complementary network.

[0213] To enhance the expressive power of the raw input data, a multimodal guided and complementary network is implemented at the data level. The raw input image is a three-channel color image; one-dimensional depth information is added to enhance its geometric perception capabilities. For pseudo-point cloud input, the pseudo-point cloud needs to be densified. Simultaneously, to prevent interference from irrelevant background information, the network input is made more suitable for 3D object detection tasks.

[0214] Given an original image input C, an instance-level mask M is obtained through labeling information. Given a pseudo-point cloud input L obtained by mapping, a depth map D corresponding to the original image input is obtained based on intrinsic and extrinsic parameters. M and D are input into a multimodal mutual guidance and complementation network at the data level to obtain an instance-level depth map F and instance-level 3D virtual points V, which are used to complement the image and the pseudo-point cloud, respectively.

[0215] In the multimodal mutual guidance and complementarity network at the data level, firstly, the input original image depth map D is densed using nearest neighbor interpolation to obtain a dense multi-view depth map D'. Then, D' is cropped using M, retaining only the depth map relevant to the instance to obtain F. Finally, F is mapped back to 3D space using intrinsic and extrinsic parameters to obtain the virtual point V in 3D space.

[0216] Concatenating F and C along the channel dimension yields a new input C', which contains instance-level depth information, allowing the 2D image to acquire more geometric information. Concatenating V and L along the point dimension yields L', which, compared to L, has a more complete and denser outline of the target, contributing to more accurate prediction of manual brake handwheels.

[0217] Next, feature extraction operations will be performed on the multimodal input data C' and L'.

[0218] High-dimensional semantic information is extracted using neural networks. Feature information F' is obtained using operations such as convolution and pooling. Three-dimensional sparse convolution is used as a feature extractor for the pseudo-point cloud to extract voxel features V'.

[0219] After extracting the feature map F' and voxel features V' corresponding to the original image, a multimodal guided complementarity network at the feature level is used to deeply fuse semantic features between different modalities. During the fusion process, guided complementarity is used to reduce semantic ambiguity caused by irrelevant semantic information (such as background and noise). Feature locations in one modality are selected, and the neural network predicts the related feature locations in another modality. Then, an attention mechanism is used to fuse the features. This approach of selecting key regions first and then fusing features not only reduces computational complexity but also facilitates network convergence.

[0220] The feature map F' of the original image is used to obtain the voxel keypoint positions in 3D space for each pixel coordinate through intrinsic and extrinsic parameters, denoted as F'. Then, the voxel coordinates are calculated using equation (2). The position offset in three-dimensional space.

[0221] (2)

[0222] in, As a fully connected network, the voxel keypoint position of each pixel in voxel space can be obtained.

[0223] Given the features of a pixel in the original image feature map And the feature set of pixel key point locations in the corresponding voxel features. ,Will and By performing fusion, the updated features of the pixel are obtained. The feature map is obtained by fusing the features of each pixel in the feature map. As shown in equations (3) and (4). Wherein, Represents the number of feature channels. (·) is the activation function.

[0224] (3)

[0225] (4)

[0226] Given a voxel feature V', the keypoint position in the pixel coordinate system for each voxel coordinate is obtained through intrinsic and extrinsic parameters, denoted as . Then, the pixel coordinates are calculated using equation (5). A set of position offsets in pixel space.

[0227]

[0228] (5)

[0229] in, As a fully connected network, the pixel keypoint locations corresponding to each voxel in the pixel space can be obtained. Given the features of a voxel in the spatial voxel feature map... and the feature set of key locations in the corresponding pixel features. ,Will and The voxel was fused to obtain the updated features. Each voxel feature in the feature map is fused to obtain the supplemented voxel feature map. As shown in equations (6) and (7). Wherein, Represents the number of feature channels. (·) is the activation function.

[0230] (6)

[0231] (7)

[0232] 4. Multimodal feature fusion

[0233] To leverage the features extracted from the original image voxel features extracted from pseudo point clouds The advantage of this patent lies in its ability to fuse two types of features in the newly constructed feature extraction network, perform cross-attention calculations, and obtain pixel-level semantic feature maps. This promotes the learning of semantic consistency between the original image features and the pseudo-point cloud features.

[0234] First, the voxel features extracted from the pseudo point cloud Perform linear projection to obtain the key. ,value Features extracted from the original image Perform linear projection to obtain the query As shown in equation (8), where , , These are linear projections, respectively. The resulting... , , The cross-attention mechanism is calculated as shown in equation (9). Wherein, It is a scaling factor.

[0235] (8)

[0236] (9)

[0237] The final eigenvalues After passing through a fully connected layer, the features extracted from the original image are... The features are then fused together to obtain the final feature map used for model training. Specific improvement details are as follows: Figure 2 As shown.

[0238] Based on practical results, this method embeds an innovative feature extraction network into the Faster R-CNN algorithm and trains it after embedding. This improves the feature extraction capability of the feature extraction network through multimodal spatial feature fusion, resulting in a recognition effect that surpasses the previous algorithm, achieving more accurate identification of manual brake handwheels and their misaligned shapes.

[0239] VI. Fault Identification

[0240] 1. Coarse positioning of components

[0241] Obtain an image of the sub-region containing the manual brake handwheel.

[0242] 2. Load the model

[0243] Load the deep learning object detection model.

[0244] 3. Generate detection results

[0245] Input the sub-image of the region to be detected into the target detection model to obtain the prediction box of whether the handwheel of the manual brake is in the correct or incorrect position.

[0246] 4. Fault Diagnosis

[0247] The detection results determine whether there is any misalignment; if the alarm criteria are met, the area information is obtained.

[0248] 5. Upload alarm

[0249] Based on the fault information, a message is generated indicating the fault location and fault type, and then uploaded to the alarm platform.

[0250] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other embodiments.

Claims

1. A method for identifying the correct position of a handwheel on a manual brake mechanism, comprising: Step 1: Input the image to be detected into the handwheel alignment detection model; The image to be detected includes a handwheel; The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map. The pseudo point cloud image is generated from the image to be detected; Step 2: The handwheel alignment detection model detects whether the handwheel in the image to be detected is in the correct position, thus completing the identification of the handwheel's correct position. The characteristic feature is that, in step 1, the process of obtaining the fused feature map is as follows: Step 11: Use the handwheel mask of the image to be detected to crop the multi-view depth map, and keep only the part corresponding to the handwheel in the multi-view depth map to obtain the handwheel depth map; Multi-view depth maps are obtained by interpolating the depth map of the image to be detected; The depth map of the image to be detected is obtained by calculating the pseudo-point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera; Step 12: Based on the camera parameters, map the handwheel depth map into voxel space to obtain the three-dimensional virtual points of the handwheel; Step 13: Use the handwheel depth map as a new channel of the image to be detected, and stitch it with the image to be detected to obtain an updated image to be detected; Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image; Step 14: Extract the updated features of the image to be detected to obtain a pixel feature map; Extract features from the updated pseudo-point cloud image to obtain a voxel feature map; Step 15: Obtain the updated pixel features and updated voxel features through the pixel feature map and voxel feature map; Step 16: Perform cross-attention calculation on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

2. The method for identifying the correct position of the handwheel of a manual brake mechanism according to claim 1, characterized in that, In step 15, the specific process of obtaining the updated pixel features and the updated voxel features is as follows: Step 151: Calculate the feature vector of the voxel keypoint corresponding to each pixel in the pixel feature map in the voxel space. : Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space. Furthermore, the feature vector of the pixel keypoint corresponding to each voxel in the voxel feature map is calculated. ; Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space. Step 152: Utilize the feature vector of the corresponding pixel keypoint in pixel space for each voxel. Constructing a feature set of pixel key point locations in voxel features ; The feature vector of each pixel in the pixel feature map Feature set of pixel key point locations in voxel features The updated pixel features are obtained by fusing them using the following formula. ; Using the feature vector of the voxel keypoint corresponding to each pixel in voxel space Constructing feature sets of key voxel locations in pixel features ; The feature vector of each voxel in the voxel feature map Feature set of key voxel locations in pixel features The updated voxel features are obtained by fusing them using the following formula. ; in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

3. The method for identifying the correct position of the handwheel of a manual brake mechanism according to claim 2, characterized in that, The specific process of step 16 is as follows: Step 161: Set the features Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query : in, , ; Step 162, put the key ,value and query The feature value out is obtained by performing cross-attention mechanism calculation: Where d is the cross-attention ratio factor; Step 163: Pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

4. The method for identifying the correct position of the handwheel of a manual brake mechanism according to claim 3, characterized in that, In step 1, the image to be detected is converted into a pseudo-point cloud image using the following formula: Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

5. A system for identifying the correct position of a manual brake handwheel, comprising: The input module is used to input the image to be detected into the handwheel alignment detection model; The image to be detected includes a handwheel; A handwheel alignment detection model is used to detect whether the handwheel in the image to be detected is in an upright position, thereby completing the identification of the handwheel's upright position. The handwheel alignment detection model is an improved Faster R-CNN, in which the feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network is used to extract the features of the image to be detected and the features of the pseudo point cloud image and fuse them to obtain a fused feature map. The pseudo point cloud image is generated from the image to be detected; The feature extraction network is characterized by comprising: The feature extraction preprocessing module is used to crop the multi-view depth map using the handwheel mask of the image to be detected, and only retain the part of the multi-view depth map corresponding to the handwheel to obtain the handwheel depth map; Multi-view depth maps are obtained by interpolating the depth map of the image to be detected; The depth map of the image to be detected is obtained by calculating the pseudo-point cloud image based on camera parameters; the camera parameters include the internal and external parameters of the train image acquisition camera; And for mapping the handwheel depth map to voxel space based on the camera parameters to obtain a three-dimensional virtual point of the handwheel; And used to use the handwheel depth map as a new channel of the image to be detected, and to stitch it with the image to be detected to obtain an updated image to be detected; Add the 3D virtual points of the handwheel to the pseudo point cloud image to obtain an updated pseudo point cloud image; And used to extract features from the updated image to be detected to obtain a pixel feature map; Extract features from the updated pseudo-point cloud image to obtain a voxel feature map; A multimodal mutual guidance and complementation network is used to obtain updated pixel features and updated voxel features through pixel feature maps and voxel feature maps; The fusion network is used to perform cross-attention calculations on the updated pixel features and the updated voxel features, and then fuse them to obtain a fused feature map.

6. The handwheel positioning identification system for a manual brake machine according to claim 5, characterized in that, Multimodal mutual guidance and complementarity networks include: The guidance module is used to calculate the feature vector of the voxel keypoint corresponding to each pixel in the pixel feature map in the voxel space. : Where i is the ordinal number of the pixel and the corresponding voxel keypoint; f1 represents the original position of each pixel in the pixel feature map and its corresponding voxel keypoint in voxel space; offset1 represents the position offset of each voxel keypoint coordinate in voxel space; f1 represents a fully connected network; and voxel keypoints are the voxel points used to register the pixel feature map from pixel space to voxel space. And used to calculate the feature vector of the pixel key point corresponding to each voxel in the voxel feature map in the pixel space. ; Where j is the ordinal number of the voxel and the corresponding pixel key point; f1 represents the original position of the corresponding pixel keypoint in the pixel space for each voxel in the voxel feature; offset2 represents the position offset of each pixel keypoint coordinate in the pixel space; f2 represents the fully connected network; and pixel keypoints are the pixels used to register the voxel feature map from the voxel space to the pixel space. The supplementary module is used to utilize the feature vectors of the corresponding pixel keypoints for each voxel in pixel space. Constructing a feature set of pixel key point locations in voxel features ; The feature vector of each pixel in the pixel feature map Feature set of pixel key point locations in voxel features The updated pixel features are obtained by fusing them using the following formula. ; Using the feature vector of the voxel keypoint corresponding to each pixel in voxel space Constructing feature sets of key voxel locations in pixel features ; The feature vector of each voxel in the voxel feature map Feature set of key voxel locations in pixel features The updated voxel features are obtained by fusing them using the following formula. ; in, This is the feature vector of each pixel in the pixel feature map. is the feature vector of each voxel in the voxel feature map; n is the total number of pixels and corresponding voxel keypoints, i=1,2,3...n; m is the total number of voxels and corresponding pixel keypoints, j=1,2,3...m; For the activation function, d k This represents the number of feature channels.

7. The handwheel positioning identification system for a manual brake machine according to claim 6, characterized in that, Converged networks include: The projection module is used to project the feature set. Perform linear projection to obtain the key. Sum ; feature set Perform linear projection to obtain the query : in, , ; The cross-attention calculation module is used to calculate the key. ,value and query The feature value out is obtained by performing cross-attention mechanism calculation: Where d is the cross-attention ratio factor; The feature map generation module is used to pass the feature values ​​'out' through a fully connected layer and then combine them with the feature set. The fusion process yields a fused feature map.

8. The handwheel positioning identification system for a manual brake machine according to claim 7, characterized in that, Multimodal feature extraction networks include: The detection preprocessing module is used to convert the image to be detected into a pseudo-point cloud image using the following formula: Where a and b represent the coordinates of pixels in the image to be detected, X, Y and Z represent points in the pseudo point cloud image, w is an adjustment factor used to convert non-homogeneous coordinates to homogeneous coordinates, K is the intrinsic parameter matrix, and Lidar→cam is the projection matrix between the pseudo point cloud image and the image to be detected.

Images