Hook lifting rod bending image recognition method based on deep learning
By adopting an improved Cascade-rcnn detection network in hook and lifting rod bending image recognition, Resnet50, FPN and ECA-Net combined with deformable convolution, the identification accuracy caused by the messy background of hook and lifting rod bending image and the many similar parts are solved. The problem of low rate achieves higher recognition accuracy.
Patent Information
- Application Number
- CN202411625328.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-21
AI Technical Summary
Due to the messy background of the bending image of the hook and lifting rod and many similar parts, the recognition accuracy is low.
Using a deep learning-based hook and lifting rod bending image recognition method, the improved Cascade-rcnn detection network, which includes Resnet50, FPN and ECA-Net, enhances feature extraction capabilities through deformable convolution and ECA-NET structures and improves the quality extraction capability. The accuracy of identification of irregular targets.
By adding ECA-NET structure after FPN and using deformable convolution, the recognition accuracy of the model for hook lifting bend images is improved, and the problem of low recognition accuracy caused by messy backgrounds and many similar parts is solved.
Smart Images
Figure CN119495015A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method for identifying a hook lifting rod bending image based on deep learning. Background Art
[0002] In the hook lift rod bending fault detection, the fault detection is performed by manually checking the image. The automatic image recognition method can improve the detection efficiency and stability. In recent years, deep learning and artificial intelligence have continued to develop and mature in technology. The use of deep learning for hook lift rod bending fault recognition can effectively improve the detection accuracy. However, due to the relatively cluttered background of the hook lift rod bending image and the large number of similar parts, the recognition accuracy is low. Summary of the invention
[0003] The purpose of the present invention is to propose a method for identifying a bent hook lifting rod image based on deep learning in order to address the problem in the prior art that the background of the bent hook lifting rod image is relatively cluttered and there are many similar parts, resulting in low recognition accuracy.
[0004] The technical solution adopted by the present invention to solve the above technical problems is:
[0005] The method for identifying the bending image of a hook lifting rod based on deep learning comprises the following steps:
[0006] Obtaining an image of a hook-lifting rod to be identified, and inputting the image of the hook-lifting rod to be identified into a trained detection network to obtain an output recognition result;
[0007] The detection network is an improved Cascade-rcnn detection network, which includes Resnet50, FPN and ECA-Net. The improved Cascade-rcnn detection network specifically performs the following steps:
[0008] The input image in Resnet50 first passes through Conv1 to extract features using deformable convolution to generate a feature map:
[0009] The feature map generated by Conv1 is input into Conv2 and deformable convolution is used to extract features to generate a feature map;
[0010] The feature map generated by Conv2 is input into Conv3 and deformable convolution is used to extract features to generate a feature map;
[0011] The feature map generated by Conv3 is input into Conv4 and deformable convolution is used to extract features to generate a feature map;
[0012] The feature map generated by Conv4 is input into Conv5 and deformable convolution is used to extract features to generate a feature map;
[0013] In FPN, the feature map generated by Conv2 is used as feature map A;
[0014] After downsampling, feature map A is fused with the feature map generated by Conv3 to form feature map B.
[0015] After downsampling, feature map B is fused with the feature map generated by Conv4 to form feature map C.
[0016] After downsampling, feature map C is fused with the feature map generated by Conv5 to form feature map D.
[0017] After upsampling, feature map D is fused with feature map C to form feature map CC.
[0018] After upsampling, feature map CC is fused with feature map B to form feature map BB.
[0019] After upsampling, feature map BB is fused with feature map A to form feature map AA.
[0020] After the feature maps AA, BB, CC, and D pass through ECA-Net, ROI pooling is performed to obtain the output category and position, that is, the image recognition result of the hook lifting rod bending.
[0021] Furthermore, the trained detection network is obtained through the following steps:
[0022] Step 1: Obtain a hook lifting rod bending image, and grayscale the hook lifting rod bending image to obtain a hook lifting rod bending grayscale image;
[0023] Step 2: Mark the bending position of the hook lifting rod in the hook lifting rod bending grayscale image to construct a sample data set;
[0024] Step 3: Use the sample data set to train the detection network to obtain a trained detection network.
[0025] Furthermore, the sample data set is a sample data set after data amplification, and the data amplification includes translation, scaling, and brightness adjustment.
[0026] Furthermore, the activation function of the improved Cascade-rcnn detection network is expressed as:
[0027]
[0028] Among them, z represents the horizontal axis coordinate of the function.
[0029] Furthermore, the activation function of the ResNet50 residual network is:
[0030]
[0031] Furthermore, the loss function of the improved Cascade-rcnn detection network is expressed as:
[0032] New_Loss=0.25*SmoothL1Loss+0.75*DIoU
[0033] Among them, SmoothL1Loss represents the smooth L1 loss function, and DIoU represents the distance loss between the predicted box and the actual box.
[0034] Furthermore, SmoothL1Loss is expressed as:
[0035]
[0036] Furthermore, DIoU is expressed as:
[0037]
[0038] Among them, IoU represents the intersection-over-union ratio between the predicted box and the actual box, b represents the center point of the predicted box, and b gt represents the center point of the real box, c represents the diagonal distance of the minimum closure area, and ρ represents the Euclidean distance between the two center points.
[0039] Furthermore, the output of each convolutional layer in ResNet50 is expressed as:
[0040]
[0041] Among them, y represents the convolution output value, p0 represents the current sampling point, pn represents the enumeration of the sampling positions in each sliding window during the convolution process, w(pn) represents the weight of pn, and x represents the input feature map.
[0042] Furthermore, the deformable convolution is expressed as:
[0043]
[0044] Among them, Δpn represents the offset of the convolution window position.
[0045] The beneficial effects of the present invention are:
[0046] This application adds an ECA-NET structure after FPN to ensure that the attention mechanism obtains more accurate feature maps, and adds deformable convolution to give full play to the characteristics of the receptive field in the network, improve the model's ability to extract features for irregular targets, and thus solve the problem of low recognition accuracy due to the relatively cluttered background and many similar parts in the image of the bent hook and lifting rod. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is the fault identification flow chart for this application;
[0048] Figure 2 This is a schematic diagram of the improved Cascade-rcnn detection network structure;
[0049] Figure 3 This is a schematic diagram of the ECA-NET structure. DETAILED DESCRIPTION
[0050] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.
[0051] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, the hook lifting rod bending image recognition method based on deep learning in this embodiment includes the following steps:
[0052] Obtaining an image of a hook-lifting rod to be identified, and inputting the image of the hook-lifting rod to be identified into a trained detection network to obtain an output recognition result;
[0053] The detection network is an improved Cascade-rcnn detection network, which includes Resnet50, FPN and ECA-Net. The improved Cascade-rcnn detection network specifically performs the following steps:
[0054] The input image in Resnet50 first passes through Conv1 to extract features using deformable convolution to generate a feature map:
[0055] The feature map generated by Conv1 is input into Conv2 and deformable convolution is used to extract features to generate a feature map;
[0056] The feature map generated by Conv2 is input into Conv3 and deformable convolution is used to extract features to generate a feature map;
[0057] The feature map generated by Conv3 is input into Conv4 and deformable convolution is used to extract features to generate a feature map;
[0058] The feature map generated by Conv4 is input into Conv5 and deformable convolution is used to extract features to generate a feature map;
[0059] In FPN, the feature map generated by Conv2 is used as feature map A;
[0060] After downsampling, feature map A is fused with the feature map generated by Conv3 to form feature map B.
[0061] After downsampling, feature map B is fused with the feature map generated by Conv4 to form feature map C.
[0062] After downsampling, feature map C is fused with the feature map generated by Conv5 to form feature map D.
[0063] After upsampling, feature map D is fused with feature map C to form feature map CC.
[0064] After upsampling, feature map CC is fused with feature map B to form feature map BB.
[0065] After upsampling, feature map BB is fused with feature map A to form feature map AA.
[0066] After the feature maps AA, BB, CC, and D pass through ECA-Net, ROI pooling is performed to obtain the output category and position, that is, the image recognition result of the hook lifting rod bending.
[0067] This application adds the ECA-NET module after FPN, and the structure is as follows Figure 3 As shown in the figure, the main idea of this module is that the image will not experience dimensionality reduction after processing. The specific measure is to enable the local cross-channel interaction strategy to autonomously select the convolution kernel as a single-kernel convolution combination when processing the image, to ensure that the image will not be deformed during the processing, to ensure that the attention mechanism obtains a more accurate feature map, and to better solve the problem that the change in the dimension of the feature map will cause the loss of some detail information.
[0068] Specific implementation method 2: This implementation method is a further explanation of specific implementation method 1. The difference between this implementation method and specific implementation method 1 is that the trained detection network is obtained by the following steps:
[0069] Step 1: Obtain a hook lifting rod bending image, and grayscale the hook lifting rod bending image to obtain a hook lifting rod bending grayscale image;
[0070] Step 2: Mark the bending position of the hook lifting rod in the hook lifting rod bending grayscale image to construct a sample data set;
[0071] Step 3: Use the sample data set to train the detection network to obtain a trained detection network.
[0072] Specific implementation method three: This implementation method is a further description of specific implementation method two. The difference between this implementation method and specific implementation method two is that the sample data set is a sample data set after data amplification, and the data amplification includes translation, scaling and brightness adjustment.
[0073] Set up high-definition equipment around the truck tracks, and obtain high-definition images after the truck passes through the equipment. The images are clear grayscale images. Truck parts may be affected by natural conditions or artificial conditions such as rain, mud stains, oil stains, black paint, etc. In addition, there may be differences in images taken at different sites. Therefore, the images of hook lifting rod bending vary greatly. Therefore, in the process of collecting hook lifting rod bending image data, diversity should be ensured, and all hook lifting rod bending images under various conditions should be collected as much as possible.
[0074] The morphology of hook lever bending varies in different types of bogies and vehicle models. However, due to the large difference in the frequency of occurrence between different types, it is difficult to collect images of hook lever bending for some less common bogie types and vehicle models. Therefore, all types of hook lever bending are collectively referred to as one category, and the sample data set is established according to all of them.
[0075] The sample data set is mainly divided into two parts: hook lifting rod bending and hook lifting rod bending, both of which are grayscale images. The hook lifting rod bending screenshots include: grayscale image set and labeled image set. The grayscale image set is a high-definition grayscale image taken by the device. The labeled image set is the bending position of the hook lifting rod. This part of the image is a grayscale image obtained by manual labeling. There is a one-to-one correspondence between the grayscale image data set and the labeled image data set, that is, each grayscale image corresponds to a labeled image.
[0076] Although the sample data set includes images under various conditions, it is still necessary to augment the sample data set to improve the stability of the model. Augmentation includes operations such as image translation, scaling, and brightness adjustment. Each operation is performed under random conditions, which can maximize the diversity and applicability of the samples.
[0077] Specific implementation method 4: This implementation method is a further explanation of specific implementation method 3. The difference between this implementation method and specific implementation method 3 is that the activation function of the improved Cascade-rcnn detection network is expressed as:
[0078]
[0079] Among them, z represents the horizontal axis coordinate of the function.
[0080] like Figure 2 To improve the Cascade-rcnn network structure, the network is improved on the basis of Faster-rcnn-resnet50, and the feature extraction network adopts resnet50, such as Figure 2In the left part, Conv1~Conv5 represent Resnet50 convolution modules. Each convolution module contains multiple convolution layers. Except for the activation function, the parameter settings of the feature extraction network, such as the number of network convolution layers and the number of convolution kernels in each convolution layer, are consistent with the feature extraction network Resnet50 of Cascade-rcnnn-resnet50.
[0081] This application uses ELU as the feature extraction network activation function, as shown in formula (1), where x is the value of the horizontal axis.
[0082]
[0083] The activation function of Faster-rcnn-resnet50 uses Relu, as shown in formula (2). When the input parameter of Relu is negative, the output is zero and its derivative is also zero, which causes the parameters to be unable to be updated when the network training gradient is reversely conducted, which reduces the learning ability of the network to a certain extent. When the input parameter of New_Relu used in this application is negative, the output of the network is not zero, and its derivative is not zero. When the network training gradient is reversely conducted, the parameters can be updated normally, which improves the learning ability of the network.
[0084]
[0085] Specific implementation method five: This implementation method is a further explanation of specific implementation method four. The difference between this implementation method and specific implementation method four is that the activation function of the ResNet50 residual network is:
[0086]
[0087] Specific implementation method 6: This implementation method is a further explanation of the specific implementation method 5. The difference between this implementation method and the specific implementation method 5 is that the loss function of the improved Cascade-rcnn detection network is expressed as:
[0088] New_Loss=0.25*SmoothL1Loss+0.75*DIoU
[0089] Among them, SmoothL1Loss represents the original loss function.
[0090] This application uses New Loss to replace the original SmoothL1 Loss. Smooth L1 Loss is as shown in formula (3). When SmoothL1 Loss is used to calculate the Bounding Box Loss of target detection, compared with other loss functions, it is insensitive to outliers (points far from the center) and outliers, and can control the magnitude of the gradient so that the loss is not easy to converge during training. DLOU Loss is shown in formula (4), where b represents the center point of the predicted box, represents the center point of the real box, represents the distance between the two center points, and c represents the diagonal distance of the minimum closure area. ρ represents the Euclidean distance between the two center points. DIoU can provide the optimization direction of the bounding box when the two boxes do not intersect; and DIoU loss minimizes the distance between the two target boxes, improves the convergence speed during training and can achieve good convergence.
[0091]
[0092]
[0093] New_Loss=0.25*SmoothL1Loss+0.75*DIoU (5)
[0094] This application improves on SmoothL1 Loss and DLOU Loss and proposes a new regression loss function New_Loss. It is used in two places in the structure: in the RPN network structure, when generating the candidate box area, New_Loss is used to locate the foreground and background; and when locating the candidate box, New_Loss is used to locate the target position.
[0095] Specific implementation method 7: This implementation method is a further description of specific implementation method 6. The difference between this implementation method and specific implementation method 6 is that SmoothL1Loss is expressed as:
[0096]
[0097] Specific implementation eight: This implementation is a further description of specific implementation seven. The difference between this implementation and specific implementation seven is that DIoU is expressed as:
[0098]
[0099] Among them, IoU represents the intersection-over-union ratio between the predicted box and the actual box, b represents the center point of the predicted box, and b gt represents the center point of the real box, c represents the diagonal distance of the minimum closure area, and ρ represents the Euclidean distance between the two center points.
[0100] Specific implementation method 9: This implementation method is a further explanation of specific implementation method 1. The difference between this implementation method and specific implementation method 1 is that the output of each convolutional layer in ResNet50 is expressed as:
[0101]
[0102] Among them, y represents the convolution output value, p0 represents the current sampling point, pn represents the enumeration of the sampling positions in each sliding window during the convolution process, w(pn) represents the weight of pn, and x represents the input feature map.
[0103] This application uses ResNet50 as the feature extraction network with an unchanged (N×N) convolution kernel, and performs standard sampling on the original image and feature map. The expression is:
[0104]
[0105] Where: y is the convolution output value; p0 is the current sampling point; pn is the enumeration of the sampling positions in each sliding window in the traditional convolution process; w(pn) is the weight of pn; x is the input feature map. For each position p0, first use the regular N×N to sample on the input feature map x, and then sum the sampling values weighted by w to obtain the convolution output value. In the hook lifting rod bending detection, it has the characteristics of non-fixed size, non-fixed position, and non-fixed angle. A good detection model should have the ability to solve the above problems. However, the traditional convolution has poor adaptability to the shape changes and feature sizes of the target and cannot meet the requirements. Inspired by deformable convolution, this application replaces the 3×3 convolution of the C1~C5 convolution layers with deformable convolution to form an improved feature extraction network D-ResNet50. The advantage of deformable convolution is that only a small number of parameters can be added to increase the receptive field, thereby improving the model's adaptability to irregular targets and feature extraction capabilities.
[0106] Specific implementation method ten: This implementation method is a further explanation of the specific implementation method one. The difference between this implementation method and the specific implementation method one is that the deformable convolution is expressed as:
[0107]
[0108] Among them, Δpn represents the offset of the convolution window position.
[0109] The feature extraction network of this application abandons the traditional regular grid points when sampling, and instead samples near the current sampling point through a learnable offset, so as to adapt to different targets and sample at a position closer to the target shape to avoid extracting irrelevant information. By automatically learning the parameters of the convolution kernel, the intervention of manual parameter adjustment is reduced, thereby simplifying the design and training of the network.
[0110]
[0111] When a railway train passes by the high-definition imaging equipment installed nearby, it obtains the train image, intercepts the hook lifting rod image, and sends the image to the trained detection network to output the detection result. If the hook lifting rod is not detected to be bent, the image is a fault image and the fault message is uploaded. Otherwise, the next image will be detected. The inspection personnel will proceed to the next step according to the fault message.
[0112] This application proposes a new activation function New_Relu to replace the original Relu activation function. When the network training gradient is reversely conducted, the parameters can be updated normally, thereby enhancing the learning ability of the network.
[0113] This application proposes a new loss function New_Loss, which combines the advantages of DIoU Loss and SmoothL1 Loss, minimizes the distance between the two target boxes, improves the convergence speed during training and can achieve good convergence, thereby improving the accuracy of the model.
[0114] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.
Claims
1. A hook lifting rod bending image recognition method based on deep learning, characterized in that The following steps are involved: Obtaining an image of a hook-lifting rod to be identified, and inputting the image of the hook-lifting rod to be identified into a trained detection network to obtain an output recognition result; The detection network is an improved Cascade-rcnn detection network, which includes Resnet50, FPN and ECA-Net. The improved Cascade-rcnn detection network specifically performs the following steps: The input image in Resnet50 first passes through Conv1 to extract features using deformable convolution to generate a feature map: The feature map generated by Conv1 is input into Conv2 and deformable convolution is used to extract features to generate a feature map; The feature map generated by Conv2 is input into Conv3 and deformable convolution is used to extract features to generate a feature map; The feature map generated by Conv3 is input into Conv4 and deformable convolution is used to extract features to generate a feature map; The feature map generated by Conv4 is input into Conv5 and deformable convolution is used to extract features to generate a feature map; In FPN, the feature map generated by Conv2 is used as feature map A; After downsampling, feature map A is fused with the feature map generated by Conv3 to form feature map B. After downsampling, feature map B is fused with the feature map generated by Conv4 to form feature map C. After downsampling, feature map C is fused with the feature map generated by Conv5 to form feature map D. After upsampling, feature map D is fused with feature map C to form feature map CC. After upsampling, feature map CC is fused with feature map B to form feature map BB. After upsampling, feature map BB is fused with feature map A to form feature map AA. After the feature maps AA, BB, CC, and D pass through ECA-Net, ROI pooling is performed to obtain the output category and position, that is, the image recognition result of the hook lifting rod bending.
2. The method for identifying hook-lift rod bending images based on deep learning according to claim 1 is characterized in that The trained detection network is obtained by the following steps: Step 1: Obtain a hook lifting rod bending image, and grayscale the hook lifting rod bending image to obtain a hook lifting rod bending grayscale image; Step 2: Mark the bending position of the hook lifting rod in the hook lifting rod bending grayscale image to construct a sample data set; Step 3: Use the sample data set to train the detection network to obtain a trained detection network.
3. The method for identifying hook-lift rod bending images based on deep learning according to claim 2 is characterized in that The sample data set is a sample data set after data amplification, and the data amplification includes translation, scaling, and brightness adjustment.
4. The method for identifying hook-lift rod bending images based on deep learning according to claim 3 is characterized in that The activation function of the improved Cascade-rcnn detection network is expressed as: Among them, z represents the horizontal axis coordinate of the function.
5. The method for identifying hook-lift rod bending images based on deep learning according to claim 4 is characterized in that The activation function of the ResNet50 residual network is:
6. The method for identifying hook-lift rod bending images based on deep learning according to claim 5 is characterized in that The loss function of the improved Cascade-rcnn detection network is expressed as: New_Loss=0.25*SmoothL1 Loss+0.75*DIoU Among them, SmoothL1 Loss represents the smooth L1 loss function, and DIoU represents the distance loss between the predicted box and the actual box.
7. The method for identifying hook-lift rod bending images based on deep learning according to claim 6 is characterized in that The SmoothL1 Loss is expressed as:
8. The method for identifying hook-lift rod bending images based on deep learning according to claim 7 is characterized in that The DIoU is expressed as: Among them, IoU represents the intersection-over-union ratio between the predicted box and the actual box, b represents the center point of the predicted box, and b gt represents the center point of the real box, c represents the diagonal distance of the minimum closure area, and ρ represents the Euclidean distance between the two center points.
9. The method for identifying hook-lift rod bending images based on deep learning according to claim 1, characterized in that The output of each convolutional layer in the ResNet50 is expressed as: Among them, y represents the convolution output value, p0 represents the current sampling point, pn represents the enumeration of the sampling positions in each sliding window during the convolution process, w(pn) represents the weight of pn, and x represents the input feature map.
10. The method for identifying hook-lift rod bending images based on deep learning according to claim 9, characterized in that The deformable convolution is expressed as: Among them, Δpn represents the offset of the convolution window position.