A truck part positioning and fault identification method, device, equipment and medium

Abstract

The present application relates to image processing technology, disclose a kind of truck parts positioning and fault identification method, comprising: through region annotation and image illumination processing generation truck original image data set's annotation image data set, according to annotation image data set, target detection network is trained, and the positioning network of component part is obtained;Label the fault type of component part in component part image data set, utilize the component part image data set and convolution attention mechanism corresponding to fault type to the training of lightweight convolution neural network model, and the component part fault identification network is obtained;The component part area of truck image to be measured is detected using component part positioning network, and the fault of component part area is identified using component part fault identification network, and the component part fault type is obtained.The present application also proposes a kind of truck parts positioning and fault identification device, electronic equipment and storage medium.The present application can improve the accuracy of truck parts positioning and identification.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for locating and identifying faults in truck parts. Background Technology

[0002] With the booming development of various sectors of the transportation industry, railway transportation has become the dominant mode of freight transport, ranking first in the world in both freight volume and freight turnover. The development trend of railway transportation towards heavy haulage and high-density operation has made railway safety issues more severe. To improve the accuracy and real-time nature of railway freight car fault detection, it is necessary to accurately locate the faulty parts of the freight cars for fault identification.

[0003] Existing truck fault identification methods are mostly based on traditional image recognition, primarily relying on manual extraction of fault features and then identifying faults based on these features. Generally, these methods only address each type of fault. However, in practical applications, truck faults may occur in multiple locations. Considering only one type of fault results in poor versatility and slow detection speed, leading to insufficient practicality for truck component fault identification and consequently low accuracy in component location and fault identification. Summary of the Invention

[0004] To address the aforementioned problems, embodiments of the present invention provide a method, apparatus, electronic device, and computer-readable storage medium for locating and identifying faults in truck parts.

[0005] In a first aspect, embodiments of the present invention provide a method for locating and identifying faults in truck parts, including:

[0006] Annotated image dataset of the original truck image dataset is generated by region annotation and image lighting processing;

[0007] The labeled image dataset is augmented using a preset data augmentation method to obtain an augmented image dataset. The augmented image dataset is then weighted by a convolutional attention mechanism to obtain an image feature set. Finally, the image feature set is nonlinearly mapped using a two-dimensional visual activation function to obtain an updated image feature set.

[0008] The target detection network is calculated based on the updated image feature set and the preset SIOU loss function until the loss value is less than the preset loss threshold, at which point the target detection network is used as the component localization network.

[0009] The component image dataset is segmented from the labeled image dataset, the fault types of the components in the component image dataset are labeled, and a lightweight convolutional neural network model is trained using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network.

[0010] An image of the truck to be tested is acquired, and the truck component areas in the image are detected using the component localization network. The component fault identification network is then used to identify faults in the truck component areas to obtain the component fault types.

[0011] According to an embodiment of the present invention, the labeled image dataset for generating the original truck image dataset through region labeling and image lighting processing includes:

[0012] Region labeling is performed on the parts in the original truck image dataset to obtain a region image dataset;

[0013] The detailed and coarse parts of the images in the region image dataset are separated using a preset bilateral filtering algorithm to obtain detailed and coarse images;

[0014] The lighting of the rough image is adjusted using a preset Retinex algorithm to obtain an enhanced rough image;

[0015] The detailed image is restored to the enhanced coarse image to obtain the labeled image dataset.

[0016] According to an embodiment of the present invention, the step of using a convolutional attention mechanism to perform attention weighting on the enhanced image dataset to obtain an image feature set includes:

[0017] The channel attention module in the convolutional attention mechanism is used to perform attention weighting on the enhanced image dataset to obtain channel attention weights. The channel attention weights are then weighted with the pixel values ​​corresponding to the enhanced image dataset to obtain a channel attention feature map.

[0018] The spatial attention module in the convolutional attention mechanism is used to perform attention weighting on the channel attention feature map to obtain spatial attention weights. The spatial attention weights are then weighted with the pixel values ​​corresponding to the augmented dataset to obtain the image feature set.

[0019] According to an embodiment of the present invention, the step of using a two-dimensional visual activation function to perform nonlinear mapping on the image feature set to obtain an updated image feature set includes:

[0020] Spatial context features are extracted from the image feature set using a preset feature extractor to obtain a spatial image feature set;

[0021] The maximum eigenvalue between the image feature set and the spatial image feature set is calculated using the two-dimensional visual activation function, wherein the two-dimensional visual activation function is:

[0022] y = max(x, T(x))

[0023] Where y is the maximum eigenvalue, x is the eigenvalue corresponding to the image feature set, T(x) is the eigenvalue corresponding to the spatial image feature set, and max is the maximum value function;

[0024] The image feature set corresponding to the largest eigenvalue is selected as the updated image feature set.

[0025] According to an embodiment of the present invention, calculating the loss value of the target detection network based on the updated image feature set and a preset SIOU loss function includes:

[0026] The updated image feature set is input into the target detection network for training to obtain the component prediction values;

[0027] The difference between the predicted value of the component and the preset actual value of the component is calculated using the SIOU loss function as follows:

[0028]

[0029] Among them, SIOU Loss The difference value is defined as follows: IOU is the intersection-union ratio, Δ is the distance loss value, and Ω is the shape loss value.

[0030] The difference value is used as the loss value of the target detection network.

[0031] According to an embodiment of the present invention, the step of training a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain a component fault identification network includes:

[0032] The component image feature set of the component image dataset is extracted using the depthwise separable convolutional layer in the lightweight convolutional neural network;

[0033] The convolutional attention mechanism is used to perform attention weighting processing on the feature set of the component images to obtain an attention image feature set;

[0034] The loss value of the lightweight convolutional neural network model is calculated based on the attention image feature set and the preset loss function.

[0035] When the loss value is less than the preset loss threshold, the current lightweight convolutional neural network model is output as the component fault identification network.

[0036] According to an embodiment of the present invention, the step of performing image data augmentation on the labeled image dataset using a preset data augmentation method to obtain an augmented image dataset includes:

[0037] Obtain the fault type of the labeled parts in the labeled image dataset;

[0038] Based on the fault type, a preset data augmentation method is selected for the labeled image dataset to obtain an updated data augmentation method;

[0039] The image data of the labeled image dataset is augmented using the updated data augmentation method to obtain the augmented image dataset.

[0040] Secondly, embodiments of the present invention provide a truck parts positioning and fault identification device, characterized in that it includes:

[0041] The image lighting processing module is used to generate an annotated image dataset of the original truck image dataset through region annotation and image lighting processing.

[0042] The image data augmentation module is used to augment the labeled image dataset using a preset data augmentation method to obtain an augmented image dataset, apply attention weighting to the augmented image dataset using a convolutional attention mechanism to obtain an image feature set, and apply a two-dimensional visual activation function to the image feature set for nonlinear mapping to obtain an updated image feature set.

[0043] The component positioning network determination module is used to calculate the loss value of the target detection network based on the updated image feature set and the preset SIOU loss function, until the loss value is less than the preset loss threshold, and then the target detection network is used as the component positioning network.

[0044] The component fault identification network training module is used to segment the component image dataset from the labeled image dataset, label the fault types of the components in the component image dataset, and train a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network.

[0045] The truck image detection module is used to acquire images of the truck under test, detect truck component areas in the truck image using the component positioning network, and identify faults in the truck component areas using the component fault identification network to obtain the component fault type.

[0046] Thirdly, embodiments of the present invention provide an electronic device, which includes:

[0047] processor;

[0048] Memory used to store the processor's executable instructions;

[0049] The processor is configured to execute the instructions to implement a truck parts location and fault identification method as described in the first aspect above.

[0050] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a truck parts positioning and fault identification method as described in the first aspect above.

[0051] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial effects:

[0052] The embodiments of this invention employ a two-stage detection approach: first, component location is performed, followed by fault identification. This allows both networks to focus more intently on the required features, reducing false positive and false negative rates. A Retinex+bilateral filtering preprocessing method is designed to address the uneven illumination problem in TFDS (Total Fault Diagnostics Detection and Distributed Data), performing illumination processing on the original image data. Different data augmentation methods are employed for different fault types. The Mosaic-4 data augmentation method in the YOLOv5 network structure is replaced with Mosaic-9, the SiLU activation function with FReLU, and the CIOU loss function with SIOU. A hybrid attention mechanism (CBAM) is embedded in the network structure. Based on the improved YOLOv5 network, the component region detection algorithm achieves the required location speed and accuracy. The SE attention module (channel attention module) in the MobileNetv3 network is replaced with the CBAM module, further improving network performance. Based on the improved MobileNetv3 network, the fault classification algorithm achieves the required fault identification accuracy. Both the component location and component fault identification algorithms utilize deep learning methods, effectively meeting the requirements for accuracy and real-time performance. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 The flowchart of the truck parts location and fault identification method according to Embodiment 1 of the present invention is shown;

[0055] Figure 2The YOLOv5s network structure diagram of the truck parts location and fault identification method of Embodiment 1 of the present invention is shown.

[0056] Figure 3 This diagram shows the structure of the CBAM attention mechanism in the truck parts localization and fault identification method of Embodiment 1 of the present invention;

[0057] Figure 4 A comparison chart of ReLU, PreLU, and FReLU activation functions of the truck part localization and fault identification method according to Embodiment 1 of the present invention is shown;

[0058] Figure 5 This diagram illustrates the depthwise separable convolution in the fault discrimination method of the truck parts positioning and fault identification method according to Embodiment 1 of the present invention.

[0059] Figure 6 This shows a diagram of the inverted residual structure in the fault discrimination method of the truck parts positioning and fault identification method according to Embodiment 1 of the present invention;

[0060] Figure 7 This diagram illustrates the flowchart of the training and application phases of the component positioning network and component fault identification network in the truck component positioning and fault identification method according to Embodiment 1 of the present invention.

[0061] Figure 8 The diagram shows the functional modules of the truck parts positioning and fault identification device according to Embodiment 3 of the present invention.

[0062] Figure 9 The diagram shows the composition of an electronic device for implementing the truck part positioning and fault identification method according to Embodiment 5 of the present invention. Detailed Implementation

[0063] The present disclosure will be further described below with reference to the embodiments shown in the accompanying drawings.

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] This invention proposes a method for truck part localization and fault identification based on image acquisition technology. Based on deep learning theory and combined with a bi-segmentation method, a target detection network and a fault identification network are constructed to achieve truck part localization and fault identification. Compared with traditional methods, the deep learning-based truck part localization and fault identification technology is more efficient and reduces the false alarm rate and false negative rate of part faults, showing great potential and application prospects in truck part fault identification.

[0066] Example 1

[0067] like Figure 1 As shown, this invention proposes a method for locating and identifying faults in truck parts, comprising the following steps:

[0068] S1. Generate an annotated image dataset of the original truck image dataset through region labeling and image lighting processing;

[0069] In this embodiment of the invention, the TFDS system can collect a raw image dataset of freight trucks. The raw image dataset should include two main categories: normal images and faulty images. To ensure image diversity, images of multiple freight trucks under different lighting conditions and from different train numbers are collected by the bottom camera of the TFDS system, and then labeled to form the raw image dataset. However, because the images collected by the TFDS are numerous but have a very low failure rate, the number of fault samples in the raw image data is too small, which will affect the performance of the target detection network. Therefore, it is necessary to collect fault samples separately and add them to the raw image dataset to adjust the sample ratio. The TFDS system is an intelligent, networked, and information-based system integrating high-speed image acquisition technology, large-capacity real-time image data processing technology, precise positioning control technology, and automatic control technology.

[0070] In this embodiment of the invention, the region annotation refers to marking potentially faulty component areas in the original truck image dataset, i.e., marking the component areas. The annotated image dataset is the image dataset obtained after region annotation and image lighting processing of the original image dataset.

[0071] In detail, since the TFDS system acquires images outdoors around the clock, it is easily affected by lighting and weather conditions, resulting in images that are too bright, too dark, or unevenly lit. The quality of TFDS images directly affects the accuracy of fault location and identification. Therefore, it is necessary to preprocess the images in the raw image data.

[0072] In this embodiment of the invention, the labeled image dataset for generating the original truck image dataset through region labeling and image illumination processing includes:

[0073] Region labeling is performed on the parts in the original truck image dataset to obtain a region image dataset;

[0074] The detailed and coarse parts of the images in the region image dataset are separated using a preset bilateral filtering algorithm to obtain detailed and coarse images;

[0075] The lighting of the rough image is adjusted using a preset Retinex algorithm to obtain an enhanced rough image;

[0076] The detailed image is restored to the enhanced coarse image to obtain the labeled image dataset.

[0077] In detail, by labeling the parts of the truck in the original truck image dataset, a region image dataset with labeled parts can be obtained.

[0078] Specifically, for images with uneven illumination, single image processing methods are often insufficient to achieve good results. While the Retinex algorithm effectively eliminates the influence of illumination, it cannot enhance the overall detail of the image. Therefore, a combination of bilateral filtering and the Retinex algorithm is used to improve the quality of TFDS images. This preserves image details while effectively suppressing the impact of highlights. Bilateral filtering is a spatial filtering algorithm that simultaneously removes noise and preserves image edge details; the Retinex algorithm estimates the image's illumination using a certain method, then eliminates or adjusts the illumination to obtain an enhanced image less affected by ambient lighting.

[0079] Furthermore, after region labeling and preprocessing the original truck image dataset, the preprocessed image dataset needs to be used as training data to train the target detection network and locate the parts that may malfunction.

[0080] S2. The labeled image dataset is augmented using a preset data augmentation method to obtain an augmented image dataset. The augmented image dataset is then weighted by a convolutional attention mechanism to obtain an image feature set. Finally, the image feature set is nonlinearly mapped using a two-dimensional visual activation function to obtain an updated image feature set.

[0081] In this embodiment of the invention, the target detection network is trained using the YOLOv5s network as the baseline network to obtain the component localization network. The YOLOv5s network is a single-stage target detection algorithm. Figure 2The diagram shows the YOLOv5s network structure. The YOLOv5s network mainly consists of a backbone network, a neck network, and a head. The backbone network uses CSPDarknet53 and is responsible for feature extraction. It mainly consists of the CBS module, C3 module, and SPPF module. The most basic module is the CBS module, which is composed of convolution (Conv), batch normalization (BN), and an activation function (SiLU). The C3 module is composed of the ResUnit module and the concatenation operation of the outer CBS module. The SPPF module sequentially passes the input through multiple 5x5 MaxPool layers, then concatenates the results before passing them through the CBS module. The neck network's role is to achieve feature fusion, concatenating and fusing intermediate feature maps with different receptive fields generated at different stages of the backbone network before inputting them into the head. The function of the detection head is to output the prediction results. The most important operation is 1×1 convolution. Specifically, assuming the input image size is 640×640×3, the three feature maps output by the neck network have specific sizes of 80×80×256, 40×40×512 and 20×20×1024, respectively.

[0082] In detail, the input of the YOLOv5s network is mainly the labeled image dataset. Image data augmentation and adaptive image scaling are performed on the input of this dataset. The original YOLOv5s network used the Mosaic-4 data augmentation method, which has a relatively high computational cost. Therefore, the Mosaic-4 input method has been replaced with Mosaic-9 in the YOLOv5s network.

[0083] In this embodiment of the invention, the step of performing image data augmentation on the labeled image dataset using a preset data augmentation method to obtain an augmented image dataset includes:

[0084] The labeled image dataset was augmented using the Mosaic-9 data augmentation method to obtain an augmented image dataset.

[0085] In detail, the Mosaic-9 data augmentation method involves randomly cropping and scaling nine images, then combining them into a single image to serve as the training image dataset for model training. Mosaic-9 significantly increases data information, including the number of small objects. During normalization, it calculates all nine images, independent of batch processing parameters, thus reducing computational load.

[0086] In this embodiment of the invention, a Hybrid Attention Mechanism (CBAM) is embedded in the YOLOv5s network structure to select more critical component images from a large amount of image information. Attention mechanisms in deep learning are essentially similar to the selective visual attention mechanism in humans; their core objective is also to select information more critical to the current task objective from a large amount of information. Given limited computing power, resources are allocated according to the importance of features, prioritizing important information and suppressing unimportant information to solve the problem of information overload. Figure 3 The diagram shown is a structural diagram of the CBAM attention mechanism, which can generally be divided into channel attention mechanism and spatial attention mechanism. The CBAM module combines the two to more effectively improve the detection performance of interest targets. It is composed of a channel attention module and a spatial attention module, with the two modules corresponding to the calculation of two different attention mechanisms.

[0087] In this embodiment of the invention, the step of using a convolutional attention mechanism to perform attention weighting on the enhanced image dataset to obtain an image feature set includes:

[0088] The channel attention module in the convolutional attention mechanism is used to perform attention weighting on the enhanced image dataset to obtain channel attention weights. The channel attention weights are then weighted with the pixel values ​​corresponding to the enhanced image dataset to obtain a channel attention feature map.

[0089] The spatial attention module in the convolutional attention mechanism is used to perform attention weighting on the channel attention feature map to obtain spatial attention weights. The spatial attention weights are then weighted with the pixel values ​​corresponding to the augmented dataset to obtain the image feature set.

[0090] In detail, the channel attention module focuses on what features are meaningful. First, it extracts channel attention from the augmented image dataset, using the sum of average pooling and max pooling weights as the weights for the entire image feature map. These channel attention weights are then weighted with the corresponding pixel values ​​from the augmented image dataset to obtain the channel attention feature map. Next, spatial attention is extracted from this feature map by performing average pooling and max pooling to obtain spatial attention weights. Finally, these spatial attention weights are weighted with the corresponding pixel values ​​from the augmented dataset to obtain the image feature set. This image feature set, based on the importance of parts in truck images, focuses on images of truck components more prone to failure.

[0091] Furthermore, to achieve pixel-level modeling capabilities, the SiLU activation function in the CBS module of the YOLOv5s backbone network is replaced with the FReLU activation function, such as... Figure 4As shown, this is a comparison chart of ReLU, PreLU, and FReLU activation functions. The FReLU activation function, by adding negligible spatial conditional overhead and setting a spatial condition T(·), transforms ReLU and PreLU activation functions into two-dimensional visual activation functions, achieving pixel-level modeling capabilities in a simple way.

[0092] In this embodiment of the invention, the step of using a two-dimensional visual activation function to perform nonlinear mapping on the image feature set to obtain an updated image feature set includes:

[0093] Spatial context features are extracted from the image feature set using a preset feature extractor to obtain a spatial image feature set;

[0094] The maximum eigenvalue between the image feature set and the spatial image feature set is calculated using the two-dimensional visual activation function, wherein the two-dimensional visual activation function is:

[0095] y = max(x, T(x))

[0096] Where y is the maximum eigenvalue, x is the eigenvalue corresponding to the image feature set, T(x) is the eigenvalue corresponding to the spatial image feature set, and max is the maximum value function;

[0097] The image feature set corresponding to the largest eigenvalue is selected as the updated image feature set.

[0098] In detail, the feature extractor is a simple and efficient context feature extractor. The FReLU activation function is extended to a two-dimensional condition that depends on the spatial context of each pixel, and the maximum value between x and the condition is obtained using max.

[0099] Specifically, the enhanced image dataset is processed through the backbone network of the YOLOv5s network to extract image feature sets. These feature sets are then input into the neck network of the YOLOv5s network for feature fusion. Intermediate feature maps with different receptive fields generated at different stages of the backbone network are spliced ​​and fused together and then input into the detection head. The detection head outputs the model prediction results to obtain the training parameters of the target detection network. The training parameters need to be continuously adjusted using a loss function to obtain the optimal parameters for the target detection network to locate truck parts.

[0100] S3. Calculate the loss value of the target detection network based on the updated image feature set and the preset SIOU loss function until the loss value is less than the preset loss threshold, then use the target detection network as the component localization network.

[0101] In this embodiment of the invention, the traditional CIOU loss function commonly used in YOLOv5s networks relies on the aggregation of bounding box regression metrics, ignoring the direction of mismatch between the desired ground truth boxes and predicted boxes. This deficiency leads to slow convergence and low efficiency because predicted boxes may "wander" during training, ultimately resulting in a worse model. Therefore, the CIOU loss function is replaced with the SIOU loss function. The novel SIOU loss function takes into account the vector angle between the desired regressions and redefines the penalty metric, which can improve training speed and inference accuracy.

[0102] In this embodiment of the invention, calculating the loss value of the target detection network based on the updated image feature set and a preset SIOU loss function includes:

[0103] The updated image feature set is input into the target detection network for training to obtain the component prediction values;

[0104] The difference between the predicted value of the component and the preset actual value of the component is calculated using the SIOU loss function as follows:

[0105]

[0106] Among them, SIOU Loss The difference value is defined as follows: IOU is the intersection-union ratio, Δ is the distance loss value, and Ω is the shape loss value.

[0107] The difference value is used as the loss value of the target detection network.

[0108] In detail, SIOU loss is used instead of CIOU loss to describe the localization loss portion of the predicted bounding box in the total loss function. SIOU loss mainly involves four parts: angle loss, distance loss, shape loss, and IOU loss. Δ in the SIOU loss calculation formula represents the distance loss value and also includes parameters describing the angle loss.

[0109] Specifically, the object detection network backpropagates based on the loss value to update the various parameters of the model, reducing the loss between the true and predicted values, and making the predicted values ​​generated by the model closer to the true values, until the loss value is less than a preset loss threshold. At this point, the current object detection network is used as the component localization network. The role of the component localization network is to locate the component area that may be faulty, which is a prerequisite for fault detection and meets the two indicators of real-time performance and accuracy.

[0110] Furthermore, after training, the component localization network is used to locate component regions that may be faulty, and then the faulty component regions are cut out from the original image for identification of the type of component fault.

[0111] S4. Segment the component image dataset from the labeled image dataset, label the fault types of the components in the component image dataset, and train the lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network.

[0112] In this embodiment of the invention, the component image dataset is used to locate potentially faulty component regions after the target detection model has been trained. These components are then cut out from the labeled images to obtain component sub-images. These sub-images are then categorized into different folders, with each component classified into two categories: normal and faulty. This constitutes the component image dataset.

[0113] In detail, the fault types of the components are labeled in the component image data for use in training the fault classification network. These component fault types include missing component faults (such as missing brake shoe pins or missing pull rings of the automatic derailment device), door closure faults (such as the automatic derailment brake valve handle being closed), and thickness exceeding limits faults (such as brake shoe thickness exceeding limits).

[0114] In this embodiment of the invention, the component fault identification network determines whether a fault exists and identifies the type of fault from the segmented component images. For each component, it is a binary classification problem. Considering real-time performance and accuracy, the lightweight network MobileNetv3 is selected as the base network for the classification model. MobileNetv3 (Lightweight Convolutional Neural Network) is a lightweight convolutional neural network model designed for mobile and embedded devices. It employs depthwise separable convolution, inverse residual structures, and SE attention mechanisms to significantly reduce the number of parameters and computational cost while maintaining high model accuracy, effectively improving the model's detection speed.

[0115] In this embodiment of the invention, the step of training a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain a component fault identification network includes:

[0116] The component image feature set of the component image dataset is extracted using the depthwise separable convolutional layer in the lightweight convolutional neural network;

[0117] The convolutional attention mechanism is used to perform attention-weighted processing on the feature set of the component images to obtain an attention image feature set;

[0118] The loss value of the lightweight convolutional neural network model is calculated based on the attention image feature set and the preset loss function.

[0119] When the loss value is less than the preset loss threshold, the current lightweight convolutional neural network model is output as the component fault identification network.

[0120] In detail, such as Figure 5 The diagram illustrates depthwise separable convolution, which consists of DW and PW convolutions. First, a single filter is applied to each input channel without increasing the number of channels. Then, 1×1 convolutions are applied point-by-point to increase the number of channels. This decomposition method significantly reduces computational cost and model size. Lightweight convolutional neural networks also include inverse residual structures, such as... Figure 6 As shown, the features are first expanded in dimension through an expansion layer, then further extracted using depthwise separable convolution, and finally compressed back to the original dimension through a projection layer.

[0121] Specifically, the SE module in the original lightweight convolutional neural network performs pooling on each channel of the obtained feature matrix, and then passes it through two fully connected layers. The output of the second fully connected layer can be understood as the weight relationship analyzed for each channel of the feature matrix, assigning larger weights to the more important ones. Embedding an SE attention mechanism within the network allows the network to dynamically adjust the weights of each channel (channel attention), effectively improving model performance. However, the SE module only focuses on channel attention. Replacing the SE channel attention module with a CBAM module can further improve network performance. The CBAM module is composed of a channel attention module and a spatial attention module, thus more effectively improving the detection performance for targets of interest, i.e., more effectively paying attention to faulty components within parts.

[0122] Furthermore, the lightweight convolutional neural network backpropagates based on the loss value to update the various parameters of the model, thereby reducing the loss between the true value and the predicted value, making the predicted value generated by the model closer to the true value, until the loss value is less than a preset loss threshold, at which point the current lightweight convolutional neural network is used as a component fault identification network.

[0123] S5. Acquire an image of the truck to be tested, use the component positioning network to detect the truck component area in the image of the truck to be tested, and use the component fault identification network to identify faults in the truck component area to obtain the component fault type.

[0124] In this embodiment of the invention, the image to be tested can be processed and passed successively through a trained object detection network and an image classification network to obtain information such as whether the image has a fault, the type of fault, and the location of the fault. The fault images obtained by the system need to be manually confirmed before being used to update the fault database and further optimize the network parameters.

[0125] In this embodiment of the invention, the step of using the component fault identification network to identify faults in the truck component area and obtain component fault types includes:

[0126] The classifier in the component fault identification network is used to determine whether there is a fault in the truck component area;

[0127] When a fault exists in the truck component area, the fault type of the truck component area is determined to obtain the component fault type;

[0128] If there is no fault in the truck parts area, the truck parts area is removed.

[0129] In detail, a component localization network is used to locate components in the image under test, thus obtaining the truck component area. Then, a component fault identification network is used to check whether there are faults in the parts within the truck component area and to confirm the type of fault. When a component is faulty, the type and location of the fault are manually confirmed; when a component is not faulty, the truck component area is removed.

[0130] Specifically, when locating components and identifying fault types in images under test, the component areas are first accurately located, and then the fault type is determined and identified. For example... Figure 7 The diagram illustrates the workflow structure of the component localization network and the component fault identification network during the training and application phases. In the training phase, a large number of truck image samples captured by TFDS are first acquired to construct an original image dataset, and the regions of components that may experience faults are labeled. The labeled sample data is then preprocessed and used to train the object detection network to obtain optimal parameters. The trained object detection network is then used to generate target bounding boxes. The selected component regions are then separated to construct a component image dataset, and each component image is labeled to indicate whether a fault exists. Finally, the component image dataset is used to train the image classification network. In the application phase, the images to be tested, after preprocessing, are passed through the trained object detection network and image classification network to obtain information such as whether the image has a fault, the type of fault, and the location of the fault. The fault images obtained by the system need to be manually verified before being used to update the fault database and further optimize the network parameters.

[0131] Example 2

[0132] To better understand the present invention, a second embodiment is provided below to further explain how the present invention enhances image data for different fault types.

[0133] In this embodiment of the invention, in order to enhance the generalization ability of the network model and alleviate the problems of overfitting and imbalanced samples, data augmentation is required. Appropriate image augmentation methods should be selected for different fault types to achieve the best results.

[0134] In this embodiment of the invention, the step of performing image data augmentation on the labeled image dataset using a preset data augmentation method to obtain an augmented image dataset includes:

[0135] Obtain the fault type of the labeled parts in the labeled image dataset;

[0136] Based on the fault type, a preset data augmentation method is selected for the labeled image dataset to obtain an updated data augmentation method;

[0137] The image data of the labeled image dataset is augmented using the updated data augmentation method to obtain the augmented image dataset.

[0138] In detail, appropriate data augmentation methods should be selected for different fault types to achieve the best image results. The term "updating data augmentation method" refers to selecting a suitable data augmentation method from among many options based on the fault type. For example, data augmentation methods may include rotation, translation, flipping, and brightness adjustment. Depending on the fault type of the component, only rotation data augmentation may be selected; that is, rotation data augmentation is the updated data augmentation method.

[0139] For example, for loss-related faults (such as missing brake shoe pins or missing automatic derailment device pull rings), the focus should be on the overall characteristics, i.e., the difference between presence and absence. Data augmentation methods could include rotation, translation, flipping, and brightness adjustment, using one or more of these methods to enhance the sample data. For gate closure-related faults (such as the automatic derailment brake valve gate handle being closed), the focus should be on the gate's opening and closing state, i.e., the handle angle. Enhancing the image through rotation would be more beneficial for the model's general applicability. For thickness exceeding limits-related faults (such as excessive brake shoe thickness), which is essentially a quantitative problem, in addition to brightness and contrast adjustment, the proportion of the original image should be carefully maintained throughout the process.

[0140] Example 3

[0141] like Figure 8 As shown in the figure, this embodiment also provides a functional module diagram of a truck parts positioning and fault identification device.

[0142] The truck parts positioning and fault identification device 100 described in this embodiment can be installed in an electronic device. Depending on the functions implemented, the truck parts positioning and fault identification device 100 may include an image illumination processing module 101, an image data enhancement module 102, a parts positioning network determination module 103, a parts fault identification network training module 104, and a truck image detection module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0143] In this embodiment, the functions of each module / unit are as follows:

[0144] The image illumination processing module 101 is used to generate an labeled image dataset of the original image dataset of the truck through region labeling and image illumination processing.

[0145] The image data enhancement module 102 is used to enhance the labeled image dataset using a preset data enhancement method to obtain an enhanced image dataset, apply attention weighting to the enhanced image dataset using a convolutional attention mechanism to obtain an image feature set, and apply a two-dimensional visual activation function to the image feature set for nonlinear mapping to obtain an updated image feature set.

[0146] The component positioning network determination module 103 is used to calculate the loss value of the target detection network based on the updated image feature set and the preset SIOU loss function, until the loss value is less than the preset loss threshold, and then the target detection network is used as the component positioning network.

[0147] The component fault identification network training module 104 is used to segment the component image dataset from the labeled image dataset, label the fault types of the components in the component image dataset, and train a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network.

[0148] The truck image detection module 105 is used to acquire an image of the truck to be tested, detect the truck component area of ​​the truck image using the component positioning network, and identify the fault in the truck component area using the component fault identification network to obtain the component fault type.

[0149] In detail, each module in the truck parts positioning and fault identification device 100 described in the embodiments of the present invention adopts the same technical means as the truck parts positioning and fault identification methods described in Embodiment 1 and Embodiment 2, and can produce the same technical effects, which will not be repeated here.

[0150] Example 4

[0151] like Figure 9 As shown, this embodiment also provides a computer electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a truck parts location and fault identification program.

[0152] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing truck parts location and fault identification programs) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0153] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code for a truck parts location and fault identification program, but also to temporarily store data that has been output or will be output.

[0154] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0155] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0156] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0157] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0158] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0159] The truck parts location and fault identification program stored in the memory 11 of the electronic device is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0160] Annotated image dataset of the original truck image dataset was generated by region annotation and image lighting processing.

[0161] The labeled image dataset is augmented using a preset data augmentation method to obtain an augmented image dataset. The augmented image dataset is then weighted by a convolutional attention mechanism to obtain an image feature set. Finally, the image feature set is nonlinearly mapped using a two-dimensional visual activation function to obtain an updated image feature set.

[0162] The target detection network is calculated based on the updated image feature set and the preset SIOU loss function until the loss value is less than the preset loss threshold, at which point the target detection network is used as the component localization network.

[0163] The component image dataset is segmented from the labeled image dataset, the fault types of the components in the component image dataset are labeled, and a lightweight convolutional neural network model is trained using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network.

[0164] An image of the truck to be tested is acquired, and the truck component areas in the image are detected using the component localization network. The component fault identification network is then used to identify faults in the truck component areas to obtain the component fault types.

[0165] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0166] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0167] Example 5

[0168] This embodiment provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the truck parts positioning and fault identification method described above.

[0169] This program code can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0170] Storage media include permanent and non-permanent, removable and non-removable media, and can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by computing devices.

[0171] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. When the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0172] It should be understood that the terms used in this way can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in a sequence other than those illustrated or described herein.

[0173] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0174] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0176] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0177] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0178] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0179] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for locating and identifying faults in truck parts, characterized in that, The method includes: The original truck image dataset was generated by region labeling and image lighting processing; the original truck image dataset was acquired using the TFDS system. The labeled image dataset is augmented using the Mosaic-9 data augmentation method to obtain an augmented image dataset. An attention-weighted convolutional attention mechanism is then applied to the augmented image dataset to obtain an image feature set. Finally, a two-dimensional visual activation function is used to perform a non-linear mapping on the image feature set to obtain an updated image feature set. The object detection network is a YOLOv5s network, and the two-dimensional visual activation function is the FReLU function, used to replace the activation function in the YOLOv5s network. The target detection network is calculated based on the updated image feature set and the preset SIoU loss function until the loss value is less than the preset loss threshold. Then, the target detection network is used as the component localization network. The SIoU loss function is used to replace the CIoU loss function in the YOLOv5s network. The component image dataset is segmented from the labeled image dataset, the fault types of the components in the component image dataset are labeled, and a lightweight convolutional neural network model is trained using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network. The process involves acquiring an image of the truck to be tested, using the component localization network to detect the truck component regions in the image, and using the component fault identification network to identify faults in the truck component regions to obtain the component fault types. The labeled image dataset, which generates the original truck image dataset through region labeling and image lighting processing, includes: Region annotation is performed on the parts in the original truck image dataset to obtain a region image dataset; The detailed and coarse parts of the images in the region image dataset are separated using a preset bilateral filtering algorithm to obtain detailed and coarse images; The lighting of the rough image is adjusted using a preset Retinex algorithm to obtain an enhanced rough image; The detailed image is restored to the enhanced coarse image to obtain the labeled image dataset.

2. The truck parts positioning and fault identification method as described in claim 1, characterized in that, The process of applying attention weights to the enhanced image dataset using a convolutional attention mechanism to obtain an image feature set includes: The channel attention module in the convolutional attention mechanism is used to perform attention weighting on the enhanced image dataset to obtain channel attention weights. The channel attention weights are then weighted with the pixel values ​​corresponding to the enhanced image dataset to obtain a channel attention feature map. The spatial attention module in the convolutional attention mechanism is used to perform attention weighting on the channel attention feature map to obtain spatial attention weights. The spatial attention weights are then weighted with the pixel values ​​corresponding to the augmented dataset to obtain the image feature set.

3. The truck parts positioning and fault identification method as described in claim 1, characterized in that, The step of using a two-dimensional visual activation function to perform a nonlinear mapping on the image feature set to obtain an updated image feature set includes: Spatial context features are extracted from the image feature set using a preset feature extractor to obtain a spatial image feature set; The maximum eigenvalue between the image feature set and the spatial image feature set is calculated using the two-dimensional visual activation function, wherein the two-dimensional visual activation function is: in, The largest eigenvalue, The feature value corresponding to the image feature set. The feature value corresponding to the spatial image feature set. It is a function for maximizing the value; The image feature set corresponding to the largest eigenvalue is selected as the updated image feature set.

4. The truck parts positioning and fault identification method as described in claim 1, characterized in that, The step of calculating the loss value of the object detection network based on the updated image feature set and the preset SIOU loss function includes: The updated image feature set is input into the target detection network for training to obtain the component prediction values; The difference between the predicted value of the component and the preset actual value of the component is calculated using the SIOU loss function as follows: in, The difference value, For intersection, union, and comparison, This is the distance loss value. This represents the shape loss value; The difference value is used as the loss value of the target detection network.

5. The truck parts positioning and fault identification method as described in claim 1, characterized in that, The step of training a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain a component fault identification network includes: The component image feature set of the component image dataset is extracted using the depthwise separable convolutional layer in the lightweight convolutional neural network; The convolutional attention mechanism is used to perform attention weighting processing on the feature set of the component images to obtain an attention image feature set; The loss value of the lightweight convolutional neural network model is calculated based on the attention image feature set and the preset loss function. When the loss value is less than the preset loss threshold, the current lightweight convolutional neural network model is output as the component fault identification network.

6. The truck parts positioning and fault identification method as described in claim 1, characterized in that, The step of performing image data augmentation on the labeled image dataset using a preset data augmentation method to obtain an augmented image dataset includes: Obtain the fault type of the labeled parts in the labeled image dataset; Based on the fault type, a preset data augmentation method is selected for the labeled image dataset to obtain an updated data augmentation method; The image data of the labeled image dataset is augmented using the updated data augmentation method to obtain the augmented image dataset.

7. A truck parts positioning and fault identification device, characterized in that, The device includes: The image illumination processing module is used to generate an labeled image dataset of the original truck image dataset through region annotation and image illumination processing; the original truck image dataset is acquired through the TFDS system. The image data augmentation module is used to augment the labeled image dataset using the Mosaic-9 data augmentation method to obtain an augmented image dataset. It then uses a convolutional attention mechanism to perform attention weighting on the augmented image dataset to obtain an image feature set. Finally, it uses a two-dimensional visual activation function to perform a non-linear mapping on the image feature set to obtain an updated image feature set. The object detection network is a YOLOv5s network, and the two-dimensional visual activation function is the FReLU function, used to replace the activation function in the YOLOv5s network. The component localization network determination module is used to calculate the loss value of the target detection network based on the updated image feature set and the preset SIoU loss function, until the loss value is less than the preset loss threshold, and then the target detection network is used as the component localization network; wherein, the SIoU loss function is used to replace the CIoU loss function in the YOLOv5s network; The component fault identification network training module is used to segment the component image dataset from the labeled image dataset, label the fault types of the components in the component image dataset, and train a lightweight convolutional neural network model using the component image dataset corresponding to the fault type and the convolutional attention mechanism to obtain the component fault identification network. The truck image detection module is used to acquire images of trucks under test, detect truck component areas in the truck image using the component positioning network, and identify faults in the truck component areas using the component fault identification network to obtain component fault types. The labeled image dataset, which generates the original truck image dataset through region labeling and image lighting processing, includes: Region labeling is performed on the parts in the original truck image dataset to obtain a region image dataset; The detailed and coarse parts of the images in the region image dataset are separated using a preset bilateral filtering algorithm to obtain detailed and coarse images; The lighting of the rough image is adjusted using a preset Retinex algorithm to obtain an enhanced rough image; The detailed image is restored to the enhanced coarse image to obtain the labeled image dataset.

8. An electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the truck parts location and fault identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the truck parts location and fault identification method as described in any one of claims 1 to 6.

Images