Truck connecting ball head fault detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of intelligent manufacturing, and discloses a truck connecting ball head fault detection method and device, electronic equipment and a computer readable storage medium. The truck connecting ball head fault detection method comprises the following steps: obtaining a detection picture comprising a truck connecting ball head; the detection picture is input into a pre-trained fault detection model, an output result of the fault detection model is obtained and serves as a fault detection result of the truck connecting ball head, the fault detection model comprises a BiMama network layer, the BiMama network layer comprises a plurality of feature layers, and the feature layers are used for detecting the fault of the truck connecting ball head. Each feature layer is used for extracting feature information of the detection picture under different resolutions. According to the truck connecting ball head fault detection method and device, the electronic equipment and the computer readable storage medium provided by the invention, the technical effects of improving the small target detection precision and ensuring the fault detection efficiency are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automobiles, in particular to a truck connecting ball head fault detection method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] As one of the main tools for land transportation, the safety and reliability of heavy trucks are directly related to the stable operation of the entire transportation system. During a long period of operation, due to factors such as vibration and impact, the linking ball head on the heavy truck may become loose or even lost, which not only affects the normal operation of the vehicle, but also may cause serious safety accidents. Heavy truck safety component linking ball head loose detection is an important measure to ensure transportation safety and efficiency. Currently, many heavy truck safety component linking ball head loose detection still relies on manual inspection. The inspection personnel observe and manually check the state of the linking ball head by naked eye.

[0003] In recent years, with the development of computer vision technology, more and more technical personnel have begun to use machine vision systems for linking ball head detection. Traditional image detection algorithms have poor environmental adaptability and are greatly affected by environmental factors such as light and weather. Using deep learning models such as convolutional neural networks, the images captured by the camera can be analyzed to adapt to different environmental conditions, such as light changes, occlusions, and automatically identify the state of the linking ball head, with high real-time performance, high precision, etc.

[0004] However, the current fault detection method using convolutional neural network and other deep learning models has poor precision for small target detection, and if the precision for small target detection is to be improved, a large number of calculations are required, resulting in reduced fault detection efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a truck connecting ball head fault detection method and device, an electronic device and a computer readable storage medium, to realize the technical effect of improving the precision of small target detection while ensuring the fault detection efficiency.

[0006] In order to achieve the above technical effect, in a first aspect, the present application provides a truck connecting ball head fault detection method, comprising: obtaining a detection picture including a truck connecting ball head; inputting the detection picture into a pre-trained fault detection model to obtain the output result of the fault detection model as the fault detection result of the truck connecting ball head, the fault detection model comprising a BiMamba network layer, the BiMamba network layer comprising a plurality of feature layers, each feature layer being used to extract feature information of the detection picture at different resolutions.

[0007] In a possible embodiment, the feature layers include a first type of feature layer and a second type of feature layer, the first type of feature layer corresponding to a resolution greater than that of the second type of feature layer. The first type of feature layer includes an MB mixed structure, and the second type of feature layer includes a Mamba structure.

[0008] In a possible embodiment, the MB mixed structure includes a BRA module, a feature division module, a bidirectional SSM module, and a logical operation module. The BRA module is configured to extract sparse features enhanced by sparse self-attention from the detection image, the feature division module is configured to divide the sparse features into first features and second features, and the bidirectional SSM module is configured to perform forward processing on the first features to obtain forward first features, and perform backward processing on the first features to obtain backward first features. The logical operation module is configured to multiply the forward first features and the second features to obtain forward product features, multiply the backward first features and the second features to obtain backward product features, and sum the forward product features and the backward product features to obtain the feature information.

[0009] In a possible embodiment, the BRA module includes a region division module, a region screening module, and an attention enhancement module. The region division module is configured to divide the detection image into a plurality of non-overlapping regions, the region screening module is configured to screen target regions including the truck connecting ball head from the plurality of non-overlapping regions, and the attention enhancement module is configured to perform attention enhancement calculation on each of the target regions to obtain the sparse features.

[0010] In a possible embodiment, the first type of feature layer includes a patch embedding feature layer and a patch merging feature layer, the patch embedding feature layer includes a patch embedding module, the patch merging feature layer includes a patch merging module, and the resolution corresponding to the patch embedding feature layer is greater than that of the patch merging feature layer.

[0011] In a possible embodiment, the fault detection model includes a cross-feature fusion network, the cross-feature fusion network includes a first type of feature fusion layer and a second type of feature fusion layer, the first type of feature fusion layer includes a plurality of first type of fusion structures connected in sequence and a first type of convolution structure arranged between two first type of fusion structures, the second type of feature fusion layer includes a plurality of second type of fusion structures connected in sequence and a second type of convolution structure arranged between two second type of fusion structures, the feature information is sequentially input into the second type of fusion structures and the second type of convolution structure via each first type of fusion structure and the first type of convolution structure, and each decoupled head corresponding to each feature information is output via each second type of fusion structure.

[0012] In a possible implementation, the fault detection model further comprises a decoupling head structure, the decoupling head structure comprising a classification branch, a regression branch and a state detection branch independent of each other; The classification branch comprises a SiLU activation module and a CLsLoss algorithm module; The regression branch comprises a SiLU activation module and a BboxLoss algorithm module; The state detection branch comprises a SiLU activation module and a StatusLoss algorithm module.

[0013] In a second aspect, the present application provides a truck connecting ball head fault detection device, comprising: An image acquisition module, configured to acquire a detection picture comprising a truck connecting ball head; A fault detection module, configured to run a fault detection model, and input the detection picture into the pre-trained fault detection model to obtain an output result of the fault detection model as a fault detection result of the truck connecting ball head, wherein the fault detection model comprises a BiMamba network layer, and the BiMamba network layer comprises a plurality of feature layers, each of which is configured to extract feature information of the detection picture at different resolutions.

[0014] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, wherein The memory is configured to store a program; The processor is coupled to the memory and configured to execute the program stored in the memory to implement the steps of the truck connecting ball head fault detection method described in any of the implementation manners.

[0015] In a fourth aspect, the present application further provides a computer readable storage medium for storing computer readable programs or instructions, which can implement the steps of the truck connecting ball head fault detection method described in any of the implementation manners when executed by a processor.

[0016] The present application has the following beneficial effects: Compared with the related art, in the truck connecting ball head fault detection method and device, the electronic device and the computer readable storage medium provided in the application, after the detection picture including the truck connecting ball head is acquired, the pre-trained fault detection model is used for image recognition on the detection picture to determine the fault detection result of the truck connecting ball head. Since the BiMamba network layer is included in the fault detection model, the feature information of the detection picture under different resolutions can be extracted through the multiple feature layers set in the BiMamba network layer. Based on the feature information of the detection picture extracted under different resolutions, for the feature information with higher resolution, rich details and position information can be better extracted for the tiny target features, and for the feature information with lower resolution, the semantic information is stronger, so that the calculation efficiency can be saved and the detection efficiency of the fault detection model is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The flowchart of the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 2 The schematic diagram of the fault detection model in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 3 The schematic diagram of the MB mixed structure in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 4 The schematic diagram of the BRA module in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 5 The schematic diagram of the cross feature fusion network in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 6 The schematic diagram of the PConv convolution structure in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 7 The schematic diagram of the decoupling head structure in the truck connecting ball head fault detection method provided by an embodiment of the application is shown in the figure. Figure 8 The structural schematic diagram of the truck connecting ball head fault detection device provided by an embodiment of the application is shown in the figure. Figure 9 The structural schematic diagram of the electronic device provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more than two. The association relationship of "and / or" describing the associated objects means that there can be three relationships, for example: A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone.

[0021] The "first", "second", and the like described in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the technical features limited by "first" and "second" can explicitly or implicitly include at least one of the features.

[0022] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The present application provides a truck connecting ball head fault detection method, a truck connecting ball head fault detection device, an electronic device and a computer readable storage medium, which are described below respectively.

[0024] Please refer to Figure 1 The truck connecting ball head fault detection method provided in the embodiment specifically includes the following steps: Step S101: Obtain a detection picture including a truck connecting ball head.

[0025] Specifically, the truck connecting ball head fault detection device can specifically include a photographing assembly. On this basis, the detection picture including the truck connecting ball head acquired in this step can be specifically a picture including the connecting ball head photographed by the photographing assembly every interval of a period of time or upon receipt of a photographing instruction. Alternatively, in some other embodiments of the present application, the truck connecting ball head fault detection device can specifically include a communication assembly. On this basis, the detection picture including the truck connecting ball head acquired in this step can be specifically a picture including the connecting ball head photographed by an external photographing device and acquired by the communication assembly through communication with the external photographing device. It can be understood that the foregoing is only an example of a specific method for acquiring a detection picture in some embodiments of the present application, and in the actual application process of the present application, a picture including a truck connecting ball head can be acquired as a detection picture by any method, and the present application does not limit the specific manner of acquiring a detection picture.

[0026] Step S102: inputting the detection picture into the pre-trained fault detection model to acquire an output result of the fault detection model as a fault detection result of the truck connecting ball head.

[0027] In this step, please refer to Figure 2 The fault detection model includes a BiMamba (bidirectional Mamba) network layer 201, and the BiMamba network layer includes a plurality of feature layers (2011, 2012, 2013, 2014) for extracting features of the detection picture. Each feature layer is respectively used to extract feature information of the detection picture at different resolutions, i.e., output feature maps feature1, feature2, feature3, and feature4. Figure 2 For example, in this embodiment, the BiMamba network layer specifically includes four feature layers (2011, 2012, 2013, 2014), and the four feature layers respectively correspond to acquiring feature information of the detection picture at four different resolutions of 160x160, 80x80, 40x40, and 20x20. Specifically, the feature layer 2014 is used to acquire feature information of the detection picture at a resolution of 20x20, the feature layer 2013 is used to acquire feature information of the detection picture at a resolution of 40x40, the feature layer 2012 is used to acquire feature information of the detection picture at a resolution of 80x80, and the feature layer 2011 is used to acquire feature information of the detection picture at a resolution of 160x160.

[0028] Further, in the present application, the feature layer can specifically include a first type of feature layer and a second type of feature layer. The first type of feature layer includes an MB (Mamba-Biformer) mixed structure 2015, and the second type of feature layer includes a Mamba structure 2016. The resolution corresponding to the first type of feature layer is greater than the resolution corresponding to the second type of feature layer. Please continue to refer to Figure 2In the embodiment, the feature layer 2011 and the feature layer 2012 are a first type of feature layer, and the feature layer 2013 and the feature layer 2014 are a second type of feature layer.

[0029] Further, please refer to Figure 3 The MB hybrid structure includes a BRA module 301, a feature division module 302, a bidirectional SSM module 303, and a logical operation module 304. The BRA module 301 is configured to extract sparse features enhanced by sparse self-attention from a detection image. The feature division module 302 is configured to divide the sparse features into first features and second features. The bidirectional SSM module 303 is configured to perform forward processing on the first features to obtain forward first features, and perform backward processing on the first features to obtain backward first features. The logical operation module 304 is configured to multiply the forward first features and the second features to obtain forward product features, multiply the backward first features and the second features to obtain backward product features, and sum the forward product features and the backward product features to obtain feature information.

[0030] In the present application, the first type of feature layer includes a patch embedding feature layer and a patch merging feature layer. The patch embedding feature layer includes a patch embedding module, and the patch merging feature layer includes a patch merging module. The resolution corresponding to the patch embedding feature layer is greater than the resolution corresponding to the patch merging feature layer. Please continue to refer to Figure 2 That is, the first type of feature layer can also include a PatchEmbedding (image block embedding) module 2017 or a PatchMerging (image block merging) module 2018. In the embodiment, the feature layer 2011 includes the PatchEmbedding (image block embedding) module 2017, which is a patch embedding feature layer, and the feature layer 2012 includes the PatchMerging (image block merging) module 2018, which is a patch merging feature layer. In addition, the second type of feature layer can also include the PatchMerging (image block merging) module 2018, and the feature layer 2013 and the feature layer 2014 include the PatchMerging (image block merging) module 2018.

[0031] Further, please refer to Figure 4In the embodiment, the BRA module includes a region division module 401, a region screening module 402, and an attention enhancement module 403. The region division module 401 is configured to divide the detection image into a plurality of non-overlapping regions. The region screening module 402 is configured to screen a target region including the truck connecting ball head from the plurality of non-overlapping regions. The attention enhancement module 403 is configured to perform attention enhancement calculation on each target region to obtain sparse features. The region screening module 402 screening the target region including the truck connecting ball head from the plurality of non-overlapping regions specifically includes: dividing the input feature map HxWxC into SxS regions, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map. Each region contains HxW / SxS tokens. The tokens in each region are averaged to obtain a region-level feature tensor X. The region-level feature X is projected to query Q, key K 2 , and value V 2 through three independent linear transformation matrices. The query Q and the key K 2 are constructed into an inter-region affinity matrix A through matrix multiplication mm&softmax. The target region is selected through the affinity matrix A. The attention enhancement module 403 performs attention enhancement calculation on each target region to obtain sparse features. The sparse features O can be obtained by screening a plurality of target regions for attention enhancement calculation through the value V 2 .

[0032] Specifically, in the present embodiment, after the detection picture input of 640x640x3 size is input into the BiMamba network layer 201, first, in the feature layer 2011, a high-dimensional feature vector of 160x160x64 size is obtained by using a convolution algorithm with a step of 4 for downsampling operation by the PatchEmbedding module 2017, and then multi-level local fine feature extraction is performed through the MB mixed structure 2015 to obtain a feature map of 160x160x64 size. The feature map has a high spatial resolution and contains rich details and position information, and is specially used for extracting micro target features. Then, the feature map of 160x160x64 size is input into the feature layer 2012. In the feature layer 2012, the feature map of 160x160x64 size is first reduced in resolution and doubled in channel number through the PatchMerging module 2018 to realize 2 times downsampling operation. Then, similarly, the feature extraction is performed through the MB mixed structure 2015 to obtain a feature map of 80x80x128 size, which is used for extracting medium and small targets. In the feature layers 2013 and 2014, similarly, the PatchMerging module 2018 is used for downsampling to reduce the resolution of the feature map and double the number of channels, and then the Mamba structure 2016 is used for feature extraction to obtain feature maps of 40x40x256 and 20x20x512 sizes, respectively. Since the spatial resolution of these two layers is low and the position feature information is weak, but the semantic information is very strong, the Mamba module is directly used for feature extraction to save the calculation efficiency and improve the model performance.

[0033] Please continue to refer to Figure 2 In the present application, the fault detection model can further include a cross-feature fusion network 202. Please refer to Figure 5 In the present embodiment, the cross-feature fusion network 202 includes a first-type feature fusion layer 501 and a second-type feature fusion layer 502. The first-type feature fusion layer 501 includes a plurality of first-type fusion structures connected in sequence and a first-type convolution structure arranged between two first-type fusion structures. The second-type feature fusion layer 502 includes a plurality of second-type fusion structures connected in sequence and a second-type convolution structure arranged between two second-type fusion structures. The feature information is input into the second-type fusion structure and the second-type convolution structure after passing through each first-type fusion structure and the first-type convolution structure in sequence, and each feature information is output through each second-type fusion structure as a decoupling head.

[0034] Please continue to refer to Figure 5 With Figure 2The four feature layers correspond to four feature layers shown in the embodiment, which can specifically include three first feature fusion layers and three first convolution structures, namely, a first feature fusion layer A, a first feature fusion layer B, a first feature fusion layer C, a first convolution structure a, a first convolution structure b, and a first convolution structure c. Correspondingly, the four feature layers also include four second feature fusion layers and three second convolution structures, namely, a second feature fusion layer Q, a second feature fusion layer W, a second feature fusion layer E, a second feature fusion layer R, a second convolution structure q, a second convolution structure w, and a second convolution structure e. In the embodiment, the four feature layers output feature maps feature1, feature2, feature3, and feature4, respectively. The feature4 is input into the first convolution structure a for convolution, and the convolution result is input into the first feature fusion layer A, the second feature fusion layer Q, and the second feature fusion layer W. The first feature fusion layer A is fused with the feature3, and then the fusion result is input into the first convolution structure b, the second feature fusion layer Q, and the second feature fusion layer W. The first convolution structure b is used to perform convolution on the input feature, and the convolution result is input into the first feature fusion layer B for fusion with the feature2. Then, the fusion result is input into the first convolution structure c, the second feature fusion layer E, and the second feature fusion layer R. The first convolution structure c is used to perform convolution on the input feature, and the convolution result is input into the first feature fusion layer C for fusion with the feature1. Then, the fusion result is input into the second feature fusion layer Q, the second feature fusion layer E, and the second feature fusion layer R. Finally, the second feature fusion layer Q, the second feature fusion layer W, the second feature fusion layer E, and the second feature fusion layer R are used to perform convolution and fusion, respectively, and the second feature fusion layer Q, the second feature fusion layer W, the second feature fusion layer E, and the second feature fusion layer R output feature maps feature1, feature2, feature3, and feature4, respectively, corresponding to decoupling heads head1, head2, head3, and head4.

[0035] Further, in the embodiment, the first convolution structure includes a SPConv (Sparse Convolution) convolution structure. The SPConv is a sub-pixel convolution structure, which disperses the pixels of each high-dimensional feature map to different positions of a low-dimensional feature map by rearranging the channels of the feature map, so as to realize up-sampling of the feature layer. The operation has high calculation efficiency, few parameters, and is not easy to lose feature information, and can be trained end-to-end with other convolution layers.

[0036] In addition, the second convolution structure includes a PConv (Partial Convolution) convolution structure. Please refer to Figure 6PConv reduces the computation and memory access by applying regular convolution only to part of the input channels and leaving the rest untouched. Further, in the embodiment, the size of the convolution kernel can be set to 3 and the stride can be set to 2 when the convolution is performed on part of the channels in the PConv convolution structure.

[0037] In the present application, after the output of each feature information corresponding to the decoupling head, the decoupling head separates the classification and regression tasks, reduces the mutual interference between tasks, and enables the model to focus more on the respective features and predictions, thereby improving the detection accuracy. For the decoupling head, the fault detection model further includes a decoupling head structure for data processing of the decoupling head. Please refer to Figure 7 The decoupling head structure includes a classification branch 701, a regression branch 702, and a state detection branch 703 which are independent of each other. Among them, the classification branch is responsible for predicting the class probability of the target; the regression branch is responsible for predicting the bounding box coordinates of the target; and the state detection branch is responsible for predicting the state probability of the target.

[0038] The classification branch includes a one-time convolution module, a BN (Batch Normalization) module, a SiLU activation module, a two-time convolution module, and a CLsLoss algorithm module connected in sequence. Among them, the one-time convolution module and the two-time convolution module are used for convolution operation; the BN module is used to standardize the input feature distribution of the one-time convolution module, accelerate the neural network training, and alleviate the gradient disappearance problem; the SiLU activation module is used to introduce nonlinear features for the neural network, so that the model can fit complex data distribution; and the core algorithm of the CLsLoss algorithm module is specifically as follows: ; Among them, N is the batch size of the feature information input into the decoupling head, y i is the true label (taking 0 or 1); is the probability predicted by the model.

[0039] The regression branch includes a one-time convolution module, a BN (Batch Normalization) module, a SiLU activation module, a two-time convolution module, and a BboxLoss algorithm module connected in sequence. The core algorithm of the BboxLoss algorithm module is specifically as follows: ; Among them, N is the batch size of the feature information input into the decoupling head, p i is the predicted bounding box coordinate distribution, t i is the true bounding box coordinate distribution.

[0040] The state detection branch includes a first convolution module, a BN (Batch Normalization) module, a SiLU activation module, a second convolution module and a StatusLoss algorithm module connected in sequence. The core algorithm of the StatusLoss algorithm module is specifically: , ; wherein, is the Euclidean distance between the center point of the real box and the center point of the prediction box, c is the diagonal distance of the minimum enclosing rectangle. v is a parameter for measuring the consistency of the aspect ratio, w g , h g are the width and height of the real box, w p , h p are the width and height of the prediction box, IOU is the overlap index of the center point of the real box and the center point of the prediction box.

[0041] Compared with the related art, in the truck connecting ball head fault detection method provided in the embodiment, after a detection picture including a truck connecting ball head is acquired, a pre-trained fault detection model is used to perform image recognition on the detection picture to determine a fault detection result of the truck connecting ball head. Since the fault detection model includes a BiMamba network layer, the feature information of the detection picture at different resolutions can be extracted through the multiple feature layers set in the BiMamba network layer. Based on the feature information of the detection picture extracted at different resolutions, for the feature information at a higher resolution, rich details and position information can be better extracted for the micro target features, and for the feature information at a lower resolution, the semantic information is stronger, so that the calculation efficiency can be saved and the detection efficiency of the fault detection model can be ensured.

[0042] In order to better implement the truck connecting ball head fault detection method in the embodiment of the present application, on the basis of the truck connecting ball head fault detection method, as shown in Figure 8 , the embodiment of the present application further provides a truck connecting ball head fault detection device. The truck connecting ball head fault detection device comprises: an image acquisition module 801, the image acquisition module 801 is configured to acquire a detection picture including a truck connecting ball head; The fault detection module 802 runs a fault detection model, and is configured to input the detection picture into the pre-trained fault detection model to obtain an output result of the fault detection model as a fault detection result of the truck connecting ball head. The fault detection model includes a BiMamba network layer, and the BiMamba network layer includes a plurality of feature layers, each of which is configured to extract feature information of the detection picture at different resolutions.

[0043] The truck connecting ball head fault detection device provided by the above embodiments can implement the technical solutions described in the truck connecting ball head fault detection method embodiments. The principles of the implementation of the above modules or units can be referred to the corresponding content in the truck connecting ball head fault detection method embodiments, which will not be described here.

[0044] Please refer to Figure 9 The present application also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902, and a display 903. Figure 9 Only part of the components of the electronic device 900 are shown, but it should be understood that all the shown components are not required, and more or less components can be alternatively implemented.

[0045] The processor 901 can be a central processing unit (CPU), a microprocessor, or other data processing chip in some embodiments, used to run program codes or process data stored in the memory 902, such as the truck connecting ball head fault detection method in the present application.

[0046] In some embodiments, the processor 901 can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor 901 can be local or remote. In some embodiments, the processor 901 can be implemented in a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-internal, a multiple cloud, etc., or any combination thereof.

[0047] The memory 902 can be an internal storage unit of the electronic device 900 in some embodiments, such as a hard disk or memory of the electronic device 900. The memory 902 can also be an external storage device of the electronic device 900 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 900.

[0048] Furthermore, the memory 902 may include both internal storage units of the electronic device 900 and external storage devices. The memory 902 is used to store application software and various types of data installed on the electronic device 900.

[0049] In some embodiments, display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 903 is used to display information from electronic device 900 and to display a visual user interface. Components 901-903 of electronic device 900 communicate with each other via a system bus.

[0050] In one embodiment, when processor 901 executes the truck connection ball joint fault detection program in memory 902, the following steps can be implemented: Obtain detection images including the truck's connecting ball joint; The detected image is input into the pre-trained fault detection model, and the output of the fault detection model is used as the fault detection result of the truck connecting ball joint. The fault detection model includes a BiMamba network layer, which includes multiple feature layers. Each feature layer is used to extract feature information of the detected image at different resolutions.

[0051] It should be understood that when the processor 901 executes the truck connection ball joint fault detection program in the memory 902, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0052] Furthermore, this application does not specifically limit the type of electronic device 900 mentioned in the embodiments. Electronic device 900 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, electronic device 900 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0053] Correspondingly, the embodiment of the present application further provides a computer readable storage medium, which is used to store computer readable programs or instructions, and the programs or instructions are executed by a processor to realize the steps or functions in the truck connecting ball head fault detection method provided by the above-mentioned method embodiments.

[0054] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware (such as a processor, a controller, etc.) to complete. The computer program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0055] The truck connecting ball head fault detection method, device, electronic equipment and storage medium provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above embodiment is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for detecting faults in truck connecting ball joints, characterized in that, include: Obtain detection images including the truck's connecting ball joint; The detected image is input into a pre-trained fault detection model, and the output of the fault detection model is obtained as the fault detection result of the truck connecting ball joint. The fault detection model includes a BiMamba network layer, and the BiMamba network layer includes multiple feature layers. Each feature layer is used to extract feature information of the detected image at different resolutions.

2. The truck connection ball joint fault detection method according to claim 1, characterized in that, The feature layer includes a first-type feature layer and a second-type feature layer, wherein the resolution corresponding to the first-type feature layer is greater than the resolution corresponding to the second-type feature layer. The first type of feature layer includes an MB hybrid structure, and the second type of feature layer includes a Mamba structure.

3. The truck connection ball joint fault detection method according to claim 2, characterized in that, The MB hybrid structure includes a BRA module, a feature partitioning module, a bidirectional SSM module, and a logic operation module. The BRA module is used to extract sparse features after sparse self-attention enhancement from the detection image. The feature division module is used to divide the sparse features into a first feature and a second feature. The bidirectional SSM module is used to perform forward processing on the first feature to obtain a forward first feature and backward processing on the first feature to obtain a backward first feature. The logic operation module is used to multiply the forward first feature and the second feature to obtain the forward product feature, multiply the backward first feature and the second feature to obtain the backward product feature, and sum the forward product feature and the backward product feature to obtain the feature information.

4. The truck connection ball joint fault detection method according to claim 3, characterized in that, The BRA module includes a region segmentation module, a region filtering module, and an attention enhancement module; The region segmentation module is used to divide the detection image into several non-overlapping regions. The region filtering module is used to filter out the target region including the truck connecting ball head from the several non-overlapping regions. The attention enhancement module is used to perform attention enhancement calculation on each of the target regions to obtain the sparse features.

5. The truck connection ball joint fault detection method according to claim 2, characterized in that, The first type of feature layer includes a patch embedding feature layer and a patch merging feature layer. The patch embedding feature layer includes a patch embedding module, and the patch merging feature layer includes a patch merging module. The resolution corresponding to the patch embedding feature layer is greater than the resolution corresponding to the patch merging feature layer.

6. The truck connection ball joint fault detection method according to claim 1, characterized in that, The fault detection model includes a cross-feature fusion network, which comprises a first-class feature fusion layer and a second-class feature fusion layer. The first-class feature fusion layer includes multiple sequentially connected first-class fusion structures and a first-class convolutional structure disposed between two of the first-class fusion structures. The second-class feature fusion layer includes multiple sequentially connected second-class fusion structures and a second-class convolutional structure disposed between two of the second-class fusion structures. The feature information is sequentially input into the second-class fusion structure and the second-class convolutional structure after passing through each of the first-class fusion structures and the second-class convolutional structure. The second-class fusion structure outputs the decoupling head corresponding to each feature information.

7. The truck connection ball joint fault detection method according to claim 6, characterized in that, The fault detection model also includes a decoupling head structure for data processing of the decoupling head, the decoupling head structure including mutually independent classification branches, regression branches and state detection branches; The classification branch includes the SiLU activation module and the CLsLoss algorithm module; The regression branch includes the SiLU activation module and the BboxLoss algorithm module; The state detection branch includes the SiLU activation module and the StatusLoss algorithm module.

8. A truck connection ball joint fault detection device, characterized in that, include: An image acquisition module, wherein the image acquisition module is used to acquire a detection image including a truck connecting ball joint; The fault detection module runs a fault detection model and is used to input the detected image into the pre-trained fault detection model and obtain the output result of the fault detection model as the fault detection result of the truck connecting ball joint. The fault detection model includes a BiMamba network layer, which includes multiple feature layers. Each feature layer is used to extract feature information of the detected image at different resolutions.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the truck connection ball joint fault detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can perform the steps in the truck connection ball joint fault detection method according to any one of claims 1 to 7.

Images