Multi-modal detection method and system for looseness of bolt fastener

Abstract

The invention belongs to the field of mechanical structure state detection, and particularly discloses a multi-modal detection method and system for bolt fastener looseness, and the method comprises the steps: collecting corresponding ultrasonic information in the axial direction of a bolt fastener under different fastening conditions of the bolt fastener, collecting corresponding RGB images in the radial direction of the bolt fastener, and carrying out the detection of the looseness of the bolt fastener; obtaining a multi-modal sample composed of a plurality of groups of RGB images and ultrasonic information; performing format adjustment and data enhancement on the multi-modal sample to obtain multi-modal enhanced data; constructing a bolt looseness detection network, and training the bolt looseness detection network by using the multi-modal enhanced data; and inputting the RGB image of the to-be-detected bolt fastener and the ultrasonic information into the trained bolt looseness detection network, and evaluating the bolt looseness level of the to-be-detected bolt fastener. According to the invention, complementarity of information among different modes of the bolt fastener can be fully utilized, and the loosening condition of the bolt can be judged in a more comprehensive view angle.

Description

Technical Field

[0001] The present application belongs to the technical field of mechanical structure status detection, and more specifically, to a multi-modal detection method and system for loosening of bolt fasteners. Background Art

[0002] For mechanical systems, the assembly quality of bolt fasteners will directly affect the quality safety, working stability and service life of the entire system. When important fastening points become loose or fail, engineering accidents may even occur. The accuracy, detection speed, convenience and application cost of bolt tightening detection methods are important evaluation references for fastener assembly detection methods. Traditional detection of fastener loosening is mainly through applying anti-loosening line and manually observing whether this line remains continuous after the bolts are tightened. This detection method is inefficient and low-precision, and relies on a lot of manual labor, which makes it difficult to promote the modernization and intelligence of the industry.

[0003] Among the existing intelligent detection methods, there are looseness detection methods based on machine vision, tightening detection based on voiceprint information, and bolt preload detection methods based on ultrasonic sensing. CN118658003A discloses a bolt loosening detection method and device based on RGB-D. After obtaining the standard image and test image of the RGB and depth image of the fastener, the bolt segmentation network is input to obtain the standard image and test image of the bolt segmentation mask. After the RGB image, depth image and segmentation mask are sent to the bolt twin classification network, the result of whether the bolt is loose can be obtained. This method adopts a pure visual method, which will be limited by the performance of the sensor and the interference of the light source noise in the actual production environment. The accuracy and stability of the system still need to be further considered and optimized. CN118571251A discloses a bolt loosening assessment method based on acoustic emission characteristic parameters. This method collects acoustic emission signals at the bolt connection, obtains the spectrum through fast Fourier transform, and performs energy spectrum analysis to obtain the distribution of energy. Subsequently, the evaluation index is dimensionless, the loosening characteristic value is established, and the state classification of the bolt loosening is completed. This method completely relies on voiceprint information for looseness detection. In practical applications, the sampling rate and resolution of its signal collection components are extremely high, and it is difficult to maintain the effective characteristics of the information when other noises appear. CN109668672A discloses an ultrasonic bolt preload measurement method, which calculates the elongation of the bolt by ultrasonically measuring the acoustic time difference between the bolt in the free state and the tightened state, and obtains the bolt preload based on the elongation to accurately measure the bolt preload. The ultrasonic method has the characteristics of high precision, but the excitation of ultrasonic signals often requires the cooperation of a coupling agent, which will face great difficulties under the requirements of large-scale and high-speed detection. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of the present application is to provide a multimodal detection method and system for loosening of bolt fasteners, aiming to solve the problems that the existing detection methods have low detection accuracy and are easily restricted by the detection environment.

[0005] To achieve the above objectives, the present application provides a multi-modal detection method for loosening of bolt fasteners, comprising:

[0006] S1, under different tightening conditions of the bolt fastener, collecting corresponding ultrasonic information along the axial direction of the bolt fastener, and collecting corresponding RGB images along the radial direction of the bolt fastener, so as to obtain a multimodal sample consisting of a plurality of groups of RGB images and the ultrasonic information;

[0007] S2 performs format adjustment and data enhancement on the multimodal samples to obtain multimodal enhanced data;

[0008] S3 constructs a bolt loosening detection network, and trains the bolt loosening detection network using the multimodal enhanced data;

[0009] S4 inputs the RGB image and ultrasonic information of the bolt fastener to be tested into the bolt loosening detection network trained in step S3 to evaluate the bolt loosening level of the bolt fastener to be tested.

[0010] The present application performs feature extraction and data fusion on multi-modal information of bolt tightening conditions, thereby making full use of the complementarity of information between different modes and judging the looseness of the bolts from a more comprehensive perspective.

[0011] Furthermore, in step S2, the step of format adjustment and data enhancement of the multimodal sample includes:

[0012] S201 adjusts the size of the RGB image and performs normalization processing;

[0013] S202 divides the normalized data into a training set, a validation set, and a test set;

[0014] S203 performs data enhancement on the RGB image data in the training set.

[0015] Furthermore, in step S3, the bolt loosening detection network includes a feature extraction module, a self-attention mechanism module, a cross-attention module and a multi-layer perceptron decision layer.

[0016] Furthermore, in step S3, the step of training a bolt loosening detection network using the multimodal enhanced data includes:

[0017] S301: inputting the ultrasound modality data in the multimodal enhanced data into the self-attention mechanism module to obtain a first attention weight; inputting the RGB image modality data in the multimodal enhanced data into the feature extraction module to obtain an RGB image feature map; inputting the RGB image feature map into the self-attention mechanism module for learning to obtain a second attention weight;

[0018] S302 uses the first attention weight as a query and the second attention weight as a key and a value to perform feature fusion in the cross attention module;

[0019] S303 inputs the fused features into the decision layer of the multi-layer perceptron to obtain a bolt fastener loosening classification decision vector.

[0020] Furthermore, in step S3, a cross entropy function is used as a loss function when training the bolt loosening detection network. The loss function is expressed as:

[0021]

[0022] Where M is the number of categories; c represents the cth label among the M category labels; if the true category of sample i is equal to c, y ic Take 1, otherwise, y ic Take 0; p ic is the predicted probability that the observed sample i belongs to the cth label, N represents the total number of samples taken to calculate the loss once, and i represents the index of the sample, ranging from 1 to N.

[0023] Furthermore, in step S301, the first attention weight is expressed as:

[0024]

[0025] Among them, Attention(Q1,K i1 ,V i1 ) ultrasound is the first attention weight, Q1 is the query vector corresponding to the ultrasound modality, K i1 is the key vector corresponding to the ultrasound modality, used to match the query vector, d k1 is the dimension of the encoded vector corresponding to the ultrasonic modality, V i1 is the value vector corresponding to the ultrasonic mode, Q1 T Represents the transpose of the query vector corresponding to the ultrasound modality.

[0026] Furthermore, in step S301, the second attention weight is expressed as:

[0027]

[0028] Among them, Attention(Q2,K i2 ,V i2 ) image is the second attention weight, Q2 is the query vector corresponding to the image modality, and K i2 is the key vector corresponding to the image modality, d k2 is the dimension of the encoded vector corresponding to the image modality, V i2 is the value vector corresponding to the image modality, Q2 T Represents the transpose of the query vector corresponding to the image modality.

[0029] Furthermore, in step S302, the first attention weight is used as a query, and the second attention weight is used as a key and a value, and the cross attention method is used to perform feature fusion. The fusion process is expressed as:

[0030] Q S34 =Attention(Q1,K i1 ,V i1 ) ultrasound

[0031] K S34 =V S34 =Attention(Q2,K i2 ,V i2 ) image

[0032]

[0033] Among them, Q S34 is the attention output from ultrasound data, as the query vector; K S34 is the attention output from the RGB image data, as the key vector; V S34 is the attention output from the RGB image data as a value vector; Q1 is the query vector corresponding to the ultrasound modality, K i1 is the bond vector corresponding to the ultrasonic mode, V i1 is the value vector corresponding to the ultrasound modality; Q2 is the query vector corresponding to the image modality, K i2 is the key vector corresponding to the image modality, V i2 is the value vector corresponding to the image modality, d k is the dimension size of the key vector.

[0034] According to another aspect of the present application, a multimodal detection system for implementing any of the multimodal detection methods described above is provided, comprising:

[0035] A sample collection module, used for collecting corresponding ultrasonic information along the axial direction of the bolt fastener and collecting corresponding RGB images along the radial direction of the bolt fastener under different tightening conditions of the bolt fastener, so as to obtain a plurality of groups of multimodal samples consisting of the RGB images and the ultrasonic information;

[0036] A data preprocessing module, used for performing format adjustment and data enhancement on the multimodal samples to obtain multimodal enhanced data;

[0037] A neural network training module, used for constructing a bolt loosening detection network, and training the bolt loosening detection network using the multimodal enhanced data;

[0038] The bolt loosening detection module is used to input the RGB image and ultrasonic information of the bolt fastener to be tested into the trained bolt loosening detection network to evaluate the bolt loosening level of the bolt fastener to be tested.

[0039] According to another aspect of the present application, the present application also provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0040] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0041] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0042] (1) This application can fully utilize the complementarity of information between different modes and judge the looseness of bolts from a more comprehensive perspective by extracting features and fusing data from multi-modal information of bolt tightening conditions. The detection method of this application has high robustness. Even if a certain degree of noise appears in the single-modal information, the system can still make accurate judgments after being supplemented by other modal information.

[0043] (2) The detection method of the present application integrates different types of data and utilizes the complementary information between different modes through multimodal enhanced data fusion to improve the accuracy of the model and its generalization ability in different situations. In decision-level fusion, since each sensor makes a decision based on its own data first, and then the fusion center completes the local decision, it has a smaller communication volume and stronger anti-interference ability. In short, the detection method of the present application has the characteristics of higher accuracy, stronger anti-interference ability, higher efficiency and speed, and can complete the loosening detection task of bolt fasteners with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1It is a schematic diagram of the process of a multi-modal detection method for loosening of bolt fasteners provided in an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the data processing and network module construction process of the multi-modal detection method for loosening of bolt fasteners provided in an embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of the actual detection process of the multi-modal detection method for loosening of bolt fasteners provided in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the multimodal neural network refinement structure of the multimodal detection method for loosening of bolt fasteners provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0050] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0051] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] In the description of the embodiments of the present application, it is otherwise stated that “multiple” means two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0053] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0054] This embodiment provides a multi-modal detection method for loosening of bolt fasteners. The process of the method is as follows: Figure 1 and Figure 2 As shown, the main steps include:

[0055] S1 collects ultrasonic information corresponding to the tightening conditions of the bolt fastener along the axial direction of the bolt fastener, and collects RGB images corresponding to the tightening conditions along the radial direction of the bolt fastener. The RGB images and ultrasonic information of the bolt fastener in each tightening condition are taken as a group of samples. After multiple collections, multimodal samples consisting of multiple groups of RGB images and ultrasonic information are obtained.

[0056] S2 performs format adjustment and data enhancement on multimodal samples to obtain multimodal enhanced data.

[0057] The steps for formatting and data augmentation of multimodal samples include:

[0058] S201 adjusts the size of the RGB image and performs normalization processing; specifically, after all data are collected, the size of all RGB images is adjusted to [224, 224] according to the bilinear interpolation method; then all data are normalized, and the normalization formula is:

[0059]

[0060] In the formula, x new is the normalized data, x is the source data, x min is the minimum value in the source data set, x max is the maximum value in the source data set.

[0061] S202 randomly divides the normalized data into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0062] S203 performs data enhancement on the RGB image data in the training set. Specifically, after the division is completed, the RGB image data in all training set samples are horizontally symmetrical and brightness adjusted, Gaussian noise is added, randomly cropped and filled, and black data masking is randomly added. After a series of processing, the data enhancement process is completed.

[0063] S3 constructs a bolt loosening detection network, which includes at least a feature extraction module (CNN backbone network), a self-attention mechanism module, a cross-attention module and a multi-layer perceptron decision layer. The bolt loosening detection network is trained using the multi-modal enhanced data after the aforementioned enhancement processing.

[0064] Specifically, the training steps include:

[0065] S301 inputs the ultrasound modality data into the self-attention mechanism module, outputs the first attention weight after learning the ultrasound modality information features through the self-attention mechanism module, and waits for the cross-attention layer processing. The formula of the ultrasound modality self-attention mechanism is as follows:

[0066]

[0067] Among them, Attention(Q1,K i1 ,V i1 ) ultrasound is the first attention weight, Q1 is the query vector corresponding to the ultrasound modality, K i1 is the key vector corresponding to the ultrasound modality, used to match the query vector, d k1 is the dimension of the encoded vector corresponding to the ultrasonic modality, V i1 is the value vector corresponding to the ultrasonic mode, Q1 T Represents the transpose of the query vector corresponding to the ultrasound modality.

[0068] The RGB image modality data in the multimodal enhanced data is input into the feature extraction module to obtain the RGB image feature map; after the RGB image feature map is input into the self-attention mechanism module for learning, the second attention weight is obtained, which is expressed as:

[0069]

[0070] Among them, Attention(Q2,K i2 ,V i2 ) image is the second attention weight, Q2 is the query vector corresponding to the image modality, and K i2 is the key vector corresponding to the image modality, V i2 is the value vector corresponding to the image modality, Q2 T represents the transpose of the query vector corresponding to the image modality, d k2 is the dimension size of the encoded vector corresponding to the image modality.

[0071] S302 uses the first attention weight as the query and the second attention weight as the key and value, and uses the cross attention method to perform feature fusion. The fusion process is expressed as:

[0072]

[0073] Among them, Q S34 is the attention output from ultrasound data, as the query vector, and Q S34 =Attention(Q1,K i1 ,V i1 ) ultrasound ; K S34is the attention output from the RGB image data, as the key vector, K S34 =V S34 =Attention(Q2,K i2 ,V i2 ) image ; V S34 is the attention output from the RGB image data as a value vector; d k is the dimension size of the key vector.

[0074] S303 inputs the fused features into the MLP multi-layer perceptron decision layer to obtain a classification decision vector for the loosening of the bolt fasteners. The classification decision vector is used for subsequent identification, prediction and evaluation of the level of the tightening condition of the bolt to be tested.

[0075] Specifically, after completing the cross-attention operation, the ultrasound and image signals are fused, and the final classification decision vector is output through the MLP layer. More specifically, Figure 4 As shown, the RGB image data is feature extracted via the CNN backbone network, which can be various traditional deep convolutional neural networks such as Resnet, which will not be described in detail in this embodiment. After convolution processing, the shallow features of the image are output, and the shallow features of the image are still two-dimensional multi-channel data. The shallow feature data of the image is embedded and encoded in the image block through the Vision Transformer module (ViT) and processed using the self-attention mechanism, and then the updated image features are output after decoding. The ultrasonic data is feature encoded through the Transformer module and feature extracted using the self-attention mechanism, and the updated ultrasonic features are output after decoding.

[0076] The main features provided by the two modal data in their respective data domains are the important high-level features extracted by the above method, namely the updated image features and the updated ultrasonic features. The updated image features and the updated ultrasonic features are then fused through the constructed cross-attention layer, so that the information of the ultrasonic modality and the visual modality can communicate with each other. The fused features are then extracted through the constructed MLP layer and integrated into a single-dimensional vector. The final classification decision vector is output through a fully connected operation to realize the judgment of the tightening condition of the bolt fasteners.

[0077] In the process of training the bolt loosening detection network, the cross entropy loss function and Adam optimizer are used for training, the validation set is used to adjust the hyperparameters, and the test set is used to detect the final performance of the network. The cross entropy function is used as a loss function and is expressed as:

[0078]

[0079] Where M is the number of categories; c represents the cth label among the M category labels; if the true category of sample i is equal to c, y ic Take 1, otherwise, y ic Take 0; p ic is the predicted probability that the observed sample i belongs to the cth label, N represents the total number of samples taken to calculate the loss once, and i represents the index of the sample, ranging from 1 to N.

[0080] The specific evaluation parameters considered are accuracy, precision, and recall, which can be expressed as:

[0081]

[0082] In the above, TP means that the positive sample is successfully predicted as positive, TN means that the negative sample is successfully predicted as negative, FP means that the negative sample is incorrectly predicted as positive, and FN means that the positive sample is incorrectly predicted as negative.

[0083] S4 inputs the RGB image and ultrasonic information of the bolt fastener to be tested into the trained bolt loosening detection network to evaluate the bolt loosening level of the bolt fastener to be tested.

[0084] like Figure 2 and 3 As shown, first create a multimodal dataset of bolts to be tested, and perform data enhancement and normalization on the multimodal dataset according to the steps of the aforementioned embodiment, and then input the enhanced and normalized data into the bolt fastener detection network trained in the aforementioned steps for actual detection. During detection, a convolutional neural network is used to collect the underlying features of the image; the high-level features of the ultrasonic modality and the visual modality are obtained based on the self-attention mechanism; the multimodal data fusion is completed based on the cross-attention operation; and finally, the MLP layer is used to output the evaluation results of the tightening condition of the bolt fasteners.

[0085] According to another aspect of the present application, in one embodiment, a multimodal detection system for implementing the multimodal detection method provided in any of the previous embodiments is provided, and the multimodal detection system includes:

[0086] A sample collection module is used to collect corresponding ultrasonic information along the axial direction of the bolt fastener and collect corresponding RGB images along the radial direction of the bolt fastener under different tightening conditions of the bolt fastener, so as to obtain multimodal samples consisting of multiple groups of RGB images and ultrasonic information;

[0087] The data preprocessing module is used to adjust the format and enhance the data of multimodal samples to obtain multimodal enhanced data;

[0088] A neural network training module is used to build a bolt loosening detection network and train the bolt loosening detection network using multimodal enhanced data;

[0089] The bolt loosening detection module is used to input the RGB image and ultrasonic information of the bolt fastener to be tested into the trained bolt loosening detection network to evaluate the bolt loosening level of the bolt fastener to be tested.

[0090] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0091] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0092] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor, a communications interface 820, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the method in the above embodiment.

[0093] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0094] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0095] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0096] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0098] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0099] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0100] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multi-modal detection method for loosening of bolt fasteners, characterized in that: include: S1, under different tightening conditions of the bolt fastener, collecting corresponding ultrasonic information along the axial direction of the bolt fastener, and collecting corresponding RGB images along the radial direction of the bolt fastener, so as to obtain a multimodal sample consisting of a plurality of groups of RGB images and the ultrasonic information; S2 performs format adjustment and data enhancement on the multimodal samples to obtain multimodal enhanced data; S3 constructs a bolt loosening detection network, and trains the bolt loosening detection network using the multimodal enhanced data; S4 inputs the RGB image and ultrasonic information of the bolt fastener to be tested into the bolt loosening detection network trained in step S3 to evaluate the bolt loosening level of the bolt fastener to be tested.

2. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 1, characterized in that: In step S2, the steps of format adjustment and data enhancement of the multimodal samples include: S201 adjusts the size of the RGB image and performs normalization processing; S202 divides the normalized data into a training set, a validation set, and a test set; S203 performs data enhancement on the RGB image data in the training set.

3. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 1, characterized in that: In step S3, the bolt loosening detection network includes a feature extraction module, a self-attention mechanism module, a cross-attention module and a multi-layer perceptron decision layer.

4. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 3, characterized in that: In step S3, the step of training a bolt loosening detection network using the multimodal enhanced data includes: S301: inputting the ultrasound modality data in the multimodal enhanced data into the self-attention mechanism module to obtain a first attention weight; inputting the RGB image modality data in the multimodal enhanced data into the feature extraction module to obtain an RGB image feature map; inputting the RGB image feature map into the self-attention mechanism module for learning to obtain a second attention weight; S302 uses the first attention weight as a query and the second attention weight as a key and a value to perform feature fusion in the cross attention module; S303 inputs the fused features into the decision layer of the multi-layer perceptron to obtain a bolt fastener loosening classification decision vector.

5. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 1, characterized in that: In step S3, a cross entropy function is used as a loss function when training the bolt loosening detection network. The loss function is expressed as: Where M is the number of categories; c represents the cth label among the M category labels; if the true category of sample i is equal to c, y ic Take 1, otherwise, y ic Take 0; p ic is the predicted probability that the observed sample i belongs to the cth label, N represents the total number of samples taken to calculate the loss once, and i represents the index of the sample, ranging from 1 to N.

6. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 4, characterized in that: In step S301, the first attention weight is expressed as: Among them, Attention(Q1,K i1 ,V i1 ) ultrasound is the first attention weight, Q1 is the query vector corresponding to the ultrasound modality, K i1 is the bond vector corresponding to the ultrasonic mode, d k1 is the dimension of the encoded vector corresponding to the ultrasonic modality, V i1 is the value vector corresponding to the ultrasonic mode, Q1 T Represents the transpose of the query vector corresponding to the ultrasound modality.

7. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 4, characterized in that: In step S301, the second attention weight is expressed as: Among them, Attention(Q2, K i2 , V i2 ) image is the second attention weight, Q2 is the query vector corresponding to the image modality, and K i2 is the key vector corresponding to the image modality, d k2 is the dimension of the encoded vector corresponding to the image modality, V i2 is the value vector corresponding to the image modality, Q2 T Represents the transpose of the query vector corresponding to the image modality.

8. A multi-modal detection method for loosening of bolt fasteners as claimed in claim 4, characterized in that: In step S302, the first attention weight is used as a query, and the second attention weight is used as a key and a value, and the cross attention method is used to perform feature fusion. The fusion process is expressed as: Q S34 =Attention(Q1,K i1 ,V i1 ) ultrasound K S34 =V S34 =Attention(Q2,K i2 ,V i2 ) image Among them, Q S34 is the attention output from ultrasound data, as the query vector; K S34 is the attention output from the RGB image data, as the key vector; V S34 is the attention output from the RGB image data as a value vector; Q1 is the query vector corresponding to the ultrasound modality, K i1 is the bond vector corresponding to the ultrasonic mode, V i1 is the value vector corresponding to the ultrasound modality; Q2 is the query vector corresponding to the image modality, K i2 is the key vector corresponding to the image modality, V i2 is the value vector corresponding to the image modality, d k is the dimension size of the key vector.

9. A multimodal detection system for implementing the multimodal detection method according to any one of claims 1 to 8, characterized in that: include: A sample collection module, used for collecting corresponding ultrasonic information along the axial direction of the bolt fastener and collecting corresponding RGB images along the radial direction of the bolt fastener under different tightening conditions of the bolt fastener, so as to obtain a plurality of groups of multimodal samples consisting of the RGB images and the ultrasonic information; A data preprocessing module, used for performing format adjustment and data enhancement on the multimodal samples to obtain multimodal enhanced data; A neural network training module, used for constructing a bolt loosening detection network, and training the bolt loosening detection network using the multimodal enhanced data; The bolt loosening detection module is used to input the RGB image and ultrasonic information of the bolt fastener to be tested into the trained bolt loosening detection network to evaluate the bolt loosening level of the bolt fastener to be tested.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

Images