A multimodal fusion mechanical defect detection method based on attention graph convolutional network

Abstract

The present invention discloses a multimodal fusion mechanical defect detection method based on an attention graph convolutional network. First, a camera is used to collect internal videos of machinery working under different defect conditions, and each section of the internal working video of the machinery is decomposed into a group of pictures, each of which corresponds to a corresponding short audio segment; an audio feature extraction encoder, an image feature extraction network, an attention graph convolutional network, and a fully connected network are connected to form a mechanical defect detection model based on multimodal fusion deep learning; the audio and images in each group of mechanical inspection data collected are used as input, and the mechanical defect status is used as output; the established classification model is trained using the collected mechanical inspection data; and finally, the defect status of the machinery is detected using the trained classification model. The present invention utilizes the complementary information of the two modes of hearing and vision, which can improve the accuracy and robustness of mechanical defect detection to a certain extent.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical defect detection, and in particular to a multimodal fusion mechanical defect detection method based on an attention graph convolutional network. Background Art

[0002] Mechanical defect detection is an essential part of the mass production process, which directly determines the value of various products produced and the safety factor of the factory. Mechanical defects refer to errors in the working process of the machine due to some factors (shaft breakage, bearing wear, bearing jamming, screw loosening, belt loosening, component damage, etc.). Such mechanical defects may lead to a decline in the quality of the final product, or even pose a safety hazard, which may cause a series of safety accidents at any time.

[0003] At present, most domestic production workshops still use human eye recognition to detect mechanical defects. However, this method requires professional inspectors who have undergone extensive training and have certain practical operation experience. In addition, in the actual inspection process, there are still problems such as low efficiency and inconsistent inspection standards, which will greatly reduce the efficiency of mechanical defect detection, and make the quality of mechanically produced products uneven, and even pose safety hazards.

[0004] In order to overcome a series of problems exposed by manual defect detection, there are currently methods that use deep learning to identify mechanical defect detection. However, most of these methods are based on images of the machine at work. In some cases, the internal structure of the machine will not show obvious faults, but it will make different noises when it is working. Deep learning methods based on image feature extraction alone cannot effectively classify such defects correctly. At the same time, since some detections need to identify small structural defects, image-based deep learning methods have high requirements for image clarity, and it is difficult to provide such accurate and high-quality images in complex industrial production environments. Therefore, introducing information of different modes (such as audio) to fill in the defect features that cannot be captured by the image and further improve the classification accuracy of the defect detection model is a feasible solution. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a multimodal fusion mechanical defect detection method based on attention graph convolutional network. A camera is used to collect the internal audio and image information of the machine when it is working; a multimodal fusion method based on attention graph convolutional network is used to combine the auditory and visual information, and the complementary modal information can greatly improve the accuracy and robustness of mechanical defect detection.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] A multimodal fusion mechanical defect detection method based on an attention graph convolutional network comprises the following steps:

[0008] Step 1: Create different types of mechanical defect detection datasets

[0009] Use a camera to collect internal videos of the machine working under different defect conditions, and collect audio information and image information of the machine under various defect conditions; the defect conditions are broken shafts, worn bearings, stuck bearings, loose screws, loose belts, and damaged parts; a video is used as a group of machine inspection data, and the defect condition of the machine is used as the label of each video, that is, each group of defective machine inspection data contains a video of the machine working and a defect condition label; all the collected groups of machine inspection data constitute a machine defect detection data set;

[0010] Step 2: Establish a multimodal fusion mechanical defect detection classification model based on attention graph convolutional network

[0011] First, each video of the internal working of the machine is decomposed into a group of pictures, and each picture corresponds to a corresponding short audio segment; the audio feature extraction encoder, image feature extraction network, attention graph convolution network and fully connected network are connected to form a mechanical defect detection model based on multimodal fusion deep learning; the audio feature extraction encoder uses the collaborative speech analysis repository COVAREP, and the image feature extraction network uses the ResNet-101 network, which respectively use the sound emitted by the machine during operation and the internal image of the machine in each group of collected mechanical inspection data as input to extract the features of the audio and internal image of the machine during operation; a T*N audio feature matrix and a T*N image feature matrix are obtained; T is each group of input The number of pictures and corresponding audios, N is the dimension; perform attention operation on the audio feature matrix and the image feature matrix to obtain a T*T attention weight matrix; input the attention weight matrix as a fully connected graph and the image feature matrix into a densely connected graph convolutional network to perform multimodal feature fusion; after fusion, perform a maximum pooling operation on the feature representation matrix output by the graph convolutional network to obtain a feature representation vector with a dimension of N; input the feature representation vector into an N*M fully connected network, and output a feature vector that represents the mechanical defect condition. The dimension of the feature vector is equal to the number of defect condition label types M contained in a set of input mechanical inspection data, and the output of each dimension of the feature vector represents the confidence of each type of mechanical defect condition;

[0012] Step 3: Train the Mechanical Defect Detection Model

[0013] The mechanical defect detection video data set obtained in step 1 is divided into a training set and a test set in a ratio of 7:3; the videos belonging to the same group of mechanical detection data in the training set are respectively input into the multimodal fusion classification model based on the attention graph convolutional network established in step 2, and then the weight values ​​in the mechanical defect detection model are trained by the back propagation algorithm; the test set is used to judge the training effect of the mechanical defect detection model to obtain a fully trained mechanical defect detection model;

[0014] Step 4: Collect the working data of the machine to be tested and input it into the trained model

[0015] Collect the internal working video of the machine to be inspected and input it into the trained mechanical defect detection model to detect the defect condition of the machine during operation. The defect condition label with the highest confidence in the feature vector representing the mechanical defect condition output by the mechanical defect detection model is the defect condition.

[0016] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0017] The present invention proposes a multimodal fusion mechanical defect detection method based on attention graph convolutional network. A camera is used to collect the internal audio and image information of the machine when it is working; a multimodal fusion method based on attention graph convolutional network is used to combine the auditory and visual information, and the complementary modal information can greatly improve the accuracy and robustness of mechanical defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention;

[0019] FIG. 2 is a schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described below with reference to specific examples and drawings.

[0021] like Figure 1 As shown, the multimodal fusion mechanical defect detection method based on the attention graph convolutional network of the present invention comprises the following steps:

[0022] Step 1: Create different types of mechanical defect detection datasets

[0023] Use a camera to collect internal videos of the machine working under different defect conditions, and collect audio and image information of the machine under various defect conditions. The defect conditions of the machine are generally broken shafts, worn bearings, stuck bearings, loose screws, loose belts, damaged parts, etc.; a video is used as a set of machine inspection data, and the defect condition of the machine is used as the label of each video, that is, each set of defective machine inspection data contains a video of the machine working and a defect condition label; all the collected sets of machine inspection data constitute a machine defect detection data set;

[0024] Step 2: Establish a multimodal fusion mechanical defect detection classification model based on attention graph convolutional network

[0025] First, each video of the internal working of the machine is decomposed into a group of pictures, and each picture corresponds to a corresponding short audio. The audio feature extraction encoder, the image feature extraction network, and the attention graph convolution network are connected to form a mechanical defect detection model based on multimodal fusion deep learning; the audio feature extraction encoder uses the collaborative speech analysis repository COVAREP, and the image feature extraction network uses the ResNet-101 network, which takes the sound emitted by the machine during operation and the internal image of the machine in each group of mechanical inspection data collected as input, respectively, to extract the features of the audio and internal image of the machine during operation. A T*N audio feature matrix and a T*N image feature matrix are obtained. The audio feature matrix and the image feature matrix are subjected to attention operation to obtain a T*T attention weight matrix. The attention weight matrix is ​​used as a fully connected graph, and the image feature matrix is ​​input into the densely connected graph convolution network for multimodal feature fusion. After fusion, the feature representation matrix is ​​subjected to maximum pooling operation to obtain a feature representation vector. The feature representation vector is input into a fully connected network, and the output is a feature vector representing the mechanical defect condition. The dimension of the feature vector is equal to the number of defect condition label types contained in a set of input mechanical inspection data. The output of each dimension of the feature vector represents the probability of the machine being in each type of defect condition.

[0026] Step 3: Train the Mechanical Defect Detection Model

[0027] The mechanical inspection video data set obtained in step 1 is divided into a training set and a test set in a ratio of 7:3; the videos belonging to the same group of mechanical inspection data in the training set are respectively input into the multimodal fusion classification model based on the attention graph convolutional network established in step 2, and then the weight values ​​in the mechanical defect detection model are trained by the back propagation algorithm; the test set is used to judge the training effect of the mechanical defect detection model to obtain a fully trained mechanical defect detection model;

[0028] Step 4: Collect the working data of the machine to be tested and input it into the trained model

[0029] Collect the internal working video of the machine to be inspected and input it into the trained mechanical defect detection model to detect the defect condition of the machine during operation. The defect condition is the defect condition label with the highest confidence in the feature vector representing the mechanical defect condition output by the mechanical defect detection model. Example

[0030] See also Figure 2 In this embodiment, a camera is used to collect internal video of the machine working under defective conditions, and a video is preprocessed to decompose it into a group of pictures of the internal working of the machine and a group of internal audio signals of the machine working. One picture corresponds to a piece of audio. Then each image and its corresponding audio signal are respectively input into the audio feature encoder and the ResNet-101 network to obtain a T*N audio feature matrix and a T*N image feature matrix. T represents the number of each group of input pictures and corresponding audios, and N represents the dimension. The audio feature matrix and the image feature matrix are subjected to attention operation to obtain a T*T attention weight matrix. The attention weight matrix is ​​used as a fully connected graph, and the image feature matrix is ​​input into a densely connected graph convolutional network to perform multimodal feature fusion. After fusion, the feature representation matrix output by the graph convolutional network is subjected to a maximum pooling operation to obtain a feature representation vector with a dimension of N; the feature representation vector is input into a fully connected network, and the output is the defect detection classification result.

[0031] The above description is only a preferred embodiment of the present invention, and certain modifications may be made thereto within the scope defined by the claims of the present invention, but all modifications will fall within the protection scope of the present invention.

Claims

1. A multimodal fusion mechanical defect detection method based on attention graph convolutional network, characterized in that: The method comprises the following specific steps: (1) Establish different types of mechanical defect detection datasets; (2) Establish a multimodal fusion mechanical defect detection and classification model based on attention graph convolutional network; (3) Training a mechanical defect detection model; (4) Collect the working data of the machine to be tested, input the trained model, and detect the defect status of the machine during operation; where: The step (2) is specifically: First, each video of the internal working of the machine is decomposed into a group of pictures, and each picture corresponds to a corresponding short audio segment; the audio feature extraction encoder, image feature extraction network, attention graph convolution network and fully connected network are connected to form a mechanical defect detection model based on multimodal fusion deep learning; the audio feature extraction encoder uses the collaborative speech analysis repository COVAREP, and the image feature extraction network uses the ResNet-101 network, which respectively use the sound emitted by the machine during operation and the internal image of the machine in each group of collected mechanical inspection data as input to extract the features of the audio and internal image of the machine during operation; a T*N audio feature matrix and a T*N image feature matrix are obtained; T is each group of input The number of pictures and corresponding audios, N is the dimension; perform attention operation on the audio feature matrix and the image feature matrix to obtain a T*T attention weight matrix; input the attention weight matrix as a fully connected graph and the image feature matrix into a densely connected graph convolutional network to perform multimodal feature fusion; after fusion, perform a maximum pooling operation on the feature representation matrix output by the graph convolutional network to obtain a feature representation vector with a dimension of N; input the feature representation vector into an N*M fully connected network, and output a feature vector that represents the mechanical defect condition. The dimension of the feature vector is equal to the number of defect condition label types M contained in a set of input mechanical inspection data, and the output of each dimension of the feature vector represents the confidence of each type of mechanical defect condition; The step (3) is specifically: The mechanical defect detection video dataset obtained in step 1 is divided into a training set and a test set in a ratio of 7:3; the videos belonging to the same group of mechanical detection data in the training set are respectively input into the multimodal fusion classification model based on the attention graph convolutional network established in step 2, and then the weight values ​​in the mechanical defect detection model are trained through the back propagation algorithm; the test set is used to judge the training effect of the mechanical defect detection model to obtain a completely trained mechanical defect detection model.

2. According to claim 1, a multimodal fusion mechanical defect detection method based on attention graph convolutional network is characterized in that: The step (1) is specifically: A camera is used to collect internal videos of machinery working under different defect conditions, and to collect mechanical audio information and image information of various defect conditions; the defect conditions are broken shafts, worn bearings, stuck bearings, loose screws, loose belts, and damaged parts; a video is used as a group of machinery inspection data, and the defect condition of the machinery is used as the label of each video, that is, each group of defective machinery inspection data contains a video of the machinery working and a defect condition label; all the collected groups of machinery inspection data constitute a machinery defect detection data set.

3. According to claim 1, a multimodal fusion mechanical defect detection method based on attention graph convolutional network is characterized in that: The step (4) is specifically as follows: Collect the internal working video of the machine to be inspected and input it into the trained mechanical defect detection model to detect the defect condition of the machine when it is working. The defect condition label with the highest confidence in the feature vector representing the mechanical defect condition output by the mechanical defect detection model is the defect condition.

Images