Defect diagnosis method and system based on multi-modal data cooperative training

Abstract

The invention discloses a defect diagnosis method and system based on multi-modal data cooperative training, and belongs to the technical field of defect diagnosis, and the method specifically comprises the steps: constructing a multi-modal cooperative diagnosis model comprising a feature extraction sub-network and a cross-modal attention module; after multi-modal data is collected and preprocessed, initial features are obtained through the feature extraction sub-network, attention weights are generated through the cross-modal attention module, weighted multi-modal features are obtained, multi-scale fusion features are obtained through the multi-scale feature extraction sub-network, and a preliminary diagnosis result is given through cascade processing. Meanwhile, the data integrity is detected, and a modal missing scene is coped with through cascade collaborative diagnosis; and finally, comparing the two types of diagnosis results with a defect labeling sample to obtain a multi-modal collaborative diagnosis model after training optimization, inputting to-be-diagnosed sample data, outputting a final diagnosis result and updating the defect labeling sample, and realizing efficient and accurate defect diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect diagnosis, and specifically relates to a defect diagnosis method and system based on collaborative training of multimodal data. Background Art

[0002] In the fields of industrial production, equipment operation and maintenance, defect diagnosis is crucial to ensuring product quality and normal equipment operation. With the development of technology, defect diagnosis based on single modal data can no longer meet the diagnostic needs in complex scenarios, and defect diagnosis methods based on collaborative training of multimodal data have gradually become a research hotspot. However, existing defect diagnosis methods based on collaborative training of multimodal data have many shortcomings. In terms of feature extraction, it is difficult to effectively focus on the multimodal feature areas that are critical to defect diagnosis, and the ability to extract key information is insufficient; in the data processing flow, there is a lack of a reasonable cascade processing method, resulting in diagnostic efficiency and accuracy that needs to be improved; when some modal data is missing, the model cannot be adaptively adjusted, and the robustness is poor; for defects of different scales, feature extraction is not accurate enough, and it is difficult to take into account the diagnostic capabilities of defects of different sizes. Therefore, there is an urgent need for a defect diagnosis method based on collaborative training of multimodal data that can solve the above problems and has multiple innovative points. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention proposes a defect diagnosis method and system based on collaborative training of multimodal data, and constructs a multimodal collaborative diagnosis model including a feature extraction subnetwork and a cross-modal attention module; after collecting and preprocessing multimodal data, the initial features are obtained by the feature extraction subnetwork, and then the attention weights are generated by the cross-modal attention module to obtain weighted multimodal features. After passing through the multi-scale feature extraction subnetwork, multi-scale fusion features are obtained, and then preliminary diagnosis results are given through cascade processing; at the same time, data integrity is detected, and modality missing scenarios are dealt with through cascaded collaborative diagnosis; finally, the two types of diagnosis results are compared with the defect annotation samples to obtain the trained and optimized multimodal collaborative diagnosis model, and then after inputting the sample data to be diagnosed, the final diagnosis result is output and the defect annotation sample is updated, thereby achieving efficient and accurate defect diagnosis.

[0004] To achieve the above object, the present invention provides the following technical solutions: A defect diagnosis method based on multimodal data collaborative training, comprising: S1: Constructing a multimodal collaborative diagnosis model integrating attention mechanism; the multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. By calculating the similarity between modal features, attention weights are generated. The multimodal initial features are multiplied by the attention weights to obtain weighted multimodal features. S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnosis results of the defect type and severity after cascade processing; S4: Check the integrity of the pre-processed multimodal data and obtain the diagnosis results of the adaptive modality missing scenario through cascade collaborative diagnosis; S5: Compare the preliminary diagnosis results and the diagnosis results of the adaptation modality missing scenario with the defect annotation samples, obtain the trained and optimized multimodal collaborative diagnosis model based on the comparison results, input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as the new defect annotation sample.

[0005] Specifically, the multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; The feature extraction subnetwork includes at least three parallel modal feature extraction branches, each modal feature extraction branch corresponds to a modal data, and the modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch; The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction subnetwork, generates an inter-modal similarity matrix by calculating the cosine similarity between modal features, and the weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

[0006] Specifically, the calculation method of the feature correlation calculation layer is: for the initial features of the three modalities of vision, acoustics, and mechanics, the mutual information values ​​of vision and acoustics, vision and mechanics, and acoustics and mechanics are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

[0007] Specifically, the multi-scale feature extraction subnetwork contains three scale branches: small-scale branch, medium-scale branch and large-scale branch. The output features of each scale branch are integrated through feature splicing and input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through back propagation to automatically adjust the fusion weights of features at different scales. During cascade processing, the multi-scale and multi-modal fusion features are first input into the preliminary defect probability judgment layer. When the output probability value is less than the preset probability threshold, it is determined to be a non-defective sample and is eliminated. The remaining samples are input into the defect classification layer and severity assessment layer to obtain the preliminary diagnosis results. The defect probability preliminary judgment layer adopts a fully connected neural network to output the probability value of the sample having a defect; the defect classification layer adopts a classification neural network to output the defect type; the severity assessment layer adopts a regression neural network to output the defect severity.

[0008] Specifically, the output features of each scale branch are integrated through feature concatenation and input into the weighted fusion layer, including: The output feature maps of the small-scale branch, the medium-scale branch, and the large-scale branch are spliced ​​according to the channel dimension to obtain a spliced ​​feature map; The spliced ​​feature map is input into the weighted fusion layer, and the output is the fusion weight of features of different scales. The weighted fusion layer adopts a fully connected neural network, the hidden layer of the fully connected neural network adopts the ReLU activation function, and the output layer adopts the softmax activation function; The weight parameters of the weighted fusion layer are dynamically optimized through the back-propagation algorithm.

[0009] Specifically, the method of detecting the integrity of the preprocessed multimodal data includes: At the data acquisition end, a data integrity check code based on the SHA-256 algorithm is generated for each modal data, and the data integrity check code is transmitted to the processing end together with the multimodal data; At the processing end, the check code of the received multimodal data is recalculated and compared with the data integrity check code transmitted from the acquisition end. If the check codes are consistent, the multimodal data is considered complete; if they are inconsistent, the multimodal data is considered missing.

[0010] Specifically, the method of detecting the integrity of the preprocessed multimodal data further includes: When missing visual modality data is detected, the backup network branch A is activated, that is, the visual modality feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified cross-modal attention module only calculates the similarity between the acoustic and mechanical features. After generating the similarity matrix, the attention weights of the acoustic and mechanical modalities are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction subnetwork. When the acoustic modality data is missing, the backup network branch B is activated, that is, the acoustic modality feature extraction branch is deleted, and a direct mapping layer of visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map the visual features to the mechanical feature space, and then calculates the similarity between the mapped visual features and the mechanical features to generate a similarity matrix, and then generates the attention weights; When the mechanical modal data is detected to be missing, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted, and a fusion compensation layer of visual and acoustic features is added after the small-scale branch of the multi-scale feature extraction subnetwork. The fusion compensation layer adopts a convolutional neural network, and adds the visual small-scale features and the acoustic small-scale features through residual connections to obtain the fused features, which are then input into the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

[0011] Specifically, the cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When there is only a single modality data, the basic diagnostic model is activated, and the output result is weightedly corrected using the historical diagnostic accuracy of the single modality data; when there are two modal data, the dual-modality collaborative model is activated, and the diagnostic reliability is improved through inter-modal feature mutual verification; when there are three modal data, the full-modality fusion model is activated, and the diagnosis result with the highest confidence is output. The switching of the basic diagnostic model, the dual-modality collaborative model, and the full-modality fusion model is automatically triggered by the modality state identifier.

[0012] A defect diagnosis system based on multimodal data collaborative training, comprising: a model building module, a data processing module, a feature extraction module, a preliminary diagnosis module, a collaborative diagnosis module, and a diagnosis and update module; The model building module is used to build a multimodal collaborative diagnosis model including feature extraction and cross-modal attention mechanism; The data processing module is used to collect multimodal data and perform preprocessing; The feature extraction module is used to extract initial features from the preprocessed multimodal data using the feature extraction subnetwork, and generate weighted multimodal features through the cross-modal attention module; The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features to obtain multi-scale multi-modal fusion features and provide preliminary diagnosis results of the defect type and severity; The collaborative diagnosis module is used to detect the integrity of the pre-processed multimodal data and obtain a diagnosis result through cascade collaborative diagnosis in the event of modality loss; The diagnosis and update module is used to input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final diagnosis result as a new defect annotation sample.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention proposes a defect diagnosis method based on collaborative training of multimodal data. By constructing a multimodal collaborative diagnosis model that integrates the attention mechanism, the feature extraction subnetwork and the cross-modal attention module are effectively integrated. When processing multimodal data, the initial features are first obtained through the feature extraction subnetwork, and then the attention weights are generated by the cross-modal attention module. The initial features are weighted, which can accurately capture the correlation and importance between different modal features, thereby obtaining more representative weighted multimodal features, thereby improving the quality and effectiveness of feature extraction.

[0014] 2. The present invention proposes a defect diagnosis method based on collaborative training of multimodal data. A multi-scale feature extraction subnetwork is designed to further process the weighted features, and multi-scale multimodal fusion features can be obtained. Preliminary diagnosis results are given through cascade processing. At the same time, data integrity is detected and cascade collaborative diagnosis is adopted to deal with modality missing scenarios, thereby enhancing the adaptability and robustness of the model. Various diagnostic results are compared with defect annotation samples to optimize the model. The final output defect diagnosis results are accurate and reliable, and the results are used as new samples to update the annotation library, forming a virtuous circle, which continuously improves the performance and practicality of the diagnosis model and has significant advantages in the field of defect diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a defect diagnosis method based on multimodal data collaborative training according to the present invention; Figure 2 This is a principle flow chart of a defect diagnosis method based on multimodal data collaborative training of the present invention; Figure 3 This is an architecture diagram of a defect diagnosis system based on multimodal data collaborative training in the present invention. DETAILED DESCRIPTION

[0016] Example 1:

[0017] See also Figure 1 and Figure 2 The present invention provides an embodiment of a defect diagnosis method based on multimodal data collaborative training, comprising the following steps: S1: Constructing a multimodal collaborative diagnosis model integrating attention mechanism; the multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; Furthermore, before building a multimodal collaborative diagnosis model that integrates the attention mechanism, the sources of multimodal data are evaluated and screened. By collecting multimodal data samples from different production batches and under different working conditions, their distribution characteristics and quality are analyzed. Then, based on the evaluation results, high-quality data samples are selected as candidate data for the training set and test set to ensure that the model can learn comprehensive defect characteristics.

[0018] At the same time, when collecting multimodal data, high-precision clock synchronization equipment is used to synchronize the acquisition clocks of visual sensors, acoustic sensors, and mechanical sensors. During the data acquisition process, the acquisition timestamp of each modal data is recorded in real time for time alignment.

[0019] Furthermore, the specific steps of constructing the multimodal collaborative diagnosis model integrating the attention mechanism include: (1) Determine the overall framework of the multimodal collaborative diagnosis model and clarify the position and role of the feature extraction subnetwork and the cross-modal attention module in the model. The feature extraction subnetwork is responsible for extracting initial features from different modal data, and the cross-modal attention module is used to explore the correlation between different modal features to achieve more effective feature fusion. (2) For different modal data, corresponding feature extraction network structures are designed respectively. For visual modal data, a convolutional neural network is used to gradually extract texture, shape, edge and other features in the image through multiple layers of convolution layers, pooling layers and activation function layers. The convolution layer uses convolution kernels of different sizes to capture feature information of different scales; the pooling layer uses maximum pooling or average pooling operations to reduce feature dimensions, reduce the amount of calculation, and enhance the translation invariance of features; for acoustic modal data, a recurrent neural network is used to process time series data and extract time series features and spectral features in acoustic signals; for mechanical modal data, a fully connected neural network is used to normalize the raw data collected by the mechanical sensor and input it into the fully connected layer, and the mechanical features are extracted through multiple layers of nonlinear transformation. The convolutional neural network, the recurrent neural network and the fully connected neural network are the existing technical contents in this field and are not the inventive solutions of this application, so they will not be described in detail here; (3) Design the structure of the cross-modal attention module, including a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer uses the cosine similarity measurement method to calculate the similarity matrix between different modal features; the weight generation layer generates an attention weight matrix based on the similarity matrix through the softmax function, so that different modal features can be weighted and fused according to their relevance to the target task. Among them, cosine similarity and softmax function are existing technical contents in this field and are not the inventive solution of this application, so they will not be described in detail here; (4) A random initialization method is used to initialize all parameters of the multimodal collaborative diagnosis model. Random initialization uses Gaussian distribution to generate initial parameter values. Pre-trained model parameter initialization uses model parameters pre-trained on relevant data sets to initialize the corresponding layers of the current model to accelerate the convergence speed of the model and improve the performance of the model.

[0020] Furthermore, when preprocessing the collected multimodal data, histogram equalization is used for visual modal data to improve the quality and clarity of the image; for acoustic modal data, wavelet transform is used to convert the sound signal into the frequency domain, remove the noise frequency band, and then convert it back to the time domain; for mechanical modal data, filtering and normalization are performed to remove high-frequency noise and scale the data to a uniform numerical range. Among them, histogram equalization, wavelet transform, filtering and normalization are the existing technical contents in this field and are not the inventive solutions of this application, so they will not be elaborated here.

[0021] It should be noted that the multimodal collaborative diagnosis model constructed in this application includes a feature extraction subnetwork and a cross-modal attention module. This unique architectural design is the core foundation of this method. The feature extraction subnetwork is responsible for extracting initial features from different modal data and providing original information for subsequent analysis, while the cross-modal attention module is the innovative highlight of this model. It can break down the barriers between different modal data and enable the model to pay attention to the correlation and importance between different modal features. Traditional models often process different modal data in isolation and cannot fully utilize the intrinsic connections between them. The architectural design of this model can dig out this hidden information, thereby more accurately identifying defect features, which is an advantage that traditional methods cannot match.

[0022] On the other hand, by integrating the feature extraction subnetwork and the cross-modal attention module into a single model, effective fusion of multimodal data is achieved. Different modal data have different characteristics and advantages. For example, visual modal data can intuitively present the appearance of an object, acoustic modal data can reflect the internal structure of an object or the sound characteristics during operation, and mechanical modal data can reflect the response of an object under stress. This model combines these three modal data and comprehensively utilizes their advantages to more comprehensively describe the characteristics of defects and improve the ability to identify various complex defects. This multimodal fusion method is groundbreaking in the field of defect diagnosis and can provide richer and more accurate diagnostic information compared to single modal or simple combined modal methods.

[0023] S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. By calculating the similarity between modal features, attention weights are generated. The multimodal initial features are multiplied by the attention weights to obtain weighted multimodal features. It should be noted that the present invention collects data in three modes: visual, acoustic, and mechanical, and can obtain information about the object being inspected from multiple dimensions. In industrial production, defects may exhibit different characteristics in different modes. For example, a surface crack on a mechanical part may be clearly visible in the visual mode, but the internal structural damage may not be directly observed visually. The acoustic modal data may be able to capture abnormal sound signals generated by internal damage when the part is running, and the mechanical modal data may reflect the abnormal deformation or vibration of the part when it is under stress. By collecting data from these three modes at the same time, it is possible to fully cover various situations in which defects may occur, providing sufficient information support for accurate diagnosis. This multimodal data acquisition method is irreplaceable by the traditional single modal data acquisition method and can greatly improve the comprehensiveness and accuracy of defect diagnosis.

[0024] At the same time, preprocessing the collected multimodal data is a key step to ensure the accuracy of subsequent model training and diagnosis. Data of different modalities may have different formats, scales and noise levels. If directly input into the model, the model will not be able to effectively learn the data features, affecting the diagnosis results. The preprocessing steps of this application include data cleaning, denoising, normalization and other operations, which can remove noise and outliers in the data to make the data cleaner and more standardized. For example, for visual modal data, there may be problems such as image blur and uneven lighting. The image quality can be improved through image enhancement technology; for acoustic modal data, there may be background noise interference, and the noise can be removed through filtering algorithms; for mechanical modal data, there may be problems such as data drift. Normalization can make the data have a unified scale. The preprocessed data can better adapt to the requirements of the model, improve the training efficiency and diagnostic accuracy of the model, which is an important link to ensure the stable and reliable operation of the entire defect diagnosis system.

[0025] The preprocessed multimodal data is then fed into the feature extraction subnetwork to obtain the multimodal initial features. This step extracts representative feature information from the raw data. The feature extraction branches for different modalities are designed based on the data characteristics of their respective modalities. For example, the visual modality feature extraction branch may use a convolutional neural network to extract image features such as texture and shape; the acoustic modality feature extraction branch may use time-frequency analysis to convert the sound signal into a time-frequency spectrum and then extract features; the mechanical modality feature extraction branch uses signal processing technology to extract the time and frequency domain features of the mechanical signal. Through the feature extraction branch, the advantages of different modal data can be fully utilized to extract more effective features.

[0026] Finally, the multimodal initial features are input into the cross-modal attention module, and the attention weights are generated by calculating the similarity between the modal features. The multimodal initial features are multiplied by the attention weights to obtain the weighted multimodal features. This process has extremely important functionality. The generation of attention weights can enable the model to automatically pay attention to the important associations between different modal features, give higher weights to key features related to defects, and give lower weights to irrelevant or interfering features. For example, in some cases, a feature in the visual modality and a feature in the acoustic modality may jointly indicate the presence of a defect. Through the cross-modal attention module, the model can identify this association and increase the importance of these two features in subsequent diagnosis. This weighted processing method can highlight key features, suppress noise and irrelevant information, and make the model more focused on features related to defect diagnosis, thereby improving the accuracy and robustness of the diagnosis. Compared with the traditional method of simply splicing or averaging different modal features, the weighted processing of this application can dynamically adjust the importance of features according to the actual association between features, and has stronger adaptability and accuracy.

[0027] S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnosis results of the defect type and severity after cascade processing; S4: Check the integrity of the pre-processed multimodal data and obtain the diagnosis results of the adaptive modality missing scenario through cascade collaborative diagnosis; The cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When only single-modal data exists, the basic diagnostic model is activated, and the output result is weightedly corrected using the historical diagnostic accuracy of the single-modal data; when two-modal data exist, the dual-modal collaborative model is activated, and the diagnostic reliability is improved through inter-modal feature mutual verification; when three-modal data exist, the full-modal fusion model is activated, and the diagnostic result with the highest confidence is output. The switching of the basic diagnostic model, dual-modal collaborative model, and full-modal fusion model is automatically triggered by the modal state identifier.

[0028] It should be noted that the use of a cascaded collaborative diagnosis method to obtain diagnostic results for scenarios with modal loss is an important innovation of this application in dealing with complex actual situations. In actual applications, due to various reasons, some modal data may be missing. Traditional methods are often unable to perform effective diagnosis in such cases. However, the cascaded collaborative diagnosis of this application can activate the corresponding backup network branch according to different modal loss situations to achieve defect diagnosis.

[0029] S5: Compare the preliminary diagnosis results and the diagnosis results of the adaptation modality missing scenario with the defect annotation samples, obtain the trained and optimized multimodal collaborative diagnosis model based on the comparison results, input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as the new defect annotation sample.

[0030] Furthermore, the new defect annotation samples adopt a three-level review mechanism. First, they are screened by the confidence of the model output, then randomly sampled and reviewed by manual reviewers, and finally the samples that pass the review are added to the annotation sample library. At the same time, a timestamp and data source identifier are added to each new sample to build a dynamically updated sample library and enhance the model's continuous learning ability.

[0031] Furthermore, the preliminary diagnosis results and the diagnosis results of the adaptation modality missing scenario are compared with the defect annotation samples. Based on the comparison results, a trained and optimized multimodal collaborative diagnosis model is obtained, including: (1) Design a suitable loss function to measure the difference between the preliminary diagnosis results, the diagnosis results of the adaptive modality missing scenario, and the defect annotation samples. For the defect classification task, the cross entropy loss function is used; for the defect severity assessment task, the mean square error loss function is used. The classification loss and regression loss are combined according to certain weights to obtain the total loss function; (2) Based on the total loss function, the back propagation algorithm is used to calculate the gradients of all parameters in the model, and then the stochastic gradient descent method is used to update the parameters of the model in the opposite direction of the gradient, so that the loss function value is continuously reduced and the performance of the model is continuously improved. Among them, the back propagation algorithm and the stochastic gradient descent method are the existing technical contents in this field and are not the inventive solutions of this application, and are not described in detail here; (3) The multimodal dataset is divided into a training set, a validation set, and a test set. The training set is used to train the model, and the validation set is used to monitor and adjust the model performance during the training process, such as adjusting the learning rate and early stopping. When the performance of the model on the validation set reaches stability, the training is stopped to obtain the multimodal collaborative diagnosis model after training optimization.

[0032] It should be emphasized that based on the comparison results, the multimodal collaborative diagnosis model can automatically adjust parameters and optimize model performance through the backpropagation algorithm. For example, if it is found that the multimodal collaborative diagnosis model has large errors in defect diagnosis, the backpropagation algorithm will adjust the weights of the relevant neurons in the multimodal collaborative diagnosis model, so that the model pays more attention to the features related to these defect types, thereby improving the diagnostic accuracy of these defects. By continuously comparing new diagnostic results with labeled samples and optimizing the model, the multimodal collaborative diagnosis model can gradually learn more accurate feature representations and diagnostic rules, and continuously improve the accuracy and generalization of diagnosis. This model optimization method based on result comparison is a self-learning and self-improvement process that enables the multimodal collaborative diagnosis model to adapt to the defect diagnosis needs in different scenarios.

[0033] The multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; The feature extraction subnetwork includes at least three parallel modal feature extraction branches, each modal feature extraction branch corresponds to a modal data, and the modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch; Furthermore, each modal feature extraction branch adopts a convolutional neural network architecture to extract modal features through multi-layer convolution and pooling operations. Among them, the visual modal feature extraction branch targets image data and uses convolution kernels to capture the texture and edge features of the image; the acoustic modal feature extraction branch processes audio data and extracts the frequency and amplitude characteristics of the sound through operations such as spectrum analysis; the mechanical modal feature extraction branch targets mechanical sensor data and extracts the characteristics of the mechanical signal through operations such as filtering and normalization.

[0034] The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction subnetwork, generates an inter-modal similarity matrix by calculating the cosine similarity between modal features, and the weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

[0035] The calculation method of the feature correlation calculation layer is as follows: for the initial features of the three modalities of vision, acoustics, and mechanics, the mutual information values ​​of vision and acoustics, vision and mechanics, and acoustics and mechanics are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

[0036] The multi-scale feature extraction subnetwork includes three scale branches: small-scale branch, medium-scale branch and large-scale branch. The output features of each scale branch are integrated through feature splicing and input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through back propagation to automatically adjust the fusion weights of features at different scales. Among them, the small-scale branch uses a 3×3 convolution kernel for feature extraction, and outputs a feature map with a resolution of 1 / 4 of the original feature; the medium-scale branch uses a 5×5 convolution kernel, and outputs a feature map with a resolution of 1 / 2 of the original feature; the large-scale branch uses a 7×7 convolution kernel, and outputs a feature map with the same resolution as the original feature.

[0037] It should be noted that the design of a multi-scale feature extraction subnetwork consisting of three scale branches—small, medium, and large—is another innovation in this patent's feature extraction. Features at different scales can reflect information at different levels of a defect. Small-scale features can capture subtle features of a defect, such as tiny cracks and scratches; medium-scale features can reveal the local structure and morphology of a defect; and large-scale features can reflect the distribution of the defect throughout an object or region. By simultaneously extracting features at these three scales, a comprehensive and detailed description of defect characteristics can be achieved, enabling analysis from multiple perspectives, from the microscopic to the macroscopic. For example, when detecting surface defects on a metal part, small-scale features may reveal tiny pits on the part's surface, medium-scale features can identify the pit's shape and size, and large-scale features can determine the distribution and density of the pits on the part's surface. This multi-scale feature extraction approach provides richer and more comprehensive defect information than single-scale feature extraction methods, helping to more accurately determine the type and severity of the defect.

[0038] The output features of each scale branch are integrated through feature concatenation and then input into the weighted fusion layer. This step effectively fuses features at different scales. Feature concatenation combines features of different scales along the channel dimension to form a comprehensive feature map containing multi-scale information. However, features of different scales may have different importance in diagnosis, so a weighted fusion layer is required to automatically adjust the fusion weights of features at different scales. The weight parameters of the weighted fusion layer are dynamically optimized through backpropagation. This means that during model training, the weighted fusion layer automatically learns and adjusts the weights of features at different scales based on actual diagnostic results, directing the model to focus on scale features that have a greater impact on the diagnostic results. For example, in some cases, small-scale features may be more critical for accurately identifying defect types, and the weighted fusion layer automatically increases the weights of small-scale features. In other cases, large-scale features may be more important for determining defect severity, and the weighted fusion layer adjusts the weights accordingly. This dynamically optimized weighted fusion approach fully leverages the advantages of features at different scales, improving the effectiveness of feature fusion and further enhancing diagnostic accuracy.

[0039] During cascade processing, the multi-scale and multi-modal fusion features are first input into the defect probability preliminary judgment layer. When the output probability value is less than the preset probability threshold, it is judged as a non-defective sample and eliminated. The remaining samples are input into the defect classification layer and the severity assessment layer, and the defect type and severity are output respectively, thereby obtaining a preliminary diagnosis result. The defect probability preliminary judgment layer adopts a fully connected neural network to output the probability value of the sample having a defect; the defect classification layer adopts a classification neural network to output the defect type; the severity assessment layer adopts a regression neural network to output the defect severity.

[0040] It should be noted that the preliminary defect probability judgment layer uses a fully connected neural network to quickly make a preliminary judgment on whether a sample has defects. By setting a probability threshold, it effectively filters out defect-free samples, reducing the computational complexity of subsequent processing and improving the efficiency of the entire diagnostic system. For example, when testing a large number of samples, many are defect-free. Performing detailed defect classification and severity assessment on all samples would consume a significant amount of computing resources and time. However, the preliminary defect probability judgment layer can quickly eliminate defect-free samples, allowing further analysis only on samples that may have defects, thereby improving diagnostic efficiency.

[0041] The defect classification layer uses a classification neural network to accurately determine the defect type based on input features. Different defect types may require different treatment methods and repair strategies, so accurate defect classification is crucial for actual production. For example, in the production of electronic products, various types of defects may exist, such as short circuits, poorly soldered components, and broken circuits. The defect classification layer can accurately identify the defect type, providing a basis for subsequent repairs and quality improvements.

[0042] The severity assessment layer uses a regression neural network to output the severity of the defect. Defect severity directly affects product quality and service life. Accurately assessing defect severity helps develop appropriate maintenance plans and quality control standards. For example, a minor crack in a mechanical part may require simple repair, while a severe crack may require part replacement. The severity assessment layer can quantify the severity of the defect, providing a scientific basis for production decisions.

[0043] The output features of each scale branch are integrated through feature concatenation and input into the weighted fusion layer, including: A1: Concatenate the output feature maps of the small-scale branch, the medium-scale branch, and the large-scale branch according to the channel dimension to obtain the concatenated feature map; A2: The spliced ​​feature maps are input into a weighted fusion layer, and the output is the fusion weights of features at different scales. The weighted fusion layer uses a fully connected neural network, the hidden layer of the fully connected neural network uses a ReLU activation function, and the output layer uses a softmax activation function. The ReLU activation function is a prior art in this field and is not an inventive solution of the present application, so it is not described in detail here. A3: Dynamically optimize the weight parameters of the weighted fusion layer through the back-propagation algorithm.

[0044] The method of detecting the integrity of the preprocessed multimodal data includes: B1: At the data acquisition end, a data integrity check code based on the SHA-256 algorithm is generated for each modal data, and the data integrity check code is transmitted to the processing end together with the multimodal data; Furthermore, the process of generating a data integrity check code based on the SHA-256 algorithm for each modal data includes: (1) Divide the original data of each modality into blocks of fixed size to facilitate subsequent hash calculation; (2) The hash value of the SHA-256 algorithm is initialized to a fixed initial value. The SHA-256 algorithm is a prior art in this field and is not an inventive solution of the present application, and is not described in detail here. (3) Perform bitwise and logical operations on each data block to update the hash value. The specific operation process includes performing XOR, shift, modular addition and other operations on the data block and the current hash value to obtain a new hash value; (4) After processing all data blocks, the hash value obtained is the data integrity check code of the modal data, and the data integrity check code is transmitted to the processing end together with the multimodal data.

[0045] B2: On the processing side, the checksum is recalculated for the received multimodal data and compared with the data integrity checksum transmitted from the acquisition side. If the checksums are consistent, the multimodal data is considered complete; if they are inconsistent, the multimodal data is considered missing. Furthermore, the specific process of B2 includes: The processing end receives the multimodal data and the data integrity check code; Recalculate the data integrity check code for the received multimodal data using the same SHA-256 algorithm as the acquisition end; The recalculated data integrity check code is compared with the data integrity check code transmitted from the acquisition end. If the check codes are consistent, the multimodal data is considered complete; if they are inconsistent, the multimodal data is considered missing.

[0046] B3: When it is detected that the visual modality data is missing, the backup network branch A is activated, that is, the visual modality feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified version of the cross-modal attention module only calculates the similarity between the acoustic and mechanical features. After generating the similarity matrix, the attention weights of the acoustic and mechanical modalities are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction sub-network. This processing method can make full use of the remaining acoustic and mechanical modal data when the visual modality data is missing, extract the correlation features between them through the cross-modal attention mechanism, and perform subsequent diagnosis. Although the visual modality data is missing, by reasonably utilizing the remaining modality data, relatively accurate diagnosis results can still be obtained, avoiding the problem of diagnosis being unable to be carried out due to the lack of single modality data; Furthermore, the specific steps of B3 include: (1) In the multimodal collaborative diagnosis model, find the visual modal feature extraction branch and delete it from the model. At this point, the model only has the acoustic and mechanical modal feature extraction branches; (2) Design a simplified version of the cross-modal attention module, whose feature correlation calculation layer only calculates the similarity between acoustic and mechanical features, specifically including: Normalize the initial eigenvectors of the acoustic and mechanical modes to make their length 1; The cosine similarity is used to calculate the similarity between the acoustic and mechanical eigenvectors to obtain the similarity matrix; According to the similarity matrix, the softmax function is used to generate the attention weight matrix of the acoustic mode and the mechanical mode; (3) Multiply the initial eigenvectors of the acoustic and mechanical modes by the corresponding attention weights to obtain the weighted eigenvectors of the acoustic and mechanical modes, and directly input the weighted eigenvectors into the multi-scale feature extraction subnetwork.

[0047] B4: When it is detected that the acoustic modal data is missing, the backup network branch B is activated, that is, the acoustic modal feature extraction branch is deleted, and a direct mapping layer of visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map the visual features to the mechanical feature space, and then calculates the similarity between the mapped visual features and the mechanical features to generate a similarity matrix, and then generates attention weights. This method can achieve cross-modal attention calculation by establishing a mapping relationship between visual and mechanical features when the acoustic modal data is missing, thereby using visual and mechanical modal data for diagnosis, making full use of the potential connection between different modal data, and compensating for the impact of the missing acoustic modal data through innovative mapping methods, ensuring the continuity and accuracy of diagnosis; Furthermore, the specific steps of B4 include: In the multimodal collaborative diagnosis model, the acoustic modal feature extraction branch is found and deleted from the model. At this point, the model only has the visual and mechanical modal feature extraction branches. A direct mapping layer between visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network and specifically includes: Design a fully connected neural network structure, including an input layer, a hidden layer, and an output layer. The input layer receives the initial feature vector of the visual modality, the hidden layer uses the ReLU activation function to introduce nonlinear factors, and the number of neurons in the output layer is the same as the characteristic dimension of the mechanical modality. A fully connected neural network is trained using training samples containing both visual and mechanical modal data. During training, backpropagation and stochastic gradient descent are used to update the network's weight parameters, enabling the network to map visual features to the mechanical feature space. Input the initial feature vector of the visual modality into the trained fully connected neural network to obtain the mapped visual feature vector; (3) Calculate the similarity between the mapped visual feature vector and the mechanical feature vector to generate a similarity matrix, and then use the softmax function to generate an attention weight matrix. The weighted features are input into the multi-scale feature extraction sub-network to complete the defect diagnosis.

[0048] B5: When the mechanical modal data is detected to be missing, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted, and a fusion compensation layer of visual and acoustic features is added after the small-scale branch of the multi-scale feature extraction subnetwork. The fusion compensation layer adopts a convolutional neural network, and adds the visual small-scale features and the acoustic small-scale features through residual connection to obtain the fused features, which are then input into the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

[0049] It should be explained that in the case of missing mechanical modal data, the small-scale features of visual and acoustic modalities are fused through the fusion compensation layer to extract richer feature information and make up for the lack of mechanical modal data. The use of residual connections can effectively avoid the gradient vanishing problem and ensure the effect of feature fusion, thereby improving the accuracy of diagnosis.

[0050] Furthermore, the fusion compensation layer includes two 3×3 convolutional sublayers and one residual connection. The first convolutional sublayer increases the visual small-scale features and acoustic small-scale features from 256 channels to 512 channels respectively, and the second convolutional sublayer reduces the 512-channel features to 256 channels. The residual connection adds the input visual and acoustic small-scale features to the output features of the second convolutional sublayer, and uses the ReLU activation function to enhance the nonlinear expression ability of the features to compensate for the feature loss caused by the lack of mechanical modes.

[0051] Furthermore, the specific steps of B5 include: In the multimodal collaborative diagnosis model, the mechanical modal feature extraction branch is found and deleted from the model. At this point, the model only has the visual and acoustic modal feature extraction branches. A fusion compensation layer for visual and acoustic features is added after the small-scale branch of the multi-scale feature extraction subnetwork. This fusion compensation layer uses a convolutional neural network and specifically includes: Design the structure of the convolutional neural network, including convolutional layers, pooling layers, and activation function layers. The convolutional layer uses a 3×3 convolution kernel to extract local information of small-scale visual and acoustic features; the pooling layer uses maximum pooling or average pooling operations to reduce feature dimensions; and the activation function layer uses the ReLU activation function to introduce nonlinear factors. The visual small-scale features and the acoustic small-scale features are added through residual connections. The specific implementation method is to splice the visual small-scale features and the acoustic small-scale features in the channel dimension, and then adjust the number of channels through a 1×1 convolution layer to make it the same as the number of channels of the input features. Finally, the adjusted features are added to the input features to obtain the fused features.

[0052] (3) The fused features are input into the multi-scale feature extraction branch and weighted fusion layer to complete subsequent feature extraction and defect diagnosis.

[0053] Example 2: See also Figure 3 Another embodiment of the present invention provides a defect diagnosis system based on multimodal data collaborative training, comprising: Model building module, data processing module, feature extraction module, preliminary diagnosis module, collaborative diagnosis module, diagnosis and update module; Model building module, used to build a multimodal collaborative diagnosis model including feature extraction and cross-modal attention mechanism; The data processing module is used to collect multimodal data and perform necessary preprocessing to provide clean and standardized data for subsequent feature extraction and diagnosis; The feature extraction module is used to extract initial features from the preprocessed multimodal data using the feature extraction subnetwork and generate weighted multimodal features through the cross-modal attention module; The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and based on this, provide preliminary diagnosis results of the defect type and severity; Collaborative diagnosis module, used to detect the integrity of pre-processed multimodal data and obtain diagnostic results through cascaded collaborative diagnosis in the event of modality loss; The diagnosis and update module is used to input the multimodal data of the samples to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis results, and use the final diagnosis results as new defect annotation samples for subsequent training.

[0054] The model building modules include: network design unit and cross-modal design unit; Network design unit, which designs a sub-network structure for extracting initial features from multimodal data; Cross-modal design unit, a module for calculating the similarity between modal features and generating attention weights to achieve interaction and enhancement between features.

[0055] The feature extraction module includes: an execution unit, a weight calculation unit, and a feature weighting unit; The execution unit is used to run the feature extraction sub-network to extract initial features from multimodal data; The weight calculation unit is used to calculate the similarity between modal features in the cross-modal attention module and generate attention weights; The feature weighting unit is used to multiply the multimodal initial features with the corresponding attention weights to obtain weighted multimodal features.

[0056] The preliminary diagnosis module includes: sub-network design unit, feature extraction unit, and preliminary diagnosis unit; Sub-network design unit, used to extract the network structure of multi-scale features; The feature extraction unit is used to run the multi-scale feature extraction sub-network to extract the weighted multimodal features; The preliminary diagnosis unit is used to fuse multi-scale features and provide preliminary diagnosis results through classification or regression models.

[0057] The diagnosis and update module includes: a diagnosis execution unit and a sample update unit; The diagnosis execution unit is used to run the trained and optimized multimodal collaborative diagnosis model to diagnose the samples to be diagnosed; The sample update unit is used to use the final diagnosis results as new defect annotation samples, update the sample library, and use it for subsequent model training and optimization.

[0058] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention. These are all protected by the present invention.

Claims

1. A defect diagnosis method based on multimodal data collaborative training, characterized in that: include: S1: Constructing a multimodal collaborative diagnosis model integrating attention mechanism; the multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. By calculating the similarity between modal features, attention weights are generated. The multimodal initial features are multiplied by the attention weights to obtain weighted multimodal features. S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnosis results of the defect type and severity after cascade processing; S4: Check the integrity of the pre-processed multimodal data and obtain the diagnosis results of the adaptive modality missing scenario through cascade collaborative diagnosis; S5: Compare the preliminary diagnosis results and the diagnosis results of the adaptation modality missing scenario with the defect annotation samples, obtain the trained and optimized multimodal collaborative diagnosis model based on the comparison results, input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as the new defect annotation sample.

2. A defect diagnosis method based on multimodal data collaborative training according to claim 1, characterized in that: The multimodal collaborative diagnosis model includes a feature extraction subnetwork and a cross-modal attention module; The feature extraction subnetwork includes at least three parallel modal feature extraction branches, each modal feature extraction branch corresponds to a modal data, and the modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch; The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction subnetwork, generates an inter-modal similarity matrix by calculating the cosine similarity between modal features, and the weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

3. The defect diagnosis method based on multimodal data collaborative training according to claim 2, characterized in that: The calculation method of the feature correlation calculation layer is as follows: for the initial features of the three modalities of vision, acoustics, and mechanics, the mutual information values ​​of vision and acoustics, vision and mechanics, and acoustics and mechanics are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

4. A defect diagnosis method based on multimodal data collaborative training according to claim 3, characterized in that: The multi-scale feature extraction subnetwork includes three scale branches: small-scale branch, medium-scale branch and large-scale branch. The output features of each scale branch are integrated through feature splicing and input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through back propagation to automatically adjust the fusion weights of features at different scales. During cascade processing, the multi-scale and multi-modal fusion features are first input into the preliminary defect probability judgment layer. When the output probability value is less than the preset probability threshold, it is determined to be a non-defective sample and is eliminated. The remaining samples are input into the defect classification layer and severity assessment layer to obtain the preliminary diagnosis results. The defect probability preliminary judgment layer adopts a fully connected neural network to output the probability value of the sample having a defect; the defect classification layer adopts a classification neural network to output the defect type; the severity assessment layer adopts a regression neural network to output the defect severity.

5. The defect diagnosis method based on multimodal data collaborative training according to claim 4, characterized in that: The output features of each scale branch are integrated through feature concatenation and input into the weighted fusion layer, including: The output feature maps of the small-scale branch, the medium-scale branch, and the large-scale branch are spliced ​​according to the channel dimension to obtain a spliced ​​feature map; The spliced ​​feature map is input into the weighted fusion layer, and the output is the fusion weight of features of different scales. The weighted fusion layer adopts a fully connected neural network, the hidden layer of the fully connected neural network adopts the ReLU activation function, and the output layer adopts the softmax activation function; The weight parameters of the weighted fusion layer are dynamically optimized through the back-propagation algorithm.

6. The defect diagnosis method based on multimodal data collaborative training according to claim 5, characterized in that: The method of detecting the integrity of the preprocessed multimodal data includes: At the data acquisition end, a data integrity check code based on the SHA-256 algorithm is generated for each modal data, and the data integrity check code is transmitted to the processing end together with the multimodal data; At the processing end, the check code of the received multimodal data is recalculated and compared with the data integrity check code transmitted from the acquisition end. If the check codes are consistent, the multimodal data is considered complete; if they are inconsistent, the multimodal data is considered missing.

7. The defect diagnosis method based on multimodal data collaborative training according to claim 6, characterized in that: The method of detecting the integrity of the pre-processed multimodal data further includes: When missing visual modality data is detected, the backup network branch A is activated, that is, the visual modality feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified cross-modal attention module only calculates the similarity between the acoustic and mechanical features. After generating the similarity matrix, the attention weights of the acoustic and mechanical modalities are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction subnetwork. When the acoustic modality data is missing, the backup network branch B is activated, that is, the acoustic modality feature extraction branch is deleted, and a direct mapping layer of visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map the visual features to the mechanical feature space, and then calculates the similarity between the mapped visual features and the mechanical features to generate a similarity matrix, and then generates the attention weights; When the mechanical modal data is detected to be missing, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted, and a fusion compensation layer of visual and acoustic features is added after the small-scale branch of the multi-scale feature extraction subnetwork. The fusion compensation layer adopts a convolutional neural network, and adds the visual small-scale features and the acoustic small-scale features through residual connections to obtain the fused features, which are then input into the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

8. The defect diagnosis method based on multimodal data collaborative training according to claim 7, characterized in that: The cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When only single-modal data exists, the basic diagnostic model is activated, and the output result is weightedly corrected using the historical diagnostic accuracy of the single-modal data; when two-modal data exist, the dual-modal collaborative model is activated; when three-modal data exist, the full-modal fusion model is activated, and the diagnosis result with the highest confidence is output. The switching of the basic diagnostic model, dual-modal collaborative model, and full-modal fusion model is automatically triggered by the modal state identifier.

9. A defect diagnosis system based on multimodal data collaborative training, which is used to implement a defect diagnosis method based on multimodal data collaborative training according to any one of claims 1 to 8, characterized in that: include: Model building module, data processing module, feature extraction module, preliminary diagnosis module, collaborative diagnosis module, diagnosis and update module; The model building module is used to build a multimodal collaborative diagnosis model including feature extraction and cross-modal attention mechanism; The data processing module is used to collect multimodal data and perform preprocessing; The feature extraction module is used to extract initial features from the preprocessed multimodal data using the feature extraction subnetwork, and generate weighted multimodal features through the cross-modal attention module; The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features to obtain multi-scale multi-modal fusion features and provide preliminary diagnosis results of the defect type and severity; The collaborative diagnosis module is used to detect the integrity of the pre-processed multimodal data and obtain a diagnosis result through cascade collaborative diagnosis in the event of modality loss; The diagnosis and update module is used to input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final diagnosis result as a new defect annotation sample.

Images