A defect diagnosis method and system based on multimodal data collaborative training

Abstract

This invention discloses a defect diagnosis method and system based on multimodal data collaborative training, belonging to the field of defect diagnosis technology. Specifically, it includes: constructing a multimodal collaborative diagnosis model comprising a feature extraction sub-network and a cross-modal attention module; collecting and preprocessing multimodal data, obtaining initial features through the feature extraction sub-network, generating attention weights through the cross-modal attention module to obtain weighted multimodal features, obtaining multi-scale fusion features through a multi-scale feature extraction sub-network, and then providing preliminary diagnostic results through cascaded processing; simultaneously detecting data integrity and addressing modality missing scenarios through cascaded collaborative diagnosis; finally, comparing the two types of diagnostic results with defect-labeled samples to obtain a trained and optimized multimodal collaborative diagnosis model, inputting the data of the sample to be diagnosed, outputting the final diagnostic results, and updating the defect-labeled samples, thus achieving efficient and accurate defect diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of defect diagnosis technology, specifically a defect diagnosis method and system based on multimodal data collaborative training. Background Technology

[0002] In industrial production and equipment maintenance, defect diagnosis is crucial for ensuring product quality and normal equipment operation. With technological advancements, single-modal data-based defect diagnosis is insufficient for complex scenarios, making multimodal data-based collaborative training defect diagnosis methods a research hotspot. However, existing multimodal data-based collaborative training defect diagnosis methods have several shortcomings. In feature extraction, they struggle to effectively focus on key multimodal feature regions crucial for defect diagnosis, resulting in insufficient extraction of key information. In data processing, they lack reasonable cascading processing methods, leading to room for improvement in diagnostic efficiency and accuracy. When some modal data is missing, the model cannot adaptively adjust, exhibiting poor robustness. Furthermore, feature extraction is not precise enough for defects of different scales, making it difficult to simultaneously address defects of varying sizes. Therefore, a defect diagnosis method based on multimodal data collaborative training that can solve these problems and possesses several innovative features is urgently needed. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a defect diagnosis method and system based on multimodal data collaborative training. It constructs a multimodal collaborative diagnosis model comprising a feature extraction sub-network and a cross-modal attention module. After collecting and preprocessing multimodal data, initial features are obtained through the feature extraction sub-network, and attention weights are generated by the cross-modal attention module to obtain weighted multimodal features. These features are then processed by a multi-scale feature extraction sub-network to obtain multi-scale fusion features, which are then cascaded to provide preliminary diagnostic results. Simultaneously, data integrity is checked, and cascaded collaborative diagnosis addresses scenarios with missing modalities. Finally, the two types of diagnostic results are compared with defect-labeled samples to obtain a trained and optimized multimodal collaborative diagnosis model. After inputting the data of the sample to be diagnosed, the final diagnostic results are output, and the defect-labeled samples are updated, achieving efficient and accurate defect diagnosis.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A defect diagnosis method based on multimodal data collaborative training includes:

[0006] S1: Construct a multimodal collaborative diagnostic model that integrates an attention mechanism; the multimodal collaborative diagnostic model includes a feature extraction sub-network and a cross-modal attention module;

[0007] S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. Generate attention weights by calculating the similarity between modal features, and multiply the multimodal initial features with the attention weights to obtain weighted multimodal features.

[0008] S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnostic results of the type and severity of defects after cascade processing;

[0009] S4: Detect the integrity of the preprocessed multimodal data and obtain diagnostic results for scenarios with missing modalities through cascaded collaborative diagnosis;

[0010] S5: Compare the preliminary diagnosis results and the diagnosis results of the adapted modality missing scenario with the defect annotation samples. Based on the comparison results, obtain the trained and optimized multimodal collaborative diagnosis model. Input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as a new defect annotation sample.

[0011] Specifically, the multimodal collaborative diagnostic model includes a feature extraction sub-network and a cross-modal attention module;

[0012] The feature extraction subnetwork contains at least three parallel modal feature extraction branches, each corresponding to a modal of data. The modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch.

[0013] The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction sub-network and generates an inter-modal similarity matrix by calculating the cosine similarity between the modal features. The weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

[0014] Specifically, the feature correlation calculation layer is calculated as follows: for the initial features of the visual, acoustic, and mechanical modalities, the mutual information values ​​between visual and acoustic, visual and mechanical, and acoustic and mechanical are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

[0015] Specifically, the multi-scale feature extraction subnetwork includes three scale branches: small-scale branch, medium-scale branch, and large-scale branch. The output features of each scale branch are integrated by feature concatenation and then input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through backpropagation to automatically adjust the fusion weights of features at different scales.

[0016] During cascaded processing, the multi-scale and multi-modal fusion features are first input to the preliminary defect probability judgment layer. When the output probability value is less than the preset probability threshold, it is judged as a defect-free sample and removed. The remaining samples are input to the defect classification layer and the severity assessment layer to obtain the preliminary diagnosis results.

[0017] The preliminary defect probability judgment layer uses a fully connected neural network to output the probability value of the presence of defects in the sample; the defect classification layer uses a classification neural network to output the defect type; and the severity assessment layer uses a regression neural network to output the severity of the defect.

[0018] Specifically, the output features of each scale branch are input to the weighted fusion layer after feature concatenation and integration, including:

[0019] The output feature maps of the small-scale branch, medium-scale branch, and large-scale branch are concatenated according to the channel dimension to obtain the concatenated feature map.

[0020] The concatenated feature map is input into a weighted fusion layer, and the output is the fusion weight of features at different scales. The weighted fusion layer adopts a fully connected neural network, the hidden layer of the fully connected neural network adopts the ReLU activation function, and the output layer adopts the softmax activation function.

[0021] The weight parameters of the weighted fusion layer are dynamically optimized using the backpropagation algorithm.

[0022] Specifically, the methods for detecting the integrity of preprocessed multimodal data include:

[0023] At the data acquisition end, a data integrity check code based on the SHA-256 algorithm is generated for each type of data, and the data integrity check code is transmitted to the processing end along with the multimodal data;

[0024] At the processing end, the checksum of the received multimodal data is recalculated and compared with the data integrity checksum transmitted from the acquisition end. If the checksums match, the multimodal data is considered complete; otherwise, the multimodal data is considered missing.

[0025] Specifically, the method for detecting the integrity of preprocessed multimodal data also includes:

[0026] When visual modal data is detected to be missing, backup network branch A is activated, that is, the visual modal feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified version of the cross-modal attention module only calculates the similarity between acoustic and mechanical features. After generating a similarity matrix, the attention weights of acoustic and mechanical modal features are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction sub-network.

[0027] When acoustic modal data is detected to be missing, backup network branch B is activated, that is, the acoustic modal feature extraction branch is deleted, and a direct mapping layer of visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map visual features to mechanical feature space, and then calculates the similarity between the mapped visual features and mechanical features to generate a similarity matrix, and then generates attention weights.

[0028] When missing mechanical modal data is detected, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted. After the small-scale branch of the multi-scale feature extraction sub-network, a fusion compensation layer of visual and acoustic features is added. The fusion compensation layer adopts a convolutional neural network, which adds the visual small-scale features and the acoustic small-scale features through residual connections to obtain the fused features, which are then input to the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

[0029] Specifically, the cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When only a single modality of data exists, the basic diagnostic model is activated, and the output result is weighted and corrected using the historical diagnostic accuracy of that single modality of data. When two modalities of data exist, the dual-modal collaborative model is activated, and the diagnostic reliability is improved through cross-verification of features between modalities. When three modalities of data exist, the full-modal fusion model is activated, and the diagnostic result with the highest confidence is output. The switching between the basic diagnostic model, the dual-modal collaborative model, and the full-modal fusion model is automatically triggered by the modality status identifier.

[0030] A defect diagnosis system based on multimodal data collaborative training includes: a model building module, a data processing module, a feature extraction module, a preliminary diagnosis module, a collaborative diagnosis module, and a diagnosis and update module;

[0031] The model building module is used to build a multimodal collaborative diagnostic model that includes feature extraction and cross-modal attention mechanisms;

[0032] The data processing module is used to collect multimodal data and perform preprocessing.

[0033] The feature extraction module is used to extract initial features from the preprocessed multimodal data using a feature extraction sub-network, and generate weighted multimodal features through a cross-modal attention module;

[0034] The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features, obtain multi-scale multi-modal fused features, and provide preliminary diagnostic results on the type and severity of the defects.

[0035] The collaborative diagnostic module is used to detect the integrity of the preprocessed multimodal data and obtain diagnostic results through cascaded collaborative diagnostics in the case of missing modalities.

[0036] The diagnosis and update module is used to input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final diagnosis result as a new defect annotation sample.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This invention proposes a defect diagnosis method based on multimodal data collaborative training. By constructing a multimodal collaborative diagnosis model that integrates an attention mechanism, it effectively integrates a feature extraction sub-network and a cross-modal attention module. When processing multimodal data, the feature extraction sub-network first obtains initial features, and then the cross-modal attention module generates attention weights to weight the initial features. This can accurately capture the correlation and importance between features of different modalities, thereby obtaining more representative weighted multimodal features and improving the quality and effectiveness of feature extraction.

[0039] 2. This invention proposes a defect diagnosis method based on multimodal data collaborative training. A multi-scale feature extraction sub-network is designed to further process weighted features, obtaining multi-scale, multimodal fusion features. These features are then processed in a cascaded manner to provide preliminary diagnostic results. Simultaneously, data integrity is checked, and a cascaded collaborative diagnosis approach is used to address scenarios with missing modalities, enhancing the model's adaptability and robustness. Various diagnostic results are compared with defect-labeled samples to optimize the model, resulting in accurate and reliable final defect diagnosis results. Furthermore, these results are used as new samples to update the labeling library, forming a virtuous cycle that continuously improves the performance and practicality of the diagnostic model, demonstrating significant advantages in the field of defect diagnosis. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a defect diagnosis method based on multimodal data collaborative training according to the present invention;

[0041] Figure 2 This is a flowchart illustrating the principle of a defect diagnosis method based on multimodal data collaborative training according to the present invention.

[0042] Figure 3 This is an architecture diagram of a defect diagnosis system based on multimodal data collaborative training according to the present invention. Detailed Implementation

[0043] Example 1:

[0044] Please see Figure 1 and Figure 2 The present invention provides an embodiment of a defect diagnosis method based on multimodal data collaborative training, comprising the following steps:

[0045] S1: Construct a multimodal collaborative diagnostic model that integrates an attention mechanism; the multimodal collaborative diagnostic model includes a feature extraction sub-network and a cross-modal attention module;

[0046] Furthermore, before constructing a multimodal collaborative diagnostic model that integrates attention mechanisms, the sources of multimodal data are evaluated and screened. By collecting multimodal data samples from different production batches and under different working conditions, their distribution characteristics and quality are analyzed. Then, based on the evaluation results, high-quality data samples are selected as candidate data for training and testing sets to ensure that the model can learn comprehensive defect features.

[0047] Meanwhile, when collecting multimodal data, a high-precision clock synchronization device is used to synchronize the acquisition clocks of the visual sensor, acoustic sensor and mechanical sensor, and the acquisition timestamp of each modal data is recorded in real time during the data acquisition process for time alignment.

[0048] Furthermore, the specific steps for constructing the multimodal collaborative diagnostic model that incorporates an attention mechanism include:

[0049] (1) Determine the overall framework of the multimodal collaborative diagnostic model, and clarify the position and role of the feature extraction sub-network and the cross-modal attention module in the model. The feature extraction sub-network is responsible for extracting initial features from different modal data, and the cross-modal attention module is used to mine the correlation between features of different modalities in order to achieve more effective feature fusion.

[0050] (2) For different modal data, corresponding feature extraction network structures are designed respectively. For visual modal data, a convolutional neural network is used to extract features such as texture, shape, and edge in the image step by step through multiple convolutional layers, pooling layers, and activation function layers. The convolutional layers use convolutional kernels of different sizes to capture feature information at different scales. The pooling layers use max pooling or average pooling operations to reduce feature dimensions, reduce computation, and enhance the translation invariance of features. For acoustic modal data, a recurrent neural network is used to process time-series data and extract time-series and spectral features from acoustic signals. For mechanical modal data, a fully connected neural network is used to normalize the raw data collected by the mechanical sensor and input it into the fully connected layer. Mechanical features are extracted through multiple nonlinear transformations. The convolutional neural network, recurrent neural network, and fully connected neural network are existing technologies in this field and are not the inventive solutions of this application. They will not be described in detail here.

[0051] (3) The structure of the cross-modal attention module is designed, including a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer uses the cosine similarity measurement method to calculate the similarity matrix between different modal features. The weight generation layer generates the attention weight matrix based on the similarity matrix and the softmax function, so that different modal features can be weighted and fused according to their relevance to the target task. Cosine similarity and softmax function are existing technologies in this field and are not the inventive solutions of this application. They will not be described in detail here.

[0052] (4) The random initialization method is used to initialize all parameters of the multimodal collaborative diagnostic model. The random initialization uses Gaussian distribution to generate initial parameter values. The pre-trained model parameter initialization uses the model parameters pre-trained on relevant datasets to initialize the corresponding layer of the current model, so as to accelerate the convergence speed of the model and improve the performance of the model.

[0053] Furthermore, when preprocessing the acquired multimodal data, histogram equalization is used to improve the image quality and clarity for visual modal data; for acoustic modal data, wavelet transform is used to convert the sound signal to the frequency domain, remove noise frequency bands, and then convert it back to the time domain; for mechanical modal data, filtering and normalization are performed to remove high-frequency noise and scale the data to a uniform numerical range. Histogram equalization, wavelet transform, filtering, and normalization are existing technologies in this field and are not inventive solutions of this application, and will not be elaborated here.

[0054] It should be noted that the multimodal collaborative diagnostic model constructed in this application includes a feature extraction subnetwork and a cross-modal attention module. This unique architectural design is the core foundation of this method. The feature extraction subnetwork is responsible for extracting initial features from different modal data to provide raw information for subsequent analysis. The cross-modal attention module is the innovative highlight of this model, which can break down the barriers between different modal data and enable the model to focus on the correlation and importance between features of different modalities. Traditional models often process different modal data in isolation and cannot make full use of their inherent connections. The architectural design of this model can uncover these hidden information and thus more accurately identify defect features, which is an advantage that traditional methods cannot match.

[0055] On the other hand, by integrating the feature extraction sub-network and the cross-modal attention module into a single model, effective fusion of multimodal data is achieved. Different modalities of data possess different characteristics and advantages. For example, visual modal data can intuitively present the appearance of an object, acoustic modal data can reflect the internal structure of an object or the sound characteristics during operation, and mechanical modal data can demonstrate the response of an object under stress. This model integrates these three modalities, comprehensively utilizing their advantages to more comprehensively describe the characteristics of defects and improve the ability to identify various complex defects. This multimodal fusion approach is groundbreaking in the field of defect diagnosis, providing richer and more accurate diagnostic information compared to single-modal or simple combination modal methods.

[0056] S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. Generate attention weights by calculating the similarity between modal features, and multiply the multimodal initial features with the attention weights to obtain weighted multimodal features.

[0057] It should be noted that this invention collects data in three modalities: visual, acoustic, and mechanical, enabling the acquisition of information about the inspected object from multiple dimensions. In industrial production, defects may exhibit different characteristics in different modalities. For example, a surface crack on a mechanical part may be clearly visible in the visual modality, but internal structural damage may not be directly observable visually. Acoustic modal data may capture abnormal sound signals generated by internal damage during operation, while mechanical modal data may reflect abnormal deformation or vibration of the part under stress. By simultaneously acquiring data in these three modalities, it is possible to comprehensively cover all possible defect scenarios, providing sufficient information support for accurate diagnosis. This multimodal data acquisition method is irreplaceable by traditional single-modal data acquisition methods, and can greatly improve the comprehensiveness and accuracy of defect diagnosis.

[0058] Meanwhile, preprocessing the collected multimodal data is a crucial step in ensuring the accuracy of subsequent model training and diagnosis. Data from different modalities may have different formats, scales, and noise levels. Directly inputting these into the model can prevent it from effectively learning data features, affecting diagnostic results. The preprocessing steps in this application include data cleaning, denoising, and normalization, which can remove noise and outliers from the data, making it cleaner and more standardized. For example, visual modal data may have issues such as image blurring and uneven lighting; image enhancement techniques can improve image quality. Acoustic modal data may have background noise interference; filtering algorithms can remove noise. Mechanical modal data may have data drift issues; normalization can ensure the data has a uniform scale. Preprocessed data can better meet the requirements of the model, improving training efficiency and diagnostic accuracy, which is an important aspect of ensuring the stable and reliable operation of the entire defect diagnosis system.

[0059] Then, the preprocessed multimodal data is input into the feature extraction subnetwork to obtain initial multimodal features. This step can extract representative feature information from the original data. The feature extraction branches for different modalities are designed according to the data characteristics of their respective modalities. For example, the visual modality feature extraction branch may use convolutional neural networks to extract features such as texture and shape of images; the acoustic modality feature extraction branch may use time-frequency analysis methods to convert sound signals into time-spectrum graphs and then extract features; the mechanical modality feature extraction branch uses signal processing techniques to extract the time-domain and frequency-domain features of mechanical signals. Through the feature extraction branches, the advantages of different modal data can be fully utilized to extract more effective features.

[0060] Finally, the initial multimodal features are input into the cross-modal attention module. Attention weights are generated by calculating the similarity between modal features, and the initial multimodal features are multiplied by the attention weights to obtain weighted multimodal features. This process has extremely important functionality. The generation of attention weights enables the model to automatically focus on important correlations between features of different modalities, giving higher weights to key features related to defects and lower weights to irrelevant or interfering features. For example, in some cases, a feature in the visual modality and a feature in the acoustic modality may jointly indicate the presence of a defect. Through the cross-modal attention module, the model can identify this correlation and increase the importance of these two features in subsequent diagnosis. This weighted processing method can highlight key features, suppress noise and irrelevant information, and make the model more focused on features related to defect diagnosis, thereby improving the accuracy and robustness of diagnosis. Compared with the traditional method of simply concatenating or averaging features of different modalities, the weighted processing of this application can dynamically adjust the importance of features according to the actual correlation between features, and has stronger adaptability and accuracy.

[0061] S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnostic results of the type and severity of defects after cascade processing;

[0062] S4: Detect the integrity of the preprocessed multimodal data and obtain diagnostic results for scenarios with missing modalities through cascaded collaborative diagnosis;

[0063] The cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When only a single modality of data exists, the basic diagnostic model is activated, and the output result is weighted and corrected using the historical diagnostic accuracy of that single modality of data. When two modalities of data exist, the bimodal collaborative model is activated, and the diagnostic reliability is improved through cross-verification of features between modalities. When three modalities of data exist, the full-modal fusion model is activated, and the diagnostic result with the highest confidence is output. The switching between the basic diagnostic model, the bimodal collaborative model, and the full-modal fusion model is automatically triggered by the modality status identifier.

[0064] It should be noted that the use of a cascaded collaborative diagnostic approach to obtain diagnostic results adapted to modality loss scenarios is a significant innovation of this application in addressing complex real-world situations. In practical applications, various reasons may lead to the loss of certain modality data. Traditional methods often fail to provide effective diagnosis in such cases. However, the cascaded collaborative diagnostic approach of this application can activate corresponding backup network branches based on different modality loss scenarios, thereby achieving defect diagnosis.

[0065] S5: Compare the preliminary diagnosis results and the diagnosis results of the adapted modality missing scenario with the defect annotation samples. Based on the comparison results, obtain the trained and optimized multimodal collaborative diagnosis model. Input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as a new defect annotation sample.

[0066] Furthermore, the new defect annotation samples adopt a three-level review mechanism. First, they are screened based on the confidence level of the model output. Second, they are randomly sampled and reviewed by human reviewers. Finally, the samples that pass the review are added to the annotation sample library. At the same time, a timestamp and data source identifier are added to each new sample to build a dynamically updated sample library and improve the model's continuous learning ability.

[0067] Furthermore, the preliminary diagnostic results and the diagnostic results for the adapted modality-deficient scenario are compared with the defect-annotated samples. Based on the comparison results, a trained and optimized multimodal collaborative diagnostic model is obtained, including:

[0068] (1) Design an appropriate loss function to measure the difference between the preliminary diagnosis results, the diagnosis results of the adapted modality missing scenario and the defect labeled samples. For the defect classification task, the cross-entropy loss function is used; for the defect severity assessment task, the mean squared error loss function is used. The classification loss and regression loss are combined according to a certain weight to obtain the total loss function.

[0069] (2) Based on the total loss function, the gradient of all parameters in the model is calculated using the backpropagation algorithm, and then the model parameters are updated along the opposite direction of the gradient using the stochastic gradient descent method, so that the loss function value is continuously reduced and the performance of the model is continuously improved. The backpropagation algorithm and the stochastic gradient descent method are existing technologies in this field and are not the inventive solutions of this application, and will not be described in detail here.

[0070] (3) Divide the multimodal dataset into training set, validation set and test set. Use the training set to train the model and use the validation set to monitor and adjust the model performance during the training process, such as adjusting the learning rate and early stopping. When the model performance on the validation set reaches stability, stop training and obtain the trained and optimized multimodal collaborative diagnostic model.

[0071] It is important to emphasize that, based on the comparison results, the multimodal collaborative diagnostic model can automatically adjust its parameters and optimize its performance through the backpropagation algorithm. For example, if the multimodal collaborative diagnostic model is found to have a large error in defect diagnosis, the backpropagation algorithm will adjust the weights of relevant neurons in the model, making the model pay more attention to features related to these defect types, thereby improving the diagnostic accuracy for these defects. By continuously comparing new diagnostic results with labeled samples and optimizing the model, the multimodal collaborative diagnostic model can gradually learn more accurate feature representations and diagnostic rules, continuously improving diagnostic accuracy and generalization ability. This model optimization method based on result comparison is a self-learning and self-improvement process, enabling the multimodal collaborative diagnostic model to adapt to defect diagnosis needs in different scenarios.

[0072] The multimodal collaborative diagnostic model includes a feature extraction subnetwork and a cross-modal attention module;

[0073] The feature extraction subnetwork contains at least three parallel modal feature extraction branches, each corresponding to a modal of data. The modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch.

[0074] Furthermore, each modal feature extraction branch adopts a convolutional neural network architecture, extracting modal features through multi-layer convolution and pooling operations. Specifically, the visual modal feature extraction branch targets image data, using convolutional kernels to capture the texture and edge features of the image; the acoustic modal feature extraction branch processes audio data, extracting the frequency and amplitude features of the sound through operations such as spectrum analysis; and the mechanical modal feature extraction branch targets mechanical sensor data, extracting the features of the mechanical signal through operations such as filtering and normalization.

[0075] The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction sub-network and generates an inter-modal similarity matrix by calculating the cosine similarity between the modal features. The weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

[0076] The calculation method of the feature correlation calculation layer is as follows: for the initial features of the three modalities of vision, acoustics, and mechanics, the mutual information values ​​of vision and acoustics, vision and mechanics, and acoustics and mechanics are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

[0077] The multi-scale feature extraction subnetwork includes three scale branches: small-scale branch, medium-scale branch, and large-scale branch. The output features of each scale branch are integrated by feature concatenation and then input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through backpropagation to automatically adjust the fusion weights of features at different scales.

[0078] The small-scale branch uses a 3×3 convolution kernel for feature extraction, and outputs a feature map with a resolution of 1 / 4 of the original feature; the medium-scale branch uses a 5×5 convolution kernel, and outputs a feature map with a resolution of 1 / 2 of the original feature; the large-scale branch uses a 7×7 convolution kernel, and outputs a feature map with the same resolution as the original feature.

[0079] It should be noted that the design of a multi-scale feature extraction subnetwork comprising three scale branches—small-scale, medium-scale, and large-scale—is another innovation in feature extraction in this patent. Features at different scales can reflect different levels of information about defects. Small-scale features can capture subtle features of defects, such as tiny cracks and scratches; medium-scale features can reflect the local structure and morphology of defects; and large-scale features can reflect the distribution of defects throughout the entire object or region. By simultaneously extracting features at these three scales, the characteristics of defects can be comprehensively and meticulously described, allowing for analysis of defects from multiple perspectives, from micro to macro. For example, when inspecting surface defects on metal parts, small-scale features may detect tiny pits on the surface, medium-scale features can identify the shape and size of the pits, and large-scale features can determine the distribution range and density of the pits on the surface. Compared to single-scale feature extraction methods, this multi-scale feature extraction approach provides richer and more comprehensive defect information, helping to more accurately determine the type and severity of defects.

[0080] The output features of each scale branch are integrated through feature concatenation and then input into the weighted fusion layer. This step enables the effective fusion of features at different scales. Feature concatenation merges features of different scales along the channel dimension to form a comprehensive feature map containing multi-scale information. However, features of different scales may have different importance in diagnosis, thus requiring a weighted fusion layer to automatically adjust the fusion weights of features at different scales. The weight parameters of the weighted fusion layer are dynamically optimized through backpropagation. This means that during model training, the weighted fusion layer can automatically learn and adjust the weights of features at different scales based on actual diagnostic results, making the model focus more on scale features that have a greater impact on the diagnostic results. For example, in some cases, small-scale features may be more critical for accurately identifying defect types, and the weighted fusion layer will automatically increase the weight of small-scale features; while in other cases, large-scale features may be more important for judging the severity of defects, and the weighted fusion layer will adjust the weights accordingly. This dynamically optimized weighted fusion method can fully leverage the advantages of features at different scales, improve the effect of feature fusion, and thus further enhance the accuracy of diagnosis.

[0081] During cascaded processing, the multi-scale, multi-modal fused features are first input to the preliminary defect probability judgment layer. When the output probability value is less than a preset probability threshold, the sample is determined to be defect-free and is removed. The remaining samples are input to the defect classification layer and the severity assessment layer, which output the defect type and severity, respectively, to obtain a preliminary diagnostic result. The preliminary defect probability judgment layer uses a fully connected neural network to output the probability value of the presence of defects in the sample. The defect classification layer uses a classification neural network to output the defect type. The severity assessment layer uses a regression neural network to output the severity of the defect.

[0082] It should be noted that the preliminary defect probability assessment layer employs a fully connected neural network, enabling rapid preliminary judgment on the presence of defects in samples. By setting probability thresholds, defect-free samples are effectively filtered out, reducing the computational load of subsequent processing and improving the overall efficiency of the diagnostic system. For example, in the detection of a large number of samples, many are defect-free. Performing detailed defect classification and severity assessment on all samples would consume significant computational resources and time. However, the preliminary defect probability assessment layer can quickly eliminate defect-free samples, allowing further analysis only on samples that may contain defects, thus improving diagnostic efficiency.

[0083] The defect classification layer employs a classification neural network, capable of accurately determining the type of defect based on input features. Different types of defects may require different handling methods and maintenance strategies; therefore, accurate defect classification is of great significance for actual production. For example, in the production of electronic products, there may be various types of defects such as short circuits, poor soldering of components, and open circuits. The defect classification layer can accurately identify the type of defect, providing a basis for subsequent maintenance and quality improvement.

[0084] The severity assessment layer employs a regression neural network to output the severity of defects. The severity of defects directly affects product quality and lifespan. Accurately assessing the severity of defects helps in developing reasonable maintenance plans and quality control standards. For example, for crack defects in mechanical parts, minor cracks may only require simple repairs, while severe cracks may require replacement of the part. The severity assessment layer can quantitatively evaluate the severity of defects, providing a scientific basis for production decisions.

[0085] The output features of each scale branch are concatenated and integrated before being input into the weighted fusion layer, including:

[0086] A1: Concatenate the output feature maps of the small-scale branch, medium-scale branch, and large-scale branch according to the channel dimension to obtain the concatenated feature map;

[0087] A2: The concatenated feature map is input into the weighted fusion layer, and the output is the fusion weight of features at different scales. The weighted fusion layer adopts a fully connected neural network. The hidden layer of the fully connected neural network adopts the ReLU activation function, and the output layer adopts the softmax activation function. The ReLU activation function is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0088] A3: The weight parameters of the weighted fusion layer are dynamically optimized using the backpropagation algorithm.

[0089] The methods for detecting the integrity of preprocessed multimodal data include:

[0090] B1: At the data acquisition end, generate a data integrity check code based on the SHA-256 algorithm for each type of data, and transmit the data integrity check code along with the multimodal data to the processing end;

[0091] Furthermore, the process of generating a data integrity check code based on the SHA-256 algorithm for each modality of data includes:

[0092] (1) Divide the raw data of each modality into blocks of a fixed size to facilitate subsequent hash calculations;

[0093] (2) The hash value of the SHA-256 algorithm is initialized to a fixed initial value. The SHA-256 algorithm is existing technology in this field and is not an inventive solution of this application. It will not be described in detail here.

[0094] (3) Perform bitwise and logical operations on each data block to update the hash value. The specific operation process includes XORing, shifting, and modulo addition of the data block with the current hash value to obtain a new hash value.

[0095] (4) After processing all data blocks, the hash value obtained is the data integrity check code of the modal data, and the data integrity check code is transmitted to the processing end along with the multimodal data.

[0096] B2: At the processing end, the checksum of the received multimodal data is recalculated and compared with the data integrity checksum transmitted from the acquisition end. If the checksums match, the multimodal data is considered complete; if they do not match, the multimodal data is considered missing.

[0097] Furthermore, the specific process of B2 includes:

[0098] The processing unit receives multimodal data and a data integrity check code;

[0099] The data integrity check code is recalculated for the received multimodal data using the same SHA-256 algorithm as the acquisition end.

[0100] The recalculated data integrity check code is compared with the data integrity check code transmitted from the acquisition end. If the check codes match, the multimodal data is considered complete; if they do not match, the multimodal data is considered missing.

[0101] B3: When visual modal data is detected to be missing, the backup network branch A is activated, that is, the visual modal feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified version of the cross-modal attention module only calculates the similarity between acoustic and mechanical features. After generating a similarity matrix, the attention weights of acoustic and mechanical modal features are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction sub-network. This processing method can make full use of the remaining acoustic and mechanical modal data when visual modal data is missing, extract the correlation features between them through the cross-modal attention mechanism, and perform subsequent diagnosis. Although visual modal data is missing, relatively accurate diagnostic results can still be obtained by making reasonable use of the remaining modal data, avoiding the problem that diagnosis cannot be performed due to the lack of a single modal data.

[0102] Furthermore, the specific steps for B3 include:

[0103] (1) In the multimodal collaborative diagnostic model, find the visual modality feature extraction branch and delete it from the model. At this time, the model only has the acoustic and mechanical modality feature extraction branches.

[0104] (2) A simplified version of the cross-modal attention module is designed, in which the feature correlation calculation layer only calculates the similarity between acoustic and mechanical features, specifically including:

[0105] The initial eigenvectors of the acoustic and mechanical modes are normalized to a length of 1.

[0106] The similarity between acoustic and mechanical feature vectors is calculated using cosine similarity to obtain a similarity matrix;

[0107] Based on the similarity matrix, the attention weight matrices for acoustic and mechanical modes are generated using the softmax function;

[0108] (3) Multiply the initial feature vectors of acoustic and mechanical modes by the corresponding attention weights to obtain the weighted acoustic and mechanical mode feature vectors, and directly input the weighted feature vectors into the multi-scale feature extraction sub-network.

[0109] B4: When acoustic modal data is detected to be missing, the backup network branch B is activated, i.e., the acoustic modal feature extraction branch is deleted, and a direct mapping layer between visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map visual features to the mechanical feature space, and then calculates the similarity between the mapped visual features and mechanical features to generate a similarity matrix, which in turn generates attention weights. This method can achieve cross-modal attention calculation by establishing a mapping relationship between visual and mechanical features when acoustic modal data is missing, thereby using visual and mechanical modal data for diagnosis. It makes full use of the potential connections between different modal data, and through innovative mapping methods, it makes up for the impact of missing acoustic modal data, ensuring the continuity and accuracy of diagnosis.

[0110] Furthermore, the specific steps of B4 include:

[0111] In the multimodal collaborative diagnostic model, the acoustic modal feature extraction branch is found and removed from the model. At this point, the model only has the visual and mechanical modal feature extraction branches.

[0112] A direct mapping layer for visual and mechanical features is added to the cross-modal attention module. This direct mapping layer employs a fully connected neural network and specifically includes:

[0113] The structure of a fully connected neural network is designed, including an input layer, a hidden layer, and an output layer. The input layer receives the initial feature vector of the visual modality, the hidden layer uses the ReLU activation function to introduce nonlinear factors, and the number of neurons in the output layer is the same as the feature dimension of the mechanical modality.

[0114] A fully connected neural network was trained using training samples containing visual and mechanical modal data. During training, backpropagation and stochastic gradient descent were used to update the network's weight parameters, enabling the network to map visual features to a mechanical feature space.

[0115] The initial feature vector of the visual modality is input into the trained fully connected neural network to obtain the mapped visual feature vector;

[0116] (3) Calculate the similarity between the mapped visual feature vector and the mechanical feature vector, generate a similarity matrix, and then use the softmax function to generate an attention weight matrix. Input the weighted features into the multi-scale feature extraction sub-network to complete the defect diagnosis.

[0117] B5: When missing mechanical modal data is detected, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted. After the small-scale branch of the multi-scale feature extraction sub-network, a fusion compensation layer of visual and acoustic features is added. The fusion compensation layer adopts a convolutional neural network. The visual small-scale features and acoustic small-scale features are added through residual connections to obtain the fused features, which are then input to the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

[0118] It should be explained that, in the absence of mechanical modal data, fusing small-scale features from visual and acoustic modalities through a fusion compensation layer can extract richer feature information, making up for the lack of mechanical modal data. The use of residual connections can effectively avoid the gradient vanishing problem, ensuring the effect of feature fusion and thus improving the accuracy of diagnosis.

[0119] Furthermore, the fusion compensation layer comprises two 3×3 convolutional sub-layers and one residual connection. The first convolutional sub-layer increases the visual and acoustic small-scale features from 256 channels to 512 channels respectively. The second convolutional sub-layer reduces the dimensionality of the 512-channel features to 256 channels. The residual connection adds the input visual and acoustic small-scale features to the output features of the second convolutional sub-layer. The ReLU activation function is used to enhance the nonlinear expressive power of the features and compensate for the feature loss caused by the lack of mechanical modes.

[0120] Furthermore, the specific steps of B5 include:

[0121] In the multimodal collaborative diagnostic model, the mechanical modal feature extraction branch is found and removed from the model. At this point, the model only has the visual and acoustic modal feature extraction branches.

[0122] A fusion compensation layer for visual and acoustic features is added after the small-scale branch of the multi-scale feature extraction subnetwork. This fusion compensation layer uses a convolutional neural network and specifically includes:

[0123] The design of the convolutional neural network includes convolutional layers, pooling layers, and activation function layers. The convolutional layers use 3×3 convolutional kernels to extract local information from small-scale visual and acoustic features. The pooling layers use max pooling or average pooling operations to reduce feature dimensionality. The activation function layers use the ReLU activation function to introduce nonlinearity.

[0124] The visual small-scale features and acoustic small-scale features are added together by residual connection. Specifically, the visual small-scale features and acoustic small-scale features are concatenated in the channel dimension, and then the number of channels is adjusted by a 1×1 convolutional layer to be the same as the number of channels of the input features. Finally, the adjusted features are added to the input features to obtain the fused features.

[0125] (3) Input the fused features into the multi-scale feature extraction branch and the weighted fusion layer to complete the subsequent feature extraction and defect diagnosis.

[0126] Example 2:

[0127] Please see Figure 3 Another embodiment of the present invention provides: a defect diagnosis system based on multimodal data collaborative training, comprising:

[0128] The module includes: model building module, data processing module, feature extraction module, preliminary diagnosis module, collaborative diagnosis module, and diagnosis and update module.

[0129] The model building module is used to build a multimodal collaborative diagnostic model that includes feature extraction and cross-modal attention mechanisms;

[0130] The data processing module is used to collect multimodal data and perform necessary preprocessing to provide clean and standardized data for subsequent feature extraction and diagnosis.

[0131] The feature extraction module is used to extract initial features from the preprocessed multimodal data using the feature extraction sub-network, and generate weighted multimodal features through the cross-modal attention module;

[0132] The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features, obtain multi-scale multi-modal fused features, and provide preliminary diagnostic results on the type and severity of defects based on these features.

[0133] The collaborative diagnostic module is used to detect the integrity of preprocessed multimodal data and obtain diagnostic results through cascaded collaborative diagnostics in the case of missing modalities.

[0134] The diagnosis and update module is used to input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final diagnosis result as a new defect annotation sample for subsequent training.

[0135] The model building module includes: network design unit and cross-modal design unit;

[0136] Network design unit, which designs sub-network structures for extracting initial features from multimodal data;

[0137] Cross-modal design unit, a module used to calculate the similarity between modal features and generate attention weights to enable interaction and enhancement between features.

[0138] The feature extraction module includes: an execution unit, a weight calculation unit, and a feature weighting unit;

[0139] The execution unit is used to run the feature extraction subnetwork to extract initial features from multimodal data;

[0140] The weight calculation unit is used to calculate the similarity between modal features in the cross-modal attention module and generate attention weights;

[0141] The feature weighting unit is used to multiply the initial multimodal features with the corresponding attention weights to obtain the weighted multimodal features.

[0142] The preliminary diagnosis module includes: a sub-network design unit, a feature extraction unit, and a preliminary diagnosis unit;

[0143] Sub-network design unit, a network structure used to extract multi-scale features;

[0144] The feature extraction unit is used to run the multi-scale feature extraction sub-network to extract the weighted multimodal features;

[0145] The preliminary diagnostic unit is used to fuse multi-scale features and provide preliminary diagnostic results through classification or regression models.

[0146] The diagnosis and update module includes: a diagnosis execution unit and a sample update unit;

[0147] The diagnostic execution unit is used to run the trained and optimized multimodal collaborative diagnostic model to diagnose the samples to be diagnosed.

[0148] The sample update unit is used to update the sample library by using the final diagnosis results as new defect annotation samples for subsequent model training and optimization.

[0149] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A defect diagnosis method based on multimodal data collaborative training, characterized in that, include: S1: Construct a multimodal collaborative diagnostic model that integrates an attention mechanism; the multimodal collaborative diagnostic model includes a feature extraction sub-network and a cross-modal attention module; S2: Collect multimodal data and preprocess it, input it into the feature extraction sub-network to obtain multimodal initial features, and then input it into the cross-modal attention module. Generate attention weights by calculating the similarity between modal features, and multiply the multimodal initial features with the attention weights to obtain weighted multimodal features. S3: Design a multi-scale feature extraction sub-network, input the weighted multi-modal features to obtain multi-scale multi-modal fusion features, and obtain preliminary diagnostic results of the type and severity of defects after cascade processing; S4: Detect the integrity of the preprocessed multimodal data and obtain diagnostic results for scenarios with missing modalities through cascaded collaborative diagnosis; S5: Compare the preliminary diagnosis results and the diagnosis results of the adapted modality missing scenario with the defect annotation samples. Based on the comparison results, obtain the trained and optimized multimodal collaborative diagnosis model. Input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final defect diagnosis result as a new defect annotation sample.

2. The defect diagnosis method based on multimodal data collaborative training as described in claim 1, characterized in that, The multimodal collaborative diagnostic model includes a feature extraction subnetwork and a cross-modal attention module; The feature extraction subnetwork contains at least three parallel modal feature extraction branches, each corresponding to a modal of data. The modal feature extraction branches include a visual modal feature extraction branch, an acoustic modal feature extraction branch, and a mechanical modal feature extraction branch. The cross-modal attention module includes a feature correlation calculation layer and a weight generation layer. The feature correlation calculation layer receives multimodal features from the feature extraction sub-network and generates an inter-modal similarity matrix by calculating the cosine similarity between the modal features. The weight generation layer takes the inter-modal similarity matrix as input and uses the softmax function to generate attention weights.

3. The defect diagnosis method based on multimodal data collaborative training as described in claim 2, characterized in that, The calculation method of the feature correlation calculation layer is as follows: for the initial features of the three modalities of vision, acoustics, and mechanics, the mutual information values ​​of vision and acoustics, vision and mechanics, and acoustics and mechanics are calculated respectively, a 3×3 inter-modal similarity matrix is ​​constructed, and the element values ​​of the inter-modal similarity matrix are normalized.

4. The defect diagnosis method based on multimodal data collaborative training as described in claim 3, characterized in that, The multi-scale feature extraction subnetwork includes three scale branches: small-scale branch, medium-scale branch, and large-scale branch. The output features of each scale branch are integrated by feature concatenation and then input into the weighted fusion layer. The weight parameters of the weighted fusion layer are dynamically optimized through backpropagation to automatically adjust the fusion weights of features at different scales. During cascaded processing, the multi-scale and multi-modal fusion features are first input to the preliminary defect probability judgment layer. When the output probability value is less than the preset probability threshold, it is judged as a defect-free sample and removed. The remaining samples are input to the defect classification layer and the severity assessment layer to obtain the preliminary diagnosis results. The preliminary defect probability judgment layer uses a fully connected neural network to output the probability value of the presence of defects in the sample; the defect classification layer uses a classification neural network to output the defect type; and the severity assessment layer uses a regression neural network to output the severity of the defect.

5. The defect diagnosis method based on multimodal data collaborative training as described in claim 4, characterized in that, The output features of each scale branch are concatenated and integrated before being input into the weighted fusion layer, including: The output feature maps of the small-scale branch, medium-scale branch, and large-scale branch are concatenated according to the channel dimension to obtain the concatenated feature map. The concatenated feature map is input into a weighted fusion layer, and the output is the fusion weight of features at different scales. The weighted fusion layer adopts a fully connected neural network, the hidden layer of the fully connected neural network adopts the ReLU activation function, and the output layer adopts the softmax activation function. The weight parameters of the weighted fusion layer are dynamically optimized using the backpropagation algorithm.

6. The defect diagnosis method based on multimodal data collaborative training as described in claim 5, characterized in that, The methods for detecting the integrity of preprocessed multimodal data include: At the data acquisition end, a data integrity check code based on the SHA-256 algorithm is generated for each type of data, and the data integrity check code is transmitted to the processing end along with the multimodal data; At the processing end, the checksum of the received multimodal data is recalculated and compared with the data integrity checksum transmitted from the acquisition end. If the checksums match, the multimodal data is considered complete; otherwise, the multimodal data is considered missing.

7. The defect diagnosis method based on multimodal data collaborative training as described in claim 6, characterized in that, The methods for detecting the integrity of preprocessed multimodal data also include: When visual modal data is detected to be missing, backup network branch A is activated, that is, the visual modal feature extraction branch is deleted, and the acoustic and mechanical modal features are directly input into the simplified version of the cross-modal attention module. The feature correlation calculation layer of the simplified version of the cross-modal attention module only calculates the similarity between acoustic and mechanical features. After generating a similarity matrix, the attention weights of acoustic and mechanical modal features are obtained through the weight generation layer. The weighted features are directly input into the multi-scale feature extraction sub-network. When acoustic modal data is detected to be missing, backup network branch B is activated, that is, the acoustic modal feature extraction branch is deleted, and a direct mapping layer of visual and mechanical features is added to the cross-modal attention module. The direct mapping layer uses a fully connected neural network to map visual features to mechanical feature space, and then calculates the similarity between the mapped visual features and mechanical features to generate a similarity matrix, and then generates attention weights. When missing mechanical modal data is detected, the backup network branch C is activated, that is, the mechanical modal feature extraction branch is deleted. After the small-scale branch of the multi-scale feature extraction sub-network, a fusion compensation layer of visual and acoustic features is added. The fusion compensation layer adopts a convolutional neural network, which adds the visual small-scale features and the acoustic small-scale features through residual connections to obtain the fused features, which are then input to the multi-scale feature extraction branch and the weighted fusion layer to complete the defect diagnosis.

8. The defect diagnosis method based on multimodal data collaborative training as described in claim 7, characterized in that, The cascaded collaborative diagnosis adopts a hierarchical decision-making mechanism. When only a single modality of data exists, the basic diagnostic model is activated, and the output result is weighted and corrected using the historical diagnostic accuracy of that single modality of data. When two modalities of data exist, the dual-modality collaborative model is activated. When three modalities of data exist, the full-modality fusion model is activated, and the diagnostic result with the highest confidence is output. The switching between the basic diagnostic model, the dual-modality collaborative model, and the full-modality fusion model is automatically triggered by the modality status identifier.

9. A defect diagnosis system based on multimodal data collaborative training, used to implement the defect diagnosis method based on multimodal data collaborative training as described in any one of claims 1-8, characterized in that, include: The module includes: model building module, data processing module, feature extraction module, preliminary diagnosis module, collaborative diagnosis module, and diagnosis and update module. The model building module is used to build a multimodal collaborative diagnostic model that includes feature extraction and cross-modal attention mechanisms; The data processing module is used to collect multimodal data and perform preprocessing. The feature extraction module is used to extract initial features from the preprocessed multimodal data using a feature extraction sub-network, and generate weighted multimodal features through a cross-modal attention module; The preliminary diagnosis module is used to design a multi-scale feature extraction sub-network to further extract and fuse the weighted multi-modal features, obtain multi-scale multi-modal fused features, and provide preliminary diagnostic results on the type and severity of the defects. The collaborative diagnostic module is used to detect the integrity of the preprocessed multimodal data and obtain diagnostic results through cascaded collaborative diagnostics in the case of missing modalities. The diagnosis and update module is used to input the multimodal data of the sample to be diagnosed into the trained and optimized multimodal collaborative diagnosis model, output the final defect diagnosis result, and use the final diagnosis result as a new defect annotation sample.

Images