Defect identification method and device for defect data, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a defect identification method and device for defect data, equipment and a medium, and the method comprises the steps: dynamically adjusting the attention weight of a defect identification model for difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in a training sample set; extracting text features and image features of the input data through a multi-mode encoder, and generating a fusion control signal; based on the fusion control signal and the attention weight, generating synthetic defect data consistent with target defect semantics through a controllable diffusion model; mixing the synthesized defect data with an original training sample set, and training a defect recognition model after a balanced data set is constructed; and carrying out defect classification and positioning on an input image by utilizing the trained defect identification model, and outputting defect category and position information. According to the method, the problems of data imbalance and sample scarcity can be effectively solved, and the accuracy and robustness of defect identification on the defect data are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a defect identification method and device for defect data, equipment and medium. BACKGROUND

[0002] In today's era of rapid development of digitization and intelligence, data, as a core production factor, plays a key role in many fields.

[0003] However, in processing various data, especially defect data, the existing technology faces many challenges. On the one hand, the problem of data imbalance is widespread, and the lack of defect samples of minority classes makes the model training prone to bias towards majority class samples, resulting in low recognition accuracy of minority class defects. Although the traditional re-weighting method can alleviate this problem, the recognition effect of difficult-to-classify samples near the class boundary is still not ideal. On the other hand, the scarcity of samples limits the training effect of the model, and the number of some samples in the complex and diverse defect categories is extremely small, making it difficult for traditional methods to obtain enough samples for effective training. In addition, when generating defect data, the generated data is difficult to guarantee semantic consistency with the target class, especially in the case of sample scarcity, the generated data often has deviations, affecting the quality and usability of the generated data. It can be seen that the existing technology has the defects of low recognition accuracy of minority class samples caused by data imbalance, limited model training effect due to sample scarcity, and poor semantic consistency of generated data.

[0004] The foregoing narrative is to provide general background information and does not necessarily constitute the prior art. SUMMARY

[0005] The embodiments of the present application provide a defect identification method, device, equipment and medium for defect data, which dynamically adjusts the attention weight and multi-modal fusion to generate synthetic data, effectively solves the problems of data imbalance and sample scarcity existing in the prior art, and significantly improves the accuracy and robustness of defect identification for defect data.

[0006] In a first aspect, the embodiments of the present application provide a defect identification method for defect data, comprising:

[0007] Based on the feature distribution and inter-class relationship of each sample in the training sample set, the attention weight of the defect identification model to the difficult-to-classify sample is dynamically adjusted;

[0008] The text features and image features corresponding to the input data are extracted by a multi-modal encoder to generate a fusion control signal;

[0009] Based on the fusion control signal and the attention weight, synthetic defect data consistent with the semantic of the target defect is generated by a controllable diffusion model;

[0010] The synthetic defect data is mixed with the original training sample set to construct a balanced data set, and a defect recognition model is trained based on the balanced data set;

[0011] The trained defect recognition model is used for defect classification and positioning of an input image, and defect category and position information are output.

[0012] Optionally, in some embodiments of the present application, the attention weight of the defect recognition model to the difficult-to-classify sample is dynamically adjusted based on the feature distribution and inter-class relationship of each sample in the training sample set, comprising:

[0013] The feature vectors of all samples in the training sample set are extracted, and the prototype features of each category are calculated;

[0014] For the feature vector of each sample, at least one out-of-class prototype is randomly selected for interpolation disturbance to generate a disturbed feature vector;

[0015] The difficult-to-classify weight of the sample is dynamically calculated according to the confidence of the disturbed feature vector in the classification model, wherein the weight increases when the confidence is lower than a preset threshold;

[0016] The loss function is reconstructed based on the difficult-to-classify weight, and the defect recognition model is dynamically weighted and trained.

[0017] Optionally, in some embodiments of the present application, the text features and image features corresponding to the input data are extracted by the multi-modal encoder, and a fusion control signal is generated, comprising:

[0018] The defect description text is converted into a text embedding vector by a text encoder;

[0019] The feature vector of the real defect image is extracted by an image encoder;

[0020] After the text embedding vector and the image feature vector are spliced, they are input into a feature cross principal stress module for cross-modal fusion to generate a fusion control signal.

[0021] Optionally, in some embodiments of the present application, the synthetic defect data consistent with the semantic of the target defect is generated by a controllable diffusion model based on the fusion control signal and the attention weight, comprising:

[0022] A controllable diffusion model is constructed based on a stable diffusion framework, and the controllable diffusion model comprises a UNet encoder, a ControlNet branch and a VAE decoder;

[0023] In the controllable diffusion model, the fusion control signal is injected into a diffusion process through the ControlNet branch to adjust a noise prediction path, wherein the fusion control signal includes the attention weight for indicating a generation priority of the difficult-to-classify defect sample;

[0024] When performing iterative denoising in the latent space, a denoising step or noise intensity is dynamically adjusted according to the attention weight, and a synthetic defect sample corresponding to the difficult-to-classify defect is preferentially generated;

[0025] The denoised latent feature is mapped to the synthetic defect sample by the VAE decoder, and the synthetic defect data consistent with the target defect semantics is output.

[0026] Optionally, in some embodiments of the present application, the synthetic defect data is mixed with the original training sample set to construct a balanced data set, and a defect recognition model is trained based on the balanced data set, comprising:

[0027] The original sample and the synthetic defect data are mixed at a preset ratio to construct a balanced data set for balancing the distribution of samples of each category;

[0028] The balanced data set is preprocessed to obtain a preprocessed balanced data set;

[0029] A joint training strategy is adopted to alternately optimize a classification loss and a generative adversarial loss of the defect recognition model based on the preprocessed balanced data set, and a trained defect recognition model is obtained.

[0030] Optionally, in some embodiments of the present application, the input image is classified and positioned for defects by using the trained defect recognition model, and defect category and position information are output, comprising:

[0031] Multi-scale feature extraction is performed on the input image to generate a corresponding feature map;

[0032] The feature map is input into a classification head and a regression head of the defect recognition model respectively, and defect category probability and defect bounding box coordinates are output.

[0033] Optionally, in some embodiments of the present application, the input image is multi-scale feature extracted to generate a corresponding feature map, comprising:

[0034] Multi-scale feature extraction is performed on the input image to generate multi-resolution feature maps through different levels of convolutional neural networks;

[0035] A corresponding feature pyramid is constructed based on the multi-resolution feature maps;

[0036] For each layer feature map in the feature pyramid, a spatial attention mechanism is used to calculate a response weight of the defect area, and an attention mask is generated.

[0037] The attention mask is multiplied element by element with the corresponding feature map to obtain an enhanced feature map.

[0038] In a second aspect, the embodiments of the present application provide a defect recognition device for defect data, comprising:

[0039] The weight adjustment module is configured to dynamically adjust the attention weight of the defect recognition model for the difficult-to-classify samples based on the feature distribution and the inter-class relationship of each sample in the training sample set.

[0040] The signal generation module is configured to extract text features and image features corresponding to the input data through a multi-modal encoder, and generate a fusion control signal.

[0041] The data generation module is configured to generate synthetic defect data consistent with the semantic of the target defect based on the fusion control signal and the attention weight through a controllable diffusion model.

[0042] The model training module is configured to mix the synthetic defect data with the original training sample set to construct a balanced data set, and train the defect recognition model based on the balanced data set.

[0043] The defect recognition module is configured to use the trained defect recognition model to classify and locate defects in the input image, and output the defect category and position information.

[0044] In a third aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the defect recognition method for defect data as described in the first aspect when executing the computer program.

[0045] In a fourth aspect, the embodiments of the present application provide a storage medium storing a computer program capable of being loaded and executed by a processor to perform the defect recognition method for defect data as described in the first aspect.

[0046] The application provides a defect recognition method, device, equipment and medium for defect data. Firstly, the attention weight of a defect recognition model to difficult-to-classify samples is dynamically adjusted based on the feature distribution and inter-class relationship of each sample in a training sample set. Then, the text features and image features corresponding to the input data are extracted by a multi-modal encoder to generate a fusion control signal. Next, the synthetic defect data consistent with the semantic of the target defect is generated by a controllable diffusion model based on the fusion control signal and the attention weight. Then, the synthetic defect data is mixed with the original training sample set to construct a balanced data set, and the defect recognition model is trained based on the balanced data set. Finally, the trained defect recognition model is used to classify and locate the defects of the input image, and the defect category and location information are output. In the defect recognition scheme for defect data provided in the embodiments of the application, the attention weight of the defect recognition model to difficult-to-classify samples is dynamically adjusted, effectively solving the problem of data imbalance and improving the recognition accuracy of minority class samples. At the same time, the multi-modal encoder is used to fuse the text and image features to generate a control signal, and the controllable diffusion model is used to generate synthetic defect data consistent with the semantic of the target defect, increasing the diversity and quantity of the training data, breaking the limitation of sample scarcity on the model training effect, and improving the recognition ability of the model for complex defects. In addition, the synthetic data is mixed with the original data to construct a balanced data set, and the defect recognition model is trained based on the balanced data set, further optimizing the model performance. Therefore, the defect recognition model can be more accurate and efficient in classifying and locating defects, thereby improving the accuracy and robustness of defect recognition for defect data, solving the problems of low recognition accuracy of minority class samples caused by data imbalance, limitation of model training effect caused by sample scarcity, and poor semantic consistency of generated data in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is an application environment diagram of the defect recognition method for defect data provided by the embodiments of the application;

[0049] Figure 2 is a flowchart of the defect recognition method for defect data provided by the embodiments of the application;

[0050] Figure 3 is a structural diagram of the defect recognition module provided by the embodiments of the application;

[0051] Figure 4is a structural schematic diagram of a controllable diffusion model provided by an embodiment of the present application;

[0052] Figure 5 is a structural schematic diagram of a defect identification device for defect data provided by an embodiment of the present application;

[0053] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same or similar components are designated by the same or similar reference numerals, and the description thereof will not be repeated. The following exemplary embodiments described in the following description are not presented to cause ambiguity with respect to the scope of the present application. Instead, they are presented merely to most effectively explain the exemplary embodiments of the present application, according to which the scope of the present application is outlined in the appended claims.

[0055] It should be noted that, in the present document, the terms "comprise", "include", or any other variant thereof are intended to mean a non-exclusive inclusion, such that processes, methods, articles, or apparatuses comprising a list of elements are not limited to those elements, but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose additional identical elements from being present in the process, method, article, or apparatus that includes the element. In addition, components, features, elements, or steps with the same name in different embodiments of the present application can have the same or different meanings, and the specific meaning thereof should be determined in the light of the explanation thereof in the specific embodiment or further in the light of the context in the specific embodiment.

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0057] In the following description, the suffixes "module", "part", or "unit" used for an element are merely intended for facilitating the description of the present application, and are not intended to limit the present application. Therefore, "module", "part", or "unit" can be mixedly used.

[0058] With the continuous development of automated detection technology, deep learning technology is widely used in image classification and defect detection fields, and convolutional neural network (CNN) and other technologies play an important role in the fields. However, the existing technology still faces many challenges in practical application. On the one hand, the problem of data imbalance is prominent, and most methods focus on adjusting the attention of the model to different class samples by using a reweighted loss function. In specific application scenarios such as nuclear power plants, due to the scarcity of minority class defect samples, the model tends to focus on the majority class samples during the training process, resulting in low recognition accuracy of minority class defects. On the other hand, the problem of sample scarcity seriously restricts the training effect of the model. The defect categories are complex and diverse, and the number of some category samples is extremely small, which makes it difficult for traditional detection methods to obtain enough samples for effective training. In addition, the existing generation model has difficulty in ensuring the semantic consistency between the generated image and the target category when generating defect images. In the case of sample scarcity, the generated images often have deviations and cannot accurately reflect the actual defect characteristics, further affecting the performance improvement of the model.

[0059] To solve the above technical problems and overcome the defects of the prior art, the embodiments of the present application provide a defect recognition method, device, equipment and medium for defect data, which dynamically adjusts the attention weight and generates synthetic data through multi-modal fusion, effectively solving the problems of data imbalance and sample scarcity existing in the prior art, and significantly improving the accuracy and robustness of defect recognition for defect data.

[0060] Figure 1 An application environment diagram for a defect recognition method for defect data in an embodiment. Referring to Figure 1 The defect recognition method for defect data is applied to a defect recognition system for defect data. The defect recognition system for defect data includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network, and the terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a notebook computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The server 120 can be used to dynamically adjust the attention weight of the defect recognition model to difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in the training sample set; generate a fusion control signal by extracting the text features and image features corresponding to the input data through a multi-modal encoder; generate synthetic defect data consistent with the semantics of the target defect based on the fusion control signal and the attention weight through a controllable diffusion model; mix the synthetic defect data with the original training sample set to construct a balanced data set, and train the defect recognition model based on the balanced data set; and use the trained defect recognition model to classify and locate defects in the input image, and output the defect category and location information.

[0061] Please refer toFigure 2 , Figure 2 is a flowchart of a defect recognition method for defect data provided by an embodiment of the present application. The embodiment mainly takes the defect recognition method for defect data applied to a computer device as an example to illustrate. The defect recognition method for defect data provided by an embodiment of the present application can specifically include the following steps:

[0062] S1. Dynamically adjusting the attention weight of the defect recognition model to difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in the training sample set;

[0063] Specifically, for step S1, by analyzing the feature distribution and inter-class relationship of the samples in the training sample set, the prototype features of each category are calculated, and the interpolation disturbance features are generated by inter-class disturbance of the feature vector of each sample. The difficult-to-classify weight is dynamically calculated according to the classification confidence of the disturbed feature vector. When the confidence is lower than the preset threshold, the weight is increased, and the loss function is reconstructed to dynamically weight the training of the model, so that the model pays more attention to the difficult-to-classify samples. In addition to the existing interpolation disturbance generation method, other feature disturbance methods such as the adversarial sample generation technology can be introduced to further enhance the learning ability of the model to the difficult-to-classify samples. At the same time, the weight calculation method can also combine more sample feature information and inter-class relationship, and use more complex machine learning algorithms to dynamically adjust the weight.

[0064] It should be noted that the difficult-to-classify samples in the embodiment refer to the samples close to the category boundary. These samples are close to the features of multiple categories due to their features, which leads to confusion in classification by the classification model and makes it difficult to accurately determine the category to which they belong. In the field of defect recognition, such samples often have complex morphology or features and have certain similarity with other defect categories, making it difficult for traditional classification methods to accurately identify and classify them. For example, in the surface defect recognition of a nuclear power plant containment, some small cracks or network cracking defects may have similar morphology to other non-defect features or slight damage features, which may lead to misjudgment or low confidence in the identification of the classification model. Such samples need to be given higher weights in the training process to enhance the learning and recognition ability of the model to them, so as to improve the accuracy and robustness of the overall classification.

[0065] S2. Extracting text features and image features corresponding to the input data through a multi-modal encoder to generate a fusion control signal;

[0066] Specifically, for step S2, the defect description text is converted into a text embedding vector using a text encoder, and the feature vector of the real defect image is extracted by an image encoder. Then, the two are spliced and input into a feature cross-attention module for cross-modal fusion to generate a fusion control signal, realizing the effective combination of text features and image features. In addition, different encoder architectures and feature fusion strategies can be combined, such as using more advanced pre-training models to extract text and image features, or using multi-layer feature fusion to better preserve and utilize feature information at different levels. In addition, techniques such as attention mechanisms can be combined to further enhance the effectiveness of feature fusion.

[0067] S3. Based on the fusion control signal and the attention weight, a controllable diffusion model is used to generate synthetic defect data consistent with the semantic of the target defect;

[0068] Specifically, for step S3, a controllable diffusion model is constructed based on a stable diffusion framework, including a UNet encoder, a ControlNet branch, and a VAE decoder. The fusion control signal is injected into the diffusion process through the ControlNet branch to adjust the noise prediction path, where the fusion control signal includes an attention weight indicating the generation priority of difficult-to-classify defect samples. When performing iterative denoising in the latent space, the denoising step or noise intensity is dynamically adjusted according to the attention weight, and synthetic defect data corresponding to difficult-to-classify defects is preferentially generated. The denoised latent features are mapped to synthetic defect samples through the VAE decoder. In addition to the stable diffusion model, other generation model architectures such as generative adversarial networks (GAN) can be used to improve and optimize the quality and diversity of synthetic defect data by combining their respective advantages. At the same time, the noise adjustment strategy in the diffusion process can be further studied to develop more effective dynamic adjustment methods.

[0069] S4. Mix the synthetic defect data with the original training sample set to construct a balanced data set, and train the defect recognition model based on the balanced data set;

[0070] Specifically, for step S4, the synthetic defect data is mixed with the original training sample set at a pre-set ratio to construct a balanced data set to balance the distribution of samples of each class. The balanced data set is preprocessed, and then a joint training strategy is used to alternately optimize the classification loss and the generative adversarial loss of the defect recognition model based on the preprocessed balanced data set, so that the model can learn the relevant features of the generation task while learning the classification task, thereby obtaining the trained defect recognition model. Data mixing strategies and preprocessing methods can be further optimized to better balance the data set and improve data quality. At the same time, in the joint training strategy, different loss function combinations and optimization algorithms can be tried to improve the training efficiency and performance of the model.

[0071] S5. Classify and locate defects in the input image using the trained defect recognition model, output the defect category and location information;

[0072] Specifically, for step S5, multi-scale feature extraction is performed on the input image to generate feature maps, which are input into the classification head and regression head of the defect recognition model respectively, and the defect category probability and defect bounding box coordinates are output to realize the classification and localization of defects in the input image. In addition, more advanced multi-scale feature extraction methods and model architectures can be used to further improve the efficiency and effectiveness of feature extraction. At the same time, more innovative techniques such as attention mechanisms can be introduced for the design and optimization of the classification head and regression head to improve the prediction performance of the model.

[0073] The embodiment solves the data imbalance problem by dynamically adjusting the attention weight of difficult-to-classify samples, improves the recognition accuracy of minority class samples; at the same time, uses multi-modal feature fusion and controllable diffusion model to generate synthetic defect data, solves the sample scarcity problem, and enriches the training data; finally, the model performance is optimized through the joint training strategy, so that the defect recognition model can realize efficient and accurate recognition in classification and positioning tasks, and has significant technical effects.

[0074] Optionally, in some embodiments, step S1 "dynamically adjusting the attention weight of difficult-to-classify samples in the defect recognition model based on the feature distribution and inter-class relationship of each sample in the training sample set" can specifically include:

[0075] S11. Extract the feature vector of each sample in the training sample set, and calculate the prototype feature of each category;

[0076] Specifically, the feature vector of each sample is extracted from the training sample set, and then the sample feature vectors of each category are averaged to obtain the prototype feature of the category. The purpose of this step is to capture the core feature representation of each category. More complex feature aggregation methods such as clustering algorithms (K-means) or deep learning-based feature encoders can be used to generate category prototype features. In addition, online clustering methods can be introduced to update the category prototype features in real time to adapt to changes in data distribution.

[0077] S12. For each sample's feature vector, randomly select at least one out-of-class prototype for interpolation perturbation to generate a perturbed feature vector;

[0078] Specifically, for each sample, a prototype feature not belonging to the category of the sample (out-of-class prototype) is randomly selected, and then a perturbed feature vector is generated by an interpolation method. The size of the interpolation coefficient γ determines the degree of perturbation. Different interpolation methods can be used, such as linear interpolation, spherical interpolation, etc., or an adversarial perturbation generation technique can be introduced to generate more challenging perturbed feature vectors. In addition, the interpolation coefficient γ can be dynamically adjusted in combination with the classification difficulty of the sample.

[0079] S13. Based on the confidence of the perturbed feature vector in the classification model, the difficult classification weight of the sample is dynamically calculated, wherein the weight is increased when the confidence is lower than a preset threshold;

[0080] Specifically, the perturbed feature vector is input into the classification model, and the classification confidence is calculated. If the confidence is lower than a preset threshold, it is considered that the sample is close to the category boundary and belongs to a difficult classification sample, and the weight is increased. More complex confidence evaluation methods can be used, such as integrating the prediction results of multiple models for integrated judgment. In addition, an adaptive threshold adjustment mechanism can be introduced to dynamically adjust the threshold according to the sample distribution in the training process.

[0081] S14. Based on the difficult classification weight, the loss function is reconstructed, and the defect recognition model is dynamically weighted and trained;

[0082] Specifically, on the basis of the traditional loss function, the difficult classification weight is introduced, and the loss of each sample is weighted. The sample with larger weight occupies a larger proportion in the loss calculation, thereby guiding the model to pay more attention to these difficult classification samples. Different loss function reconstruction methods can be used, such as combining focal loss function (Focal Loss) or other adaptive loss functions. In addition, a multi-task learning framework can be introduced to simultaneously optimize the classification task and the feature learning task.

[0083] The embodiment accurately identifies and quantifies the features of difficult classification samples by extracting category prototype features and generating perturbed feature vectors, dynamically adjusts the weight of the samples in model training, effectively solves the data imbalance problem, makes the model pay more attention to the difficult classification samples near the category boundary, and thereby improves the recognition ability of the classifier for minority class samples and the overall classification performance.

[0084] Optionally, in some embodiments, step S2 "extracting text features and image features corresponding to the input data through a multi-modal encoder to generate a fusion control signal" can specifically include:

[0085] S21. Converting the defect description text into a text embedding vector through a text encoder;

[0086] Specifically, the input defect description text is converted into a fixed-length text embedding vector using a pre-trained text encoder such as CLIP text encoder. This process converts text information into numerical form that the model can process. In addition to CLIP, other advanced text encoders such as BERT, RoBERTa, etc. can also be used to obtain more rich text semantic information. In addition, the text encoder can be fine-tuned in combination with domain knowledge to better adapt to the characteristics of specific defect descriptions.

[0087] S22. Extract the feature vector of the real defect image through the image encoder;

[0088] Specifically, the feature vector of the real defect image is extracted using a pre-trained image encoder such as ResNet, EfficientNet, etc. This process converts the visual information of the image into numerical features for fusion with features from other modalities. Alternatively, other image encoders such as Vision Transformer (ViT) or Swin Transformer can be used to obtain higher quality image feature representations. In addition, data augmentation techniques such as random cropping, rotation, etc. can be combined to improve the robustness of image features.

[0089] S23. After concatenating the text embedding vector and the image feature vector, input them into the feature cross principal stress module for cross-modal fusion to generate a fusion control signal;

[0090] Specifically, the text embedding vector and the image feature vector are concatenated to form a multi-modal feature vector. Then, the concatenated feature vector is input into the feature cross attention module, which dynamically adjusts the weights of different modal features through the attention mechanism to generate a fusion control signal. In addition, different feature fusion strategies can be used, such as weighted concatenation, multi-layer perception (MLP) fusion, etc. In addition, multi-modal feature alignment techniques can be introduced to ensure the consistency of text and image features in the semantic space.

[0091] This embodiment encodes the text description and image features into embedding vectors respectively, and uses the feature cross attention module for cross-modal fusion to generate a fusion control signal that can accurately guide the generation of defect data, effectively integrating the multi-modal information of text and image, improving the semantic consistency and quality of the generated data, and providing more rich and accurate data support for the training of the defect recognition model, thereby improving the recognition performance and generalization ability of the model.

[0092] Optionally, in some embodiments, the step S3 "generating synthetic defect data consistent with the target defect semantics based on the fusion control signal and the attention weight through the controllable diffusion model" can specifically include:

[0093] S31. Construct a controllable diffusion model based on the stable diffusion framework, the controllable diffusion model comprising a UNet encoder, a ControlNet branch and a VAE decoder;

[0094] Specifically, the controllable diffusion model comprises a UNet encoder, a ControlNet branch and a VAE decoder. The UNet encoder is used to extract latent features, the ControlNet branch is used to inject control signals, and the VAE decoder is used to map latent features to image space. In addition, model compression techniques such as quantization or pruning can be combined to improve the running efficiency of the model.

[0095] S32. In the controllable diffusion model, inject the fusion control signal through the ControlNet branch into the diffusion process to adjust the noise prediction path, wherein the fusion control signal comprises an attention weight for indicating the generation priority of the difficult-to-classify defect sample;

[0096] Specifically, in the diffusion process, the ControlNet branch receives and processes the fusion control signal, including text embedding vectors, semantic segmentation maps and image features. These signals are injected into different levels of UNet to adjust the noise prediction path and guide the generation process. Different signal injection methods can be used, such as multi-level injection or dynamic weight adjustment injection, to further improve the influence of the control signal. In addition, attention mechanisms can be introduced to make the model pay more attention to important parts of the control signal.

[0097] S33. When performing iterative denoising in latent space, dynamically adjust the denoising step or noise intensity according to the attention weight, and preferentially generate synthetic defect samples corresponding to difficult-to-classify defects;

[0098] Specifically, in the iterative denoising process in latent space, the denoising step or noise intensity is dynamically adjusted according to the attention weight in the fusion control signal. Samples with higher attention weights are given more attention and optimization during the generation process. An adaptive adjustment strategy can be introduced to adjust the denoising step or noise intensity in real time according to feedback information during the generation process. In addition, reinforcement learning methods can be combined to optimize the adjustment strategy to improve the generation quality.

[0099] S34. Map the denoised latent features to synthetic defect samples through the VAE decoder, and output synthetic defect data consistent with the semantics of the target defect;

[0100] Specifically, after the denoising process is completed, the latent features are mapped to the image space using the VAE decoder to generate synthetic defect samples consistent with the target defect semantics. The VAE decoder plays a key role in converting low-dimensional features to high-dimensional images in this process. More advanced decoder architectures such as PixelCNN or Transformer decoder can be used to improve the details and quality of the generated images. In addition, super-resolution techniques can be combined to further improve the resolution of the generated images.

[0101] The embodiment realizes precise control of the synthetic defect data generation process by constructing a controllable diffusion model based on the stable diffusion framework and combining ControlNet branch and VAE decoder. By dynamically adjusting the denoising step or noise intensity, the model can generate high-quality, semantically consistent synthetic defect samples according to the attention weight in the fusion control signal, effectively solving the sample scarcity problem, improving the diversity and quality of the model training data, and significantly improving the recognition accuracy and generalization ability of the defect recognition model, which has important practical application value.

[0102] Optionally, in some embodiments, the step S4 "mixing synthetic defect data with the original training sample set to construct a balanced data set, and training a defect recognition model based on the balanced data set" can specifically include:

[0103] S41. Mix the original samples and synthetic defect data in a predetermined proportion to construct a balanced data set for balancing the distribution of samples of each category;

[0104] Specifically, according to the number of samples of each category in the data set, a suitable mixing ratio is determined, and the synthetic defect data is added to the original training sample set in proportion to balance the number of samples of each category. A dynamic mixing ratio can be used to adjust the proportion of synthetic data and original data in real time according to the performance of the model during training (such as loss value, accuracy, etc.). In addition, clustering analysis can be combined to group and mix data to better balance the local data distribution.

[0105] S42. Preprocess the balanced data set to obtain a preprocessed balanced data set;

[0106] Specifically, the preprocessing step usually includes data cleaning, normalization, data augmentation, etc. Data cleaning is used to remove noise and outliers; normalization adjusts the data to a uniform numerical range; data augmentation expands the diversity of the data set through rotation, flipping, cropping, etc. Data augmentation techniques such as intelligent cropping, style conversion, etc. can also be introduced, combined with deep learning generative adversarial networks (GAN) for data augmentation. In addition, adaptive normalization methods can be used to dynamically adjust the normalization parameters according to the data distribution.

[0107] S43. Adopting a joint training strategy to alternately optimize the classification loss and the generative adversarial loss of the defect recognition model based on the preprocessed balanced dataset, to obtain the trained defect recognition model;

[0108] Specifically, the joint training strategy combines the loss functions of the classification task and the generation task. During training, the classification loss (such as cross-entropy loss) and the generative adversarial loss (such as GAN loss) are alternately optimized, so that the model can learn both classification features and generation features. Different joint training strategies can be used, such as multi-task learning, progressive training, etc. In addition, reinforcement learning techniques are introduced to dynamically adjust the loss weights during training through a reward mechanism.

[0109] This embodiment constructs a balanced dataset by reasonably mixing original samples and synthetic defect data, and optimizes the defect recognition model using a joint training strategy, effectively solving the data imbalance problem and improving the model's recognition ability for all classes. At the same time, the combination of preprocessing and joint training strategy further improves the model's generalization ability and robustness, making the defect recognition model more stable and accurate in defect classification and positioning in practical applications.

[0110] Optionally, in some embodiments, the step S5 "utilizing the trained defect recognition model to perform defect classification and positioning on the input image, and outputting defect class and position information" can specifically include:

[0111] S51. Perform multi-scale feature extraction on the input image to generate corresponding feature maps;

[0112] Specifically, a convolutional neural network (CNN) is used to perform multi-scale feature extraction on the input image, generating multi-resolution feature maps through different levels of convolution operations. These feature maps can capture feature information at different scales, providing rich feature representations for subsequent classification and positioning tasks. In addition, attention mechanisms such as channel attention or spatial attention can be combined to further enhance the expressiveness of the feature maps.

[0113] S52. Input the feature maps into the classification head and regression head of the defect recognition model respectively, and output the defect class probability and defect bounding box coordinates;

[0114] Specifically, the generated multi-scale feature maps are input into the classification head and regression head of the model respectively. The classification head is used to predict the class probability of the defect, and the regression head is used to predict the bounding box coordinates of the defect. The classification head usually uses a fully connected layer or a convolutional layer combined with a softmax function to output the class probability, and the regression head uses a fully connected layer or a convolutional layer to output the coordinate values of the bounding box. In addition, a multi-task learning framework can be introduced to enable the classification and regression tasks to share more feature information, improving the overall performance of the model.

[0115] The classification head outputs the probability of each pixel or region belonging to different defect categories, and the regression head outputs the bounding box coordinates of each detected defect. Through post-processing steps such as non-maximum suppression (NMS), redundant bounding boxes are removed, and finally the class and location information of each defect is obtained.

[0116] This embodiment realizes efficient classification and accurate positioning of defects in the input image through multi-scale feature extraction and separate processing of classification and regression. Multi-scale feature extraction ensures that the model can capture defect features of different scales, improving the accuracy of recognition. The separate processing of classification and regression enables the model to output the class and location information of the defect simultaneously, enhancing the comprehensiveness and practicality of the model.

[0117] Optionally, in some embodiments, step S51 "performing multi-scale feature extraction on the input image to generate corresponding feature maps" can specifically include:

[0118] S511. Perform multi-scale feature extraction on the input image to generate multi-resolution feature maps through different levels of convolutional neural networks;

[0119] Specifically, different levels of convolutional neural networks (CNN) are used to extract features from the input image to generate multi-resolution feature maps. These feature maps can capture feature information of the image at different scales, providing rich feature representations for subsequent classification and positioning tasks. More advanced feature extraction networks such as ResNet, DenseNet or EfficientNet can be used, which can extract higher quality features by introducing residual connections, dense connections or more efficient convolution operations. In addition, attention mechanisms such as channel attention or spatial attention can be combined to further enhance the expression ability of the feature maps.

[0120] S512. Construct a corresponding feature pyramid based on the multi-resolution feature maps;

[0121] Specifically, the feature maps of different levels are combined into a feature pyramid, each level corresponding to a feature map of different resolution. The feature pyramid can provide multi-scale feature representation from global to local. For example, feature pyramid network (FPN), panoramic feature pyramid (PANet) and the like, to better integrate feature information of different levels. In addition, dynamic feature fusion technology can be introduced to dynamically adjust the weights of features of different levels according to task requirements.

[0122] S513. For each layer feature map in the feature pyramid, calculate the response weight of the defect region using a spatial attention mechanism to generate an attention mask;

[0123] Specifically, at each level of the feature pyramid, a spatial attention mechanism is applied to calculate the response weight of each position, generating an attention mask. The attention mask is used to highlight the features of the defect area and suppress the features of the background area. Attention mechanisms such as channel attention, dual attention mechanism, etc. are adopted to further enhance the model's focusing ability on the defect area. In addition, an adaptive attention mechanism can be introduced to dynamically adjust the attention weight according to the complexity of the defect.

[0124] S514. Element-wise multiplication of the attention mask and the corresponding feature map is performed to obtain an enhanced feature map;

[0125] Specifically, the generated attention mask is element-wise multiplied with the corresponding feature map to enhance the features of the defect area and suppress the features of the background area. The enhanced feature map can more prominently represent the feature information of the defect area. Feature fusion strategies such as weighted fusion, multi-layer perception (MLP) fusion, etc. can be used to further improve the effect of feature enhancement. In addition, context information such as global average pooling or maximum pooling can be combined to enhance the semantic information of the features.

[0126] This embodiment realizes precise positioning and feature enhancement of the defect area in the input image through multi-scale feature extraction and feature pyramid construction, combined with a spatial attention mechanism. Multi-scale feature extraction and feature pyramid construction ensure that the model can capture defect features of different scales, improving the accuracy of recognition. The introduction of the spatial attention mechanism further highlights the features of the defect area, enhancing the model's robustness to complex backgrounds.

[0127] In order to facilitate understanding of the defect recognition method for defect data provided by the present embodiment, the present application is described taking the containment defect intelligent recognition system of a nuclear power plant as an example,

[0128] The containment defect intelligent recognition system of a nuclear power plant is based on deep learning and image processing technology, aiming to efficiently and accurately recognize various defects on the surface of the containment. Its core includes an adaptive focus loss function based on inter-class interaction (ICIL) and a defect generation algorithm based on a stable diffusion model (DD) to solve problems such as data imbalance and sample scarcity. The specific implementation process is as follows:

[0129] (1) Data preparation

[0130] Collect image data: Regularly use drones, robots, and other equipment to take panoramic photos of the containment to obtain a large number of surface images, ensuring that the images cover different lighting, angles, and defect types.

[0131] Label defect information: Organize experts to label the collected images, marking the location and category of cracks, holes, and other defects to form the labeled data required for training.

[0132] (2) Model training

[0133] Feature extraction and class prototype calculation: Use deep learning framework to extract image features and calculate class prototype features, providing a basis for subsequent feature interpolation attacks.

[0134] Apply ICIL loss function: Dynamically adjust loss weights according to the distance of samples to classification boundaries, so that the model pays more attention to difficult-to-classify samples, alleviates data imbalance problems, and improves the model's recognition ability for minority defects.

[0135] Build DD model: Combine text semantics and image features to generate control signals, use stable diffusion model to generate high-quality synthetic defect images, expand defect dataset, especially increase the number of samples of rare defect categories.

[0136] (3) Data augmentation

[0137] Image enhancement: Enhance the original image through random cropping, rotation, flipping, etc., to improve the model's adaptability to different angles and morphological defects.

[0138] Synthetic defect data generation: Use DD model to generate synthetic defect data and mix it with original data to form a balanced dataset for further model training.

[0139] (4) Model evaluation and optimization

[0140] Evaluation index calculation: Calculate accuracy, recall rate, F1 value, etc. on the validation set to evaluate model performance.

[0141] Model optimization: Adjust model parameters and training strategies according to evaluation results, such as learning rate adjustment, regularization parameter optimization, etc., to improve model recognition accuracy and generalization ability.

[0142] (5) Deployment and application

[0143] Deploy the model: Deploy the trained model to the actual detection system, such as installing it on a detection robot or edge computing device.

[0144] Real-time monitoring and analysis: The detection device regularly takes images of the containment surface and transmits them to the system, which uses the deployed model to analyze the images in real time, quickly identifies defects and issues alerts, helping to handle timely maintenance.

[0145] Through the above implementation process, the nuclear power plant containment defect intelligent identification system can efficiently and accurately identify defects on the containment surface, effectively improving the safety of nuclear power plants. The system uses ICIL and DD methods to solve the problems of data imbalance and sample scarcity, improving the model's recognition accuracy and robustness, ensuring that nuclear power plants can timely discover and handle defects on the containment surface, ensuring its safe and stable operation.

[0146] In some embodiments, the ICIL loss function implementation process is as follows:

[0147] Step 1: Construct category prototype features, extract feature vectors of all training samples, and calculate prototype features of each category

[0148]

[0149] where N j is the number of samples of category j.

[0150] Step 2: Inter-class perturbation feature interpolation, sample feature f(x i ), randomly select an out-of-class prototype p k (k≠y i ), generate interpolation perturbation features:

[0151]

[0152] Where γ is the minimum interpolation coefficient so that the model The classification confidence of is lower than the threshold θ.

[0153] Step 3: Sample adaptive reweighting, calculate sample difficulty weight:

[0154] w i =1-γ i

[0155] where γ i To satisfy the minimum interpolation condition, the weight w i The larger the value, the more difficult it is to classify the sample.

[0156] Reconstructing the focal loss function:

[0157]

[0158] The core process of the ICIL algorithm is as follows:

[0159]

[0160] In some embodiments, the DD defect generation algorithm architecture includes:

[0161] Module 1: Defect Perception Prompt Extraction (DAPE)

[0162] Input defect description text (such as "transverse crack") and generate text embeddings through CLIP text encoder t .

[0163] Extract the real defect image features f imgAfter splicing with the text embedding, a fine-grained control signal c = [e t ; f img ] is generated.

[0164] Module 2: Controllable diffusion generation network

[0165] Based on the Stable Diffusion framework, the ControlNet branch is introduced:

[0166] Conditional input: text embedding e t , semantic segmentation map S, image feature f img .

[0167] Control branch: inject the conditional signal into the middle layer of UNet through a zero convolution layer to adjust the diffusion process.

[0168] Loss function design:

[0169]

[0170] Where ∈ θ is the noise prediction network, and c is the control signal.

[0171] In some embodiments, as Figure 3 shown, Figure 3 is the core process of defect data generation and recognition in the nuclear power plant containment defect intelligent recognition system, which includes two main parts: the label model and the controllable T2I diffusion model, which are explained as follows:

[0172] (a) Label model

[0173] Background image and defect image input: the model receives the background image and the defect image as input, where the background image is the part of the containment surface without defects, and the defect image contains a specific type of defect.

[0174] Image encoder: the image encoder extracts features from the input image and converts it into a feature vector representation. For example, after processing the input "background image and defect image" through the image encoder, a feature vector is output.

[0175] Category text: the category text provides the type information of the defect, such as "cracks". This text information is combined with the image features for subsequent feature fusion and label generation.

[0176] Output label: the feature vector processed by the image encoder is combined with the category text to output the corresponding defect label, such as "cracks", completing the recognition and classification of defects in the image.

[0177] (b) Controllable T2I diffusion model

[0178] Background image input: The background image is input to generate a defect map related to it.

[0179] Label model: The label model also receives the background image and defect map in this section, extracts image features, and outputs the category text "cracks".

[0180] Image encoder: Feature extraction is performed on the background image to generate an image feature vector.

[0181] Text encoder: The category text "cracks" is converted into a text embedding vector for fusion with image features.

[0182] TCA (Text Cross-Attention Module): Receives text embedding vectors and image feature vectors, and through cross-attention mechanism, enables text features to interact with image features, enhancing feature fusion effect.

[0183] RCA (Feature Cross-Attention Module): Further processes features from TCA, optimizes feature fusion through feature cross-attention mechanism, and highlights important feature information.

[0184] Image branch, text embedding branch, image feature branch: Represent the processing paths of image features, text embedding features, and fused image features, respectively.

[0185] Controllable T2I diffusion model: Core part, uses the above fused features to generate a defect map related to the input background image. This model generates images step by step through a diffusion process guided by a control signal to generate a defect image that meets the requirements.

[0186] Generate defect map: The final output defect map has the same background as the input background image, but contains specific types of defects such as cracks.

[0187] The overall process of the nuclear power plant containment defect intelligent identification system is as follows:

[0188] Data input: The system receives the background image and defect map as input.

[0189] Feature extraction: Extract image features through image encoder and text features through text encoder.

[0190] Feature fusion: Use TCA and RCA modules to deeply fuse image features and text features.

[0191] Diffusion model generation: Fuse features into controllable T2I diffusion model to generate a defect map related to the background image.

[0192] Output results: The generated defect map can be used as training data to improve the performance of the defect recognition model.

[0193] In some embodiments, as Figure 4 shown, Figure 4 is a system architecture for a controllable T2I diffusion model, used to generate synthetic defect data based on text and image features, suitable for containment surface defect recognition scenarios in nuclear power plants. Key components include:

[0194] Text encoder: used to convert text descriptions (such as "cracks") into text embedding vectors, representing the semantic features of the text.

[0195] Image encoder: converts input image data into image feature vectors, representing the visual features of the image.

[0196] Unet encoder: composed of multiple encoding layers, extracts image features and forms a feature pyramid representation. Each encoding layer contains convolution operations and activation functions for feature extraction and nonlinear transformation. The image features are initialized to provide an initial latent representation for the diffusion model.

[0197] ControlNet: a control network that works in parallel with the Unet encoder, used to receive and process signals from the text embedding and image feature branches, aiming to fuse text and image features and guide the generation process of the diffusion model.

[0198] Image branch, text embedding branch, image feature branch: represent the processing paths of image features, text embedding features and image features respectively.

[0199] TCA (Text Cross Attention Module), RCA (Feature Cross Attention Module): used for feature interaction and fusion between different feature branches. The TCA module interacts text features with image features, and the RCA module further fuses and optimizes these features.

[0200] VAE decoder: used to decode latent features into image data, restoring the latent feature representation generated by the diffusion model into image form.

[0201] Controllable T2I diffusion model: the core model for generating synthetic images based on text and image features. Through the diffusion process, the image is gradually generated, and the fusion of text and image features is used to control the content of the generated image.

[0202] The entire processing process starts from text and image input, extracts features through the encoder, interacts and fuses features through the cross attention module, controls the diffusion process through the ControlNet, and finally generates synthetic defect images through the VAE decoder, realizing image generation based on text description, used for data augmentation and defect recognition model training.

[0203] Compared with the prior art, the defect identification method for defect data provided by the embodiment can dynamically improve the sensitivity of the model to rare defects through the ICIL loss function, avoid the bias of the classifier to the majority class, and solve the data imbalance problem; the DD algorithm is used to supplement the rare defect data, break through the modal collapse and controllability bottleneck of the traditional generative model (such as GAN), and generate high-quality defect samples; the feature learning and data enhancement are jointly optimized, the recognition ability of the model to complex defects (such as micro cracks and network cracking) is enhanced, and the detection precision and generalization are improved.

[0204] It should be understood that, although Figure 2 The steps in the flowchart of the method for identifying defects of defect data are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the method for identifying defects of defect data can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0205] In order to better implement the method for identifying defects of defect data, the embodiment of the application also provides a defect identification device for defect data based on the above-mentioned method for identifying defects of defect data. The meanings of the terms are the same as in the above-mentioned method for identifying defects of defect data, and the specific implementation details can be referred to the description in the method embodiment.

[0206] Please refer to Figure 5 , Figure 5 The structure diagram of the defect identification device for defect data provided by the embodiment of the application, which can specifically include a weight adjustment module 201, a signal generation module 202, a data generation module 203, a model training module 204 and a defect identification module 205, and can be as follows:

[0207] The weight adjustment module 201 is configured to dynamically adjust the attention weight of the defect identification model to difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in the training sample set.

[0208] The signal generation module 202 is configured to extract text features and image features corresponding to input data through a multi-modal encoder, and generate a fusion control signal.

[0209] The data generation module 203 is configured to generate synthetic defect data consistent with the semantic of the target defect based on the fusion control signal and the attention weight through a controllable diffusion model.

[0210] The model training module 204 is configured to mix the synthetic defect data with the original training sample set to construct a balanced data set, and train the defect recognition model based on the balanced data set.

[0211] The defect recognition module 205 is configured to use the trained defect recognition model to classify and locate defects in an input image, and output defect category and location information.

[0212] Optionally, in some embodiments, the weight adjustment module 201 can specifically include:

[0213] The feature calculation unit is configured to extract feature vectors of all samples in the training sample set, and calculate prototype features of each category.

[0214] The perturbation unit is configured to randomly select at least one out-of-class prototype to perform interpolation perturbation on the feature vector of each sample, to generate a perturbed feature vector.

[0215] The weight calculation unit is configured to dynamically calculate the difficult classification weight of the sample according to the confidence of the perturbed feature vector in the classification model, wherein the weight increases when the confidence is lower than a preset threshold.

[0216] The reconstruction unit is configured to reconstruct a loss function based on the difficult classification weight, and dynamically weight train the defect recognition model.

[0217] Optionally, in some embodiments, the signal generation module 202 can specifically include:

[0218] The conversion unit is configured to convert the defect description text into a text embedding vector through a text encoder.

[0219] The extraction unit is configured to extract a feature vector of a real defect image through an image encoder.

[0220] The fusion unit is configured to splice the text embedding vector and the image feature vector, and input them into a feature cross principal stress module for cross-modal fusion to generate a fusion control signal.

[0221] Optionally, in some embodiments, the data generation module 203 can specifically include:

[0222] The construction unit is configured to construct a controllable diffusion model based on a stable diffusion framework, wherein the controllable diffusion model includes a UNet encoder, a ControlNet branch, and a VAE decoder.

[0223] The injection unit is configured to inject a fusion control signal into a diffusion process through a ControlNet branch in a controllable diffusion model to adjust a noise prediction path, wherein the fusion control signal comprises an attention weight indicating a generation priority of a difficult-to-classify defect sample.

[0224] The generation unit is configured to dynamically adjust a denoising step or a noise intensity according to the attention weight when performing iterative denoising in the latent space, and to generate a synthetic defect sample corresponding to the difficult-to-classify defect preferentially.

[0225] The output unit is configured to map the denoised latent feature to the synthetic defect sample through a VAE decoder, and to output synthetic defect data consistent with the target defect semantics.

[0226] Optionally, in some embodiments, the model training module 204 can specifically include:

[0227] The dataset unit is configured to mix the original samples and the synthetic defect data at a preset ratio to construct a balanced dataset for balancing the distribution of samples of various categories.

[0228] The preprocessing unit is configured to preprocess the balanced dataset to obtain a preprocessed balanced dataset.

[0229] The training unit is configured to adopt a joint training strategy to alternately optimize a classification loss and a generative adversarial loss of the defect recognition model based on the preprocessed balanced dataset, and to obtain a trained defect recognition model.

[0230] Optionally, in some embodiments, the defect recognition module 205 can specifically include:

[0231] The feature extraction unit is configured to perform multi-scale feature extraction on the input image to generate a corresponding feature map.

[0232] The recognition unit is configured to input the feature map into a classification head and a regression head of the defect recognition model respectively, and to output a defect category probability and a defect bounding box coordinate.

[0233] Optionally, in some embodiments, the feature extraction unit is specifically configured to:

[0234] perform multi-scale feature extraction on the input image to generate multi-resolution feature maps through different levels of a convolutional neural network;

[0235] construct a corresponding feature pyramid based on the multi-resolution feature maps;

[0236] calculate response weights of defect regions for each layer feature map in the feature pyramid by using a spatial attention mechanism to generate an attention mask;

[0237] multiply the attention mask and the corresponding feature map element by element to obtain an enhanced feature map.

[0238] The specific limitations of the defect identification apparatus for defect data can refer to the limitations of the defect identification method for defect data described above, which will not be repeated here. Each module in the defect identification apparatus for defect data described above can be realized by software, hardware and their combination in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls to execute the operations corresponding to the above-mentioned modules.

[0239] The defect identification apparatus for defect data provided by the embodiment effectively solves the data imbalance problem by dynamically adjusting the attention weight of the defect identification model to the difficult-to-classify samples, and improves the recognition accuracy of the minority class samples. At the same time, the control signal is generated by fusing the text and image features through the multi-modal encoder, and the synthetic defect data consistent with the semantic of the target defect is generated by using the controllable diffusion model, which increases the diversity and quantity of the training data, breaks through the limitation of the model training effect caused by the sample scarcity, and improves the recognition ability of the model to complex defects.

[0240] In addition, the embodiment of the present application also provides an electronic device, as shown in Figure 6 The electronic device structure involved in the embodiment of the present application is shown in the figure, specifically:

[0241] The electronic device can include a processor 301 with one or more processing cores, a memory 302 with one or more computer readable storage media, a power supply 303 and an input unit 304, etc. Those skilled in the art can understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. Among them:

[0242] The processor 301 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, executes the software programs and / or modules stored in the memory 302 and the data stored in the memory 302, performs various functions and processes data of the electronic device, and thus monitors the whole electronic device. Optionally, the processor 301 can include one or more processing cores; preferably, the processor 301 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 301.

[0243] The memory 302 can be used to store software programs and modules, and the processor 301 executes various function applications and the defect identification method for defect data by running the software programs and modules stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 302 can also include a memory controller to provide access for the processor 301 to the memory 302.

[0244] The electronic device also includes a power supply 303 for powering the various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply status indicator, and the like.

[0245] The electronic device can also include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0246] Although not shown, the electronic device can also include a display unit and the like, which will not be described here. Specifically, in the present embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and run the application programs stored in the memory 302 by the processor 301, so as to realize various functions, as follows:

[0247] Based on the feature distribution and inter-class relationship of each sample in the training sample set, the attention weight of the defect recognition model to the difficult-to-classify samples is dynamically adjusted; the text features and image features corresponding to the input data are extracted through the multi-modal encoder to generate a fusion control signal; based on the fusion control signal and the attention weight, a synthetic defect data consistent with the semantic of the target defect is generated through a controllable diffusion model; the synthetic defect data is mixed with the original training sample set to construct a balanced data set, and the defect recognition model is trained based on the balanced data set; the trained defect recognition model is used to classify and locate the defects of the input image, and the defect category and position information are output.

[0248] The specific implementation of the above operations can refer to the previous embodiments, which will not be described here again.

[0249] The embodiments of the present application effectively solve the data imbalance problem by dynamically adjusting the attention weight of the defect recognition model to the difficult-to-classify samples, and improve the recognition accuracy of the minority class samples. Meanwhile, the text and image features are fused by the multi-modal encoder to generate a control signal, and the controllable diffusion model is used to generate synthetic defect data consistent with the semantics of the target defect, thereby increasing the diversity and quantity of the training data, breaking the limitation of sample scarcity on the model training effect, and improving the recognition ability of the model to complex defects.

[0250] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0251] Therefore, the embodiments of the present application provide a storage medium, which stores a plurality of instructions. The instructions can be loaded by a processor to execute the steps in any of the defect recognition methods for defect data provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0252] Based on the feature distribution and inter-class relationship of each sample in the training sample set, the attention weight of the defect recognition model to the difficult-to-classify samples is dynamically adjusted. The text features and image features corresponding to the input data are extracted by the multi-modal encoder to generate a fusion control signal. Based on the fusion control signal and the attention weight, the controllable diffusion model generates synthetic defect data consistent with the semantics of the target defect. The synthetic defect data and the original training sample set are mixed to construct a balanced data set, and the defect recognition model is trained based on the balanced data set. The trained defect recognition model is used to classify and locate defects in the input image, and the defect category and position information are output.

[0253] The specific implementation of the above operations can refer to the previous embodiments, which will not be described here again.

[0254] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0255] Since the instructions stored in the storage medium can execute the steps in any of the defect recognition methods for defect data provided by the embodiments of the present application, the beneficial effects of any of the defect recognition methods for defect data provided by the embodiments of the present application can be achieved. Details are described in the previous embodiments, which will not be described here again.

[0256] The above describes in detail the defect identification method, device, equipment and medium for defect data provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A defect identification method for defect data, characterized in that: The steps include: Based on the feature distribution and inter-class relationship of each sample in the training sample set, the defect recognition model dynamically adjusts the attention weight of difficult-to-classify samples; The multimodal encoder extracts the text features and image features corresponding to the input data and generates a fusion control signal; Based on the fused control signal and the attention weight, generating synthetic defect data that is semantically consistent with the target defect through a controllable diffusion model; Mixing the synthetic defect data with the original training sample set to construct a balanced data set, and training a defect recognition model based on the balanced data set; The trained defect recognition model is used to classify and locate defects in the input image, and output defect category and location information.

2. The defect identification method for defect data according to claim 1, characterized in that: The method dynamically adjusts the attention weight of the defect recognition model on difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in the training sample set, including: Extract the feature vectors of all samples in the training sample set and calculate the prototype features of each category; For each sample’s feature vector, randomly select at least one out-of-class prototype for interpolation perturbation to generate the perturbed feature vector; Dynamically calculating the difficulty-to-classify weight of the sample based on the confidence of the perturbed feature vector in the classification model, wherein the weight increases when the confidence is lower than a preset threshold; Based on the difficult-to-classify weight reconstruction loss function, the defect recognition model is dynamically weighted trained.

3. The defect identification method for defect data according to claim 1, characterized in that: The method of extracting text features and image features corresponding to input data by a multimodal encoder and generating a fusion control signal includes: Convert the defect description text into a text embedding vector through a text encoder; Extract the feature vector of the real defect image through the image encoder; After the text embedding vector and the image feature vector are spliced ​​together, they are input into the feature cross principal stress module for cross-modal fusion to generate a fusion control signal.

4. The defect identification method for defect data according to claim 1, characterized in that: The step of generating synthetic defect data that is semantically consistent with the target defect through a controllable diffusion model based on the fused control signal and the attention weight includes: Building a controllable diffusion model based on a stable diffusion framework, the controllable diffusion model includes a UNet encoder, a ControlNet branch, and a VAE decoder; In the controllable diffusion model, the fusion control signal is injected into the diffusion process through the ControlNet branch to adjust the noise prediction path, wherein the fusion control signal includes the attention weight, which is used to indicate the generation priority of difficult-to-classify defect samples; When performing iterative denoising in the latent space, dynamically adjusting the denoising step size or noise intensity according to the attention weight, and preferentially generating synthetic defect samples corresponding to difficult-to-classify defects; The denoised latent features are mapped to synthetic defect samples through the VAE decoder, and synthetic defect data with consistent semantics with the target defect is output.

5. The defect identification method for defect data according to claim 1, characterized in that: The step of mixing the synthetic defect data with the original training sample set to construct a balanced data set, and training a defect recognition model based on the balanced data set, includes: Mixing the original samples and the synthetic defect data in a preset ratio to construct a balanced data set for balancing the distribution of samples of each category; Preprocessing the balanced data set to obtain a preprocessed balanced data set; A joint training strategy is adopted based on the preprocessed balanced data set to alternately optimize the classification loss and the generative adversarial loss of the defect recognition model to obtain a trained defect recognition model.

6. The defect identification method for defect data according to claim 1, characterized in that: The defect recognition model after training is used to classify and locate defects in the input image and output defect category and location information, including: Performing multi-scale feature extraction on the input image to generate a corresponding feature map; The feature map is input into the classification head and regression head of the defect recognition model respectively, and the defect category probability and defect bounding box coordinates are output.

7. The defect identification method for defect data according to claim 6, characterized in that: The performing multi-scale feature extraction on the input image to generate a corresponding feature map includes: Performing multi-scale feature extraction on the input image and generating multi-resolution feature maps through different layers of a convolutional neural network; Constructing a corresponding feature pyramid based on the multi-resolution feature map; For each layer of feature maps in the feature pyramid, a spatial attention mechanism is used to calculate the response weight of the defect area and generate an attention mask; The attention mask is multiplied element-wise with the corresponding feature map to obtain the enhanced feature map.

8. A defect identification device for defect data, characterized in that: include: The weight adjustment module is used to dynamically adjust the attention weight of the defect recognition model for difficult-to-classify samples based on the feature distribution and inter-class relationship of each sample in the training sample set; A signal generation module is used to extract text features and image features corresponding to input data through a multimodal encoder to generate a fusion control signal; a data generation module, configured to generate synthetic defect data semantically consistent with the target defect through a controllable diffusion model based on the fused control signal and the attention weight; A model training module, configured to mix the synthetic defect data with an original training sample set to construct a balanced data set, and train a defect recognition model based on the balanced data set; The defect recognition module is used to classify and locate defects in the input image using the trained defect recognition model, and output defect category and location information.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the defect identification method for defect data according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: A computer program is stored which can be loaded by a processor and executes the defect recognition method for defect data according to any one of claims 1 to 7.

Images