Unsupervised industrial defect detection method, system, and device based on multimodal expert system
By generating high-quality pseudo-labels through a multimodal expert system and self-supervised learning, the problem of sample scarcity and labeling difficulties in industrial defect detection is solved, achieving efficient and accurate defect detection, and applicable to a variety of industrial visual inspection scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV OF TECH
- Filing Date
- 2025-06-24
- Publication Date
- 2026-06-30
AI Technical Summary
Existing industrial defect detection methods suffer from problems such as scarce samples, difficulty in annotation, and weak model generalization ability, resulting in unstable detection accuracy and insufficient utilization of multimodal information, making them difficult to adapt to complex industrial scenarios.
An unsupervised detection method based on a multimodal expert system is adopted. Through multiple parallel multimodal target detection models and self-supervised learning, combined with text prompts and prior constraints, high-quality pseudo-labels are generated, multimodal information is fused and self-supervised training is performed, and the model is gradually optimized.
It achieves improved detection accuracy and efficiency without requiring extensive manual annotation, enhances the model's generalization ability and robustness, and adapts to defect detection in complex industrial scenarios.
Smart Images
Figure CN120823457B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and industrial automation inspection technology, specifically relating to an unsupervised industrial defect detection method, system and device based on a multimodal expert system. Background Technology
[0002] Industrial defect detection, as a crucial component of intelligent manufacturing, directly impacts product quality and the automation level of production lines. Driven by the wave of industrial intelligence, deep learning-based automated defect detection technology, with its powerful feature learning and pattern recognition capabilities, has rapidly become a research hotspot in both industry and academia. However, the highly variable industrial environment makes it difficult to control the output of defect samples. Furthermore, the difficulty in acquiring defect samples and the high cost of manual annotation result in a limited number of usable labeled samples, making it challenging to guarantee the diversity of defects and the accuracy of large-scale manual annotation. This makes it difficult to directly apply conventional supervised learning methods.
[0003] Pseudo-labeling methods utilize self-supervised or semi-supervised learning. The model is pre-trained on a small amount of labeled data, and then applied to unlabeled data to generate pseudo-labels. As the model trains and optimizes, the quality of the generated pseudo-labels continuously improves. In each iteration, the model uses the updated pseudo-labels for training, further optimizing its parameters. Through multiple iterations, the model can more accurately identify defects, improving detection precision. This method significantly reduces reliance on manual annotation, lowering annotation costs. Developing a method capable of high-precision defect detection independent of labeled data is currently a hot topic and a key focus, addressing the specific needs of industrial defect detection. However, existing pseudo-labeling methods often rely on an initial model to provide initial label information. This often leads to unstable pseudo-label quality in the defect detection field; if the initial model is unstable, false positives or false negatives can easily introduce noise and cause the propagation and accumulation of erroneous information, resulting in training non-convergence. Furthermore, existing industrial defect detection methods do not make sufficient use of multimodal information, often generating pseudo-labels based solely on single-modal image data, ignoring the complementarity between textual semantics, structural information, and visual features; and they lack mechanisms for incorporating knowledge from the defect detection domain (such as defect size and morphological information). In situations with few samples or complex industrial scenarios, they have poor adaptability to the diversity of defects and environmental changes, weak model generalization ability, and difficulty in ensuring the stability and reliability of detection accuracy. Summary of the Invention
[0004] The purpose of this invention is to address the above-mentioned problems by proposing an unsupervised industrial defect detection method, system, and device based on a multimodal expert system, in order to solve the problems of sample scarcity, labeling difficulties, and weak model generalization ability in industrial defect detection, thereby improving the accuracy and efficiency of detection.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] This invention proposes an unsupervised industrial defect detection method based on a multimodal expert system, comprising the following steps:
[0007] S1. Obtain an unlabeled industrial defect image dataset and set corresponding text prompts for different defect categories in the industrial defect image dataset;
[0008] S2. Establish a multimodal expert system composed of multiple parallel multimodal target detection models, and input the industrial defect image dataset and text prompts into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the predicted bounding boxes and confidence scores of candidate defect regions.
[0009] S3. After fusing the prediction results of each multimodal target detection model, the preliminary annotation results are selected.
[0010] S4. Filter the preliminary annotation results based on prior constraints to obtain refined labels;
[0011] S5. Input the industrial defect image dataset and the fine labels into the self-supervised model for training, and output pseudo labels after training is completed;
[0012] S6. Use dynamic thresholds and prior constraints to perform dual filtering of pseudo-labels;
[0013] S7. Input the industrial defect image dataset and the pseudo-labels after double filtering in the current training into the self-supervised model after the previous training, train it again, output pseudo-labels, return to execute step S6, until convergence or the preset number of training times is reached, and obtain the trained self-supervised model.
[0014] S8. Use the trained self-supervised model to predict the industrial defect image to obtain the corresponding industrial defect detection results.
[0015] Preferably, the text prompt consists of several strings, including at least one of the defect's shape, color, texture, and brightness.
[0016] Preferably, the multimodal expert system includes three multimodal target detection models: the Grounding DINOv2 model, the Florences2 model, and the YOLOE model.
[0017] Preferably, the prediction results of each multimodal target detection model are fused, as follows:
[0018] Overlap judgment is performed on the prediction results of each multimodal target detection model. If the predicted bounding box of the candidate defect region appears in more than half of the multimodal target detection models and the Intersection over Union (IOU) between at least two multimodal target detection models exceeds the set threshold, the result is considered. If the corresponding predicted box is located in the candidate defect region, it is considered a valid candidate and is retained as a candidate target region. IOU stands for Intersection over Union.
[0019] The coordinates and confidence scores of the predicted bounding boxes are fused for the retained candidate target regions, as shown in the following formula:
[0020]
[0021]
[0022]
[0023]
[0024]
[0025] in, This represents the x-coordinate of the center point of the fused prediction box. This represents the ordinate of the center point of the fused prediction box. This indicates the width of the fused prediction box. This indicates the height of the fused prediction box. This represents the confidence level of the fused prediction box. This represents the total number of multimodal target detection models. The number of multimodal target detection models participating in the fusion, i.e. A multimodal target detection model generates a prediction box for the same candidate defect region. i =1~ N , As an enhancing factor, the first The predicted bounding box of a multimodal object detection model is represented as follows: ,in, Indicates the first The x-coordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The ordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The width of the predicted bounding box of a multimodal object detection model. Indicates the first The height of the predicted bounding box in a multimodal object detection model. Indicates the first The confidence level of the predicted bounding box of a multimodal object detection model.
[0026] Preferably, the preliminary annotation results are selected, as follows:
[0027] Determine the confidence level of the fused prediction boxes. Is it greater than the fusion threshold? If so, retain the fused prediction bounding box and use the defect category and coordinates of the fused prediction bounding box as the preliminary annotation result; otherwise, delete the fused prediction bounding box.
[0028] Preferably, the self-supervised model is one of the following: YOLO series models, RT-DETR model, Efficient Teacher model, Unbiased Teacher model, Instant-Teaching model, or SSDA-YOLO model.
[0029] Preferably, a dual filtering method using dynamic thresholds and prior constraints is employed for pseudo-labels, as detailed below:
[0030] Determine if the confidence level of the pseudo-label is lower than the dynamic threshold of the current training. If so, delete the pseudo-label; otherwise, retain the pseudo-label. The dynamic threshold increases linearly according to the number of training iterations of the self-supervised model. It is initially set between [0.1, 0.3] and then increases by 0.05 for each subsequent training iteration.
[0031] The pseudo-labels after dynamic threshold screening are filtered again using prior constraints.
[0032] Preferably, the prior constraints include:
[0033] (1) Area constraint: Remove areas smaller than a preset threshold The prediction box;
[0034] (2) Shape constraints: retain aspect ratio within The prediction box inside, The preset minimum aspect ratio, This is the preset maximum aspect ratio.
[0035] An unsupervised industrial defect detection system based on a multimodal expert system is provided for executing the unsupervised industrial defect detection method based on a multimodal expert system as described above. The system includes a data acquisition and annotation module, a multimodal prediction module, a fusion screening module, a filtering module, a self-supervised training module, and a detection module, wherein:
[0036] The data acquisition and annotation module is used to acquire unlabeled industrial defect image datasets and set corresponding text prompts for different defect categories in the industrial defect image datasets;
[0037] The multimodal prediction module is used to build a multimodal expert system composed of multiple parallel multimodal target detection models. The industrial defect image dataset and text prompts are input into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the predicted bounding boxes and confidence scores of candidate defect regions.
[0038] The fusion and filtering module is used to perform a fusion operation on the prediction results of various multimodal target detection models and then filter out the preliminary annotation results.
[0039] The filtering module is used to filter the initial annotation results based on prior constraints to obtain refined labels;
[0040] The self-supervised training module is used to input the industrial defect image dataset and the fine labels into the self-supervised model for training, and output pseudo labels after training. The pseudo labels are double-filtered using dynamic threshold and prior constraints. The industrial defect image dataset and the double-filtered pseudo labels of the current training are input into the self-supervised model after the previous training for retraining, and pseudo labels are output. The double filtering is then performed again until convergence or the preset number of training times is reached to obtain the trained self-supervised model.
[0041] The detection module is used to predict the industrial defect images under test using a trained self-supervised model, and obtain the corresponding industrial defect detection results.
[0042] An unsupervised industrial defect detection device based on a multimodal expert system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the unsupervised industrial defect detection method based on a multimodal expert system as described above.
[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0044] This invention eliminates the need for extensive manual annotation. It achieves high-quality automatic pseudo-label generation and continuous optimization through a combination of multimodal expert systems, prior constraints, and self-supervised learning. Specifically, it introduces a multimodal expert system composed of multiple parallel multimodal target detection models and prior constraints to ensure the semantic accuracy of the labels. Furthermore, it incorporates self-supervised learning to achieve accurate defect identification and localization without manual annotation, and can better learn defect feature patterns to accurately detect defects in new images. Through the collaboration of each stage, it combines multimodal information fusion with self-supervised learning techniques, mining the value of unlabeled data, reducing reliance on manual annotation, and effectively identifying common complex defect morphologies in actual detection processes. It improves robustness to blurred, occluded, or deformed targets, while enhancing model generalization ability and training stability while reducing data preparation costs. This effectively solves the problem of requiring extensive defect annotation in traditional supervised learning and the issues of unstable initial label quality, single modality, and insufficient generalization in traditional pseudo-label methods. It significantly improves the accuracy and efficiency of industrial defect detection and is suitable for deployment in various industrial vision inspection scenarios. Attached Figure Description
[0045] Figure 1 This is a flowchart of the unsupervised industrial defect detection method based on a multimodal expert system according to the present invention.
[0046] Figure 2 This is a comparison chart of the results of manual annotation in the prior art (a) and the detection results of the self-supervised model of the present invention (b). Detailed Implementation
[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.
[0049] Example 1:
[0050] like Figures 1-2 As shown, an unsupervised industrial defect detection method based on a multimodal expert system includes the following steps:
[0051] S1. Obtain an unlabeled industrial defect image dataset and set corresponding text prompts for different defect categories in the industrial defect image dataset.
[0052] In one embodiment, the text prompt consists of several strings, including at least one of the shape, color, texture, and brightness of the defect.
[0053] The process involves acquiring an unlabeled industrial defect image dataset, including several industrial defect images, and setting targeted prompt texts for different defect categories. These prompt texts consist of several natural language phrases related to perception direction, composed of several strings, used to guide the subsequent multimodal expert system in perceiving the corresponding type of defect region. Each prompt text is designed based on the typical visual features of a specific defect category, such as shape, color, texture, and brightness. Specifically, in this embodiment, the prompt text "Local Black Patch Defects" is set for black local patch defects; and "WhiteStrip Scratch Defects" is set for white stripe scratch defects. It is easy to understand that the prompt texts can also be adjusted according to the actual needs of those skilled in the art. Before inputting the images into the multimodal expert system, the industrial defect images in the dataset can be preprocessed, including image resizing, brightness adjustment, contrast enhancement, and denoising, to improve image quality and enhance the perception effect of the input images to the multimodal expert system. Preprocessing is a standard procedure for those skilled in the art and will not be elaborated further here.
[0054] S2. Establish a multimodal expert system composed of multiple parallel multimodal target detection models, and input the industrial defect image dataset and text prompts into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the prediction box and confidence level of the candidate defect region.
[0055] In one embodiment, the multimodal expert system includes three multimodal target detection models: the Grounding DINOv2 model, the Florences2 model, and the YOLOE model.
[0056] In this embodiment, an industrial defect image dataset and text prompts are input into a multimodal expert system. The system detects industrial defects in the dataset and uses the provided text prompts to make preliminary judgments about defect regions. The multimodal expert system consists of multiple multimodal object detection models (such as a vision-text model). The specific number and type can be adjusted according to actual needs. Each model outputs its prediction of defect regions based on the industrial defect image and text prompts, such as a predicted bounding box containing candidate defect regions and a confidence score. In this embodiment, the multimodal expert system used includes three models: Grounding DINOv2, Florences2, and YOLOE. These models are preferably pre-trained models from existing technologies. Multimodality refers to both text and visual image modes; the multimodal object detection model can guide visual detection through text.
[0057] S3. After fusing the prediction results of each multimodal target detection model, the preliminary annotation results are selected.
[0058] In one embodiment, the prediction results of each multimodal target detection model are fused, as follows:
[0059] Overlap judgment is performed on the prediction results of each multimodal target detection model. If the predicted bounding box of the candidate defect region appears in more than half of the multimodal target detection models and the Intersection over Union (IOU) between at least two multimodal target detection models exceeds the set threshold, the result is considered. If the corresponding predicted box is located in the candidate defect region, it is considered a valid candidate and is retained as a candidate target region. IOU stands for Intersection over Union.
[0060] The coordinates and confidence scores of the predicted bounding boxes are fused for the retained candidate target regions, as shown in the following formula:
[0061]
[0062]
[0063]
[0064]
[0065]
[0066] in, This represents the x-coordinate of the center point of the fused prediction box. This represents the ordinate of the center point of the fused prediction box. This indicates the width of the fused prediction box. This indicates the height of the fused prediction box. This represents the confidence level of the fused prediction box. This represents the total number of multimodal target detection models. The number of multimodal target detection models participating in the fusion, i.e. A multimodal target detection model generates a prediction box for the same candidate defect region. i =1~ N , As an enhancing factor, the first The predicted bounding box of a multimodal object detection model is represented as follows: ,in, Indicates the first The x-coordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The ordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The width of the predicted bounding box of a multimodal object detection model. Indicates the first The height of the predicted bounding box in a multimodal object detection model. Indicates the first The confidence level of the predicted bounding box of a multimodal object detection model.
[0067] In one embodiment, preliminary annotation results are selected, as follows:
[0068] Determine the confidence level of the fused prediction boxes. Is it greater than the fusion threshold? If so, retain the fused prediction bounding box and use the defect category and coordinates of the fused prediction bounding box as the preliminary annotation result; otherwise, delete the fused prediction bounding box.
[0069] First, the IOU (Intersection over Union) overlap is determined between the prediction results of different multimodal object detection models. If the predicted bounding box of the candidate defect region appears in more than half of the multimodal object detection models and at least two multimodal object detection models have an IOU exceeding the set intersection over Union threshold, the prediction is considered successful. If the corresponding predicted bounding box is located in a candidate defect region, it is considered a valid candidate and retained as a candidate target region. In this embodiment, if there are three multimodal target detection models, then at least two multimodal target detection models must have an Intersection over Union (IOU) exceeding a set threshold. In the case of [a specific condition], it is considered a valid candidate.
[0070] Subsequently, the coordinates and confidence scores of the predicted bounding boxes of the retained candidate target regions are fused according to the multimodal target detection models, and a fusion threshold is set. The preliminary annotation results are filtered out, and a unified pseudo-label annotation format is output. Specifically, the retained candidate target regions are fused and filtered. The fusion operation mainly includes coordinate fusion of predicted bounding boxes and confidence fusion: For coordinate fusion of predicted bounding boxes, the fusion method is as follows: Let there be... If a multimodal target detection model forms a prediction box for the same candidate defect region, then the... The predicted bounding box of a multimodal object detection model is represented as follows: ,in, Indicates the first The x-coordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The ordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The width of the predicted bounding box of a multimodal object detection model. Indicates the first The height of the predicted bounding box in a multimodal object detection model. Indicates the first The confidence scores of the predicted bounding boxes of each multimodal object detection model are used as the weighted average of the coordinates and confidence scores of the predicted bounding boxes of all multimodal object detection models, as shown in the formula above. As an enhancing factor, The default setting is 1.5, which results in higher confidence when multiple multimodal target detection models achieve consistent detection.
[0071] During the filtering process, the confidence level of the fused prediction bounding box is determined. Is it greater than the fusion threshold? If yes, retain the fused predicted bounding box; otherwise, delete the fused predicted bounding box. The pseudo-label annotation format is the preliminary annotation result, including the defect category and the coordinates of the fused predicted bounding box (including center point coordinates, width, and height).
[0072] S4. Filter the preliminary annotation results based on prior constraints to obtain refined labels.
[0073] In one embodiment, the prior constraints include:
[0074] (1) Area constraint: Remove areas smaller than a preset threshold The prediction box;
[0075] (2) Shape constraints: retain aspect ratio within The prediction box inside, The preset minimum aspect ratio, This is the preset maximum aspect ratio.
[0076] In this embodiment, prior constraints are used to filter the initial annotation results, eliminating annotations that do not conform to the actual industrial defect characteristics, and obtaining accurate annotation labels (precise labels) to ensure the accuracy, practicality and robustness of the annotation information. The prior constraints in this embodiment include the following two types: (1) Area constraint: removing areas smaller than a preset threshold. (2) Shape constraints: preserve aspect ratio within the predicted bounding box; The prediction box inside, The preset minimum aspect ratio, This is the preset maximum aspect ratio. Preset threshold. Preset minimum aspect ratio Preset maximum aspect ratio All of these can be built based on different industrial product specifications and historical defect feature libraries, and parameters need to be configured according to the actual scenario. It's easy to understand that prior constraints can also be set according to the actual needs of technical personnel.
[0077] S5. Input the industrial defect image dataset and precise labels into the self-supervised model for training, and output pseudo-labels after training.
[0078] In one embodiment, the self-supervised model is one of the following: YOLO series models, RT-DETR model, Efficient Teacher model, Unbiased Teacher model, Instant-Teaching model, and SSDA-YOLO model.
[0079] The process involves training a self-supervised model using an industrial defect image dataset and finely labeled data. The self-supervised model can employ various deep learning architectures, such as the YOLO series, RT-DETR, or other existing self-supervised models like the Efficient Teacher, Unbiased Teacher, Instant-Teaching, and SSDA-YOLO models. The number of rounds for pseudo-label output is set to "Round." After rounds of iterative training, the self-supervised model makes predictions and outputs pseudo-labels, which are then used for training in the next round. Self-supervision is a training method combining supervised learning and pseudo-labels. The number of rounds needs to be adjusted based on the specific task requirements. As the number of rounds increases, the self-supervised model gradually transitions from coarse perception to fine-grained target recognition, significantly improving the stability and accuracy of pseudo-labels. The number of rounds is determined based on the dataset size, model complexity, and training results.
[0080] S6. Use dynamic thresholds and prior constraints to perform dual filtering of pseudo-labels.
[0081] In one embodiment, a dual filtering method using dynamic thresholds and prior constraints is employed for pseudo-labels, as detailed below:
[0082] Determine if the confidence level of the pseudo-label is lower than the dynamic threshold of the current training. If so, delete the pseudo-label; otherwise, retain the pseudo-label. The dynamic threshold increases linearly according to the number of training iterations of the self-supervised model. It is initially set between [0.1, 0.3] and then increases by 0.05 for each subsequent training iteration.
[0083] The pseudo-labels after dynamic threshold screening are filtered again using prior constraints.
[0084] The process involves a dual filtering of pseudo-labels using dynamic thresholds and prior constraints. The first layer uses the set dynamic threshold to determine if a pseudo-label's confidence level is lower than the dynamic threshold corresponding to the current round, at which point it is discarded. The second layer applies prior constraints to ensure the quality of the pseudo-labels. The dynamic threshold is the confidence threshold used by the self-supervised model during prediction, controlling the sensitivity of the model's detection. This threshold increases linearly with the number of rounds the self-supervised model passes through, initially set between [0.1, 0.3], with an initial confidence threshold of 0.2 to increase label diversity and recall. After each round, the confidence threshold increases linearly by 0.05 to enhance label accuracy. After passing the dynamic threshold filtering, pseudo-labels undergo a second filtering using prior constraints to ensure the rationality of their structure and semantics. The prior constraints in this dual filtering are the same as those in step S4 (specific parameters may be the same or different), used to filter pseudo-labels and ensure they conform to the actual characteristics of industrial defects, thus improving their quality. The confidence threshold of pseudo-labels is adaptively adjusted during the current training phase of the self-supervised model, and unreliable labels are removed again in combination with prior constraints to ensure continuous optimization of pseudo-labels.
[0085] S7. Input the industrial defect image dataset and the pseudo-labels after double filtering in the current training into the self-supervised model after the previous training, train it again, output pseudo-labels, return to execute step S6, until convergence or the preset number of times is reached, and obtain the trained self-supervised model.
[0086] In this process, the pseudo-labels after double filtering are re-introduced into the self-supervised model for training. Step S6 is repeated until the model accuracy no longer improves, at which point the model is considered to have converged and training can be stopped, or training can be stopped after reaching a preset number of training iterations. A preset number of training iterations can be set... To determine whether a self-supervised model has converged, when the self-supervised model has completed a preset number of training iterations... If the accuracy does not increase after a certain number of training iterations, the model is considered to have converged, and training is stopped. In this embodiment, the number of training iterations is preset. The value is set to 100, and a Round is executed for each training iteration. In this embodiment, the self-supervised model is a YOLO12 model. Compared to the traditional supervised learning training method used in the existing YOLO12 model, this invention first generates pseudo-labels based on the industrial defect image dataset and precise labels, and then continuously trains the self-supervised model based on the industrial defect image dataset and pseudo-labels. This self-supervised model does not require manual labeling and is trained under self-supervision throughout the entire process.
[0087] S8. A pre-trained self-supervised model is used to predict the industrial defect image under test, obtaining the corresponding industrial defect detection results. Based on the industrial defect detection results, the corresponding defect category and defect area can be obtained, which helps improve detection efficiency and accuracy. This invention has a wide range of applications, such as metal defect detection, chip defect detection, wafer defect detection, and concrete defect detection. Concrete defect detection can include roads, bridges, walls, etc.
[0088] This embodiment uses the SDNET-2018 dataset, a publicly available dataset of cement surface crack defects on the Kaggle platform, for experiments. The SDNET-2018 dataset contains 800 images. 20% of these images are manually labeled and used as the test set to verify the accuracy of the self-supervised model. The remaining 80% of the images are used as the training set. First, the training set data from the SDNET-2018 dataset is read. After all steps S1 to S7, the optimal weights are retained and tested on the test set. For example... Figure 2 As shown, Figure 2 Figure (a) in the figure is the result of manual annotation. Figure 2 Figure (b) shows the detection results of the self-supervised model of this invention. For example, Crack 0.63 indicates that the defect category is crack and the confidence level is 0.63, and so on. Precision, recall, and mean AP50 (mAP50) with IOU of 0.5 were used as the main evaluation metrics for the detection results on the test set. The results are shown in Table 1, which compares the detection results of different methods on the SDNET-2018 dataset.
[0089] Table 1
[0090]
[0091] As shown in Table 1, the method of the present invention has high accuracy, meets the requirements of use, has significant advantages over existing unsupervised learning, and is comparable to traditional supervised learning methods, greatly saving the cost of manual annotation.
[0092] In summary, addressing the data bottleneck problem of supervised learning in defect detection in existing technologies, and considering the difficulties in obtaining defect samples, high annotation costs, and the unstable accuracy and limited generalization ability of existing pseudo-label strategies, this invention proposes an unsupervised industrial defect detection method based on a multimodal expert system. This method does not rely on a large amount of manually labeled data and can stably output high-quality detection results in an unsupervised environment. Experimental verification on a publicly available industrial defect dataset demonstrates that the method has strong feasibility, scalability, and practicality.
[0093] Example 2:
[0094] An unsupervised industrial defect detection system based on a multimodal expert system is provided for executing the unsupervised industrial defect detection method based on a multimodal expert system as described in Example 1. The system includes a data acquisition and annotation module, a multimodal prediction module, a fusion screening module, a filtering module, a self-supervised training module, and a detection module, wherein:
[0095] The data acquisition and annotation module is used to acquire unlabeled industrial defect image datasets and set corresponding text prompts for different defect categories in the industrial defect image datasets;
[0096] The multimodal prediction module is used to build a multimodal expert system composed of multiple parallel multimodal target detection models. The industrial defect image dataset and text prompts are input into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the predicted bounding boxes and confidence scores of candidate defect regions.
[0097] The fusion and filtering module is used to perform a fusion operation on the prediction results of various multimodal target detection models and then filter out the preliminary annotation results.
[0098] The filtering module is used to filter the initial annotation results based on prior constraints to obtain refined labels;
[0099] The self-supervised training module is used to input the industrial defect image dataset and the fine labels into the self-supervised model for training, and output pseudo labels after training. The pseudo labels are double-filtered using dynamic threshold and prior constraints. The industrial defect image dataset and the double-filtered pseudo labels of the current training are input into the self-supervised model after the previous training for retraining, and pseudo labels are output. The double filtering is then performed again until convergence or the preset number of training times is reached to obtain the trained self-supervised model.
[0100] The detection module is used to predict the industrial defect images under test using a trained self-supervised model, and obtain the corresponding industrial defect detection results.
[0101] It should be noted that the specific limitations of the unsupervised industrial defect detection system based on a multimodal expert system can be found in the limitations of the unsupervised industrial defect detection method based on a multimodal expert system in Example 1, and will not be repeated here. Each module in the unsupervised industrial defect detection system based on a multimodal expert system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0102] To address the data bottleneck problem of supervised learning in defect detection in existing technologies, and considering the difficulties in obtaining defect samples, the high cost of annotation, and the unstable accuracy and limited generalization ability of existing pseudo-label strategies, this invention proposes an unsupervised industrial defect detection system based on a multimodal expert system. This system does not rely on a large amount of manually labeled data and can stably output high-quality detection results in an unsupervised environment.
[0103] Example 3:
[0104] An unsupervised industrial defect detection device based on a multimodal expert system includes a memory and a processor. The memory is used to store a computer program, and the processor is used to implement any of the unsupervised industrial defect detection methods based on a multimodal expert system in Embodiment 1 when executing the computer program.
[0105] The memory is used to store computer programs, and when the processor executes the computer programs, it sequentially completes the steps in Example 1.
[0106] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor. By running the computer program stored in the memory, the processor implements the unsupervised industrial defect detection method based on a multimodal expert system as described in Embodiment 1 of the present invention.
[0107] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). The memory stores computer programs, and the processor executes these programs after receiving execution instructions.
[0108] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods and steps disclosed in Embodiment 1 of this invention. The general-purpose processor can be a microprocessor or any conventional processor.
[0109] To address the data bottleneck problem of supervised learning in defect detection in existing technologies, and considering the difficulties in obtaining defect samples, the high cost of annotation, and the unstable accuracy and limited generalization ability of existing pseudo-label strategies, the unsupervised industrial defect detection device based on a multimodal expert system proposed in this invention does not rely on a large amount of manually labeled data and can stably output high-quality detection results in an unsupervised environment.
[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0111] The above embodiments are merely illustrative of more specific and detailed examples described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.
Claims
1. An unsupervised industrial defect detection method based on a multimodal expert system, characterized in that: Includes the following steps: S1. Obtain an unlabeled industrial defect image dataset and set corresponding text prompts for different defect categories in the industrial defect image dataset; S2. Establish a multimodal expert system composed of multiple parallel multimodal target detection models, and input the industrial defect image dataset and text prompts into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the prediction box and confidence level of the candidate defect region. S3. After fusing the prediction results of each multimodal target detection model, preliminary annotation results are selected. The specific steps for fusing the prediction results of each multimodal target detection model are as follows: Overlap judgment is performed on the prediction results of each multimodal target detection model. If the predicted bounding box of the candidate defect region appears in more than half of the multimodal target detection models and the Intersection over Union (IOU) between at least two multimodal target detection models exceeds the set threshold, the result is considered. If the corresponding predicted box is located in the candidate defect region, it is considered a valid candidate and is retained as a candidate target region. IOU stands for Intersection over Union. The coordinates and confidence scores of the predicted bounding boxes are fused for the retained candidate target regions, as shown in the following formula: in, This represents the x-coordinate of the center point of the fused prediction box. This represents the ordinate of the center point of the fused prediction box. This indicates the width of the fused prediction box. This indicates the height of the fused prediction box. This represents the confidence level of the fused prediction box. This represents the total number of multimodal target detection models. The number of multimodal target detection models participating in the fusion, i.e. A multimodal target detection model forms a prediction box for the same candidate defect region, i=1~N. As an enhancing factor, the first The predicted bounding box of a multimodal object detection model is represented as follows: ,in, Indicates the first The x-coordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The ordinate of the center point of the predicted bounding box in a multimodal object detection model. Indicates the first The width of the predicted bounding box of a multimodal object detection model. Indicates the first The height of the predicted bounding box in a multimodal object detection model. Indicates the first The confidence scores of the predicted bounding boxes of a multimodal target detection model; S4. Filter the preliminary annotation results based on prior constraints to obtain refined labels; S5. Input the industrial defect image dataset and the fine labels into the self-supervised model for training, and output pseudo labels after training is completed; S6. Use dynamic thresholds and prior constraints to perform dual filtering of pseudo-labels; S7. Input the industrial defect image dataset and the pseudo-labels after double filtering in the current training into the self-supervised model after the previous training, train it again, output pseudo-labels, return to execute step S6, until convergence or the preset number of training times is reached, and obtain the trained self-supervised model. S8. Use the trained self-supervised model to predict the industrial defect image to obtain the corresponding industrial defect detection results.
2. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 1, characterized in that: The text prompt consists of several strings, including at least one of the defects' shape, color, texture, and brightness.
3. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 1, characterized in that: The multimodal expert system includes three multimodal target detection models: Grounding DINOv2, Florences2, and YOLOE.
4. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 1, characterized in that: The preliminary annotation results are as follows: Determine the confidence level of the fused prediction boxes. Is it greater than the fusion threshold? If so, retain the fused prediction bounding box and use the defect category and coordinates of the fused prediction bounding box as the preliminary annotation result; otherwise, delete the fused prediction bounding box.
5. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 1, characterized in that: The self-supervised model is one of the following: YOLO series models, RT-DETR model, Efficient Teacher model, UnbiasedTeacher model, Instant-Teaching model, or SSDA-YOLO model.
6. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 1, characterized in that: The method employs a dual filtering of pseudo-labels using dynamic thresholds and prior constraints, as detailed below: Determine whether the confidence level of the pseudo-label is lower than the dynamic threshold of the current training. If so, delete the pseudo-label; otherwise, retain the pseudo-label. The dynamic threshold increases linearly according to the number of training iterations of the self-supervised model. It is initially set between [0.1, 0.3] and then increases by 0.05 for each subsequent training iteration. The pseudo-labels after dynamic threshold screening are filtered again using prior constraints.
7. The unsupervised industrial defect detection method based on a multimodal expert system as described in claim 6, characterized in that: The prior constraints include: (1) Area constraint: Remove areas smaller than a preset threshold The prediction box; (2) Shape constraints: retain aspect ratio within The prediction box inside, The preset minimum aspect ratio, This is the preset maximum aspect ratio.
8. An unsupervised industrial defect detection system based on a multimodal expert system, used to execute the unsupervised industrial defect detection method based on a multimodal expert system as described in any one of claims 1-7, characterized in that: The unsupervised industrial defect detection system based on a multimodal expert system includes a data acquisition and annotation module, a multimodal prediction module, a fusion screening module, a filtering module, a self-supervised training module, and a detection module, wherein: The data acquisition and annotation module is used to acquire an unlabeled industrial defect image dataset and set corresponding text prompts for different defect categories in the industrial defect image dataset. The multimodal prediction module is used to establish a multimodal expert system composed of multiple parallel multimodal target detection models, and input industrial defect image datasets and text prompts into the multimodal expert system to obtain the prediction results of the corresponding multimodal target detection models. The prediction results include the prediction bounding boxes and confidence scores of candidate defect regions. The fusion and filtering module is used to perform a fusion operation on the prediction results of each multimodal target detection model and then filter out the preliminary annotation results. The filtering module is used to filter the preliminary annotation results based on prior constraints to obtain refined labels; The self-supervised training module is used to input the industrial defect image dataset and the fine labels into the self-supervised model for training, and output pseudo labels after training; the pseudo labels are double-filtered using dynamic thresholds and prior constraints; the industrial defect image dataset and the double-filtered pseudo labels of the current training are input into the self-supervised model after the previous training for retraining, output pseudo labels, return to execute double filtering, until convergence or the preset number of training times is reached, and a well-trained self-supervised model is obtained. The detection module is used to predict the industrial defect image under test using a trained self-supervised model to obtain the corresponding industrial defect detection result.
9. An unsupervised industrial defect detection device based on a multimodal expert system, characterized in that: It includes a memory and a processor, the memory being used to store a computer program, and the processor being used to implement, when executing the computer program, the unsupervised industrial defect detection method based on a multimodal expert system as described in any one of claims 1-7.