A defect detection method and device, electronic equipment and storage medium

Abstract

The present application provides a kind of defect detection method, device, electronic equipment and computer readable storage medium, it is related to nondestructive testing technical field.Defect detection method includes: the original image is input into preset computer vision model, detects out defect area, first defect category and its corresponding first confidence degree;From the original image, the target image corresponding to the defect area is cut out;With first defect category, target image is input into preset multi-modal big model, detects out defect grade, defect text description, second defect category and its corresponding second confidence degree.At least to solve the problem that computer vision model precision is insufficient in related art, multi-modal big model inference speed is slow and the problem that the consumption of large amount of computing resources is big.Adapt to defect detection scene.

Description

Technical Field

[0001] This invention relates to the field of nondestructive testing technology, and in particular to a defect detection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the manufacturing, industrial quality inspection, and line inspection sectors of the communications equipment industry, defect detection is a crucial step in ensuring product quality and operational safety. Taking a communications equipment production line as an example, optical modules, PCBs (Printed Circuit Boards), and cabinet components may develop defects such as cracks, incomplete soldering, scratches, and foreign matter residue during production. If these defects are not detected and addressed in a timely manner, they will directly affect the performance and stability of the equipment, and may even cause large-scale communication outages. Therefore, the accuracy and efficiency of defect detection are directly related to a company's production costs and product quality.

[0003] Traditional defect detection methods primarily rely on manual quality inspection or a single computer vision (CV) model. While manual inspection is flexible, it is inefficient, costly, and highly subjective, easily influenced by the experience and fatigue of inspectors, leading to missed or incorrect detections. A single CV model has an advantage in speed, but it has significant shortcomings in the semantic understanding, classification, and grading of complex defects. For example, a CV model can detect solder joint anomalies on a PCB board, but it struggles to accurately determine whether the anomaly is a "minor cold solder joint" or a "serious cold solder joint," and it cannot provide natural language explanations to describe the cause and risk level of the defect.

[0004] In recent years, multimodal large models have demonstrated powerful capabilities in cross-modal understanding and reasoning, enabling them to process both image and text information simultaneously, thus possessing stronger semantic understanding and interpretation capabilities. However, directly applying multimodal large models to industrial quality inspection scenarios presents two main problems: the sheer size of multimodal large models makes it difficult to meet the real-time requirements of production lines when used alone; and high-quality image-text pair datasets are extremely scarce in the industrial quality inspection field, making manual annotation costly.

[0005] In summary, while computer vision models are fast, they often suffer from insufficient accuracy in industrial defect detection, especially in recognizing small targets or complex defects. Multimodal large models, although possessing strong recognition and interpretation capabilities, have slow inference speeds and consume a lot of computing resources, making them unsuitable for real-time detection scenarios on industrial production lines. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a defect detection method, apparatus, electronic device and computer-readable storage medium, which can achieve efficient and accurate defect detection.

[0007] In a first aspect, the present invention provides a defect detection method, comprising: inputting an original image into a preset computer vision model to detect a defect region, a first defect category and its corresponding first confidence level; cropping a target image corresponding to the defect region from the original image; inputting the target image and the first defect category into a preset multimodal large model to detect a defect level, a defect text description, a second defect category and its corresponding second confidence level.

[0008] Preferably, after inputting the original image into a preset computer vision model to detect the defect region, the first defect category and its corresponding first confidence level, and before cropping the target image corresponding to the defect region from the original image, the defect detection method further includes: matching the cropping threshold of the original image from a preset mapping table according to the first defect category, wherein the mapping table includes the defect category and its corresponding cropping threshold; and inputting the defect region and the cropping threshold into a non-maximum suppression (NMS) function to adjust the defect region.

[0009] Preferably, after inputting the target image and the first defect category into a preset multimodal large model to detect the defect level, defect text description, second defect category and its corresponding second confidence level, the defect detection method further includes: weighting and summing the first confidence level and the second confidence level to obtain a comprehensive confidence level; and generating a defect detection report based on the comprehensive confidence level, defect level, defect text description and second defect category.

[0010] Preferably, after generating a defect detection report based on the comprehensive confidence level, defect level, defect text description, and second defect category, the defect detection method further includes: determining whether the first defect category and the second defect category are consistent; and in response to the inconsistency between the first defect category and the second defect category, marking the defect detection report as uncertain.

[0011] Preferably, after marking the defect detection report as uncertain, the defect detection method further includes: acquiring user feedback data, wherein the user feedback data includes the true defect category and its corresponding reference image; evaluating the loss of the computer vision model and the multimodal large model respectively based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image; and optimizing the computer vision model and the multimodal large model based on the loss.

[0012] Preferably, the loss of the computer vision model and the multimodal large model are evaluated based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image, respectively. Specifically, this includes: calculating the similarity between the first defect category and the true defect category, the second defect category and the true defect category, and the target image and the reference image, respectively; and evaluating the loss of the computer vision model and the multimodal large model based on the similarity.

[0013] Preferably, the computer vision model includes a convolutional neural network (CNN) feature extraction model, and the multimodal large model includes a vision-language model.

[0014] Secondly, the present invention also provides a defect detection device, including a first detection module, a cropping module, and a second detection module. The first detection module is used to input the original image into a preset computer vision model to detect the defect region, the first defect category, and its corresponding first confidence level. The cropping module is used to crop the target image corresponding to the defect region from the original image. The second detection module is used to input the target image and the first defect category into a preset multimodal large model to detect the defect level, the defect text description, the second defect category, and its corresponding second confidence level.

[0015] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the defect detection method provided in the first aspect above.

[0016] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the defect detection method provided in the first aspect above.

[0017] This invention provides a defect detection method, apparatus, electronic device, and computer-readable storage medium. By combining a pre-set computer vision model and a multimodal large-scale model, the computer vision model provides an image-text pair dataset for training the multimodal large-scale model, reducing reliance on large-scale, high-quality image-text pair datasets and thus lowering the cost of manual annotation. The multimodal large-scale model provides more accurate defect level judgments and natural language descriptions, enhancing the semantic understanding and interpretation capabilities of defect detection. Therefore, this invention can achieve efficient and accurate defect detection. Attached Figure Description

[0018] Figure 1 This is a flowchart of a defect detection method according to Embodiment 1 of the present invention;

[0019] Figure 2 This is an example diagram of the detected defect area, the first defect category, and the corresponding first confidence level in Embodiment 1 of the present invention;

[0020] Figure 3 This is an example image showing the detection of defect level, defect text description, second defect category and corresponding second confidence level in Embodiment 1 of the present invention;

[0021] Figure 4 This is an example diagram of generating defect text descriptions in Embodiment 1 of the present invention;

[0022] Figure 5 This is an example diagram illustrating the generation of user feedback data in Embodiment 1 of the present invention;

[0023] Figure 6 This is a flowchart of a defect detection method according to Embodiment 2 of the present invention;

[0024] Figure 7 This is a schematic diagram of the structure of a defect detection device according to Embodiment 3 of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0027] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0029] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0030] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0031] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0032] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0033] Example 1:

[0034] like Figure 1 As shown, this embodiment provides a defect detection method. The defect detection method includes:

[0035] S101, input the original image into the preset computer vision model to detect the defect area, the first defect category and its corresponding first confidence level.

[0036] Specifically, computer vision models include convolutional neural network (CNN) feature extraction models.

[0037] In this embodiment, the original image refers to the unprocessed image input to the computer vision model, typically acquired from a camera or other image acquisition device. The original image contains all the visual information of the object to be analyzed. The defect region refers to a specific area in the image detected by the model that contains a defect. Defect regions are usually represented by bounding boxes, segmentation masks, or other forms, indicating which parts of the image are considered defective. The defect category refers to the result of classifying the defect. The defect category is usually a label indicating the type of defect, such as "scratch," "dent," or "stain." The computer vision model determines the specific type of defect based on training data. Confidence represents the model's confidence in its detection results. Confidence is typically between 0 and 1; a higher value indicates stronger confidence in the detection result. For example, a confidence of 0.85 means the model is 85% certain that its judgment of the defect category is correct.

[0038] like Figure 2 As shown, this embodiment uses a lightweight CNN (Convolutional Neural Networks) feature extraction model to detect the defect region, the first defect category, and its corresponding first confidence level (i.e., Figure 2The confidence level (in the formula) specifically includes: inputting the original image I into the formula. The feature map F containing spatial and semantic information is extracted, and then the feature map F is processed by the detection head of the CNN feature extraction model to output the result. To ensure efficient operation even in edge devices or real-time detection scenarios, among which, Indicates the first One defective area, , They represent the first The center coordinates, width, and height of each defect area. Indicates the first The first defect category Indicates the first The first confidence level corresponding to the first defect category, where N represents the total number of defect regions. CNN feature extraction models include, but are not limited to: YOLOv5-Nano, MobileNet-SSD, and EfficientDet-D0.

[0039] In this embodiment, the original image is a PCB circuit board image. The computer vision model detects the defect areas, first defect categories, and corresponding first confidence levels of two defects as follows: Defect 1: coordinates (120,80,30,30), defect category "crack", confidence level 0.85; Defect 2: coordinates (300,200,40,40), defect category "solder joint cold solder joint", confidence level 0.90.

[0040] Optionally, after S101: inputting the original image into a preset computer vision model to detect the defect region, the first defect category, and its corresponding first confidence level, and before S102: cropping the target image corresponding to the defect region from the original image, the defect detection method further includes:

[0041] S104, based on the first defect category, match the cropping threshold of the original image from a preset mapping table, wherein the mapping table includes the defect category and its corresponding cropping threshold.

[0042] In this embodiment, the cropping threshold refers to the threshold used in image processing to determine which pixels will be retained or discarded. The cropping threshold includes, but is not limited to, the IoU (Intersection over Union) threshold. The mapping table is shown in Table 1.

[0043] Table 1 Mapping Table

[0044]

[0045] S105: Input the defect region and the trimming threshold into the Non-Maximum Suppression (NMS) function to adjust the defect region.

[0046] In this embodiment, the defect region and the clipping threshold are input into the NMS (Non-Maximum Suppression) function. In the middle, remove redundant bounding boxes from the defective areas, where, Indicates the defect area. This indicates the clipping threshold (e.g., 0.5). This indicates the defect area after removing redundant boxes (i.e.) Figure 2 (Defect coordinates in the image).

[0047] S102, crop out the target image corresponding to the defect area from the original image.

[0048] In this embodiment, the original image is cropped based on clear and accurate boundaries, i.e., defect areas, to obtain a clearer and more accurate target image (i.e., Figure 2 (defect areas in the data), thereby providing clearer and more accurate data support for further defect detection.

[0049] S103, input the target image and the first defect category into the preset multimodal large model, and detect the defect level, defect text description, second defect category and its corresponding second confidence level.

[0050] Specifically, multimodal large models include vision-language models.

[0051] In this embodiment, the defect level is the result of evaluating and classifying the detected defects. It is typically used to indicate the severity or impact of the defect. Defect levels can be divided into multiple levels, such as: Minor, Moderate, Severe, and Critical. The defect level helps users quickly understand the impact of the defect and decide on subsequent handling measures. The assessment of the defect level is usually based on the size, location, type of the defect, and its impact on the product or system functionality. The defect text description is a detailed description of the detected defect, usually presented in natural language. It provides specific information about the defect, including its characteristics, possible causes, and scope of impact. For example, for a scratch defect, the defect text description might be: "A scratch approximately 5 cm long was found in the upper left corner, possibly caused by a collision during transportation." The defect text description helps users better understand the nature and background of the defect.

[0052] like Figure 3As shown, the detection process identifies the defect level, defect text description, second defect category, and their corresponding second confidence level. Specifically, it includes: matching the defect level from a preset mapping relationship between defect categories and defect levels based on the first defect category; and generating the defect text description (i.e., based on the preset mapping relationship between the first defect category, defect region, defect category, and basic defect description). Figure 3 Text annotations in the image). The target image (i.e. Figure 3 (Input formula for cropping defective images) Extract image feature vectors, where, This indicates a preset visual encoder (such as ResNet or ViT). Represents the image feature vector. This represents the preset feature dimension. Represent real numbers. Obtain other defect text descriptions, and input the defect text descriptions and other defect text descriptions into the formula. Extract the first text feature vector and its second text feature vector, where, This indicates a preset text encoder (such as BERT or Transformer). Represents the text feature vector. This represents the first defect category. The image feature vector is concatenated with the first and second text feature vectors respectively to construct a multimodal large-scale model image-text pair dataset (i.e.,...). Figure 3 Image-text pairs in the dataset). Input the image-text pair dataset of the multimodal large model into the InfoNCE contrastive loss formula. In the process, the similarity between the image feature vector and the first text feature vector and the second text feature vector is calculated (i.e., Figure 3 (Image-text alignment loss in the image), where, This represents the total number of image feature vectors in the image-text pair dataset. Represents the first image-text pair in the dataset. Image feature vectors, Represents the first image-text pair in the dataset. Each text feature vector This represents a preset similarity function, including but not limited to the cosine similarity calculation formula. , This represents a preset temperature parameter used to adjust the distribution smoothness. If the image feature vector matches the first text feature vector, the similarity between the image feature vector and the first text feature vector should be higher than the similarity between the image feature vector and the second text feature vector. If the image feature vector matches the second text feature vector, the similarity between the image feature vector and the first text feature vector should be lower than the similarity between the image feature vector and the second text feature vector. Therefore, in this embodiment, the defect category corresponding to the maximum similarity is determined as the second defect category, and the second confidence level corresponding to the second defect category is calculated. For example: Defect 1: Category = Crack, Confidence Level = 0.95; Defect 2: Category = Poor Welding Point, Confidence Level = 0.92.

[0053] like Figure 4 As shown, generating a defect text description specifically includes: based on the first defect category (i.e. Figure 4 The system uses category labels to match the basic defect description from a preset mapping relationship between defect categories and defect descriptions. For example, "Crack" → "Medium", "Crack Found", "Poor Solder Joint" → "Severe", "Poor Solder Joint Detected". The defect area is then input into the formula. Determine the defect size (Right now Figure 4 (the geometric information of the defect box in the image), where... This represents the preset scaling factor from pixels to millimeters. For example, the width of the defect area is 20 pixels and the height is 10 pixels. 0.1mm / pixel The defect dimensions are concatenated with the basic defect description to obtain the defect text description (i.e., Figure 4 The rules described in the text, such as "crack length is about 2mm" and "solder joint poor weld area is about 1.5mm²", are suitable for industrial scenarios with fewer defect types and clear characteristics.

[0054] It should be noted that, for scenarios with complex defect types that require semantic interpretation, this embodiment can also use the target image... Second Defect Category Further input into the vision-language large model Generate defect explanation (Right now Figure 4The visual-language large model (including semantic or model descriptions) is further concatenated with the defect explanation and text description. For example, "crack length approximately 2mm" is interpreted as "a crack of approximately 2mm length exists on the circuit board surface"; "solder joint cold solder joint area approximately 1.5mm²" is interpreted as "uneven solder joint, cold solder joint exists." The visual-language large model includes, but is not limited to, BLIP and CLIP+GPT. To avoid inconsistencies between the basic defect description and the defect explanation, when there are differences in size or location descriptions, the basic defect description takes precedence, with the defect explanation serving as a supplementary description. This ensures that the final output defect text description contains both numerical information and semantic explanation, guaranteeing the accuracy and completeness of the data.

[0055] To reduce the risk of missed detections by the computer vision model, this embodiment also supports secondary verification of the entire original image using a multimodal large model. That is, if the computer vision model fails to detect defects, the entire original image can be input into the multimodal large model for low-frequency scanning to further identify potential defects. Simultaneously, the function of allowing users to report additional defects can also compensate for the computer vision model's missed detections, ensuring system robustness.

[0056] Optionally, in S103: after inputting the target image and the first defect category into a preset multimodal large model to detect the defect level, defect text description, second defect category and its corresponding second confidence level, the defect detection method further includes:

[0057] S107, the first confidence level and the second confidence level are weighted and summed to obtain the comprehensive confidence level.

[0058] In this embodiment, the first confidence level is used. Second confidence level For example, and Enter formula Calculate the overall confidence level , This represents the preset weight coefficient, which controls the relative importance of the computer vision model and the multimodal large model. It usually depends more on the confidence level of the multimodal large model and is generally set to 0.3.

[0059] S108 generates a defect detection report based on the overall confidence level, defect level, defect text description, and second defect category.

[0060] In this embodiment, in addition to the fusion between the first confidence level and the second confidence level, if the first defect category and the second defect category are inconsistent, the fusion between the first defect category and the second defect category is also involved. Compared with computer vision models, multimodal large models have higher accuracy. Therefore, in this embodiment, the first defect category and the second defect category are fused, that is, the second defect category is used as the standard. Then, based on the comprehensive confidence level, defect level, defect text description, and second defect category, a defect detection report is generated. For example: Defect 1: Defect category "crack", defect level "medium", defect text description "There is a crack on the surface of the circuit board, with a length of about 2mm", comprehensive confidence level = 0.92; Defect 2: Defect category "solder joint cold solder joint", defect level "severe", defect text description "The area of ​​the cold solder joint is about 1.5mm², the solder joint is uneven, and there is a cold solder joint", comprehensive confidence level = 0.91.

[0061] It should be noted that the defect detection report may take the form of, but is not limited to, a structured data table. This embodiment can also combine the original image and the defect detection report for visualization. For example, the defect area can be drawn on the original image, and a defect text description such as "Crack - Medium: Length approximately 2mm" can be added near the defect area. Colors can be used to distinguish the defect level, where green indicates minor, yellow indicates medium, and red indicates severe.

[0062] Optionally, after generating the defect detection report based on the comprehensive confidence level, defect level, defect text description, and second defect category in S108, the defect detection method further includes:

[0063] S109, determine whether the first defect category and the second defect category are consistent.

[0064] S110, in response to the inconsistency between the first defect category and the second defect category, mark the defect detection report as uncertain.

[0065] In this embodiment, the computer vision model predicts the defect category as "crack" and the multimodal large model predicts the defect category as "poor weld". The defect detection report is marked as "poor weld (there is a defect category conflict, manual review is required)".

[0066] Optionally, after marking the defect detection report as uncertain, the defect detection method further includes: acquiring user feedback data, wherein the user feedback data includes the true defect category and its corresponding reference image; evaluating the loss of the computer vision model and the multimodal large model based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image; and optimizing the computer vision model and the multimodal large model based on the loss.

[0067] In this embodiment, as Figure 5As shown, users can provide feedback on the detection results, generating user feedback data. User feedback data includes the following types: confirmation results, correction results, and new defects. Confirmation results indicate that the detection was correct, correction results indicate that the detection was incorrect, and correction results include, but are not limited to, the true defect category, the true defect level, and the true defect text description. New defects indicate missed detections, and new defects include, but are not limited to, the true defect category manually marked by the user.

[0068] In this example, the user confirms that defect 1 is correct and the actual defect level of defect 2 is "medium". The user feedback data is as follows: Defect 2: Defect category "solder joint cold solder joint", defect level "medium", defect text description "solder joint cold solder joint area is about 1.5mm², the solder joint is uneven and there is cold solder joint".

[0069] The existence of corrected results or new defects means that one or more of the computer vision model and the multimodal large model have insufficient accuracy. Therefore, the user feedback data in this embodiment also includes the reference image corresponding to the true defect category. The reference image refers to the image after the user manually crops the original image, and the image is also marked with defect area and true defect area. The loss of the computer vision model and the multimodal large model is calculated by using the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image for corresponding optimization.

[0070] Calculate the loss of the computer vision model and the multimodal large model, specifically including: according to the formula ,formula The losses of the computer vision model and the multimodal large model were calculated separately. This represents the classification loss of a computer vision model or a large multimodal model. Let represent the true category of the i-th defect, and let represent either the first or second defect category. This represents the localization loss of the computer vision model. This indicates the defect area marked in the comparison image. The loss of the computer vision model is a weighted sum of the classification loss and the localization loss, representing the actual defect area marked in the comparison image. The loss of the multimodal large model is the classification loss, which includes, but is not limited to, cross-entropy loss and IoU loss.

[0071] It should be noted that user feedback data is typically small in quantity but high in quality, and therefore given high weight. Through continuous iteration, the model gradually adapts to the actual production environment. This embodiment employs an offline batch retraining mechanism, storing user feedback data in a feedback sample library and retraining it uniformly at set intervals (such as daily or weekly) to avoid catastrophic forgetting caused by online learning. Simultaneously, this embodiment supports lightweight incremental fine-tuning to quickly adapt to newly emerging defect types, ensuring the model's stability and adaptability in long-term operation.

[0072] Specifically, based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image, the loss of the computer vision model and the multimodal large model are evaluated respectively, including: calculating the similarity between the first defect category and the true defect category, the second defect category and the true defect category, and the target image and the reference image respectively; and evaluating the loss of the computer vision model and the multimodal large model based on the similarity.

[0073] This embodiment provides a defect detection method that combines a pre-set computer vision model with a multimodal large model. The computer vision model provides an image-text pair dataset for training the multimodal large model, reducing the dependence on large-scale, high-quality image-text pair datasets and thus lowering the cost of manual annotation. The multimodal large model provides more accurate defect level judgment and natural language description, enhancing the semantic understanding and interpretation capabilities of defect detection, and achieving efficient and accurate defect detection.

[0074] Example 2:

[0075] like Figure 6 As shown, this embodiment provides a defect detection method. The defect detection method includes:

[0076] S201, Input the original image into the preset computer vision model to detect the defect area, the first defect category and its corresponding first confidence level.

[0077] In this embodiment, the original image is... Figure 6 User-uploaded images, computer vision models Figure 6 The small model in the middle, the defect area is Figure 6 The defect box in the image.

[0078] S202, crop out the target image corresponding to the defect area from the original image.

[0079] In this embodiment, the target image corresponding to the defect area is... Figure 6 The defective area in the middle.

[0080] S203, input the target image and the first defect category into the preset multimodal large model, and detect the defect level, defect text description, second defect category and its corresponding second confidence level.

[0081] In this embodiment, the multimodal large model is... Figure 6 The large model in the middle, the second defect category is... Figure 6 The output category, defect level, is... Figure 6 The level and defect text description in the text are Figure 6 The explanation in the text.

[0082] S204: The first confidence level and the second confidence level are weighted and summed to obtain the comprehensive confidence level; based on the comprehensive confidence level, defect level, defect text description, and second defect category, a defect detection report is generated.

[0083] In this embodiment, the defect detection report is... Figure 6 The unified testing report in China.

[0084] This embodiment provides a defect detection method that combines a pre-set computer vision model with a multimodal large model. The computer vision model provides an image-text pair dataset for training the multimodal large model, reducing the dependence on large-scale, high-quality image-text pair datasets and thus lowering the cost of manual annotation. The multimodal large model provides more accurate defect level judgment and natural language description, enhancing the semantic understanding and interpretation capabilities of defect detection, and achieving efficient and accurate defect detection.

[0085] Example 3:

[0086] like Figure 7 As shown, this embodiment also provides a defect detection device, including a first detection module 31, a cropping module 32, and a second detection module 33. The first detection module 31 is used to input the original image into a preset computer vision model to detect the defect region, the first defect category, and its corresponding first confidence level. The cropping module 32 is used to crop the target image corresponding to the defect region from the original image. The second detection module 33 is used to input the target image and the first defect category into a preset multimodal large model to detect the defect level, the defect text description, the second defect category, and its corresponding second confidence level.

[0087] Optionally, the defect detection device further includes a matching module 34 and an adjustment module 35. The matching module 34 is used to match the cropping threshold of the original image from a preset mapping table according to the first defect category, wherein the mapping table includes the defect category and its corresponding cropping threshold. The adjustment module 35 is used to input the defect region and the cropping threshold into the non-maximum suppression (NMS) function to adjust the defect region.

[0088] Optionally, the defect detection device further includes: a summation module 36 and a generation module 37. The summation module 36 is used to perform a weighted summation of the first confidence level and the second confidence level to obtain a comprehensive confidence level. The generation module 37 is used to generate a defect detection report based on the comprehensive confidence level, defect level, defect text description, and second defect category.

[0089] Optionally, the defect detection device further includes a judgment module 38 and a marking module 39. The judgment module 38 is used to determine whether the first defect category and the second defect category are consistent. The marking module 39 is used to mark the defect detection report as uncertain in response to the inconsistency between the first defect category and the second defect category.

[0090] Optionally, the defect detection device further includes: an acquisition module 40, an evaluation module 41, and an optimization module 42. The acquisition module 40 is used to acquire user feedback data, wherein the user feedback data includes the true defect category and its corresponding reference image. The evaluation module 41 is used to evaluate the loss of the computer vision model and the multimodal large model respectively based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image. The optimization module 42 is used to optimize the computer vision model and the multimodal large model based on the loss.

[0091] Specifically, the evaluation module 41 includes a calculation unit 411 and an evaluation unit 412. The calculation unit 411 is used to calculate the similarity between the first defect category and the true defect category, the second defect category and the true defect category, and the target image and the control image, respectively. The evaluation unit 412 is used to evaluate the loss of the computer vision model and the multimodal large model based on the similarity.

[0092] Understandably, the defect detection device provided above performs the defect detection method corresponding to Embodiment 1 provided above. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects of the scheme corresponding to the defect detection method of Embodiment 1 above, which will not be repeated here.

[0093] Example 4:

[0094] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the defect detection method in Embodiment 1 or Embodiment 2 above.

[0095] Example 5:

[0096] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the defect detection method in Embodiment 1 or Embodiment 2 above.

[0097] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A defect detection method, characterized in that, include: The original image is input into a preset computer vision model to detect the defect region, the first defect category and its corresponding first confidence level; The target image corresponding to the defect area is cropped from the original image; The target image and the first defect category are input into a preset multimodal large model to detect the defect level, defect text description, second defect category and its corresponding second confidence level.

2. The defect detection method according to claim 1, characterized in that, After inputting the original image into a preset computer vision model to detect the defect region, the first defect category, and its corresponding first confidence level, and before cropping the target image corresponding to the defect region from the original image, the method further includes: Based on the first defect category, the cropping threshold of the original image is matched from a preset mapping table, wherein the mapping table includes the defect category and its corresponding cropping threshold; The defect region and the trimming threshold are input into the Non-Maximum Suppression (NMS) function to adjust the defect region.

3. The defect detection method according to claim 1, characterized in that, After inputting the target image and the first defect category into a preset multimodal large model to detect the defect level, defect text description, second defect category, and its corresponding second confidence level, the method further includes: The first and second confidence levels are weighted and summed to obtain the overall confidence level; A defect detection report is generated based on the overall confidence level, defect level, defect text description, and secondary defect category.

4. The defect detection method according to claim 3, characterized in that, After generating the defect detection report based on the comprehensive confidence level, defect level, defect text description, and second defect category, the process also includes: Determine whether the first defect category and the second defect category are consistent; In response to the inconsistency between the first defect category and the second defect category, the defect detection report is marked as uncertain.

5. The defect detection method according to claim 4, characterized in that, Following the marking of the defect detection report as uncertain, the following is also included: Obtain user feedback data, which includes the actual categories of defects and their corresponding comparison images; Based on the first defect category, the second defect category, the target image, the true defect category and its corresponding comparison image, the loss of the computer vision model and the multimodal large model are evaluated respectively. Optimize computer vision models and multimodal large models based on loss.

6. The defect detection method according to claim 5, characterized in that, The step of evaluating the loss of the computer vision model and the multimodal large model based on the first defect category, the second defect category, the target image, the true defect category and its corresponding reference image, specifically includes: Calculate the similarity between the first defect category and the true defect category, the second defect category and the true defect category, and the target image and the control image, respectively; Based on similarity, the loss of the computer vision model and the multimodal large model are evaluated separately.

7. The defect detection method according to claim 1, characterized in that, Computer vision models include convolutional neural network (CNN) feature extraction models, while multimodal large models include vision-language models.

8. A defect detection device, characterized in that, It includes a first detection module, a cutting module, and a second detection module. The first detection module is used to input the original image into a preset computer vision model to detect the defect region, the first defect category, and its corresponding first confidence level. The cropping module is used to crop the target image corresponding to the defect area from the original image. The second detection module is used to input the target image and the first defect category into a preset multimodal large model to detect the defect level, defect text description, second defect category and its corresponding second confidence level.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement a defect detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a defect detection method as described in any one of claims 1 to 7.

Images