Industrial defect detection method and device, storage medium and computer equipment

Abstract

The application discloses an industrial defect detection method and device, a storage medium and computer equipment. The method comprises the following steps: in response to a defect identification instruction of a target image, analyzing the defect identification instruction to obtain the target image, contrast requirement information of the target image and a reference image without defects; inputting the target image, the contrast requirement information and the reference image into a fine-tuned visual language model; the visual language model takes the semantics of the contrast requirement information as a constraint, compares the reference image with the target image, and outputs difference region information of the target image relative to the reference image; and based on the difference region information, highlighting the difference region and the defect type on the target image. Thus, the defect detection problem can be converted into an image difference detection problem, without the need to collect a large number of samples of different types of defects for model training, so that the model can take a single normal image without defects as a reference to realize the targeted difference identification between the target image and the reference image, and the accuracy and pertinence of image defect identification are improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and industrial artificial intelligence, and in particular to an industrial defect detection method, apparatus, storage medium and computer equipment. Background Technology

[0002] With the rapid development of industrial production, automated quality inspection based on machine vision plays a crucial role in manufacturing. Currently, industrial quality inspection primarily relies on deep learning technology to ensure product consistency. For example, a large number of labeled defect images are collected to train deep learning-based detection or segmentation models to locate defect areas in the images. However, supervised methods are highly dependent on defect samples; once a new defect appears or a product model changes, the original model becomes ineffective, requiring retraining and data collection. This approach is not very adaptable to the ever-changing image data in the industrial field. Moreover, traditional methods are "black box" inspections, meaning users cannot dynamically adjust inspection standards using natural language, making it difficult to meet the flexible needs of on-site attention to or neglect of specific defects. Summary of the Invention

[0003] In view of this, this application provides an industrial defect detection method, apparatus, storage medium, and computer equipment, which uses a normal image and a visual language model, and guides the model to perform difference detection and screen target defects through text prompts.

[0004] According to a first aspect of this application, an industrial defect detection method is provided, the method comprising: In response to a defect identification instruction for a target image, the defect identification instruction is parsed to obtain the target image, its comparison requirement information, and a defect-free reference image; The target image, the comparison requirement information, and the reference image are input into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs information on the difference regions in the target image relative to the reference image. Based on the difference region information, the difference region and defect type are highlighted on the target image.

[0005] According to a second aspect of this application, an industrial defect detection device is provided, the device comprising: The acquisition module is used to respond to the defect recognition instruction of the target image, parse the defect recognition instruction to acquire the target image and its comparison requirement information, and a defect-free reference image; The defect detection module is used to input the target image, the comparison requirement information and the reference image into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs the difference region information in the target image relative to the reference image. The display module is used to highlight the difference regions and defect types on the target image based on the difference region information.

[0006] According to a third aspect of this application, a readable storage medium is provided on which a program or instructions are stored, which, when executed by a processor, implement the steps of the above-described industrial defect detection method.

[0007] According to a fourth aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described industrial defect detection method.

[0008] Using the above technical solution, the target image, comparison requirement information, and a defect-free reference image, parsed from the user's defect identification command, are input into a finely tuned visual language model. The visual language model then performs image comparison under the semantic constraints of the comparison requirement information. Finally, based on the difference region information output by the visual language model, the difference regions and defect types are highlighted on the target image. On the one hand, relying on the semantic understanding and image analysis capabilities of the visual language model, anomalies or deviations related to the comparison requirements in the target image relative to the defect-free reference image are identified, efficiently locating the difference regions where defects are located and clarifying the defect type, thus improving the accuracy and specificity of image defect identification. On the other hand, the defect detection problem can be transformed into an image difference detection problem. While ensuring the model's generalization ability, it saves the time spent collecting samples during model training, allowing the model to achieve targeted difference identification between the target image and the reference image using a single defect-free normal image as a benchmark. Even when the product model changes, only the reference image needs to be replaced without retraining the model.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1A flowchart illustrating the industrial defect detection method provided in an embodiment of this application is shown; Figure 2 This illustration shows a schematic diagram of the practical application of the industrial defect detection method provided in the embodiments of this application; Figure 3 A structural block diagram of the industrial defect detection device provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the electronic structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0011] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0012] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0013] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “attached” to another element, it can be directly connected or attached to the other element, or there may be intermediate elements. Furthermore, “connected” or “attached” as used herein can include wireless connections or wireless interconnections. The term “and / or” as used herein includes all or any unit and all combinations of one or more associated listed items.

[0014] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. It should be understood that these embodiments are provided so that the disclosure of this application is thorough and complete, and that the concept of these exemplary embodiments is fully conveyed to those skilled in the art.

[0015] This embodiment provides an industrial defect detection method, such as Figure 1 As shown, the method includes: Step 101: In response to the defect recognition instruction of the target image, parse the defect recognition instruction to obtain the target image and its comparison requirement information, and a defect-free reference image.

[0016] The target image is the image to be detected, and the reference image is a normal image with the same subject as the image to be detected, but without defects.

[0017] It is understandable that, such as Figure 2 As shown, the system provides a human-machine interface (HMI) for defect detection, including a registration area, a command area, and a display area. When a user needs to identify defects in a target image, they can click the controls in the registration area to trigger a one-click capture or upload of a reference image, and generate comparison requirement information by entering or selecting natural language prompts in the command area. The final detection results and defect bounding boxes are displayed in real time in the display area.

[0018] For example, if a user enters "stains and reflections in the background area" in the ignore field and "structural defects of the product body" in the detection field, the system will combine these into a comparison requirement message: "Please ignore stains and reflections in the background area and only detect structural defects of the product body." Alternatively, the user can directly enter the complete comparison requirement message in the command area, such as "Focus on detecting scratches and dents on the metal surface, ignoring minor color differences."

[0019] Furthermore, the system also includes a voice-to-text conversion component, which can convert the user's voice input into the corresponding comparison requirement information, greatly reducing the time required for the user to input the information.

[0020] Step 102: Input the target image, the comparison requirement information, and the reference image into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs the information on the difference regions in the target image relative to the reference image.

[0021] The difference region information includes the bounding box coordinates of the difference region where defects exist, as well as the possible defect types.

[0022] In this embodiment, the visual language model can understand the user's specific concerns and standards for defect identification. Based on this, it identifies anomalies or deviations in the target image, i.e., potential defects, by comparing it with a defect-free reference image. This not only eliminates the need for training with a large number of defect images with different defect annotations, reducing the complexity of model training and data collection costs, but also adapts to the specific quality inspection needs of different products, defect types, and industries, greatly improving the accuracy and robustness of detection and providing precise location for subsequent accurate analysis, rework, or repair.

[0023] In practical applications, the visual language model adopts an architecture combining a visual encoder and a multimodal understanding network. The visual encoder may include a standard preprocessing module, a visual feature extraction network (such as VisionTransformer), and a global average pooling module. The multimodal understanding network adopts a composite network architecture with a large language model (LLM) at its core, integrating various modal encoders (speech / audio, etc.). In step 102, the visual language model uses the semantics of the comparison requirement information as a constraint, compares the reference image and the target image, and outputs defect information describing the difference regions. Specifically, this includes the following steps: Step 102-1: The reference image and the target image are divided into image blocks by a visual feature extraction network, and the image blocks are converted into corresponding visual feature sequences.

[0024] Step 102-2: The comparison requirement information is converted into a text feature sequence through the text embedding layer.

[0025] Step 102-3: The visual feature sequence is mapped to the same feature dimension as the text feature sequence through the projection layer, and the mapped reference image feature sequence, target image feature sequence and text feature sequence are concatenated to form a multimodal input sequence.

[0026] Step 102-4: Input the multimodal input sequence into the multimodal understanding network, use the multi-head self-attention mechanism to capture the relationship between image features and text semantics, analyze the differences between the reference image and the target image under the constraint of contrast requirement information, and generate a text or identifier sequence containing information about the difference regions.

[0027] In this embodiment, the visual language model no longer relies on complex cross-attention interactions between separate image and text processing units. Instead, it employs a more advanced Decoder-only architecture or Encoder-Decoder fusion architecture. By treating images as a "visual language," its features are serialized and concatenated with text sequences. Leveraging the powerful contextual understanding and attention mechanisms of large language models, difference comparisons are performed directly within a unified semantic space. This approach more fully integrates visual and semantic information, not only accurately locating differences but also understanding complex comparison instructions (such as "ignore illumination" or "focus on structure") through the reasoning capabilities of the language model, thus outputting detection results that are more consistent with human logic.

[0028] Understandably, to align visual features with textual semantics, visual feature extraction networks can employ architectures such as ViT (Vision Transformer) and ResNet, learning general visual representations through pre-training on large amounts of image data. Multimodal understanding networks can utilize language models based on Transformer Decoders, such as those built upon LLaMA, Qwen, and Vicuna. By placing a projection layer (such as a linear layer or MLP) between the two, visual features are "translated" into token embeddings that the language model can understand, enabling the model to "read" images like it reads text, thereby achieving pixel-level fine-grained difference perception and logical reasoning.

[0029] Furthermore, in one embodiment, if the comparison requirement information includes difference information to be ignored, the visual language model attenuates or masks the attention weights associated with the difference features based on the difference information to be ignored during the attention calculation process of the multimodal understanding network.

[0030] In this embodiment, intelligent filtering of non-focused differences is achieved by adjusting the attention weight matrix in the self-attention mechanism or by modifying the Prompt to guide the model to ignore specific features during inference.

[0031] For example, when given the instruction to "ignore changes in illumination," the model reduces its attention response to the brightness feature channel or the corresponding visual token, preventing it from dominating the final difference determination. This allows the model to more accurately capture the essential similarities defined in the instruction, enabling intelligent filtering.

[0032] Understandably, the differences to be ignored in the comparison request information can also be directly identified through end-to-end semantic understanding. For example, the user inputs the comparison request information as "Please find the defects on the objects in these two images; color can be ignored." The multimodal understanding network suppresses the activation of color-related visual features during internal feature interactions through semantic analysis. This mechanism can be achieved either through explicit attention masks or through implicit attention distribution adjustments learned during model training. This ensures that the final output difference region focuses on the structural defects that the user is concerned with.

[0033] Step 103: Highlight the difference regions and defect types on the target image based on the difference region information.

[0034] The industrial defect detection method provided in this application involves inputting a target image parsed from a user-input defect identification command, comparison requirement information, and a defect-free reference image into a finely tuned visual language model. The visual language model then performs image comparison under the semantic constraints of the comparison requirement information. Finally, based on the difference region information output by the visual language model, the difference regions and defect types are highlighted on the target image. On one hand, relying on the semantic understanding and image analysis capabilities of the visual language model, it identifies anomalies or deviations in the target image relative to the defect-free reference image that are related to the comparison requirements, efficiently locating the difference regions where defects are located and clarifying the defect type, thus improving the accuracy and specificity of image defect identification. On the other hand, it transforms the defect detection problem into an image difference detection problem, saving the time spent collecting samples during model training while ensuring the model's generalization ability. The model can achieve targeted difference identification between the target image and the reference image using a single defect-free normal image as a benchmark. Even when product models change, only the reference image needs to be replaced without retraining the model.

[0035] For example, before online inspection, for each type of product to be inspected, a defect-free sample image is identified and registered as a Reference Image. This image does not need to be collected in large quantities; a single image is sufficient.

[0036] During the online inspection phase, images of the products to be inspected on the production line are captured by cameras and denoted as Target Images. The Reference Image, Target Image, and Prompt (comparison requirement information) are combined. For example, "Please carefully compare the two images to find the true differences. Please ignore differences caused only by viewpoint / perspective changes or lighting / brightness changes. Output the bounding box coordinates of each true difference region in the second image, in the format [x0,y0,x1,y1]; if there is no difference, output [0,0,0,0]." This combined information is input into a finely tuned visual language model. Internally, the model uses a cross-attention mechanism to compare the feature maps of the two images, combined with the semantic constraints of the text instructions, to calculate the differences and regress the bounding boxes. The text string output by the model is parsed. If the parsed coordinates are [x0,y0,x1,y1], it is judged as NG, and a defect box is displayed on the interface. If the output is [0,0,0,0], it is judged as OK.

[0037] In practical applications, the information about the difference region includes the bounding box coordinates and defect type of the difference region. Step 103 specifically includes: drawing the bounding box of the difference region on the target image based on the bounding box coordinates; rendering the difference region based on preset parameters associated with the defect type; and marking the difference region with the defect type.

[0038] In this embodiment, by drawing dedicated bounding boxes for the discrepancy regions on the target image, abnormal discrepancy regions in the image can be located intuitively and accurately, making the location and extent of the regions clearly identifiable. Simultaneously, by combining preset parameters associated with defect types, the discrepancy regions are rendered specifically and labeled with the corresponding defect type. This achieves both visual differentiation of different defect types and precise correspondence between defect types and region features, greatly improving the intuitiveness, recognizability, and information completeness of image defect recognition results. This facilitates rapid identification and judgment of the location, extent, and specific type of defects by staff, effectively improving the efficiency of defect detection and review.

[0039] In one embodiment, prior to step 102, the visual language model can be fine-tuned to adapt it to industrial defect scenarios. Specifically, the following method can be used to fine-tune the visual language model.

[0040] Method 1: Monitoring and fine-tuning, specifically including: A sample dataset is created based on a defect database in the industrial field; the defect coordinates and types of positive samples and the defect-free labels (such as coordinates [0,0,0,0]) of negative samples in the sample dataset are converted into labels in text sequence format; the visual language model is fine-tuned with the goal of minimizing the autoregressive generation loss (Next TokenPrediction Loss) between the predicted text sequence output by the model and the label text sequence.

[0041] The sample dataset includes positive and negative samples. Positive samples consist of associated normal and defective images. Defective images have text prompts and defect labels. The text prompts record the comparison requirements input during defect detection, while the defect labels record the defect type and location obtained after detection. Negative samples include defect-free baseline normal images and perturbed normal images.

[0042] It should be noted that the proportion of negative samples in the sample dataset should be controlled between 20% and 50%, for example, 30% or 40%. This not only avoids insufficient learning of defect features by the model due to too few positive samples, greatly reducing the model's need for defect samples, but also prevents the model from over-focusing on normal patterns and ignoring minor defects when there are too many negative samples. Thus, it balances the model's defect recall rate and false positive rate.

[0043] In this embodiment, a database containing positive and negative samples is constructed. Positive samples provide the model with visual-text aligned defect feature references, while negative samples construct clear boundaries between defective and non-defective scenes. The model learns from the positive and negative samples through a unified text generation task, mapping image difference features to specific coordinate token sequences. Whether locating defects or determining the absence of defects, the problem is essentially unified into predicting the probability distribution of the next token, allowing the model to understand the feature patterns of "difference" and "no difference" within a unified semantic space. This not only enables the model to focus on the essential features of defects, reduce background interference, and improve generalization across defect categories, but also enhances the model's immunity to non-defect factors such as noise and lighting changes by utilizing negative sample perturbations, reducing false positives.

[0044] It is worth mentioning that, during the process of selecting positive samples from the defect database, a large language model can be used to generate prompt texts that have the same meaning as the original comparison requirement information but with different expressions, and multiple text prompt labels can be generated. During model training, each image data is randomly matched with a text to prevent the model from overfitting to specific sentence structures, enhance the visual language model's adaptability to different user expression styles, and improve the model's practicality.

[0045] In one embodiment, generating a sample dataset specifically includes: selecting multiple defect-free normal images from a defect database; superimposing linear structure noise on the normal images along random directions or incorporating preset defect images into the normal images to generate defect images associated with the normal images, and generating defect labels for the defect images based on the positions of the linear structure noise or preset defect images in the normal images; performing unstructured geometric and photometric transformations on the normal images as reference normal images to generate perturbed normal images, and generating defect-free labels for the perturbed normal images.

[0046] Among them, non-structural geometric and photometric transformations include, but are not limited to, background adjustment methods that are unrelated to defects, such as rotation, scaling, translation, lighting, perspective, and contrast.

[0047] In this embodiment, for positive samples, new defect images are synthesized by superimposing linear structural noise or incorporating preset defects, and defect labels are automatically generated. For negative samples, perturbed normal images are automatically obtained without defect labels by perturbing normal images under different imaging conditions. This automatically synthesizes sample images covering a variety of possible small-sized, subtle, or specific-shaped defects, as well as factors that may occur in actual production environments, such as lighting changes, viewing angle shifts, and slight vibrations. While ensuring that the model is trained and optimized specifically for industrial defects, i.e., the actual environment, this significantly reduces the arduous work of manually collecting samples and labeling defects, saving considerable time and labor costs.

[0048] Method two, reinforcement learning optimization, specifically includes: Multiple candidate actions are sampled from the action probability distribution output by the visual language model for the sample dataset; scores for the candidate actions are generated based on a preset reward criterion, and relative reward signals between the candidate actions are generated; based on the relative reward signals, policy gradients that do not depend on the independent value function network are calculated, and the model parameters of the visual language model are updated using the policy gradients.

[0049] The candidate actions include bounding box coordinates sampled based on probability distribution, as well as variants generated by applying random perturbations to the bounding box coordinates.

[0050] In this embodiment, by sampling candidate actions from the action probability distribution output by the visual language model and generating its perturbation variants, the effective exploration range of the action space can be fully explored, improving the diversity and robustness of the model's action predictions, such as bounding box coordinates. Furthermore, scores and relative reward signals are generated based on preset reward criteria, and the model parameters are updated by calculating the policy gradient independent of the independent value function network based on the relative reward signal. This increases the probability of generating candidate actions with higher relative rewards and suppresses the probability of generating candidate actions with low relative rewards, making the policy optimization of the visual language model more targeted and efficient. This avoids the complexity and overhead of training an independent value function network while further reducing the false alarm rate of the visual language model and improving its sensitivity to subtle differences.

[0051] It should be noted that Method 1 and Method 2 can be used to optimize the model individually or jointly.

[0052] In one embodiment, after step 103, the industrial defect detection method further includes: extracting differential image features from the target image based on the differential region, and matching the differential image features and defect type with a pre-built defect cause knowledge base; based on the matching result, locating and outputting the cause of the defect in the target image.

[0053] The defect cause knowledge base stores the mapping relationship between different defect types, their corresponding typical feature patterns, and potential production causes.

[0054] Specifically, differential image features may include quantitative features such as texture, size, area, shape, or intensity used to assess the severity of defects.

[0055] In this embodiment, the matching logic of the knowledge base achieves a precise association between the features, types, and causes of differing images. This eliminates the need for manual analysis and investigation, significantly improving the efficiency and accuracy of locating the causes of industrial defects and providing a direct and effective basis for subsequent defect rectification and problem tracing.

[0056] In one embodiment, after determining the cause of the defect in the target image, the target image, reference image, information about the difference region, and the cause can be stored as historical cases in the defect database. This achieves standardized retention of defect-related information across all dimensions, facilitating defect case resources for subsequent task tracing or reuse.

[0057] Furthermore, prior to step 102, the industrial defect detection method also includes: searching the defect database based on the target image; if there are historical cases that match the target image, displaying the cause and comparison requirement information in the historical cases.

[0058] In this embodiment, before identifying defects, the target image indicated by the defect identification command is first matched with defect images in the defect database based on their similarity. Information regarding the causes of defects and comparison requirements during detection, recorded in historical cases of defect images with high similarity to the target image, is displayed. This allows users to quickly grasp the key points, comparison standards, and known causes of similar defects or similar detection subjects, significantly improving the targeting and efficiency of defect identification and reducing the time cost of ineffective analysis and investigation. Furthermore, after completing the current detection, existing information from historical cases can be used to assist in verifying the potential defect types and causes of the target image, reducing identification bias.

[0059] The industrial defect detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0060] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0061] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0062] Furthermore, such as Figure 3 As shown, as a specific implementation of the above-mentioned industrial defect detection method, this application provides an industrial defect detection device 300, which includes: an acquisition module 301, a defect detection module 302, and a display module 303.

[0063] The acquisition module 301 is used to respond to the defect recognition instruction of the target image, parse the defect recognition instruction to acquire the target image and its comparison requirement information, and a defect-free reference image; The defect detection module 302 is used to input the target image, the comparison requirement information and the reference image into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs the difference region information in the target image relative to the reference image. Display module 303 is used to highlight the difference region and defect type on the target image based on the difference region information.

[0064] Furthermore, the visual language model includes a visual feature extraction network and a multimodal understanding network; the defect detection module 302 is specifically used to convert the reference image and the target image into visual feature sequences through the visual feature extraction network; to embed the contrast requirement information to generate a text feature sequence; to map the visual feature sequence and the text feature sequence to the same feature space and to concatenate the sequences to generate a multimodal input sequence; to input the multimodal input sequence into the multimodal understanding network, to use the attention mechanism to extract the difference features between the reference image and the target image under the constraint of the contrast requirement information, and to decode and output the difference region information.

[0065] Furthermore, the defect detection module 302 is also used to attenuate or mask the attention weights associated with the difference features based on the difference information to be ignored during the attention calculation process of the multimodal understanding network in the visual language model if the comparison requirement information contains difference information to be ignored.

[0066] Furthermore, the industrial defect detection device 300 also includes: The sample construction module (not shown in the figure) is used to create a sample dataset from an industrial defect database. The sample dataset includes positive samples and negative samples. Positive samples include associated normal images and defective images. The defective images have text prompt labels and defect labels. Negative samples include a baseline normal image without defects and a perturbed normal image. The first model fine-tuning module (not shown in the figure) is used to optimize the model parameters of the visual language model with the goal of minimizing the autoregressive generation loss between the predicted text sequence output by the visual language model and the label text sequence.

[0067] Furthermore, negative samples account for 20% to 50% of the sample dataset.

[0068] Furthermore, the sample construction module is specifically used to select multiple defect-free normal images from the defect database; to superimpose linear structure noise on the normal images along random directions or to integrate preset defect images into the normal images to generate defect images associated with the normal images, and to generate defect labels for the defect images based on the position of the linear structure noise or the preset defect images in the normal images; to perform unstructured geometric and photometric transformations on the normal images as reference normal images to generate perturbed normal images, and to generate defect-free labels for the perturbed normal images.

[0069] Furthermore, the industrial defect detection device 300 also includes: The second model fine-tuning module (not shown in the figure) is used to sample multiple candidate actions from the action probability distribution output by the visual language model for the sample dataset. The candidate actions include bounding box coordinates sampled based on the probability distribution, as well as variants generated by applying random perturbations to the bounding box coordinates. The module generates scores for the candidate actions based on a preset reward criterion and generates relative reward signals between the candidate actions. Based on the relative reward signals, the module calculates the policy gradient, which is independent of the independent value function network, and uses the policy gradient to update the model parameters of the visual language model.

[0070] Furthermore, the difference region information includes the bounding box coordinates and defect type of the difference region; the display module 303 is specifically used to draw the bounding box of the difference region on the target image based on the bounding box coordinates; the difference region is rendered based on preset parameters associated with the defect type, and the difference region is marked with the defect type.

[0071] Furthermore, the industrial defect detection device 300 also includes: The attribution module (not shown in the figure) is used to extract differential image features from the target image based on the differential regions, and match the differential image features and defect types with a pre-built defect cause knowledge base; based on the matching results, it locates and outputs the cause of the defect in the target image.

[0072] Furthermore, the industrial defect detection device 300 also includes: The storage module is used to associate and store the target image, reference image, information on the difference area, and the cause as historical cases in the defect database.

[0073] The retrieval module is used to search the defect database based on the target image; The display module 303 is also used to display the reasons and comparison requirements of historical cases that match the target image if such cases exist.

[0074] Specific limitations regarding industrial defect detection devices can be found in the limitations of industrial defect detection methods described above, and will not be repeated here. Each module in the aforementioned industrial defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0075] Based on the above, Figure 1 Accordingly, embodiments of this application also provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 The industrial defect detection method shown.

[0076] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0077] Based on the above, Figure 1 The method shown, and Figure 3 The virtual device embodiment shown is designed to achieve the above objectives, such as... Figure 4 As shown in the figure, this application embodiment also provides a computer device 400, which includes a processor 401 and a memory 402. The memory 402 stores a program or instructions that can run on the processor 401. When the program or instructions are executed by the processor 401, they implement the above-mentioned... Figure 1 The industrial defect detection method shown.

[0078] The memory 402 can be used to store software programs and various data. The memory 402 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 402 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 402 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0079] Processor 401 may include one or more processing units; optionally, processor 401 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 401.

[0080] Computer equipment can specifically include personal computers, servers, network devices, etc.

[0081] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0082] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0085] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. An industrial defect detection method, characterized in that, The method includes: In response to a defect identification instruction for a target image, the defect identification instruction is parsed to obtain the target image, its comparison requirement information, and a defect-free reference image; The target image, the comparison requirement information, and the reference image are input into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs information on the difference regions in the target image relative to the reference image. Based on the difference region information, the difference region and defect type are highlighted on the target image.

2. The industrial defect detection method according to claim 1, characterized in that, The visual language model includes a visual feature extraction network and a multimodal understanding network; the visual language model uses the semantics of the comparison requirement information as constraints to compare the reference image and the target image, and outputs information on the difference regions in the target image relative to the reference image, including: The visual feature extraction network encodes image patches in the reference image and the target image into visual features, respectively. The comparison requirement information is embedded to generate a text feature sequence; The visual feature sequence and the text feature sequence are mapped to the same feature space and then concatenated to generate a multimodal input sequence; The multimodal input sequence is input into the multimodal understanding network, and the attention mechanism is used to extract the difference features between the reference image and the target image under the constraint of the contrast requirement information, and the difference region information is decoded and output.

3. The industrial defect detection method according to claim 2, characterized in that, The method further includes: If the comparison requirement information includes difference information to be ignored, the visual language model attenuates or masks the attention weights associated with the difference features based on the difference information to be ignored during the attention calculation process of the multimodal understanding network.

4. The industrial defect detection method according to claim 1, characterized in that, Before inputting the target image, the contrast requirement information, and the reference image into the fine-tuned visual language model, the method further includes: A sample dataset is created using a defect database in the industrial field. The sample dataset includes positive samples and negative samples. The positive samples include associated normal images and defective images. The defective images have text prompt labels and defect labels. The negative samples include a baseline normal image without defects and a perturbed normal image. The model parameters of the visual language model are optimized with the goal of minimizing the autoregressive generation loss between the predicted text sequence output by the visual language model and the labeled text sequence.

5. The industrial defect detection method according to claim 4, characterized in that, The creation of a sample dataset using an industrial defect database includes: Select multiple defect-free normal images from the defect database; Linear structure noise is superimposed on the normal image along a random direction or a preset defect image is incorporated into the normal image to generate a defect image associated with the normal image, and a defect label is generated for the defect image based on the position of the linear structure noise or the preset defect image in the normal image; The normal image is used as the reference normal image to undergo unstructured geometric and photometric transformations to generate the perturbed normal image, and a defect-free label is generated for the perturbed normal image.

6. The industrial defect detection method according to claim 1, characterized in that, Before inputting the target image, the contrast requirement information, and the reference image into the fine-tuned visual language model, the method further includes: Multiple candidate actions are sampled from the action probability distribution output by the visual language model for the sample dataset. The candidate actions include bounding box coordinates sampled based on the probability distribution, and variants generated by applying random perturbations to the bounding box coordinates. The candidate actions are scored based on a preset reward criterion, and relative reward signals are generated between the candidate actions. Based on the relative reward signal, the policy gradient, which is independent of the independent value function network, is calculated, and the model parameters of the visual language model are updated using the policy gradient.

7. The industrial defect detection method according to claim 1, characterized in that, The difference region information includes the bounding box coordinates and defect type of the difference region; highlighting the difference region and defect type on the target image based on the difference region information includes: Based on the bounding box coordinates, draw the bounding box of the difference region on the target image; The difference region is rendered based on the preset parameters associated with the defect type, and the difference region is marked with the defect type.

8. An industrial defect detection device, characterized in that, The device includes: The acquisition module is used to respond to the defect recognition instruction of the target image, parse the defect recognition instruction to acquire the target image and its comparison requirement information, and a defect-free reference image; The defect detection module is used to input the target image, the comparison requirement information and the reference image into the fine-tuned visual language model. The visual language model uses the semantics of the comparison requirement information as a constraint to compare the reference image and the target image, and outputs the difference region information in the target image relative to the reference image. The display module is used to highlight the difference regions and defect types on the target image based on the difference region information.

9. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the industrial defect detection method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the industrial defect detection method as described in any one of claims 1 to 7.

Images