Defect detection method based on multi-view training and single-view reasoning

Abstract

The invention belongs to the technical field of appearance defect detection of industrial manufacturing scenes, and particularly relates to a defect detection model training method and a defect detection method, and the method comprises the following steps: collecting a multi-view image of a target object; extracting image features of the multi-view image through an image encoder; based on the multi-view image, obtaining a corresponding view description text; extracting text features of the view description text through a text encoder; fusing the image features and the text features to obtain image-text fusion features; inputting the image-text fusion feature into a defect detection head, and outputting a category prediction result; and calculating classification loss according to the difference between the classification prediction result and the real label, and optimizing the parameters of the image encoder, the text encoder and the defect detection head based on the classification loss. According to the invention, the accuracy and robustness of hygristor chip appearance defect detection can be improved.

Description

Technical Field

[0001] This invention belongs to the technical field of appearance defect detection in industrial manufacturing scenarios, specifically targeting resistive chips of humidity sensors, and relates to a defect detection method based on multi-view training and single-view inference. Background Technology

[0002] In humidity sensor devices, resistive chips serve as the basic component, reflecting changes in environmental humidity information through changes in resistance, thereby realizing the sensor's sensing function. Some manufacturers directly perform functional electrical signal testing on the integrated sensors during the sensor production line. However, in the early stages of the production line, the chips may already have some significant appearance defects, including but not limited to scratches, carbon wire short circuits, carbon wire breaks, and incomplete coverage of the moisture-sensing film, leading to unqualified final products. If appearance defect detection can be performed on the chips in the early stages and unqualified chips can be screened out, defective products can be prevented from entering the subsequent production line, thereby saving costs. Appearance defect detection has always been a key research topic in the industrial production chain. Early reliance on manual visual inspection consumed a lot of manpower costs. Automated Visual Inspection (AVI) technology has significantly improved the defect detection capability and efficiency in industrial production.

[0003] For the appearance inspection of humidity-sensitive resistor chips, the current mainstream technologies are divided into two categories: one is based on digital image processing technology, and the other is based on deep learning technology.

[0004] The core idea of ​​this digital image processing-based approach is to identify defects through manually designed image features and a rule-based logical judgment process. Specifically, for overly bright or dark areas in the original image, homomorphic filtering or adaptive histogram equalization (CLAHE) is used for correction; median filtering is used to remove salt-and-pepper noise; then, artificial visual features are extracted from the optimized image—the outlines of the chip body, moisture-sensing film, and carbon lines are extracted using the Canny operator or Sobel gradient operator; grayscale histograms are calculated for grayscale images; and chromaticity components are further analyzed in HSV or Lab color spaces for color images. Morphological operations (such as opening and closing operations) are used to process the binarized image and apply it for connected component analysis. Based on the extracted artificial features, a hierarchical decision-making process is performed: the first layer determines whether the chip as a whole is complete and meets geometric constraints (such as aspect ratio, symmetry, and spacing consistency); the second layer determines the integrity of individual structures (such as whether carbon lines, moisture-sensing films, and pins are complete and qualified); and the third layer focuses on the surface condition, checking for defects such as scratches, stains, or oxide spots.

[0005] Deep learning-based methods take a data-driven approach, allowing the model to automatically learn the intrinsic representation of defects from samples without manual intervention in feature design. In supervised learning paradigms, classification networks (such as ResNet and EfficientNet) are typically used to classify the entire chip image as good or bad, or to classify it into specific defect types with fine granularity. If defect location information needs to be obtained simultaneously, object detection models (such as the YOLO series and DETR series) are used to output bounding box coordinates and category labels. For applications requiring pixel-level accuracy, such as measuring scratch length or moisture-sensitive membrane coverage area, semantic segmentation models (such as U-Net, Mask R-CNN, or YOLO-seg) are used. However, in real-world industrial scenarios, defect samples are often scarce and diverse, making it difficult to collect labeled data covering all anomalies. To address this, unsupervised or weakly supervised deep learning approaches are often adopted. These require only a large number of normal samples for training, constructing a distribution model of normal images in a deep feature space, and classifying samples that deviate from this distribution during testing as anomalies.

[0006] Traditional image processing has limited generalization capabilities, only allowing for the setting of judgment logic based on specific conditions. Once a new type of defect not covered by the judgment rules appears, detection performance drops significantly. Deep learning-based methods are gradually becoming the mainstream choice for high-precision, high-reliability electronic component appearance inspection, but their detection robustness still has room for improvement. Feature enhancement is particularly necessary when the number of samples is limited and conventional data augmentation methods are insufficient. Future development trends will focus on the effective fusion of multimodal features to further improve the model's generalization and discriminative capabilities. Summary of the Invention

[0007] This application discloses a training method for a defect detection model and a defect detection method, which can improve the accuracy and robustness of appearance defect detection for humidity-sensitive resistor chips.

[0008] The first aspect of this application discloses a method for training a defect detection model, comprising the following steps: Acquire multi-view images of the target object; The image features of the multi-view images are extracted using an image encoder; Based on the multi-view images, corresponding view description text is obtained; The text features of the perspective description text are extracted using a text encoder. The image features and the text features are fused to obtain image-text fusion features; The image-text fusion features are input into the defect detection head, and the category prediction result is output. The classification loss is calculated based on the difference between the category prediction result and the true label, and the parameters of the image encoder, the text encoder and the defect detection head are optimized based on the classification loss.

[0009] As an optional implementation, in the first aspect of this application embodiment, the acquisition of multi-view images of the target object and the extraction of image features from the multi-view images using an image encoder include the following steps: Bright-field images of the target object are acquired using a bright-field camera; Dark-field images of the target object are acquired using a dark-field camera; The bright-field features of the bright-field image and the dark-field features of the dark-field image are extracted using an image encoder.

[0010] As an optional implementation, in the first aspect of this application, fusing the image features and the text features to obtain image-text fused features includes the following steps: The image features and text features are aggregated using a Cross-Attention structure. The semantic prior of the viewpoint description file is used as a guiding signal to drive different branches to perform targeted feature extraction on the image. This is achieved through the following calculations: , , ,

[0011] in, For text representation, The length of the text sequence. For feature dimensions; For image feature maps, The length of the image patch sequence; Q indicates that the query is based on image features; K indicates that the text representation is the key; V indicates that the text representation is the value; It is a learnable projection matrix.

[0012] After obtaining the image-text fusion features, the following steps are also included: The image-text fusion features are aligned by calculating the symmetric KL divergence loss.

[0013] As an optional implementation, in the first aspect of the embodiments of this application, aligning the image-text fusion features by calculating the symmetric KL divergence loss includes the following steps: Based on the Softmax function, the image-text fusion features are mapped to the probability space to obtain a normalized bright-field feature distribution. Dark field feature distribution : , , in, It is a bright field characteristic. Features of dark fields and Represent the eigenvectors of the bright field and dark field respectively at the th... Values ​​in each dimension ; Minimize the difference between the bright field features and the dark field features using KL divergence: Calculate the forward KL divergence: , Calculate the reverse KL divergence: , Calculate the symmetric KL divergence of the bright-field characteristic distribution and the dark-field characteristic distribution:

[0014] By constraining the bright field feature distribution and the dark field feature distribution to tend to be consistent, the alignment of the image-text fusion features is achieved.

[0015] As an optional implementation, in the first aspect of the embodiments of this application, the step of calculating the classification loss based on the difference between the category prediction result and the true label includes the following steps: The cross-entropy loss is calculated by comparing the predicted category results with the true labels, as follows: , Where N is the number of samples in a batch; C is the number of categories; for the th Each sample has a true label that is a one-hot vector. The unnormalized logits predicted by the model are .

[0016] As an optional implementation, in the first aspect of the embodiments of this application, the image features and the text features are fused through an image-text feature aggregation module to obtain image-text fused features; Based on the classification loss, the parameters of the image encoder, the text encoder, the image and text feature aggregation module, and the defect detection head are updated through backpropagation algorithm, and iterative optimization is performed until convergence.

[0017] The second aspect of this application discloses a defect detection method based on multi-view training and single-view reasoning, employing a model training method disclosed in the first aspect of this application, including: Multi-view training phase: Acquire multi-view images of the target object; Extract image features from the multi-view images; Based on the multi-view images, corresponding view description text is obtained; Extract the text features of the perspective description text; The image features and the text features are fused to obtain image-text fusion features; Align the image-text fusion features; The image-text fusion features are input into the defect detection head, and the category prediction result is output. The classification loss is calculated based on the difference between the predicted category and the true label, and the parameters of the defect detection head are optimized based on the classification loss. The parameters of the defect detection head are iteratively optimized to obtain a defect detection model; Inference phase: The target image and its corresponding view description text are input into the trained model using a single viewpoint, and the defect category prediction results are output.

[0018] As an optional implementation, in a second aspect of the embodiments of this application, aligning the image-text fusion features includes the following steps: The image-text fusion features are mapped to a probability space to obtain bright-field feature distributions and dark-field feature distributions. The symmetric KL divergence between the bright-field feature distributions and the dark-field feature distributions is calculated to constrain the bright-field feature distributions and the dark-field feature distributions to tend to be consistent, so as to achieve the alignment of the image-text fusion features.

[0019] The third aspect of this application discloses a defect detection system, which employs a defect detection method based on multi-view training and single-view reasoning disclosed in the second aspect of this application, including: The image acquisition module is used to acquire perspective images of the target product; The image encoding module is used to extract image features from the viewpoint image; A text generation module is used to generate viewpoint description text for the viewpoint image; A text encoding module is used to extract text features from the perspective description text; The image and text feature aggregation module is used to fuse the image features and the text features to generate image-text fusion features; The alignment loss module is used to align the distribution of the image-text fusion features; The detection output module outputs prediction scores for each category based on the image-text fusion features.

[0020] Compared with related technologies, the embodiments of this application have the following beneficial effects: 1. This invention combines complementary information from bright-field oblique side view and dark-field top view, making full use of the differences under different optical conditions, and jointly models the two perspectives during the training phase, which significantly improves the model's ability to learn the defect features of humidity-sensitive resistor chips.

[0021] 2. By incorporating textual descriptions related to the imaging perspective into the network training process, the model can adaptively focus on the structures and anomalous regions most relevant to the current perspective during the feature extraction stage, based on semantic prior knowledge.

[0022] 3. In the training phase, this invention utilizes multi-view data for joint learning to fully explore cross-view consistency and complementarity; while in the inference phase, it relies on only a single view input, which is in line with the actual conditions of industrial sites where only a single camera is usually configured, and achieves high-precision detection without increasing hardware costs. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a training method for a defect detection model disclosed in an embodiment of this application; Figure 2 This is one of the flowcharts of a defect detection method based on multi-view training and single-view reasoning disclosed in the embodiments of this application; Figure 3 This is a second flowchart illustrating a defect detection method based on multi-view training and single-view reasoning disclosed in an embodiment of this application. Figure 4 This is a diagram of the multi-view training-single-view inference deep learning framework disclosed in the embodiments of this application; Figure 5 This is a schematic diagram of the dual-view image acquisition hardware system disclosed in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a defect detection system disclosed in an embodiment of this application.

[0025] The module consists of: 1. Image acquisition module; 2. Image encoding module; 3. Text generation module; 4. Text encoding module; 5. Image and text feature aggregation module; 6. Alignment loss module; and 7. Detection output module. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms “comprising” and “having”, and any variations thereof, in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0028] In the embodiments of this application, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0029] This invention proposes a deep learning framework based on multi-view training and single-view inference: different geometric perspectives and multimodal features are integrated during the training phase, while only a single-view input is required during the inference phase, thereby improving the accuracy and robustness of appearance defect detection for humidity-sensitive resistor chips.

[0030] This invention introduces multi-view (e.g., dark-field illumination top view, bright-field illumination oblique view) and multi-modal (e.g., text, image) approaches during the training phase, enriching the representational information of samples from multiple dimensions such as geometric structure, optical response, and semantic priors. However, in real-world industrial inspection scenarios, production lines operate rapidly with limited space, typically deploying only a single fixed-view camera system, making it difficult to acquire multi-view or multi-modal images in real time. Therefore, using single-view input during the inference phase is more practical. The multi-view training-single-view inference strategy is effective because the model learns more discriminative and robust shared feature representations through cross-view and cross-modal feature alignment during training, enabling even a single-view input to activate fully trained deep semantics, thus balancing performance and deployment feasibility.

[0031] Capture a top view of the dark field lighting and a side view of the bright field lighting: The difference between dark field and bright field illumination lies in the relative positions of the light source and the camera. When the camera is positioned in the specular reflection path (i.e., the direction of the emitted light), it is bright field illumination; otherwise, it is dark field illumination. Light primarily undergoes specular reflection on smooth areas such as thin films on chip surfaces, while diffuse reflection occurs in rougher areas. Therefore, bright field images effectively capture information from the surface of thin films, while dark field images are better suited for obtaining structural details beneath the thin film. Figure 5 As shown, in this invention, the humidity-sensitive resistor chip (the object to be detected) is placed on a horizontal plane, the light source is set on one side at a 45° angle to the horizontal plane, the bright field camera is placed at a 135° position on the opposite side, and the dark field camera is placed directly above at a 90° angle.

[0032] Feature extraction using a two-branch network: During training, a weight-sharing dual-branch network is used to process both bright-field and dark-field data. Each perspective includes both images and text. For text data, a large language model is first used to enhance basic cue words (such as dark-field and top-down view) to generate specialized perspective descriptions (e.g., "A top-down dark-field illumination image is acquired with the camera positioned perpendicular to the sample plane, while illumination is delivered at an oblique angle."). Then, a Transformer Encoder is used as the text encoder to extract its semantic features. Since the Transformer Encoder's pre-trained model is based on a large-scale English corpus, the text data in this invention is uniformly presented in English. For image data, a Vision Transformer (ViT) is used as the image encoder.

[0033] Image and text feature aggregation module 5: A text encoder (Transformer Encoder) outputs a text representation. ,in The length of the text sequence. The feature dimension is used to output the image feature map. The image encoder (ViT) outputs the image feature map. ,in This refers to the length of the image patch sequence. This invention employs a Cross-Attention-based structure to aggregate image and text features, using the semantic prior of the perspective-description text as a guiding signal to drive different branches to extract more targeted features from the image. Specifically, using image features as the query and text representation as keys and values, text-guided image feature enhancement is achieved through the following calculations: , , , , in This is a learnable projection matrix. The image-text feature aggregation module of this invention enables each image patch to dynamically focus on the text description fragments most relevant to the current viewpoint semantics, thereby integrating geometric and optical prior knowledge at the feature level and improving the accuracy and robustness of the subsequent defect discrimination network.

[0034] Two-branch alignment loss: This invention employs a weight-sharing network to extract data features from different perspectives; therefore, a mechanism is needed to align the feature representations of each branch. and ,in and These represent the bright field and dark field features, respectively. This invention employs a method based on Kullback-Leibler Divergence (KL divergence) to align the feature distributions of the two branches. The specific calculation process is as follows: First, the output of the image and text feature aggregation module 5 should be mapped to the probability space using the Softmax function to obtain a normalized feature distribution. and : , , in and The eigenvector represents the eigenvector at the th... Values ​​in each dimension Next, we use KL divergence as a metric to minimize the difference between the two distributions. Similarly, we calculate the inverse KL divergence: , , To simultaneously consider the distribution differences in both directions, this invention employs symmetric KL divergence, which is the average of the two: .

[0035] During training, the goal of this invention is to minimize the symmetric KL divergence, thereby making the distributions of bright-field features and dark-field features as close as possible, helping the model learn a consistent representation across perspectives, enabling the model to better utilize multi-perspective information, and enhancing the consistency and generalization ability of features.

[0036] Dual-view image and text data preparation: This example deploys a hardware data acquisition system as follows: Figure 5 As shown, two industrial cameras are used to acquire images, with lens focal lengths continuously adjustable from 8 to 50 mm. One camera is mounted directly above the base to acquire dark-field images; the other camera is positioned at an angle to the side to acquire bright-field images. A 12 cm diameter ring light source is mounted on the opposite side of the dark-field camera to achieve the required lighting configuration.

[0037] In terms of text data processing, this example utilizes Google's Gemini-3 large model product to refine the basic cue words, generating professional and semantically rich perspective description text. A fixed mapping relationship between the basic cue words and the generated text is established and saved for consistent use during the training and inference phases, ensuring the stability of the semantic alignment between the image and text. For example, the dark-field, top-down view is mapped as: A top-down dark-field illumination image is acquired with the camera positioned perpendicular to the sample plane, while illumination is delivered at an oblique angle.

[0038] Multi-view training: This example employs the text encoder and image encoder from the CLIP architecture of a pre-trained multimodal vision and language model. Specifically, the text encoder is a standard 12-layer Transformer encoder, and the image encoder is ViT-B / 32. The input image, with a size of 224×224, is divided into a sequence of 32×32 patches, and image features are extracted using a basic-scale Vision Transformer encoder. The open-source CLIP pre-trained model utilizes 400 million image-text pairs for contrastive learning, mapping images and text to a unified semantic vector space, providing an effective initialization and semantic alignment foundation for the proposed image-text feature aggregation module 5.

[0039] Building upon this, this invention employs a Cross-Attention-based architecture to aggregate image and text features, using the perspective description text as a semantic prior to guide image feature extraction. The fused features are fed into a classification detection head, which outputs the category prediction result and calculates the cross-entropy loss with the ground truth label. There are five categories (in this example: qualified products, scratches, carbon wire short circuits, carbon wire open circuits, and moisture-sensitive film with incomplete coverage). For the first... Each sample has a true label that is a one-hot vector. The unnormalized logits predicted by the model are The cross-entropy loss is calculated as follows: , in This represents the number of samples in a batch.

[0040] Furthermore, this example inputs image and text data from both bright and dark field perspectives into the same weight-sharing network. After image-text feature aggregation, bright field fusion features and dark field fusion features are obtained separately. Then, symmetric KL divergence is further introduced. To align the distributions of the two and promote cross-perspective feature consistency.

[0041] This invention fine-tunes the pre-trained model by using more than 3,000 images of humidity-sensitive resistor chips to optimize the parameters of the text encoder, image encoder, image and text feature aggregation module 5, and defect detection head, making the overall model more suitable for the task of detecting defects in humidity-sensitive resistor chips.

[0042] Single-perspective reasoning: This invention employs a single-branch network (i.e., single-view input) for defect detection. Each inference iteration requires only one viewpoint image and its corresponding viewpoint description text (which can be a bright-field oblique view or a dark-field top-down view). After feature extraction by the image encoder and text encoder, the input data undergoes cross-modal fusion via the image-text feature aggregation module 5. Finally, the defect detection head outputs the chip's defect category. Notably, this invention is a specific application in the task of classifying defects at the image level of an entire chip. Therefore, each image used for training and inference strictly contains one and only one complete humidity-sensitive resistor chip sample, ensuring a one-to-one correspondence between the label and the image content and avoiding interference from multiple targets.

[0043] The technical solution of this application will be described in detail below with reference to specific embodiments.

[0044] In some embodiments, a method for training a defect detection model includes the following steps: S1. Acquire multi-view images of the target object, wherein the target object can be a humidity-sensitive resistor chip, and the multi-view images include a dark field image acquired according to view one (top view, dark field illumination) and a bright field image acquired according to view two (oblique side view, bright field illumination). S2. Extract image features from multi-view images using an image encoder. Input images from two different perspectives into the image encoder to extract image features, providing an effective initialization and semantic alignment basis for the image and text feature aggregation module 5. S3. Based on multi-view images, obtain view description text corresponding to each view, and use the Gemini-3 large model to generate specialized view description text corresponding to dark field images and bright field images. S4. Through the text encoder, extract the text features of the perspective description text, input the description texts from the two perspectives into the text editor to extract the text features, and provide an effective initialization and semantic alignment basis for the image and text feature aggregation module 5. S5. Fuse image features and text features to obtain image-text fusion features. Fuse image and text features from corresponding perspectives to achieve text-guided image feature enhancement. The image-text fusion features from two perspectives are aligned by calculating the symmetric KL divergence loss. S6. Input the image-text fusion features into the defect detection head and output the category prediction results; S7. Calculate the classification loss based on the difference between the category prediction results and the true labels, and optimize the parameters of the image encoder, text encoder and defect detection head based on the classification loss; The two viewpoint images are processed by the detection head to output predicted scores for each category. The category with the highest score is taken as the detection result, and the classification loss is calculated with the category label. Multiple losses are backpropagated to iteratively optimize the model weights until training converges.

[0045] In some embodiments, multi-view images of the target object are acquired; image features of the multi-view images are extracted using an image encoder, including the following steps: Bright-field images of the target object are acquired using a bright-field camera; Acquire dark-field images of the target object using a dark-field camera; The bright-field features of the bright-field image and the dark-field features of the dark-field image are extracted using an image encoder. The image and text feature fusion module fuses bright-field features with their corresponding bright-field view description files and dark-field features with their corresponding dark-field view description files.

[0046] In some embodiments, image features and text features are fused to obtain image-text fused features, including the following steps: The image and text features are aggregated using a Cross-Attention structure. The semantic prior of the viewpoint description document is used as a guiding signal to drive different branches to perform targeted feature extraction on the image. This is achieved through the following calculations: , , ,

[0047] in, For text representation, The length of the text sequence. For feature dimensions; For image feature maps, The length of the image patch sequence; Q indicates that the query is based on image features; K indicates that the text representation is the key; V indicates that the text representation is the value; It is a learnable projection matrix.

[0048] In some embodiments, after obtaining the image-text fusion feature, the following steps are also included: By calculating the symmetric KL divergence loss, the image-text fusion features are aligned. The symmetric KL divergence between the bright field feature distribution and the dark field feature distribution is calculated to constrain the bright field feature distribution and the dark field feature distribution to tend to be consistent, so as to achieve the alignment of image-text fusion features.

[0049] In some embodiments, the text-image fusion features are aligned using symmetric KL divergence loss calculation, including the following steps: Based on the Softmax function, the image-text fusion features are mapped to the probability space to obtain the normalized bright-field feature distribution. Dark field feature distribution : , , in, It is a bright field characteristic. Features of dark fields and These represent the eigenvectors of the bright field and the dark field at the th... Values ​​in each dimension ; Minimize the difference between the bright-field and dark-field feature distributions using KL divergence: Calculate the forward KL divergence: , Calculate the reverse KL divergence: , Calculate the symmetric KL divergence of the bright-field and dark-field characteristic distributions:

[0050] By constraining the distribution of bright-field features and dark-field features to tend to be consistent, the alignment of image-text fusion features can be achieved.

[0051] In some embodiments, the classification loss is calculated based on the difference between the category prediction result and the true label, including the following steps: The cross-entropy loss is calculated by comparing the category prediction results with the true labels, as follows: , Where N is the number of samples in a batch; C is the number of categories; for the th Each sample has a true label that is a one-hot vector. The unnormalized logits predicted by the model are .

[0052] The image and text features are fused together using the image and text feature aggregation module 5 to obtain image-text fused features. Based on classification loss, the parameters of the image encoder, text encoder, image and text feature aggregation module 5, and defect detection head are updated through backpropagation algorithm. The optimization is iteratively performed until convergence. Fine-tuning is carried out on the pre-trained model. Using more than 3,000 images of humidity-sensitive resistor chips, the parameters of the text encoder, image encoder, image and text feature aggregation module 5, and defect detection head are optimized to make the overall model more suitable for the task of humidity-sensitive resistor chip defect detection.

[0053] In some embodiments, a defect detection method based on multi-view training and single-view reasoning includes: 100. Multi-view training phase: 101. Acquire multi-view images of the target object; 102. Extract image features from multi-view images using an image encoder; 103. Based on multi-view images, obtain view description text corresponding to each view; 104. Extract text features of perspective-description texts using a text encoder; 105. Through the image and text feature aggregation module 5, image features and text features are fused to obtain image-text fused features; 106. Align the image and text fusion features; 107. Input the image-text fusion features into the defect detection head and output the category prediction results; 108. Calculate the classification loss based on the difference between the category prediction results and the true labels, and optimize the parameters of the defect detection head based on the classification loss; 109. Iteratively optimize the parameters of the image encoder, text encoder, image and text feature aggregation module 5, and defect detection head to obtain a defect detection model; 200. Reasoning Stage: 201. Input a target image from a single perspective and its corresponding perspective description text into a trained defect detection model, and output the defect category prediction result.

[0054] In some embodiments, aligning the image-text fusion features includes the following steps: The image-text fusion features are mapped to the probability space to obtain the bright field feature distribution and the dark field feature distribution. The symmetric KL divergence between the bright field feature distribution and the dark field feature distribution is calculated to constrain the bright field feature distribution and the dark field feature distribution to tend to be consistent, so as to achieve the alignment of the image-text fusion features.

[0055] In some embodiments, a defect detection system includes: Image acquisition module 1 is used to acquire perspective images of the target product; Image encoding module 2 is used to extract image features from the viewpoint image; Text generation module 3 generates view description text for the view image based on the view image; Text encoding module 4 is used to extract text features from the perspective description text; Image and text feature aggregation module 5 is used to fuse image features and text features to generate image-text fusion features; Alignment loss module 6 is used to align the distribution of image-text fusion features; The detection output module 7 outputs predicted scores for each category based on image-text fusion features.

[0056] In some embodiments, the image acquisition module 1 can be a bright-field camera and a dark-field camera; the image encoding module 2 can be a Vision Transformer encoder; the text generation module 3 can be Google's large model product Gemini-3; the text encoding module 4 can be a Transformer encoder; the image and text feature aggregation module 5 can be implemented through a Cross-Attention architecture; the alignment loss module 6 can align the feature distributions of the two branches (image and text) based on the KL divergence method; and the detection output module 7 can be a defect detection head.

[0057] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the model training method or the defect detection method based on multi-view training and single-view inference disclosed in the above embodiments; wherein the processor may be a central processing unit (CPU) or a graphics processing unit (GPU).

[0058] This application also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer performs some or all of the steps of the methods described in the above method embodiments.

[0059] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0060] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0061] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0062] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0063] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0064] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0065] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the invention is not limited to the specific embodiments described above, and any obvious improvements, substitutions, or modifications made by those skilled in the art based on this invention are within the scope of protection of this invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the invention.

Claims

1. A training method for a defect detection model, characterized in that, Includes the following steps: Acquire multi-view images of the target object; The image features of the multi-view images are extracted using an image encoder; Based on the multi-view images, corresponding view description text is obtained; The text features of the perspective description text are extracted using a text encoder. The image features and the text features are fused to obtain image-text fusion features; The image-text fusion features are input into the defect detection head, and the category prediction result is output. The classification loss is calculated based on the difference between the category prediction result and the true label, and the parameters of the image encoder, the text encoder and the defect detection head are optimized based on the classification loss.

2. The training method for the defect detection model as described in claim 1, characterized in that, The acquisition of multi-view images of the target object; extraction of image features from the multi-view images using an image encoder, including the following steps: Bright-field images of the target object are acquired using a bright-field camera; Dark-field images of the target object are acquired using a dark-field camera; The bright-field features of the bright-field image and the dark-field features of the dark-field image are extracted using an image encoder.

3. The training method for the defect detection model as described in claim 1, characterized in that, The process of fusing the image features and the text features to obtain image-text fused features includes the following steps: The image features and text features are aggregated through a Cross-Attention structure, and the semantic prior of the viewpoint description file is used as a guiding signal to drive different branches to perform targeted feature extraction on the image.

4. The training method for the defect detection model as described in claim 2, characterized in that, After obtaining the image-text fusion features, the following steps are also included: The image-text fusion features are aligned by calculating the symmetric KL divergence loss.

5. The training method for the defect detection model as described in claim 4, characterized in that, The process of aligning the image-text fusion features using symmetric KL divergence loss includes the following steps: The image-text fusion features are mapped to the probability space to obtain the bright field feature distribution and the dark field feature distribution; Calculate the symmetric KL divergence between the bright field feature distribution and the dark field feature distribution, and constrain the bright field feature distribution and the dark field feature distribution to tend to be consistent, so as to achieve the alignment of the image and text fusion features.

6. The training method for the defect detection model as described in claim 1, characterized in that, The calculation of classification loss based on the difference between the category prediction result and the true label includes the following steps: The cross-entropy loss is calculated by comparing the predicted category results with the true labels, as follows: ， Where N is the number of samples in a batch; C is the number of categories; for the th Each sample has a true label that is a one-hot vector. The unnormalized logits predicted by the model are .

7. The training method for the defect detection model as described in claim 1, characterized in that, The image features and text features are fused together using the image-text feature aggregation module to obtain image-text fused features; Based on the classification loss, the parameters of the image encoder, the text encoder, the image and text feature aggregation module, and the defect detection head are updated through backpropagation algorithm, and iterative optimization is performed until convergence.

8. A defect detection method based on multi-view training and single-view reasoning, characterized in that, include: Multi-view training phase: Acquire multi-view images of the target object; Extract image features from the multi-view images; Based on the multi-view images, corresponding view description text is obtained; Extract the text features of the perspective description text; The image features and the text features are fused to obtain image-text fusion features; Align the image-text fusion features; The image-text fusion features are input into the defect detection head, and the category prediction result is output. The classification loss is calculated based on the difference between the predicted category and the true label, and the parameters of the defect detection head are optimized based on the classification loss. The parameters of the defect detection head are iteratively optimized to obtain a defect detection model; Inference phase: The target image and its corresponding view description text are input into the trained model using a single viewpoint, and the defect category prediction results are output.

9. The defect detection method as described in claim 8, characterized in that, Aligning the image-text fusion features includes the following steps: The image-text fusion features are mapped to a probability space to obtain bright-field feature distributions and dark-field feature distributions. The symmetric KL divergence between the bright-field feature distributions and the dark-field feature distributions is calculated to constrain the bright-field feature distributions and the dark-field feature distributions to tend to be consistent, so as to achieve the alignment of the image-text fusion features.

10. A defect detection system, characterized in that, include: The image acquisition module is used to acquire perspective images of the target product; The image encoding module is used to extract image features from the viewpoint image; A text generation module is used to generate viewpoint description text for the viewpoint image; A text encoding module is used to extract text features from the perspective description text; The image and text feature aggregation module is used to fuse the image features and the text features to generate image-text fusion features; The alignment loss module is used to align the distribution of the image-text fusion features; The detection output module outputs prediction scores for each category based on the image-text fusion features.

Images