Small sample fault image detection method, system and device and storage medium

Abstract

The invention discloses a small sample fault image detection method, system and device and a storage medium, which are mutually corresponding schemes, and in the scheme, a high-quality defect-free sample is generated through a variational quantization generative adversarial network, more diversified defect-free images can be generated in combination with the mean value and variance adjustment of a potential space, and meanwhile, the detection accuracy is improved. High consistency between the generated image and actual distribution is kept through alignment processing, and the problem of sample simplification caused by mode collapse in a traditional generation method is avoided; moreover, the generated diversified defect-free image samples and the collected diversified defect samples are combined to form an enhanced sample set, an abnormal image detection model is trained, and the abnormal detection capability is improved in combination with a deep learning model, so that the method has real-time performance and high robustness, and meets diversified and complicated defect detection requirements in an industrial scene. Compared with a traditional deep learning method, the method does not need to depend on a large number of labeled samples, and can be deployed in an actual industrial scene more quickly.

Description

Technical Field

[0001] The present invention relates to the technical field of fault image detection, and in particular to a small sample fault image detection method, system, device and storage medium. Background Art

[0002] With the rapid development of intelligent industrial manufacturing, the importance of product quality inspection has become increasingly prominent, especially in the field of semiconductor manufacturing. The semiconductor manufacturing process involves multiple complex process steps, each of which may introduce different types of defects. For example, pattern distortion may occur in the photolithography process, shape defects may be introduced in the etching process, and particle contamination or residue may occur in the cleaning stage. The morphology, size, and distribution of these defects are highly diverse. Traditional methods are often unable to establish unified detection standards for all defect types, which can easily lead to missed detections or false detections. Although these defects are subtle, they can have a significant impact on the electrical performance, reliability, and overall yield of semiconductor products. Undetected defects may cause circuit shorts, opens, or performance degradation, leading to serious product quality issues.

[0003] Current mainstream industrial defect detection methods rely primarily on deep learning technology, which typically requires a large number of labeled samples for model training. However, in the semiconductor field, the applicability of traditional methods is significantly limited due to the scarcity of defect samples, the high cost of labeling, and, particularly in new process stages, the lack of sufficient historical data. Furthermore, semiconductor defect types are diverse, ranging from significant macroscopic defects to microscopic pattern distortions, limiting the accuracy of traditional methods.

[0004] Existing training datasets suffer from sample imbalance, manifested in a situation where defective samples exhibit diverse features while non-defective samples exhibit a single, more homogeneous set. This data skewness leads to overfitting of the model's decision boundary for normal samples in the feature space, resulting in key technical bottlenecks such as increased false positive rates and degraded generalization performance during defect detection. While existing sample data augmentation techniques can expand the sample size through simple transformations, these generated samples are essentially repetitions of the original data and fail to incorporate the representations of non-defective samples. Consequently, their improvement in model generalization is limited, especially when dealing with novel or unseen defect types. Generative Adversarial Networks (GANs) excel at generating non-defective samples, but their training process often suffers from instability and mode collapse, where the generator may focus on generating specific non-defective samples, resulting in insufficient diversity. Therefore, it is crucial to develop a generation and detection method that can effectively expand data diversity and improve the quality of generated samples in small sample sizes, while also exhibiting high stability and generalization capabilities.

[0005] In view of this, the present invention is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a small sample fault image detection method, system, device and storage medium, which can generate diverse defect-free image samples and realize efficient anomaly detection based on the generated enhanced samples containing diverse defect-free images.

[0007] The purpose of the present invention is achieved through the following technical solutions: A small sample fault image detection method, comprising: Combined with the input image, a diverse set of images is generated by adjusting the mean and variance of the variational quantization generative adversarial network. The optimal hint corresponding to the image category of the input image is selected based on semantic relevance, and the optimal hint is aligned with the diverse images to obtain diverse defect-free image samples. The generated diverse defect-free image samples are combined with the collected diverse defect samples to form an enhanced sample set, the enhanced sample set is used to train the abnormal image detection model, and the trained abnormal image detection model outputs the abnormality detection results of the image to be detected.

[0008] A small sample fault image detection system, comprising: A diversified defect-free image sample generation unit is configured to combine an input image, generate diversified images by adjusting the mean and variance of a variational quantization generative adversarial network, select the best hint corresponding to the image category of the input image based on semantic relevance, and align the best hint with the diversified images to obtain diversified defect-free image samples; The model training and anomaly detection unit is used to combine the generated diverse defect-free image samples with the collected diverse defect samples to form an enhanced sample set, use the enhanced sample set to train the abnormal image detection model, and output the anomaly detection results of the image to be detected through the trained abnormal image detection model.

[0009] A processing device comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that high-quality defect-free samples are generated by the variational quantization generative adversarial network, and combined with the mean and variance adjustment of the latent space, more diverse defect-free images can be generated. At the same time, the generated image is highly consistent with the actual distribution through alignment processing, avoiding the problem of single sample caused by mode collapse in traditional generation methods; and, based on the defect-free image samples, the anomaly detection model is trained to achieve efficient anomaly detection and positioning; compared with traditional deep learning methods, the present invention does not need to rely on a large number of labeled samples and can be deployed more quickly in actual industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a small sample fault image detection method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of a small sample fault image detection method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for generating diverse defect-free image samples provided by an embodiment of the present invention; Figure 4 A schematic diagram of an abnormal image detection model provided by an embodiment of the present invention; Figure 5 A schematic diagram of a small sample fault image detection system provided by an embodiment of the present invention; Figure 6 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] First, the following terms may be used in this article: The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles)" should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0016] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0017] The following describes in detail the small-sample fault image detection method, system, device, and storage medium provided by the present invention. Any information not described in detail in the embodiments of the present invention represents prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, the procedures were performed in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Instruments used in the embodiments of the present invention, where the manufacturer is not specified, are all commercially available conventional products.

[0018] Example 1 The embodiment of the present invention provides a small sample fault image detection method, such as Figure 1 As shown, it mainly includes the following steps: Step 1: Diversified defect-free image sample generation stage.

[0019] In an embodiment of the present invention, in combination with an input image, a diverse image is generated by adjusting the mean and variance of a variational quantization generative adversarial network, the best hint of the image category corresponding to the input image is selected based on semantic relevance, and the best hint is aligned with the diverse images to obtain diverse defect-free image samples.

[0020] Preferably, the variational quantization generative adversarial network includes: a first encoder and a first decoder; After the input image is enhanced, it is encoded into a continuous latent vector through the first encoder, which is expressed as: ; in, is the first encoder, is the image after the input image is enhanced, i is the sequence number of the input image, for The quantized latent vector of is the potential vector of the dth dimension, and D is the dimension of the potential vector; Represents the quantization process of the latent vector.

[0021] Calculate the mean and variance of the latent vector and pass it through the first decoder to generate the corresponding image, which is expressed as: ; in, Indicates the generated images; is the first decoder, is the mean of the latent vectors of all input images, which represents the center of the latent vector distribution. The variance of the latent vectors of all input images is calculated to indicate the spread of the latent vector distribution.

[0022] Preferably, the method of selecting the best prompt for the image category corresponding to the input image based on semantic relevance includes: generating multiple candidate prompts for the image category corresponding to the input image through a predefined word set in a semantic database; calculating the semantic relevance between each candidate prompt and the input image respectively, and selecting the candidate prompt with the highest semantic relevance to the input image as the best prompt.

[0023] Preferably, the optimal prompt is aligned with a variety of images to obtain a variety of defect-free image samples, including: extracting text features from the optimal prompt to obtain a text latent feature vector; for each group of images generated, after extracting the visual features, using the visual feature mean to calculate the expected value of the visual features of each group of images; calculating the similarity between the expected value of the visual features of each group of images and the text latent feature vector, and extracting a group of images with the highest similarity as the diverse defect-free image samples.

[0024] Step 2: Model training and anomaly detection phase.

[0025] In an embodiment of the present invention, the generated diverse defect-free image samples are combined with the collected diverse defect samples to form an enhanced sample set, and the enhanced sample set is used to train an abnormal image detection model. The trained abnormal image detection model outputs the abnormality detection results of the image to be detected.

[0026] In the embodiment of the present invention, the original diverse defect samples can be obtained through conventional methods, which will not be elaborated in the present invention.

[0027] In the above-mentioned solution provided by the embodiment of the present invention, high-quality defect-free samples are generated by the variational quantization generative adversarial network. Combined with the mean and variance adjustment of the latent space, more diverse defect-free images can be generated. At the same time, the generated image is highly consistent with the actual distribution through alignment processing, avoiding the problem of single sample caused by mode collapse in traditional generation methods; and, the anomaly detection model is trained based on defect-free image samples, thereby achieving efficient anomaly detection and positioning; compared with traditional deep learning methods, there is no need to rely on a large number of labeled samples, and the present invention can be deployed more quickly in actual industrial scenarios.

[0028] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0029] 1. Overall overview of the plan.

[0030] The solution provided by this embodiment aims to address the issues of scarce defect samples and limited detection accuracy in industrial scenarios. This solution uses text-guided variational image generation, combined with generative adversarial network technology, to achieve high-quality defect-free image generation and optimization. It also aligns the latent distributions of generated images and textual hints to generate samples consistent with semantic descriptions.

[0031] In an embodiment of the present invention, in the defect-free image generation stage, the input of the model is the image and the image category. For example, for the hole defect detection data on the wafer, the input to the model is the image of the hole and the text "hole".

[0032] like Figure 2 As shown in the figure, a set of object feature description prompts is generated according to the image category, and the best prompt is selected from it, which contains information equivalent to the input image in the generated image. The image generator based on the variational quantization generative adversarial network obtains a set of defect-free images, and optimizes the quality and training effect of the generated images by comparing the potential distribution of the generated images and text prompts. Afterwards, the optimally generated diverse defect-free image samples are used as additional training sets for the anomaly detection model and combined with diverse defect samples as an enhanced sample set for model training. The deep learning model is combined to improve the anomaly detection capability, with real-time and high robustness, to meet the diverse and complex defect detection needs in industrial scenarios. Among them, the anomaly detection model adopts conditional normalization flow, extracts image features through multi-scale pyramid pooling, and combines the conditional flow decoder to model the feature distribution and density estimation, so as to achieve efficient unsupervised anomaly detection and positioning.

[0033] 2. Detailed introduction of the plan.

[0034] 1. Diverse defect-free image sample generation stage.

[0035] like Figure 3 As shown in the figure, the diverse defect-free image sample generation stage mainly includes three parts: image generation, optimal hint generation, and text-image alignment.

[0036] (1) Image generation.

[0037] In this embodiment of the present invention, a variational quantization generative adversarial network (VQGAN) is used to generate a sufficient number of defect-free images. The VQGAN comprises a first encoder and a first decoder. The output features of the first encoder follow the approximate posterior distribution of the latent vector, enabling the first decoder to effectively reconstruct a given input image. Finally, by predicting and utilizing the variance of the latent space, the diversity of the generated images is enhanced, while ensuring that the generated images conform to a normal distribution.

[0038] Specifically: the input image is enhanced, and a continuous latent vector is output through the first encoder. The input image is first passed through a convolutional neural network (CNN) encoder to map it to the latent space. Indicates that the image is The features on the potential dimensions are called latent vectors.

[0039] In the variational quantization generative adversarial network, the latent space is mapped to a discrete embedding space. For each latent vector , find the closest embedding space vector in the pre-generated discrete numerical embedding space And replace, the potential vector is quantized in the vector space The calculation is as follows: ; in, are the embedding space vectors in the vector space, is the size of the vector space.

[0040] The overall features of the image are decomposed into multiple local features to facilitate capturing the image information of the local area. The encoding process can be expressed as follows: ; in, is the first encoder, is the image after the input image is enhanced, i is the sequence number of the input image, For images The quantized latent vector of is the potential vector of the dth dimension, and D is the dimension of the potential vector; Represents the quantization process of the latent vector.

[0041] In order to improve the appearance diversity of defect-free images, the generation of a set of images is controlled by the mean and variance of the latent vector. The generation of images can be expressed as follows: ; in, Indicates the generated images; is the mean of the latent vectors of all input images, which represents the center of the latent vector distribution. The variance of the latent vectors of all input images is calculated to indicate the degree of dispersion of the latent vector distribution. For the first decoder, the mean and square of the latent distribution are used to generate images.

[0042] (2) Optimal prompt generation.

[0043] In the embodiment of the present invention, in order to create a prompt that can effectively express the characteristics of the target object, a plurality of candidate prompts corresponding to the input image are generated by using a predefined word set in the semantic database WordNet. , where each item is a hint and T is the number of hints; calculate each candidate hint separately With the input image The candidate hint with the highest semantic relevance to the input image is selected as the best hint.

[0044] For example, the CLIP model (a multimodal model based on contrastive learning) can be used to extract image embeddings and text embedding , according to the cosine similarity, the hint closest to the image semantics is selected. The selection calculation of the best hint P is as follows: ; Among them, cos represents the calculation of cosine similarity.

[0045] (3) Text-image alignment.

[0046] In the embodiment of the present invention, the text feature extraction is performed on the best prompt P to obtain the text potential feature vector , for example, can be implemented by the text encoder in the CLIP model, expressed as: ; in, represents the text embedding of the best prompt P extracted by the text encoder G.

[0047] For each set of images generated, the visual features of each image are calculated through the image encoder, and then the expected value of the visual features of each set of images is calculated using the mean of the visual features. , expressed as: ; in, For the images The visual characteristics of To calculate the expected value of visual features using the mean value of visual features.

[0048] For example, image embeddings can be extracted as visual features through the image encoder in the CLIP model.

[0049] When text and image semantic alignment is performed, the text latent feature vector The cosine similarity calculation is performed on the expected value of the visual features of each group of images to obtain the generated The semantic consistency between the group image and the text prompt is the strongest, and its calculation is as follows: ; in, is the index of the optimal image set (i.e. ), is the total number of image groups generated, For the The expected value of the visual features of the group of images. The optimal image set selected is , The total number of images generated for each group, the optimal image set was selected for further training of the anomaly detection model.

[0050] In this embodiment of the present invention, a new set of images is generated in each iteration, while the variational quantization generative adversarial network is trained and its parameters are updated. This enhances the diversity of the generated images and ensures that the generated images more closely resemble the distribution characteristics of the input defect-free image. In this embodiment of the present invention, the mean squared error (MSE) loss is used to ensure visual consistency between the generated images and the input images. This loss is calculated based on the difference between the generated and input images and is expressed as: .

[0051] Combined with variational quantization to generate the original loss of the adversarial network , obtain the training loss function , expressed as: ; in, is a weight hyperparameter used to balance the two losses.

[0052] Considering that the process of training the variational quantization generative adversarial network in combination with the above-mentioned training loss function can be implemented by referring to conventional technology, it will not be described in detail.

[0053] 2. Model training and anomaly detection stage.

[0054] In the embodiment of the present invention, the diverse defect-free image samples generated in the first stage are combined with the diverse defect samples collected to form an enhanced sample set, which is combined with a deep learning model to improve the anomaly detection capability, with real-time and high robustness, to meet the diverse and complex defect detection needs in industrial scenarios. Figure 4 As shown, the abnormal image detection model includes: a second encoder and a second decoder; the training image samples input on the left side include the defect-free image samples generated above and the original training samples.

[0055] In an embodiment of the present invention, the second encoder may use a convolutional neural network (CNN) as a basic architecture, combined with multi-scale pyramid pooling to implement feature extraction, and the second decoder may use conditional normalization popular feature distribution modeling and anomaly detection.

[0056] (1) Second encoder.

[0057] In an embodiment of the present invention, the second encoder extracts a multi-scale feature vector and a spatial position condition vector of the input defect-free image sample; wherein the multi-scale feature vector is a feature map extracted by multi-scale pyramid pooling, and the spatial position condition vector is generated by two-dimensional position encoding, encoding the position information of the feature vector of the corresponding scale in space.

[0058] Specifically, the second encoder uses a pre-trained CNN model as a feature extractor. For example, a ResNet (residual network) can be used as a feature extractor to map each block of the input training image sample into a feature vector z, which is used to extract global and local semantic information. Pre-trained CNN models based on natural images can generalize well to anomaly detection tasks. Because anomalies vary in size and shape, they must be processed using a variable receptive field. Multi-scale pyramid pooling can extract multi-scale features to accommodate anomalies of varying sizes and shapes. Pyramid pooling generates feature vectors through multiple downsampling layers, allowing the model to capture both local details and global contextual information.

[0059] like Figure 4 As shown, examples of two scales are provided. The features of each scale are transformed and then enter the decoder of the corresponding scale for processing. The feature transformation scheme involved can refer to conventional technology and will not be described in detail in the present invention.

[0060] In addition, the second encoder includes an independent encoder corresponding to each scale, and each independent encoder is responsible for extracting features of the corresponding scale.

[0061] (2) Second decoder.

[0062] The second decoder includes an independent decoder corresponding to each scale, that is, each scale is processed independently, and there are K scales and corresponding K independent decoders. Figure 4 An example of two independent decoders (Independent Decoder 1, Independent Decoder 2) is provided. The second decoder performs probability density estimation through a conditional normalization flow, and the distribution modeling is derived from a single scale feature vector to the standard Gaussian distribution The core is to achieve step-by-step transformation through a series of coupled layers. The second decoder performs probability density estimation through conditional normalization flow, and distribution modeling is performed from a single scale feature vector to the standard Gaussian distribution At its core is a step-by-step transformation implemented through a series of coupled layers.

[0063] With a single scale feature vector For example, it corresponds to an independent decoder , decoders corresponding to different scales have different parameters Small-scale features are used to capture global contextual information, while large-scale features focus on fine-grained local details. The decoder at each scale independently estimates the log-likelihood of its input features and generates a probability value.

[0064] The decoder corresponding to a single scale processes in a coupling layer: Copy it into two and record it as and ; Among them, the first part Remain unchanged as conditional information, the second part Adjustment is performed through a transformation function, expressed as: ; in, is a single scale feature vector The corresponding spatial position condition vector, For the second part The transformed features, is a scaling function used to control the change of feature amplitude; is a translation function used to adjust the change of the feature center position; is the dot product symbol. The coupling layer is designed and The Jacobian matrix of the transformation The determinant of is easy to compute, allowing for efficient estimation of the log-likelihood.

[0065] An independent decoder of the second decoder is composed of Each coupling layer performs a reversible transformation. Assuming the input feature ,go through Second transformation to obtain , No. Jacobian matrix Expressed as: ; in, is the symbol of the partial derivative.

[0066] go through After the coupling layer, Calculating the log-likelihood can be expressed as: ; Among them, det represents the determinant of the calculation matrix.

[0067] Number of coupling layers in independent decoders The user can set it according to the actual situation or experience, and the present invention does not limit it. As an example, you can set .

[0068] In the embodiment of the present invention, the transformed features Perform log-likelihood estimation to obtain the result , and converted into probability values Finally, the probability values ​​obtained by the independent decoders corresponding to each scale are upsampled and aggregated to obtain the anomaly detection results.

[0069] In the embodiment of the present invention, the second decoder maximizes the transformed feature of each independent decoder. The log-likelihood is trained and the training loss function is for: ; in, represents the KL divergence, Represents the feature vector of the i-th sample (i.e., the i-th input image) output by an independent encoder of the second encoder at the k-th scale, Indicates the probability value. is the norm symbol, represents the number of samples in the training dataset, represents the characteristics of the i-th sample transformed by an independent decoder of the second decoder, is the Jacobian matrix of the i-th sample transformed in the second decoder, and det represents the determinant of the calculated matrix.

[0070] Considering that the process of training the second decoder based on the above-mentioned training loss function can be implemented with reference to conventional technology, it will not be described in detail.

[0071] After training, it can be used to detect the input image to be detected and output the anomaly detection result. Specifically, after the second encoder completes the process, the second decoder estimates the log-likelihood of the position feature of each pixel, converts the log-likelihood into a probability value, and upsamples and aggregates the probability values ​​of all scales to generate the final anomaly detection result (for example, an anomaly score map).

[0072] Through the above description of the embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented via software or by utilizing software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) and includes a number of instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention.

[0073] Example 2 The present invention also provides a small sample fault image detection system, which is mainly used to implement the method provided in the above embodiment, such as Figure 5 As shown, the system mainly includes: A diversified defect-free image sample generation unit is configured to combine an input image, generate diversified images by adjusting the mean and variance of a variational quantization generative adversarial network, select the best hint corresponding to the image category of the input image based on semantic relevance, and align the best hint with the diversified images to obtain diversified defect-free image samples; The model training and anomaly detection unit is used to combine the generated diverse defect-free image samples with the collected diverse defect samples to form an enhanced sample set, use the enhanced sample set to train the abnormal image detection model, and output the anomaly detection results of the image to be detected through the trained abnormal image detection model.

[0074] Considering that the main technical details of each unit involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0075] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0076] Example 3 The present invention also provides a processing device, such as Figure 6As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0077] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0078] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example: The input device can be a touch screen, image acquisition device, physical button or mouse; The output device may be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0079] Example 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0080] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0081] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A small sample fault image detection method, characterized in that: include: Combined with the input image, a diverse set of images is generated by adjusting the mean and variance of the variational quantization generative adversarial network. The optimal hint corresponding to the image category of the input image is selected based on semantic relevance, and the optimal hint is aligned with the diverse images to obtain diverse defect-free image samples. The generated diverse defect-free image samples are combined with the collected diverse defect samples to form an enhanced sample set, the enhanced sample set is used to train the abnormal image detection model, and the trained abnormal image detection model outputs the abnormality detection results of the image to be detected.

2. A small sample fault image detection method according to claim 1, characterized in that: The variational quantization generative adversarial network includes: a first encoder and a first decoder; After the input image is enhanced, it is encoded into a continuous latent vector through the first encoder, which is expressed as: ； in, is the first encoder, is the image after the input image is enhanced, i is the sequence number of the input image, for The quantized latent vector of is the potential vector of the dth dimension, and D is the dimension of the potential vector; Represents the quantization process of the latent vector; Calculate the mean and variance of the latent vector and pass it through the first decoder to generate the corresponding image, which is expressed as: ； in, Indicates the generated images; is the first decoder, is the mean of the latent vectors of all input images, which represents the center of the latent vector distribution. The variance of the latent vectors of all input images is calculated to indicate the spread of the latent vector distribution.

3. The small sample fault image detection method according to claim 1, characterized in that: The selecting the best hint of the input image based on semantic relevance comprises: Generate multiple candidate hints for the image category corresponding to the input image through a predefined word set in a semantic database; The semantic relevance between each candidate cue and the input image is calculated separately, and the candidate cue with the highest semantic relevance to the input image is selected as the best cue.

4. A small sample fault image detection method according to claim 1 or 3, characterized in that: Aligning the optimal hint with diverse images to obtain diverse defect-free image samples includes: Perform text feature extraction on the best prompt to obtain the text potential feature vector; For each set of generated images, after extracting the visual features, the expected value of the visual features of each set of images is calculated using the mean of the visual features; The similarity between the expected value of the visual features of each group of images and the text latent feature vector is calculated, and the group of images with the highest similarity is extracted as a diverse set of defect-free image samples.

5. A small sample fault image detection method according to claim 1 or 2, characterized in that: Also includes: The variational quantization generative adversarial network is trained, and the training loss function is Expressed as: ； in, is the mean squared error loss calculated using the difference between the generated image and the input image, Generate the original loss for the adversarial network for variational quantization, is the weight hyperparameter.

6. The method for detecting small sample fault images according to claim 1, characterized in that: The abnormal image detection model includes: a second encoder and a second decoder; The second encoder extracts a multi-scale feature vector and a spatial position condition vector of the input defect-free image sample; wherein the multi-scale feature vector is a feature map extracted by multi-scale pyramid pooling, and the spatial position condition vector is generated by two-dimensional position encoding, encoding the position information of the feature vector of the corresponding scale in space; The second decoder includes an independent decoder corresponding to each scale, with a total of K scales and corresponding K independent decoders; the independent decoder processing process corresponding to a single scale includes: converting a single scale feature vector Copy it into two and record it as and ; Among them, the first part As conditional information, the second part Adjustment is performed through a transformation function, expressed as: ； in, is a single scale feature vector The corresponding spatial position condition vector, For the second part The transformed features, is the scaling function, is the translation function; is the dot product symbol; The transformed features Perform log-likelihood estimation and convert it into probability value; After upsampling and aggregating the probability values ​​obtained by the independent decoders corresponding to each scale, the anomaly detection results are obtained.

7. The method for detecting small sample fault images according to claim 6, characterized in that: The second decoder is trained by maximizing the log-likelihood of the transformed features, and the training loss function is for: ； in, represents the number of samples in the training dataset, Indicates the information of the i-th input image transformed by an independent decoder of the second decoder, is the Jacobian matrix of the i-th input image transformed in the second decoder, and det represents the determinant of the calculated matrix.

8. A small sample fault image detection system, characterized in that: include: A diversified defect-free image sample generation unit is configured to combine an input image, generate diversified images by adjusting the mean and variance of a variational quantization generative adversarial network, select the best hint corresponding to the image category of the input image based on semantic relevance, and align the best hint with the diversified images to obtain diversified defect-free image samples; The model training and anomaly detection unit is used to combine the generated diverse defect-free image samples with the collected diverse defect samples to form an enhanced sample set, use the enhanced sample set to train the abnormal image detection model, and output the anomaly detection results of the image to be detected through the trained abnormal image detection model.

9. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images