Anomaly image generation method, training sample generation method, and detection method

Abstract

The present application provides an anomaly image generation method, a training sample generation method for a detection model, a detection method, a related apparatus, an electronic device, and a medium, which are applied to the technical field of computer vision. The anomaly image generation method comprises: inputting a reference anomaly image and an initialized embedding vector into an image generation model to update the initialized embedding vector on the basis of the reference anomaly image, so as to obtain an updated embedding vector; then, inputting a reference normal image and the updated embedding vector into the image generation model to apply the updated embedding vector to the reference normal image, so as to obtain a generated anomaly image. Since the updated embedding vector comprises the anomaly image feature distribution obtained by learning on the basis of the reference anomaly image, the generated anomaly image can reflect the features of the anomaly image in the real world, and further, the generated anomaly image can serve as a negative sample for training a detection algorithm model, thereby facilitating improvement of the detection performance of the detection algorithm model.

Description

Abnormal image generation method, training sample generation method and detection method

[0001] Priority information

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on February 1, 2024, with application number 2024101583133 and invention name “Abnormal image generation method, training sample generation method and detection method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of computer vision technology, and in particular to a method for generating abnormal images, a method for generating training samples of a detection model, a detection method, related devices, electronic equipment, and a computer-readable storage medium. Background Art

[0004] With the upgrade of equipment manufacturing automation, the yield (normal product) rate on industrial production lines can reach a higher level, for example, it can reach more than 90%. Although the defective (abnormal product) rate can reach a lower level, a higher accuracy rate (for example, greater than 98%) is still required to improve the overall quality of the product. Artificial intelligence algorithm models can be used to predict whether industrial products are good or defective to improve detection efficiency. However, negative samples of abnormal products are difficult to obtain on a large scale, which makes the intelligent abnormal product detection algorithm model a practical and challenging task in industrial quality inspection.

[0005] Related technologies address this issue by synthesizing abnormal features of industrial products with noise or external data to generate abnormal images, which are then used as negative samples. However, there is a significant semantic gap between these synthesized abnormal images and real-world abnormal images (reference abnormal images), which affects the performance of the detection algorithm model. Summary of the Invention

[0006] The present application provides a method for generating abnormal images, a method for generating training samples for a detection model, a detection method, related devices, electronic equipment, and a computer-readable storage medium. The generated abnormal images can reflect the characteristics of abnormal images in the real world. The abnormal images determined by the solution provided by the present application are further used to train the detection model, which is conducive to improving the detection performance of the detection algorithm model.

[0007] In a first aspect, the present application provides a method for generating an abnormal image, which is applied to an electronic device, and the method includes: obtaining a reference abnormal image, wherein the reference abnormal image is obtained by capturing an image of a first preset object, and the first preset object has a first degree of abnormality; obtaining an initialized embedding vector, wherein the embedding vector is a numerical vector representing image features in a multidimensional space; obtaining a reference normal image, wherein the reference normal image includes a second preset object, the second preset object belongs to the same object type as the first preset object, and the second preset object has a second degree of abnormality lower than the first degree of abnormality; inputting the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector; inputting the reference normal image and the updated embedding vector into the image generation model, wherein the image generation model is used to apply the updated embedding vector to the reference normal image to obtain a generated abnormal image.

[0008] In one implementation, the reference abnormal image and the initialized embedding vector are input into an image generation model, and the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector, including: inputting the reference abnormal image and the initialized embedding vector into the image generation model, and the image generation model is used to update the initialized embedding vector based on the reference abnormal image while keeping the parameters of the image generation model unchanged to optimize the embedding vector, thereby obtaining an updated embedding vector.

[0009] In one implementation, based on the aforementioned solution, the image generation model includes a pixel space network and a latent space network;

[0010] The above-mentioned reference abnormal image and the above-mentioned initialized embedding vector are input into the image generation model, and the above-mentioned image generation model is used to update the above-mentioned initialized embedding vector based on the above-mentioned reference abnormal image, including: inputting the above-mentioned reference abnormal image into the encoder of the above-mentioned pixel space network, so as to convert the above-mentioned reference abnormal image into a reference abnormal feature through the above-mentioned encoder, wherein the above-mentioned reference abnormal feature will be processed by the denoising network of the above-mentioned latent space network to obtain the corresponding noisy feature; through the denoising network in the above-mentioned latent space network, based on the initialized embedding vector, the above-mentioned noisy feature is denoised to obtain the noise predicted by the denoising network, wherein the above-mentioned initialized embedding vector is mapped to the above-mentioned denoising network in a cross-attention manner; determining the first objective function according to the noise predicted by the above-mentioned denoising network; updating the parameters in the above-mentioned initialized embedding vector through the above-mentioned first objective function to obtain the updated embedding vector.

[0011] In one implementation, based on the above-mentioned scheme, the denoising network includes T U-Nets corresponding to T denoising processes, where T is a positive integer; the method further includes: determining the j-th key vector based on the initialized embedding vector and the first parameter of the j-th U-net, where j is a positive integer less than or equal to T; determining the j-th value vector based on the initialized embedding vector and the second parameter of the j-th U-net; and mapping the initialized embedding vector to the j-th U-net based on the j-th key vector and the j-th value vector.

[0012] In one implementation, based on the aforementioned solution, determining the first objective function based on the noise predicted by the denoising network includes: determining the first objective function based on the noise predicted by the denoising network and the noise sampled from the Gaussian distribution; and updating the parameters in the initialized embedding vector using the first objective function to obtain the updated embedding vector includes:

[0013] By minimizing the first objective function, the values ​​of the parameters in the embedding vector are updated; and the embedding vector when the first objective function takes the minimum value is determined as the updated embedding vector.

[0014] In one implementation, based on the aforementioned scheme, the reference abnormal image and the initialized embedding vector are input into an image generation model, and the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector, including: inputting the reference abnormal image and its corresponding mask image into a pre-trained image generation model; while keeping the parameters of the image generation model unchanged, determining a second objective function based on the features in the reference abnormal image that are inside the boundary of the mask; updating the parameters in the initialized embedding vector by means of the second objective function to obtain an updated embedding vector; wherein the inside of the boundary of the mask corresponds to the abnormal area in the reference abnormal image.

[0015] In one implementation, based on the aforementioned scheme, while keeping the parameters of the above-mentioned image generation model unchanged, the second objective function is determined based on the features in the above-mentioned reference abnormal image that are inside the boundary of the above-mentioned mask, including: determining the difference between the noise sampled by the Gaussian distribution and the noise predicted by the denoising network in the above-mentioned image generation model; determining the second objective function based on the above-mentioned difference and the features in the above-mentioned reference abnormal image that are inside the boundary of the above-mentioned mask; updating the parameters in the above-mentioned initialized embedding vector through the above-mentioned second objective function to obtain an updated embedding vector, including: updating the values ​​of the parameters in the above-mentioned embedding vector by minimizing the above-mentioned second objective function; and determining the embedding vector when the above-mentioned second objective function takes the minimum value as the above-mentioned updated embedding vector.

[0016] In one implementation, based on the aforementioned scheme, the reference normal image and the updated embedding vector are input into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to obtain a generated abnormal image, including: inputting the reference normal image, the bounding box corresponding to the reference normal image, and the updated embedding vector into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image while keeping the parameters of the image generation model unchanged to obtain the generated abnormal image; wherein, the abnormal area inside the bounding box in the generated abnormal image is the abnormal area.

[0017] In one implementation, based on the aforementioned solution, the reference normal image and the updated embedding vector are input into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to generate an abnormal image, including: inputting the reference normal image into the image generation model, performing T-step noise processing on the reference normal image through the noise addition network of the image generation model, and obtaining the noise addition feature of the reference normal image, wherein the noise addition feature includes the noise addition feature of t steps, where T is a positive integer and t+1 is a positive integer not greater than T; and performing noise addition processing on the reference normal image through the noise addition network of the image generation model. A denoising network is injected with the updated embedding vector, and the denoising features of the reference normal image are subjected to T-step denoising processing to apply the updated embedding vector to the reference normal image; wherein the generated abnormal features obtained by the t-step denoising processing include: a first portion of features outside the bounding box of the loaded features of the t-step, and a second portion of features inside the bounding box of the denoised features corresponding to the loaded features of the t+1-step; the generated abnormal features after the T-step denoising processing are decoded by a decoder of the pixel space network in the image generation model to obtain the generated abnormal image.

[0018] In a second aspect, an abnormal image generation device is provided, which is configured in an electronic device, and the device includes: a first acquisition module, used to obtain a reference abnormal image, wherein the reference abnormal image is obtained by capturing an image of a first preset object, and the first preset object has a first abnormality degree; a second acquisition module, used to obtain an initialized embedding vector, wherein the embedding vector is a numerical vector representing image features in a multidimensional space; a third acquisition module, used to obtain a reference normal image, wherein the reference normal image includes a second preset object, the second preset object belongs to the same object type as the first preset object, and the second preset object has a second abnormality degree lower than the first abnormality degree; a first processing module, used to input the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector; a second processing module, used to input the reference normal image and the updated embedding vector into the image generation model, wherein the image generation model is used to apply the updated embedding vector to the reference normal image to obtain a generated abnormal image.

[0019] In one implementation, based on the aforementioned scheme, the above-mentioned first processing module is specifically used to: input the above-mentioned reference abnormal image and the above-mentioned initialized embedding vector into the image generation model, and the above-mentioned image generation model is used to update the above-mentioned initialized embedding vector based on the above-mentioned reference abnormal image while keeping the parameters of the above-mentioned image generation model unchanged to optimize the above-mentioned embedding vector and obtain an updated embedding vector.

[0020] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned image generation model includes a pixel space network and a latent space network;

[0021] The first processing module includes: a first input unit, a denoising unit, a first determining unit, and a second determining unit;

[0022] Among them, the above-mentioned first input unit is used to: input the above-mentioned reference abnormal image into the encoder of the above-mentioned pixel space network, so as to convert the above-mentioned reference abnormal image into a reference abnormal feature through the above-mentioned encoder, wherein the above-mentioned reference abnormal feature will be processed by the denoising network of the above-mentioned latent space network to obtain the corresponding noisy feature; the above-mentioned denoising unit is used to: through the denoising network in the above-mentioned latent space network, based on the initialized embedding vector, the above-mentioned noisy feature is denoised to obtain the noise predicted by the denoising network, wherein the above-mentioned initialized embedding vector is mapped to the above-mentioned denoising network in a cross-attention manner; the above-mentioned first determination unit is used to: determine the first objective function according to the noise predicted by the above-mentioned denoising network; the above-mentioned second determination unit is used to: update the parameters in the above-mentioned initialized embedding vector through the above-mentioned first objective function to obtain the updated embedding vector.

[0023] In an exemplary embodiment, based on the above-mentioned scheme, the denoising network includes T U-nets corresponding to T denoising processes, where T is a positive integer; the embedding vector acquisition module also includes a mapping unit; wherein the mapping unit is used to: determine the j-th key vector based on the initialized embedding vector and the first parameter of the j-th U-net, where j is a positive integer less than or equal to T; determine the j-th value vector based on the initialized embedding vector and the second parameter of the j-th U-net; and, based on the j-th key vector and the j-th value vector, map the initialized embedding vector to the j-th U-net.

[0024] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned first determination unit is specifically used to: determine the first objective function based on the noise predicted by the above-mentioned denoising network and the noise sampled by the Gaussian distribution; the above-mentioned second determination unit is specifically used to: update the values ​​of the parameters in the above-mentioned embedding vector by minimizing the above-mentioned first objective function; and determine the embedding vector when the above-mentioned first objective function takes the minimum value as the above-mentioned updated embedding vector.

[0025] In an exemplary embodiment, based on the above-mentioned scheme, the above-mentioned first processing module includes: a second input unit, a third determination unit and a fourth determination unit; wherein the above-mentioned second input unit is used to: input the reference abnormal image and its corresponding mask image into a pre-trained image generation model; the above-mentioned third determination unit is used to: while keeping the parameters of the above-mentioned image generation model unchanged, determine the second objective function based on the features in the above-mentioned abnormal image that are inside the boundary of the above-mentioned mask; the above-mentioned fourth determination unit is used to: update the parameters in the above-mentioned initialized embedding vector through the above-mentioned second objective function to obtain an updated embedding vector; wherein the inside of the boundary of the above-mentioned mask corresponds to the abnormal area in the above-mentioned reference abnormal image.

[0026] In one implementation, based on the aforementioned scheme, the third determination unit is configured to: determine the difference between the noise sampled by the Gaussian distribution and the noise predicted by the denoising network; and determine the second objective function based on the difference and the features of the reference abnormal image that are inside the boundary of the mask; the fourth determination unit is specifically configured to: update the values ​​of the parameters in the embedding vector by minimizing the second objective function; and determine the embedding vector when the second objective function takes the minimum value as the updated embedding vector.

[0027] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned second processing module is specifically used to: input the reference normal image, the bounding box corresponding to the above-mentioned reference normal image and the above-mentioned updated embedding vector into the above-mentioned image generation model, and the above-mentioned image generation model is used to apply the above-mentioned updated embedding vector to the above-mentioned reference normal image while keeping the parameters of the above-mentioned image generation model unchanged, so as to obtain a generated abnormal image; wherein, the area inside the above-mentioned bounding box in the above-mentioned generated abnormal image is the abnormal area.

[0028] In an exemplary embodiment, based on the aforementioned solution, the second processing module includes: a first generation unit and a second generation unit; wherein the first generation unit is configured to: input the reference normal image into the image generation model, perform T-step denoising processing on the target image reference normal image through the denoising network of the image generation model, and obtain denoised features of the target image reference normal image, wherein the denoised features include t-step denoising features, where T is a positive integer and t+1 is a positive integer not greater than T; perform T-step denoising processing on the denoised features of the target image reference normal image through a denoising network mapped with the updated embedding vector, so as to apply the updated embedding vector to the reference normal image; wherein the generated abnormal features obtained by the t-step denoising processing include: a first portion of features outside the bounding box according to the loaded features of the t-step, and a second portion of features inside the bounding box according to the denoised features corresponding to the loaded features of the t+1-step; and the second generation unit is configured to: decode the generated abnormal features after the T-step denoising processing through the decoder of the pixel space network to obtain the generated abnormal image.

[0029] In a third aspect, a method for generating training samples of a detection model is provided, which is applied to an electronic device. The method includes: determining a positive sample image, wherein the positive sample image is a reference normal image of a second preset object; determining a generated abnormal image of the first preset object by the method provided in the first aspect or any one of its implementations to obtain a negative sample image; wherein the first preset object has a first degree of abnormality, and the second preset object has a second degree of abnormality lower than the first degree of abnormality.

[0030] In a fourth aspect, a training sample generation device for a detection model is provided, which is configured in an electronic device, and the device includes: a first determination module, used to determine a positive sample image, wherein the positive sample image is a reference normal image of a second preset object; a second determination module, used to determine a generated abnormal image of the first preset object through the method provided by the first aspect or any one of its implementations to obtain a negative sample image; wherein the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree.

[0031] In a fifth aspect, a detection method is provided, which is applied to an electronic device, and the method includes: detecting an object to be detected by a trained detection model to obtain a detection result; wherein, the detection result includes whether the type of the object to be detected is a first preset object or a second preset object, and when the detection result is that the object to be detected is the first preset object, the detection result also includes a positioning result of the abnormal area; wherein, the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree, and the detection model is obtained by sample training determined by the third aspect.

[0032] In a sixth aspect, a detection device is provided, which is configured in an electronic device, and the device includes: a detection module, which is used to detect the object to be detected through a trained detection model to obtain a detection result; wherein, the detection result includes whether the type of the object to be detected is a first preset object or a second preset object, and when the detection result is that the object to be detected is the first preset object, the detection result also includes a positioning result of the abnormal area; wherein, the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree, and the detection model is obtained by sample training determined according to the third aspect.

[0033] In the seventh aspect, an electronic device is provided, comprising a processor and a memory; the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the abnormal image generation method in the above-mentioned first aspect and its various implementation methods, or to execute the training sample generation method of the detection model provided in the above-mentioned third aspect, or to execute the detection method provided in the above-mentioned fifth aspect.

[0034] In an eighth aspect, a chip is provided for implementing the method of any aspect or its implementations described in the first aspect. Specifically, the chip includes a processor configured to retrieve and execute a computer program from a memory, causing a device equipped with the chip to execute the abnormal image generation method described in the first aspect and its implementations, or to execute the method for generating training samples for the detection model described in the third aspect, or to execute the detection method described in the fifth aspect.

[0035] In the ninth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the abnormal image generation method in the above-mentioned first aspect and its various implementation methods, or to execute the training sample generation method of the detection model provided in the above-mentioned third aspect, or to execute the detection method provided in the above-mentioned fifth aspect.

[0036] In the tenth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the abnormal image generation method in the above-mentioned first aspect and its various implementation methods, or to execute the training sample generation method of the detection model provided in the above-mentioned third aspect, or to execute the detection method provided in the above-mentioned fifth aspect.

[0037] In the eleventh aspect, a computer program is provided, which, when running on a computer, enables the computer to execute the abnormal image generation method in the above-mentioned first aspect and its various implementation methods, or to execute the training sample generation method of the detection model provided by the above-mentioned third aspect, or to execute the detection method provided by the above-mentioned fifth aspect.

[0038] In summary, in the solution provided by the embodiments of the present application, a reference abnormal image and an initialized embedding vector are input into an image generation model to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector. The reference abnormal image is obtained by capturing an image of a first preset object with a first degree of abnormality, and the embedding vector is a numerical vector representing image features in a multidimensional space. A reference normal image and the updated embedding vector are then input into the image generation model to apply the updated embedding vector to the reference normal image to obtain a generated abnormal image. The reference normal image includes a second preset object, which belongs to the same object type as the first preset object and has a second degree of abnormality lower than the first degree of abnormality. Because the updated embedding vector can reflect the characteristics of a true abnormal image of that object type, the generated abnormal image obtained after applying the updated embedding vector to the reference normal image contains the characteristics of the true abnormal image. Furthermore, the generated abnormal image can be used as a negative sample to train a detection model, thereby improving the detection performance of the detection algorithm model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a schematic diagram of a system framework applicable to an embodiment of the present application;

[0040] FIG2 is a schematic diagram of a flow chart of a method for generating an abnormal image according to an embodiment of the present application;

[0041] FIG3 is a schematic diagram of a process for determining an updated embedding vector according to an embodiment of the present application;

[0042] FIG4 is a schematic flow chart of a method for determining an updated embedding vector according to an embodiment of the present application;

[0043] FIG5 is a schematic diagram of a process for determining an updated embedding vector according to an embodiment of the present application;

[0044] FIG6 is a schematic diagram of a process for determining an updated embedding vector according to another embodiment of the present application;

[0045] FIG7 is a schematic diagram of a process for determining an updated embedding vector according to another embodiment of the present application;

[0046] FIG8 is a schematic diagram of a process for determining and generating an abnormal image based on an updated embedding vector according to an embodiment of the present application;

[0047] FIG9 is a schematic diagram of a process for determining and generating an abnormal image based on an updated embedding vector according to an embodiment of the present application;

[0048] FIG10 is a schematic diagram of a process for determining and generating an abnormal image based on an updated embedding vector according to another embodiment of the present application;

[0049] FIG11 is a schematic diagram of generating negative samples of a detection model based on a small number of reference abnormal images according to an embodiment of the present application;

[0050] FIG12 is a schematic structural diagram of an abnormal image generating device provided in an embodiment of the present application;

[0051] FIG13 is a schematic diagram of the structure of a training sample generating apparatus for a detection model provided in an embodiment of the present application;

[0052] FIG14 is a schematic diagram of the structure of a detection device provided in an embodiment of the present application;

[0053] FIG15 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In the embodiments of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A, but that B can also be determined based on A and / or other information. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise specified, "plurality" refers to two or more than two.

[0056] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0057] The solutions provided in the embodiments of this application involve artificial intelligence computer vision technology, machine learning and other technologies, which are specifically illustrated by the following embodiments.

[0058] For example, Figure 1 is a schematic diagram of a system framework that can be applied to an embodiment of the present application. As shown in Figure 1, the object type in the embodiment of the present application is illustrated using hazelnuts as an example. Specifically, image 11 represents a normal hazelnut, also known as a good product, i.e., a product with almost no defects or whose defects are difficult to directly determine with the naked eye; image 12 represents an abnormal hazelnut, also known as a defective product, i.e., a product with defects. In the embodiment of the present application, the image of the defective product is denoted as an "abnormal image." In the embodiment of the present application, image 11 can serve as a second preset object with a second degree of abnormality, and image 12 can serve as a first preset object with a first degree of abnormality, where the second degree of abnormality is less than the first degree of abnormality. The measurement of the above degree of abnormality is related to the object type. For example, if the object type is hazelnut, the above degree of abnormality can be the ratio of the damaged surface area of ​​the hazelnut to the total surface area of ​​the hazelnut. The second degree of abnormality can be a ratio of less than 1%, and the above first degree of abnormality can be a ratio of greater than 10%, etc.

[0059] As mentioned above, because negative samples of abnormal products are difficult to obtain on a large scale, related technologies employ unsupervised abnormal product detection algorithms and models for intelligent inspection of industrial products. Specifically, unsupervised detection algorithms and models typically use only features from normal images to learn a normal distribution or reconstruct normal features for anomaly detection. However, due to the lack of discriminative representation, these models perform suboptimal performance and are unable to locate abnormal regions within defective products, for example, failing to locate anomalies at the pixel level. To address this issue, related technologies employ synthetic methods to obtain abnormal images as negative samples for supervised intelligent detection algorithms. For example, the paper "Draem: A Discriminatively Trained Reconstruction Embedding for Surface Anomaly Detection" utilizes an external texture dataset to construct anomaly images; the paper "Cutpaste: Self-supervised Learning for Anomaly Detection and Localization" creates anomaly images by randomly cropping image blocks and pasting them onto other parts; and the paper "Simplenet: A Simple Network for Image Anomaly Detection and Localization" directly perturbs feature maps to simulate the characteristics of abnormal images. However, there is a significant distribution gap between synthetic abnormal images or features and real-world abnormal images or features. Clearly, the detection performance of models trained using synthetic abnormal images as negative samples needs to be improved. It should be noted that in the embodiments of this application, the synthetic abnormal images described above, i.e., abnormal images that are not determined using the image features of real defective or abnormal products, are referred to as "synthetic abnormal images." Clearly, there is a significant distribution gap between "synthetic abnormal images" and real-world abnormal images or features.

[0060] To address the aforementioned technical issues in related technologies, the present embodiment utilizes an embedded vector to learn abnormal semantic features in reference abnormal images, and further generates abnormal images based on the updated embedded vectors. It should be noted that in the present embodiment, abnormal images determined using the real-world image features of defective or abnormal products are referred to as "generated abnormal images." "Generated abnormal images" can reflect abnormal images or features in the real world.

[0061] Exemplarily, the server 104 is equipped with a pre-trained image generation model, and the user can input the reference abnormal image and the initialized embedding vector into the pre-trained image generation model through the terminal 102. Furthermore, the control server 104 updates the parameter values ​​of the initialized embedding vector based on the reference abnormal image while keeping the parameters of the image generation model unchanged to learn the feature distribution of the reference abnormal image (which can be called abnormal semantic features), thereby obtaining an updated embedding vector representing the feature distribution of the abnormal image. Since the embodiment of the present application does not optimize the image generation model with more parameters, but optimizes the embedding vector with fewer parameters, it is easy to implement without a large number of reference abnormal images to conveniently meet the sample requirements in the abnormal image generation process.

[0062] For example, after determining the updated embedding vector, the user can also input the reference normal image and the updated embedding vector into the image generation model through terminal 102. Because the updated embedding vector contains abnormal semantic features learned from the reference abnormal image, the abnormal image features learned from the updated embedding vector can be applied to the reference normal image to generate a generated abnormal image. The generated abnormal image can be used as a negative sample to train the detection algorithm model, which is conducive to improving the detection performance of the detection algorithm model.

[0063] Through the above-mentioned embodiments of the present application, negative samples of the algorithm model for detecting abnormal products can be generated, thereby solving the problem that negative samples of abnormal products are difficult to obtain on a large scale. At the same time, compared with the above-mentioned synthetic abnormal images, the semantic gap between the generated abnormal images and the abnormal images (reference abnormal images) existing in the real world can be reduced, which is conducive to improving the detection performance of the detection algorithm model, thereby making the trained model more robust.

[0064] Exemplarily, by using a trained detection algorithm model on server 104, after a user inputs an image to be tested via terminal 102, server 104 performs anomaly detection on the image to be tested. The detection results can then be displayed on terminal 102 for the user to review. The detection results, as shown in FIG1 , can be the second preset object (i.e., abnormal product, defective product) or the first preset object (normal product, good product). Exemplarily, the predicted category of the current image to be tested is indicated by whether the circle is filled with color.

[0065] Among them, the above-mentioned terminal 102 is a smart phone, a tablet, an intelligent audio and video interactive device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a wearable smart device, a medical device, etc. As mentioned above, the above-mentioned terminal 102 is often equipped with a display device, which can also be a display, a display screen, a touch screen, etc. The touch screen can also be a touch screen, a touch panel, etc. However, it is not limited to this. The above-mentioned server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited thereto.

[0066] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0067] Figure 2 is a schematic flow chart of an abnormal image generation method P200 provided in an embodiment of the present application. Method P200 can be executed by a server, such as terminal 104 in Figure 1 ; terminal 102 in Figure 1 ; or an electronic device storing a relevant algorithm, such as electronic device 1500 shown in Figure 15 . Method P200 is described using an electronic device as an example. Referring to Figure 2 , method P200 includes steps S210 through S250.

[0068] In S210 , the electronic device obtains a reference abnormal image, where the reference abnormal image is obtained by capturing an image of a first preset object, and the first preset object has a first abnormality degree.

[0069] The above-mentioned reference abnormal image can reflect the abnormality of an object in the real world, that is, the image carries the image features of the abnormality of the object in the real event.

[0070] As mentioned above, the standard for measuring the degree of abnormality is related to the object type. Taking hazelnuts as an example, the degree of abnormality of hazelnuts can be measured based on the ratio of the surface defect area of ​​the hazelnuts to the total surface area of ​​the hazelnuts. Specifically, for objects with an abnormality degree higher than or equal to the above-mentioned first abnormality degree (such as the above-mentioned first preset object), it can be considered that the object has visible defects to the naked eye and is an abnormal product or defective product in the current object type, such as the hazelnut image 12 shown in Figure 1. Conversely, for objects with an abnormality degree lower than or equal to the second abnormality degree (such as the second preset object below), it can be considered that the object does not have visible defects to the naked eye and is a normal product or good product in the current object type, such as the hazelnut image 11 shown in Figure 1.

[0071] In the embodiment of the present application, the abnormal reference image may be obtained by capturing an image of the first preset object (e.g., the hazelnut image 12 shown in FIG. 1 ). The normal reference image described below includes the second preset object (e.g., the hazelnut image 11 shown in FIG. 1 ), and may also be obtained by capturing an image of the second preset object.

[0072] The first preset object and the second preset object are of the same object type, for example, both are hazelnuts.

[0073] In S220 , an initialized embedding vector is obtained, where the embedding vector is a numerical vector representing image features in a multi-dimensional space.

[0074] In this embodiment, an initialized embedding vector is used to learn image feature information (specifically, abnormal semantic features) in a reference abnormal image. For example, the parameters in the initialized embedding vector are all initial values ​​(e.g., 0). In this embodiment, the values ​​of the parameters in the embedding vector are updated based on the reference abnormal image, so that the updated embedding vector learns semantic features that can reflect the abnormality of the current object in the real world.

[0075] In S230, the reference abnormal image and the initialized embedding vector are input into an image generation model, and the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector.

[0076] In an exemplary embodiment, FIG3 is a flow chart of determining the updated embedding vector provided by an embodiment of the present application. Referring to FIG3, in an embodiment of the present application, a reference abnormal image 30 and an initialized embedding vector 32 are input into a pre-trained image generation model 300. The image generation model 300 is used to update the initialized embedding vector 32 based on the reference abnormal image 30 while keeping the parameters of the image generation model unchanged, thereby optimizing the embedding vector 34. It should be noted that, when the number of reference abnormal images 30 is limited, it is impractical to optimize an image generation model with millions of parameters so that the model learns the abnormal semantic features of the current object in the real world; however, if an embedding vector with fewer parameters (e.g., only containing hundreds of parameters) is updated so that it learns the above-mentioned abnormal semantic features, the optimization process becomes relatively simple, that is, even if the number of reference abnormal images is limited, the demand for optimizing the embedding vector can be met. It can be seen that the embodiment of the present application focuses on learning embeddings that effectively capture real abnormal semantic features, rather than relying on fine-tuning complex models.

[0077] Exemplarily, the above-mentioned image generation model may be a diffusion model (DM) or a latent diffusion model (LDM). Of course, other types of generation models are also possible, which are not limited in the embodiments of the present application.

[0078] The abnormal image generation method in the embodiments of this application is implemented based on a pretrained image generation model. Therefore, the pretraining process for the aforementioned image generation model will be described below, using the diffusion model as an example. The diffusion model is a type of generative model that generates new samples by gradually adding noise to the data and then learning to reverse this process.

[0079] Exemplarily, the pre-training process of using the diffusion model to perform an image generation task may include steps S31 to S36.

[0080] In step S31, a forward diffusion process is defined, which may specifically include a discrete time step and a noise scheduling function.

[0081] The discrete time step is used to divide the entire process into T discrete time steps, where t = 1, 2, ..., T. At each time step t, Gaussian noise is added to the current data according to a certain noise level. A noise scheduling function is used to control the amount of noise added at each step, such as linear or cosine scheduling.

[0082] In step S32, an inverse generation process is constructed, which may specifically include denoising network design and parameterized inverse process.

[0083] In order to reverse the forward diffusion process, it is necessary to build a neural network (specifically a deep convolutional neural network) that can predict the noise components in a given noisy image and try to remove these noises to restore the original image. The goal of the parameterized inverse process is to learn an inverse process q(x t-1 ∣x t )≈p θ (x t-1 ∣x t ), that is, given a noisy image x t In the case of , predict the image x of the previous step t-1 Or directly predict the noise ∈.

[0084] In step S33, a loss function is defined. For example, it can be a mean squared error loss function that directly minimizes the difference between the predicted noise and the actual added noise. Specifically, for each time step t, the mean squared error between the predicted noise and the actual noise ∈ is calculated.

[0085] In step S34, a pre-training process is performed.

[0086] In step S34-1, a large number of high-quality images are collected as a training set. Ensure that the dataset is diverse enough so that the model can learn a wide range of data distribution characteristics.

[0087] In step S34 - 2 , a batch of images are randomly sampled from the training set at each iteration, and a forward diffusion process is applied to the batch of images to generate samples with different noise levels.

[0088] In step S34-3, the back-propagation algorithm is used to calculate the gradient of the loss function relative to the model parameters, and the optimizer (such as Adam) is used to update the parameters and gradually adjust the model weights to reduce the loss value.

[0089] In step S34-4, the above steps are repeated several times until the model converges or reaches a predetermined maximum number of iterations. As the training progresses, the model gradually learns how to effectively remove noise and reconstruct a clear image.

[0090] In step S35, sampling is performed to generate a new image.

[0091] Initially, a pure noise input (e.g., a random vector sampled from a standard normal distribution) is provided. A stepwise denoising process is then performed. Specifically, the noise is gradually reduced by following the inverse generative process, with each step relying on the results of the previous steps. After T steps, a new composite image is output.

[0092] In an exemplary embodiment, as a specific implementation of step S230, FIG4 is a flow chart of method P400 for determining an updated embedding vector provided in an embodiment of the present application. Method P400 may be executed by a server, such as terminal 104 in FIG1 ; terminal 102 in FIG1 ; or an electronic device storing a relevant algorithm, such as electronic device 1500 shown in FIG15 . Method P400 is described using an electronic device as an example. Referring to FIG4 , the embodiment shown includes steps S410-S450.

[0093] In S410 , the electronic device inputs the reference abnormal image into an encoder of a pixel space network of an image generation model, so that the encoder converts the reference abnormal image into a reference abnormal feature.

[0094] Exemplarily, the above-mentioned image generation model in method P400 takes the latent diffusion model as an example to describe an optimization embodiment of the embedding vector. Referring to Figure 5, the image generation model 50 includes a pixel space network 51 and a latent space network 52. The pixel space network 51 includes an autoencoder model (AutoEncoder) that needs to be trained. The autoencoder model includes an encoder ε and a decoder D. The input image can be compressed by the encoder ε, and then a diffusion operation is performed on the latent representation space. Finally, the decoder D is used to reconstruct the image to restore the image to the pixel space network. By transferring the image to the latent space network for denoising and denoising, only important and basic features can be retained while ignoring the imperceptible high-frequency information in the image, thereby greatly reducing the computational complexity of the training and sampling stages, allowing the generation task to generate images on a consumer-grade GPU in 10 seconds, thereby greatly reducing the landing threshold.

[0095] For example, referring to FIG5, the abnormal image I is referred to as T Input pixel space network 51, after being encoded by encoder ε, obtains reference abnormal feature z in latent space network 52, which can also be recorded as ε(I T ).

[0096] In S420, the electronic device processes the reference abnormal feature through the noise adding network of the latent space network to obtain a noise adding feature corresponding to the reference abnormal feature.

[0097] For example, referring to FIG5 , the reference abnormal feature z is processed by the noise adding network of the latent space network 52 to obtain the noise adding feature z corresponding to the reference abnormal feature T .

[0098] In an exemplary embodiment, the number of reference abnormal images may be one or more, for example, the number of reference abnormal images may be any one from 1 to 10. In the case where the number of reference abnormal images is more than 1, the multiple reference abnormal images input into the image generation model at the same time may belong to the same type of abnormality. For example, taking hazelnuts as an example, the abnormal type of hazelnuts are actually hazelnuts with defects. More specifically, the types of defects may include notches caused by extrusion or defects caused by wear and tear, insect bites, and so on. In order to enable the embedding vector to finely learn the true semantic features, in an embodiment of the present application, reference abnormal images belonging to the same abnormal type are input into the model, for example, different abnormal images of notches caused by extrusion are input into the pre-trained image generation model at the same time, so that the embedding vector learns the semantic features of the notches caused by extrusion.

[0099] In S430, the electronic device performs denoising on the noisy features based on the initialized embedding vector through the denoising network in the latent space network to obtain noise predicted by the denoising network, wherein the initialized embedding vector is mapped to the denoising network in a cross-attention manner.

[0100] Exemplarily, the initialized embedding vector is first mapped to the denoising network in the latent space network in a cross-attention manner.

[0101] In the embodiment of the present application, the denoising network in the latent space network is enhanced by the cross attention mechanism, that is, the conditional denoising autoencoder is realized. Conditional Time Series Denoising Autoencoder The goal is to freeze the model parameters and embed the variables Optimize to learn semantic features in reference anomaly images that reflect true anomalies.

[0102] In the case where the denoising network in the latent space network consists of T U-Nets corresponding to T denoising processes, the embedding variables are transformed through the cross attention layer. Mapped to the jth (positive integer less than or equal to T) middle layer of U-Net, instantiated, according to the embedding vector and the first parameter of the j-th U-net Determine the j-th key vector K; according to the embedding vector and the second parameter of the j-th U-net Determine a j-th value vector V; and map the initialized embedding vector to the j-th U-net based on the j-th key vector K and the j-th value vector V.

[0103] Specifically, mapping the initialized embedding vector to the j-th U-net can be achieved based on formula (1):

[0104] in, represents the initialized embedding vector, ε() represents the automatic encoder of the pixel space network, I T represents the reference abnormal image, is the feature expression of the middle layer of the j-th U-net, d represents the matrix dimension of K and Q, and is the matrix associated with the j-th U-Net parameters.

[0105] It should be noted that in the embodiments of this application and Belongs to the matrix related to the model parameters, in the process of optimizing the embedding vector and This is beneficial for obtaining embedded features that reflect real anomalies using a smaller number of reference anomaly images. It can be understood that, referring to Figure 5 , the denoising network can include T U-nets corresponding to T denoising processes, and the same mapping process is performed for each U-net.

[0106] Exemplarily, the noisy features corresponding to the reference abnormal features are denoised by mapping the denoising network with the embedded vector to obtain the noise predicted by the denoising network.

[0107] For example, referring to FIG5, the noise feature z corresponding to the reference abnormal feature is mapped by the denoising network with the embedded vector T After denoising, we can get the noise z0 predicted by the denoising network, which can also be recorded as

[0108] In S440 , the electronic device determines a first objective function according to the noise predicted by the denoising network.

[0109] Exemplarily, the first objective function is determined based on the noise predicted by the denoising network and the noise sampled by the Gaussian distribution; specifically, the first objective function can be determined according to the following formula (2):

[0110] Among them, ∈ represents the noise sampled from Gaussian distribution, represents the noise predicted by the denoising network.

[0111] In S450, the electronic device updates the parameters in the initialized embedding vector by using the first objective function to obtain an updated embedding vector.

[0112] Exemplarily, by minimizing the first objective function, the values ​​of the parameters in the embedding vector are updated. The embedding vector when the first objective function takes the minimum value is determined as the updated embedding vector. Specifically, the updated embedding vector is determined according to the following formula (3):

[0113] According to the solution provided by method P400, since fewer reference abnormal images are required to optimize the embedding vector, the updated embedding vector can be obtained based on a smaller number of reference abnormal images. Specifically, in the embodiment provided by method P400, it is not necessary to learn the parameters of the entire network of the image generation model, but rather to start embedding and freeze the parameters of the image generation model, so that the embedding parameters can be updated separately during the denoising and de-noising processes. In this way, the learned embedding captures the distribution of the provided real-world anomalies and can be used to guide image generation in subsequent stages. In a further embodiment, since the updated embedding vector It represents the semantic features in the reference abnormal image that reflect the real abnormality, and thus can be applied to the normal image of the good product to obtain the generated abnormal image (in the embodiment of the present application, the "generated abnormal image" can reflect the characteristics of the abnormal image in the real world, and is therefore different from the "synthesized abnormal image" in the related art that cannot reflect the characteristics of the abnormal image in the real world).

[0114] In an exemplary embodiment, another specific implementation of step S230 is described below. That is, the updated embedding vector It can be determined not only based on the image generation model 50 shown in FIG5 , but also based on the image generation model 60 shown in FIG6 . Specifically, in the embodiment shown in FIG6 , the input image is directly subjected to denoising in the pixel space network. For example, referring to the abnormal image I T (can also be written as z), input image generation model 60.

[0115] As mentioned above, the number of reference abnormal images can be one or more, for example, the number of reference abnormal images can be any one from 1 to 10. In the case where the number of reference abnormal images is more than 1, the multiple reference abnormal images input into the image generation model at the same time can belong to the same type of abnormality. In order to enable the embedding vector to finely learn the true semantic features, in an embodiment of the present application, reference abnormal images belonging to the same abnormality type are input into the model, for example, different abnormal images of gaps caused by extrusion are input into the pre-trained image generation model at the same time, so that the embedding vector learns the semantic features of the gaps caused by extrusion.

[0116] The noise z is obtained by processing the reference abnormal image z through the noise adding network of the image generation model 60T In this embodiment, the embedding vector to be initialized is still mapped to the denoising network in a cross-attention manner, that is, the conditional denoising autoencoder is implemented. Conditional Time Series Denoising Autoencoder The goal is to freeze the model parameters and embed the variables Optimize to learn the semantic features that reflect the real anomaly in the reference anomaly image. In the case where the denoising network in the latent space network includes T U-Nets corresponding to T denoising processes, the embedding variables are transformed through the cross attention layer. Mapped to the middle layer of the j-th U-Net, where the cross attention layer is implemented as in formula (4).

[0117] in, represents the initialized embedding vector, ε() represents the automatic encoder of the pixel space network, I T represents the reference abnormal image, is the feature expression of the middle layer of the j-th U-net, d represents the matrix dimension of K and Q, and is the matrix associated with the j-th U-Net parameters.

[0118] It should be noted that in the embodiments of this application and Belongs to the matrix related to the model parameters, in the process of optimizing the embedding vector and This is beneficial for obtaining embedded features that reflect real anomalies using a smaller number of reference anomaly images. It can be understood that, referring to Figure 6, the denoising network can include T U-nets corresponding to T denoising processes, and the same mapping process is performed for each U-net.

[0119] Referring to Figure 6, the noise z corresponding to the reference abnormal image is mapped by the denoising network with the embedded vector T After denoising, we can get the noise z0 predicted by the denoising network, which can also be recorded as A third objective function is determined according to the noise predicted by the denoising network, and an updated embedding vector is determined according to the third objective function.

[0120] The third objective function is determined according to the following formula (5):

[0121] Among them, ∈ represents the noise sampled from Gaussian distribution, represents the noise predicted by the denoising network.

[0122] The updated embedding vector is determined according to the following formula (6):

[0123] In an exemplary embodiment, the updated embedding vector It can not only be determined based on the method shown in Figure 5 or Figure 6. Since the abnormal area in the reference abnormal image is usually a small part of the entire object. Optimizing the above embedding vector on the entire reference abnormal image may cause the data distribution learned by the updated embedding vector to be biased towards the characteristics of the object itself, rather than focusing on the characteristics of the abnormal area. For example, a hole on a carpet, where the abnormal area "hole" is a small part of the entire object "carpet" image; optimizing the embedding vector on the abnormal image about the carpet may cause the data distribution learned by the updated embedding vector to be biased towards the entire carpet, rather than focusing on the feature distribution at the abnormal area "hole". However, one of the main problems solved by the embodiment of the present application is to determine the generated abnormal image, focusing on the semantic features at the abnormal area. In order to solve this problem, the embodiment of the present application combines the reference abnormal image I T The segmentation mask M T To optimize the initialized embedding vector This can be used as another specific implementation of step S230.

[0124] In an exemplary embodiment, referring to FIG7 , the image generation model used in the embodiment shown in the figure is consistent with the image generation model used in FIG5 . Exemplarily, the content of the pixel space network 51 of the input model includes the reference abnormal image I T and its corresponding mask M T For example, FIG7 shows a reference abnormal image and its corresponding mask. In other embodiments, the input of the model may include multiple reference abnormal images and their corresponding masks, such as the i-th mask. The boundary corresponds to the i-th reference abnormal image For example, referring to FIG7 , the reference abnormal image I of hazelnut T , the corresponding mask M T The boundary is the damaged edge of the hazelnut, and the model outputs a reconstructed image based on the abnormal image.

[0125] In this embodiment, after the reference abnormal image and its corresponding mask image are input into the pre-trained image generation model, the second objective function is determined based on the features of the reference abnormal image that are inside the boundary of the corresponding mask while keeping the parameters of the image generation model unchanged. Exemplarily, the difference between the noise sampled by the Gaussian distribution and the noise predicted by the denoising network is determined; and then the second objective function is determined based on the difference and the features of the reference abnormal image that are inside the boundary of the mask. Specifically, the objective function of the model (which can be recorded as the second objective function) is determined according to formula (7)

[0126] Among them, ∈ represents the noise sampled from Gaussian distribution, represents the noise predicted by the denoising network, I T represents the reference abnormal image, M T Denotes the reference abnormal image I T mask.

[0127] Furthermore, the updated embedding vector is determined according to the second objective function The following formula (8):

[0128] In the implementation method of determining the updated embedding vector as shown in Figure 7, a mask of the reference abnormal image is introduced to guide the capture of local details of the object in a targeted manner. That is, by guiding the learning of the abnormal area, it is possible to ensure that the generated image exhibits semantic controllability while also controlling spatial controllability. Specifically, at the semantic level, the reference abnormal image is intended to generate an image that is consistent with the abnormalities in the real world, maintaining the consistency of the object or texture (for example, a bottle) and the abnormal type (for example, breakage); at the spatial level, the position, size and number of the abnormal area can be accurately controlled by providing a mask. Based on semantic controllability and spatial controllability, the updated embedding vector can be prompted to accurately learn the real abnormal semantic features, which is conducive to generating high-quality abnormal images in the following embodiments.

[0129] It can be understood that the method of determining the embedding vector based on the reference abnormal image and its corresponding mask can be applied not only to the image generation model shown in Figure 5, but also to the image generation model shown in Figure 6, which will not be repeated here.

[0130] 2 , in S240 , the electronic device obtains a reference normal image, wherein the reference normal image includes a second preset object, the second preset object and the first preset object belong to the same object type, and the second preset object has a second abnormality degree lower than the first abnormality degree.

[0131] The reference normal image, the second predicted object, and the second abnormality level have been described in detail in the embodiment corresponding to step S210 and will not be repeated here.

[0132] Continuing with reference to FIG2 , after determining the updated embedding vector and obtaining the reference normal image, S250 is executed: the electronic device inputs the reference normal image and the updated embedding vector into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to generate an abnormal image.

[0133] In an exemplary embodiment, since the above updated embedding vector The real abnormal semantic features are learned, so the updated embedding vector can be used in a cross-attention manner Mapped to the denoising network of the model to The real abnormal semantic features carried are applied to the reference normal image, that is, the abnormal region is generated based on the reference normal image, and then the generated abnormal image is output. For example, referring to FIG8 , the reference normal image 80 and the updated embedding vector 34 inputs the pre-trained image generation model 300, and the output of the model is to generate an abnormal image.

[0134] For example, referring to FIG9 , in S250-1, the electronic device refers to the normal image I normal (i.e., an image without abnormal regions), the encoder ε of the pixel space network of the input image generation model 50 is input so that the encoder ε can generate the input image I normal After compression, the target feature z in the latent space network 52 is obtained after encoding by the encoder ε, which can also be recorded as ε(I normal Then, the target feature z is diffused in the potential representation space, and the decoder D is used to reconstruct the image. To restore the image to the pixel space network. The reference normal image I normal By transferring the image to the latent space network 52 for denoising and adding noise, only important and basic features can be retained while ignoring the imperceptible high-frequency information in the image, thereby significantly reducing the computational complexity of the training and sampling stages.

[0135] For example, referring to FIG9 , in S250 - 2 , the electronic device processes the target feature z through the noise adding network of the latent space network 52 to obtain the noise adding feature z corresponding to the target feature. T In S250-3, the electronic device embeds the updated vector in a cross-attention manner. Mapped to the denoising network of model 50. In the embodiment of the present application generating abnormal images, the updated embedding vector is frozen and injected into the image as a condition through the cross-attention module, thereby generating the expected anomaly. Specifically, the denoising network in the latent space network is enhanced by the cross-attention mechanism, that is, the conditional temporal denoising autoencoder is realized. Conditional Time Series Denoising Autoencoder The goal is to embed the vector after updating Under the influence of T Predict a corresponding denoised variable or predict noise, where z T is the input ε(I normal ) to generate abnormal images output by model 50 Contains semantic features that reflect real anomalies.

[0136] In the case where the denoising network of model 50 is U-Net, the updated embedding vector is updated through the cross attention layer. Mapped to the middle layer of each U-Net (taking the h-th U-Net as an example, h is a positive integer less than or equal to T), the cross attention layer implements the following formula (9):

[0137] in, represents the updated embedding vector, ε() represents the automatic encoder of the pixel space network, I normal represents the reference normal image, is the feature expression of the middle layer of the h-th U-net, d represents the matrix dimension of K and Q, and is a matrix related to the hth U-Net parameter. It can be understood that the denoising network can include T U-nets corresponding to T denoising processes, and the same mapping process is performed for each U-net.

[0138] For example, in S250-4, the electronic device maps the updated embedding vector The denoising network, the noise feature z corresponding to the target feature T After denoising, we can get the noise z0 predicted by the denoising network, which can also be recorded as

[0139] It should be noted that in the embodiments of this application and It is a matrix related to the model parameters. In the process of determining the generation of abnormal images, all the parameters of the model (including and ) remains unchanged and keeps the updated embedding vector Under the condition of no change, due to Carrying real abnormal semantic features, thus the updated embedding vector is mapped The denoising network can apply or pass the real abnormal semantic features to the noise-added features z corresponding to the target features T , and then the noise z0 predicted by the denoising network also carries the real abnormal semantic features, which ultimately guides the reference normal image I normal Generate the corresponding abnormal image effect.

[0140] In S250-5, the electronic device reconstructs the image through the noise z0 predicted by the encoder D denoising network to generate an abnormal image

[0141] Through the solution provided by the embodiment of the present application, since the embedded vector We learn the distribution of real-world anomalies provided by the reference anomaly images, and thus embed the updated vectors through the cross-attention mechanism. A denoising network is introduced and applied to normal images of good products to guide the generation of abnormal images in the image generation stage, so that the generated abnormal images can contain the semantic features of real abnormalities.

[0142] In an exemplary embodiment, in order to adjust any one of the position, size, and number of abnormal regions in the generated abnormal image, a bounding box can be introduced. Specifically, the bounding box can be input into the pre-trained image generation model together with the reference normal image. For example, referring to FIG10, the image generation model used in the embodiment shown in the figure is consistent with the image generation model used in FIG5. For example, the content of the pixel space network 51 of the input model includes the reference normal image I normal and its corresponding bounding box M box , where the bounding box M box The position of the reference normal image I normal At any position in the corresponding generated abnormal image In the middle, the bounding box M box For example, referring to FIG10 , the reference normal image I of hazelnut is normal , along with the bounding box M of the input model box The inner side shows the damaged area of ​​hazelnut in the generated abnormal image. box

[0143] In the embodiment of the present application, for each step of the de-noising process, the inference image (as shown in FIG. 10 ) T -1…z0), the bounding box M box The inner area will be preserved, and the bounding box M boxThe outer area will be referenced to the normal image I normal In this way, the embodiment of the present application can control the generated abnormal area to be located in the specified area of ​​the reference normal image while keeping other areas as unchanged as possible.

[0144] Exemplarily, a reference normal image is input into the image generation model, and the reference normal image is subjected to T-step denoising processing by the denoising network of the image generation model to obtain the denoised features of the reference normal image, wherein the denoised features include t-step denoising features, where T is a positive integer and t+1 is a positive integer not greater than T; and the denoising network mapped with the updated embedding vector is used to remove the T-step denoising processing from the denoised features of the reference normal image; wherein the generated abnormal feature zt obtained in the t-step denoising processing can be determined according to the following formula (10):

[0145] in, represents ε(I normal ) The noise feature at step t, z' t Indicates zt +1 Denoising features, M box It can be seen that the generated abnormal feature zt obtained in the t-step denoising process includes: the first part of the features outside the bounding box in the loaded features of step t, that is, And, the second part of the feature z' inside the bounding box in the denoised feature corresponding to the loaded feature of step t+1 t ⊙M box .

[0146] Further, referring to FIG10 , the abnormal feature z0 is decoded by the decoder D of the pixel space network 51 to obtain the above-mentioned reference normal image I normal The corresponding generated abnormal image

[0147] In the embodiments of the present application, images (i.e., the aforementioned generated abnormal images) are generated from random noise based on an image generation model such as a diffusion model. Therefore, the generated abnormal images exhibit a certain degree of semantic diversity. Furthermore, to further enhance diversity, the aforementioned bounding boxes can be introduced during the generation of abnormal images in the present application to adjust the position, size, and number of abnormal regions, thereby increasing the diversity of the generated abnormal images.

[0148] In an exemplary embodiment, the present application also provides a method for generating training samples for a detection model. The method can be executed by a server, such as terminal 104 in Figure 1 ; terminal 102 in Figure 1 ; or an electronic device storing a relevant algorithm, such as electronic device 1500 shown in Figure 15 . In this embodiment, the method for generating training samples for a detection model is described using an electronic device as an example.

[0149] The method includes: S1. The electronic device determines a positive sample image, wherein the positive sample image is a reference normal image of a second preset object. That is, the positive sample image refers to an image of a good product (normal product) with a low degree of abnormality, and it can be considered that there are no abnormal areas in the image. S2. The electronic device determines the generation of an abnormal image of the first preset object using the abnormal image generation method provided in the above embodiment to obtain a negative sample image. The first preset object has a first degree of abnormality, and the second preset object has a second degree of abnormality lower than the first degree of abnormality.

[0150] As mentioned above, negative samples of abnormal products are difficult to obtain on a large scale, and the abnormal images obtained by synthesis in the related art have a significant distribution gap with the abnormal images or features in the real world, which is not conducive to training abnormality detection models in practical applications. In the sample generation scheme provided in the embodiment of the present application, the distribution of image features about abnormal areas in the reference abnormal image is learned by embedding vectors, and the generated abnormal image is obtained based on the embedded features. Therefore, the embodiment of the present application can solve the problem that negative samples of abnormal products are difficult to obtain on a large scale. At the same time, compared with the above-mentioned synthetic abnormal images, the semantic gap between the generated abnormal images and the abnormal images (reference abnormal images) in the real world can be reduced, which is conducive to improving the detection performance of the detection algorithm model, thereby making the trained model more robust.

[0151] The above generally introduces the abnormal image generation method and the detection model training sample generation method provided in the embodiments of the present application. The following further introduces the abnormal image generation method provided in the embodiments of the present application through specific examples.

[0152] FIG11 is a schematic diagram of generating negative samples of a detection model based on a small number of reference abnormal images provided in an embodiment of the present application.

[0153] Because negative samples of abnormal products are difficult to obtain on a large scale, in the embodiments of the present application, negative samples containing the distribution of image features related to abnormal regions in the reference abnormal images can be generated based on a small number of reference abnormal images. Referring to FIG11 , the above goal is achieved using only three reference abnormal images 111 . Specifically, based on the specific implementation of S210 in FIG2 , the three reference abnormal images 111 can be used to determine the embedding vector for learning the distribution of image features related to abnormal regions in the reference abnormal images.

[0154] Furthermore, for the randomly acquired image of a normal product and bounding box 112, based on the specific implementation of S220 in FIG. 2 , an abnormal image 114 can be generated. For the randomly acquired image of a normal product and bounding box 113, based on the specific implementation of S220 in FIG. 2 , an abnormal image 115 can be generated. It is understood that the image 112 of the normal product can serve as a positive sample for the detection algorithm model, and the abnormal image 114, obtained based on the specific implementation of S220 in FIG. 2 , can serve as a negative sample for the detection algorithm model.

[0155] It can be seen that the embodiment of the present application proposes a solution that can drive the generation of more realistic and diverse abnormal images through a small number of reference abnormal images. Specifically, based on the embedding vectors of some given real abnormalities, the real abnormality distribution is learned. Furthermore, the updated embedding vectors and the given bounding boxes are used to guide the generation model to generate realistic and diverse abnormalities on specific objects, thereby solving the technical problem that abnormal samples are difficult to obtain on a large scale. At the same time, compared with the synthetic abnormal images in the solutions provided by the related technologies, the solution provided by the embodiment of the present application can determine the generated abnormal images that can reflect the semantic features of the real abnormalities, so that the intelligent detection algorithm obtained by training the generated abnormal images in the embodiment of the present application as negative samples has higher detection performance.

[0156] In an exemplary embodiment, the embodiment of the present application also provides a detection method. Specifically, the method includes: detecting the object to be detected by the trained detection model to obtain a detection result; wherein, the detection result includes whether the type of the object to be detected is a first preset object or a second preset object, and when the detection result is that the object to be detected is the first preset object, the detection result also includes the positioning result of the abnormal area. Wherein, the detection model is obtained by training the samples determined according to the training sample generation method of the detection model of the above embodiment. Since the training sample generation method of the above detection model can provide rich negative samples and positive samples, and the negative samples are generated abnormal images containing real abnormal features, compared with the synthetic abnormal images in the solutions provided by the related technologies, the intelligent detection algorithm obtained by training the generated abnormal images as negative samples in the embodiment of the present application has higher detection performance.

[0157] The above describes in detail the method embodiment of the present application in conjunction with Figures 1 to 11, and the following describes in detail the device embodiment of the present application in conjunction with Figures 12 to 14.

[0158] Figure 12 is a schematic diagram of the structure of an abnormal image generation device 1200 provided in an embodiment of the present application. Abnormal image generation device 1200 can be deployed on a server, such as terminal 104 in Figure 1 ; it can also be deployed on terminal 102 in Figure 1 ; or it can be deployed on an electronic device storing a relevant algorithm, such as electronic device 1500 shown in Figure 15 . This embodiment of the present application is not limited to this.

[0159] 12, the abnormal image generating device 1200 includes: a first obtaining module 1210, a second obtaining module 1220, a third obtaining module 1230, a first processing module 1240 and a second processing module 1250; wherein the first obtaining module 1210 is used to obtain a reference abnormal image, the reference abnormal image is obtained by collecting an image of a first preset object, and the first preset object has a first abnormality degree; the second obtaining module 1220 is used to obtain an initialized embedding vector, the embedding vector is a numerical vector representing an image feature in a multidimensional space; the third obtaining module 1230 is used to obtain a reference normal image, the reference normal image includes the second preset image. object, the second preset object and the first preset object belong to the same object type, and the second preset object has a second abnormality degree lower than the first abnormality degree; the first processing module 1240 is used to input the reference abnormal image and the initialized embedding vector into the image generation model, and the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector; and the second processing module 1250 is used to input the reference normal image and the updated embedding vector into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to obtain a generated abnormal image.

[0160] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned first processing module 1240 is specifically used to: input the above-mentioned reference abnormal image and the above-mentioned initialized embedding vector into the image generation model, and the above-mentioned image generation model is used to update the above-mentioned initialized embedding vector based on the above-mentioned reference abnormal image while keeping the parameters of the above-mentioned image generation model unchanged to optimize the above-mentioned embedding vector and obtain an updated embedding vector.

[0161] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned image generation model includes a pixel space network and a latent space network;

[0162] The first processing module 1240 includes: a first input unit, a denoising unit, a first determining unit, and a second determining unit;

[0163] Among them, the above-mentioned first input unit is used to: input the above-mentioned reference abnormal image into the encoder of the above-mentioned pixel space network, so as to convert the above-mentioned reference abnormal image into a reference abnormal feature through the above-mentioned encoder, wherein the above-mentioned reference abnormal feature will be processed by the denoising network of the above-mentioned latent space network to obtain the corresponding noisy feature; the above-mentioned denoising unit is used to: through the denoising network in the above-mentioned latent space network, based on the initialized embedding vector, the above-mentioned noisy feature is denoised to obtain the noise predicted by the denoising network, wherein the above-mentioned initialized embedding vector is mapped to the above-mentioned denoising network in a cross-attention manner; the above-mentioned first determination unit is used to: determine the first objective function according to the noise predicted by the above-mentioned denoising network; the above-mentioned second determination unit is used to: update the parameters in the above-mentioned initialized embedding vector through the above-mentioned first objective function to obtain the updated embedding vector.

[0164] In an exemplary embodiment, based on the above-mentioned scheme, the denoising network includes T U-nets corresponding to T denoising processes, where T is a positive integer; the embedding vector acquisition module also includes a mapping unit; wherein the mapping unit is used to: determine the j-th key vector based on the initialized embedding vector and the first parameter of the j-th U-net, where j is a positive integer less than or equal to T; determine the j-th value vector based on the initialized embedding vector and the second parameter of the j-th U-net; and, based on the j-th key vector and the j-th value vector, map the initialized embedding vector to the j-th U-net.

[0165] In an exemplary embodiment, based on the aforementioned scheme, the above-mentioned first determination unit is specifically used to: determine the first objective function based on the noise predicted by the above-mentioned denoising network and the noise sampled by the Gaussian distribution; the above-mentioned second determination unit is specifically used to: update the values ​​of the parameters in the above-mentioned embedding vector by minimizing the above-mentioned first objective function; and determine the embedding vector when the above-mentioned first objective function takes the minimum value as the above-mentioned updated embedding vector.

[0166] In an exemplary embodiment, based on the above-mentioned scheme, the above-mentioned first processing module 1240 includes: a second input unit, a third determination unit and a fourth determination unit; wherein the above-mentioned second input unit is used to: input the reference abnormal image and its corresponding mask image into a pre-trained image generation model; the above-mentioned third determination unit is used to: while keeping the parameters of the above-mentioned image generation model unchanged, determine the second objective function based on the features in the above-mentioned abnormal image that are inside the boundary of the above-mentioned mask; the above-mentioned fourth determination unit is used to: update the parameters in the above-mentioned initialized embedding vector through the above-mentioned second objective function to obtain an updated embedding vector; wherein the inside of the boundary of the above-mentioned mask corresponds to the abnormal area in the above-mentioned reference abnormal image.

[0167] In one implementation, based on the aforementioned scheme, the third determination unit is configured to: determine the difference between the noise sampled by the Gaussian distribution and the noise predicted by the denoising network; and determine the second objective function based on the difference and the features of the reference abnormal image that are inside the boundary of the mask; the fourth determination unit is specifically configured to: update the values ​​of the parameters in the embedding vector by minimizing the second objective function; and determine the embedding vector when the second objective function takes the minimum value as the updated embedding vector.

[0168] In an exemplary embodiment, based on the aforementioned scheme, the second processing module 1250 is specifically used to: input the reference normal image, the bounding box corresponding to the reference normal image, and the updated embedding vector into the image generation model; the image generation model is used to apply the updated embedding vector to the reference normal image while keeping the parameters of the image generation model unchanged, so as to obtain a generated abnormal image; wherein, the area inside the bounding box in the generated abnormal image is the abnormal area.

[0169] In an exemplary embodiment, based on the aforementioned solution, the second processing module 1250 includes: a first generation unit and a second generation unit; wherein the first generation unit is configured to: input the reference normal image into the image generation model, perform T-step denoising on the reference normal image through the denoising network of the image generation model, and obtain a denoised feature of the reference normal image, wherein the denoised feature includes t-step denoising features, where T is a positive integer and t+1 is a positive integer not greater than T; perform T-step denoising on the denoised feature of the reference normal image through a denoising network mapped with the updated embedding vector, and apply the updated embedding vector to the reference normal image; wherein the generated abnormal feature obtained by the t-step denoising includes: a first portion of features outside the bounding box according to the loaded feature of the t-step, and a second portion of features inside the bounding box according to the denoised feature corresponding to the loaded feature of the t+1-step; and the second generation unit is configured to: decode the generated abnormal feature after the T-step denoising through the decoder of the pixel space network to obtain a generated abnormal image.

[0170] In the abnormal image generation scheme provided in an embodiment of the present application, a reference abnormal image and an initialized embedding vector are input into an image generation model to update the initialized embedding vector based on the reference abnormal image, thereby obtaining an updated embedding vector. The reference abnormal image is obtained by capturing an image of a first predetermined object with a first abnormality level, and the embedding vector is a numerical vector representing image features in a multidimensional space. A reference normal image and the updated embedding vector are then input into the image generation model to apply the updated embedding vector to the reference normal image, thereby obtaining a generated abnormal image. The reference normal image includes a second predetermined object, which belongs to the same object type as the first predetermined object and has a second abnormality level lower than the first abnormality level. Because the updated embedding vector can reflect the characteristics of a true abnormal image of that object type, the generated abnormal image obtained by applying the updated embedding vector to the reference normal image contains the characteristics of the true abnormal image. Furthermore, the generated abnormal image can be used as a negative sample to train a detection algorithm model, thereby improving the detection performance of the detection algorithm model.

[0171] It should be understood that the abnormal image generation device embodiment and the abnormal image generation method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they are not described here in detail. Specifically, the abnormal image generation device shown in Figure 12 can execute the above-mentioned abnormal image generation method embodiment, and the aforementioned and other operations and / or functions of the various modules in the device are respectively for implementing the abnormal image generation method embodiment. For the sake of brevity, they are not described here in detail.

[0172] Figure 13 is a schematic diagram of the structure of a detection model training sample generation device 1300 provided in an embodiment of the present application. The detection model training sample generation device 1300 can be configured on a server, such as terminal 104 in Figure 1 ; it can also be configured on terminal 102 in Figure 1 ; or it can be configured on an electronic device storing a relevant algorithm, such as electronic device 1500 shown in Figure 15 . This embodiment of the present application is not limited to this.

[0173] Referring to FIG13 , the training sample generation apparatus 1300 of the detection model includes: a first determination module 1310 and a second determination module 1320 ; wherein the first determination module 1310 is configured to determine a positive sample image, wherein the positive sample image is a reference normal image of a second preset object; and the second generation module 1320 is configured to determine a generated abnormal image of the first preset object using the abnormal image generation method provided in the above embodiment to obtain a negative sample image. The first preset object has a first abnormality level, and the second preset object has a second abnormality level lower than the first abnormality level.

[0174] It is difficult to obtain negative samples of abnormal products on a large scale, and the abnormal images obtained by synthesis in the related art have a significant distribution gap with the abnormal images or features in the real world, which is not conducive to training abnormality detection models in practical applications. In the sample generation scheme provided in the embodiment of the present application, the distribution of image features about abnormal areas in the reference abnormal image is learned by embedding vectors, and the generated abnormal image is obtained based on the embedded features. Therefore, the embodiment of the present application can solve the problem that negative samples of abnormal products are difficult to obtain on a large scale. At the same time, compared with the above-mentioned synthetic abnormal images, the semantic gap between the generated abnormal images and the abnormal images (reference abnormal images) in the real world can be reduced, which is conducive to improving the detection performance of the detection algorithm model, thereby making the trained model more robust.

[0175] It should be understood that the embodiments of the apparatus for generating training samples of the detection model and the embodiments of the method for generating training samples of the detection model may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be described here. Specifically, the apparatus for generating training samples of the detection model shown in FIG13 can execute the embodiments of the method for generating training samples of the detection model described above, and the aforementioned and other operations and / or functions of the various modules in the apparatus are respectively for implementing the embodiments of the method for generating training samples of the detection model, which will not be described here for the sake of brevity.

[0176] Figure 14 is a schematic diagram of the structure of a detection device 1400 provided in an embodiment of the present application. Detection device 1400 can be deployed on a server, such as terminal 104 in Figure 1 ; it can also be deployed on terminal 102 in Figure 1 ; or it can be deployed on an electronic device that stores a relevant algorithm, such as electronic device 1500 shown in Figure 15 . This embodiment of the present application is not limited to this.

[0177] Referring to Figure 14, the detection device 1400 includes: a detection module 1410; wherein the above-mentioned detection module 1410 is used to detect the object to be detected through the trained detection model to obtain a detection result; wherein, the detection result includes whether the type of the object to be detected is a first preset object or a second preset object, and when the detection result is that the object to be detected is the second preset object, the detection result also includes a positioning result of the abnormal area; wherein, the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree, and the detection model is obtained by sample training determined according to claim 10.

[0178] It should be understood that the detection device embodiment and the detection method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they are not described here. Specifically, the detection device shown in Figure 14 can perform the above-mentioned detection method embodiment, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the detection method embodiment. For the sake of brevity, they are not described here.

[0179] The above text describes the apparatus for generating an embodiment of the building information model of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in the form of hardware, or can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules. Specifically, the various steps of the method embodiments in the embodiments of the present application can be completed by hardware integrated logic circuits and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in conjunction with its hardware.

[0180] FIG15 is a schematic block diagram of an electronic device 1500 provided in an embodiment of the present application. The electronic device 1500 in FIG15 can be used to execute the above-mentioned abnormal image generation method, or execute the above-mentioned detection model training sample generation method, or execute the above-mentioned detection method. As shown in FIG15 , the electronic device 1500 may include:

[0181] The memory 1510 and the processor 1520 are configured to store a computer program 1530 and transmit the program code 1530 to the processor 1520. In other words, the processor 1520 can call and execute the computer program 1530 from the memory 1510 to implement the method in the embodiment of the present application.

[0182] For example, the processor 1520 may be configured to execute the steps of the above method according to the instructions in the computer program 1530 .

[0183] In some embodiments of the present application, the processor 1520 may include but is not limited to:

[0184] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0185] In some embodiments of the present application, the memory 1510 includes but is not limited to:

[0186] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0187] In some embodiments of the present application, the computer program 1530 may be divided into one or more modules, which are stored in the memory 1510 and executed by the processor 1520 to implement the above-mentioned abnormal image generation method provided in the present application, or to execute the above-mentioned method for generating training samples for the detection model, or to execute the above-mentioned detection method. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 1530 in the electronic device.

[0188] As shown in FIG15 , the electronic device 1500 may further include:

[0189] The transceiver 1540 may be connected to the processor 1520 or the memory 1510 .

[0190] The processor 1520 may control the transceiver 1540 to communicate with other devices. Specifically, the processor 1520 may send information or data to other devices or receive information or data sent by other devices. The transceiver 1540 may include a transmitter and a receiver. The transceiver 1540 may further include one or more antennas.

[0191] It should be understood that the various components in the electronic device 1530 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0192] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiment. Alternatively, the present application also provides a computer program product containing instructions. When the computer is executed by the instructions, the computer is enabled to perform the method of the above-mentioned method embodiment.

[0193] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of the above-described method embodiment.

[0194] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0195] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0197] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0198] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for generating an abnormal image, characterized in that: Applied to electronic equipment, the method includes: Obtaining a reference abnormal image, where the reference abnormal image is obtained by capturing an image of a first preset object, and the first preset object has a first abnormality degree; Obtaining an initialized embedding vector, wherein the embedding vector is a numerical vector representing image features in a multidimensional space; Obtaining a reference normal image, wherein the reference normal image includes a second preset object, the second preset object and the first preset object belong to the same object type, and the second preset object has a second abnormality level lower than the first abnormality level; Inputting the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector; The reference normal image and the updated embedding vector are input into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to generate an abnormal image.

2. The method according to claim 1, characterized in that Inputting the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector, including: The reference abnormal image and the initialized embedding vector are input into an image generation model. The image generation model is used to update the initialized embedding vector based on the reference abnormal image to optimize the embedding vector while keeping the parameters of the image generation model unchanged, thereby obtaining an updated embedding vector.

3. The method according to claim 1 or 2, characterized in that The image generation model includes a pixel space network and a latent space network; Inputting the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image, includes: Inputting the reference abnormal image into an encoder of the pixel space network to convert the reference abnormal image into reference abnormal features through the encoder, wherein the reference abnormal features are processed by the denoising network of the latent space network to obtain corresponding noisy features; Denoising the noisy features based on an initialized embedding vector using a denoising network in the latent space network to obtain noise predicted by the denoising network, wherein the initialized embedding vector is mapped to the denoising network in a cross-attention manner; Determining a first objective function based on the noise predicted by the denoising network; The parameters in the initialized embedding vector are updated using the first objective function to obtain an updated embedding vector.

4. The method according to claim 3, characterized in that The denoising network includes T U-Nets corresponding to T denoising processes, where T is a positive integer; and the method further includes: Determine a j-th key vector based on the initialized embedding vector and a first parameter of the j-th U-net, where j is a positive integer less than or equal to T; Determining a j-th value vector based on the initialized embedding vector and the second parameter of the j-th U-net; Mapping the initialized embedding vector to the j-th U-net based on the j-th key vector and the j-th value vector.

5. The method according to claim 3 or 4, characterized in that Determining a first objective function according to the noise predicted by the denoising network includes: Determining a first objective function based on the noise predicted by the denoising network and the noise sampled by the Gaussian distribution; Updating the parameters in the initialized embedding vector by using the first objective function to obtain an updated embedding vector includes: Updating the values of the parameters in the embedding vector by minimizing the first objective function; The embedding vector when the first objective function takes a minimum value is determined as the updated embedding vector.

6. The method according to claim 1 or 2, characterized in that Inputting the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is used to update the initialized embedding vector based on the reference abnormal image to optimize the embedding vector to obtain an updated embedding vector, including: Input the reference abnormal image and its corresponding mask image into the pre-trained image generation model; determining a second objective function based on features within the boundary of the mask in the reference abnormal image while maintaining parameters of the image generation model unchanged; Updating parameters in the initialized embedding vector using the second objective function to obtain an updated embedding vector; The inner side of the boundary of the mask corresponds to the abnormal area in the reference abnormal image.

7. The method according to claim 6, characterized in that The determining of a second objective function based on features within the boundary of the mask in the reference abnormal image while keeping parameters of the image generation model unchanged comprises: Determining the difference between the noise sampled from the Gaussian distribution and the noise predicted by the denoising network in the image generation model; determining a second objective function based on the difference and features in the reference abnormal image that are inside the boundary of the mask; Updating parameters in the initialized embedding vector by using the second objective function to obtain an updated embedding vector includes: Updating the values of the parameters in the embedding vector by minimizing the second objective function; The embedding vector when the second objective function takes a minimum value is determined as the updated embedding vector.

8. The method according to claim 6 or 7, characterized in that The step of inputting the reference normal image and the updated embedding vector into the image generation model, wherein the image generation model is configured to apply the updated embedding vector to the reference normal image to generate an abnormal image, comprises: Inputting a reference normal image, a bounding box corresponding to the reference normal image, and the updated embedded features into the image generation model, wherein the image generation model is configured to apply the updated embedded vector to the reference normal image while keeping parameters of the image generation model unchanged, to obtain the generated abnormal image; The abnormal area in the generated abnormal image is located inside the boundary box.

9. The method according to claim 6 or 7, characterized in that The step of inputting the reference normal image and the updated embedding vector into the image generation model, wherein the image generation model is configured to apply the updated embedding vector to the reference normal image to generate an abnormal image, comprises: Inputting the reference normal image into the image generation model, performing T-step noise processing on the reference normal image through the noise addition network of the image generation model, and obtaining a noise feature of the reference normal image, wherein the noise feature includes the noise feature of t steps, where T is a positive integer and t+1 is a positive integer not greater than T; Performing a T-step denoising process on the noisy features of the reference normal image using a denoising network mapped with the updated embedded features, so as to apply the updated embedded vector to the reference normal image; wherein the generated abnormal features obtained in the t-step denoising process include: a first portion of features outside the bounding box based on the loaded features of the t-step, and a second portion of features inside the bounding box based on the denoised features corresponding to the loaded features of the t+1-step; The abnormal features generated after T-step denoising are decoded by a decoder of the pixel space network in the image generation model to obtain the generated abnormal image.

10. A method for generating training samples for a detection model, characterized in that: Applied to electronic equipment, the method includes: Determining a positive sample image, wherein the positive sample image is a reference normal image of a second preset object; Determining a generated abnormal image about the first preset object by the method according to any one of claims 1 to 9 to obtain a negative sample image; The first preset object and the second preset object belong to the same object type, the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree.

11. A detection method, characterized in that: Applied to electronic equipment, the method includes: The trained detection model is used to detect the object to be tested and obtain the detection result; Wherein, the detection result includes whether the type of the object to be detected is the first preset object or the second preset object. When the detection result shows that the object to be detected is the first preset object, the detection result also includes a positioning result of the abnormal area; The first preset object and the second preset object belong to the same object type, the first preset object has a first abnormality degree, the second preset object has a second abnormality degree lower than the first abnormality degree, and the detection model is obtained by sample training determined according to claim 10.

12. An abnormal image generating device, characterized in that: Configured in an electronic device, the device includes: a first obtaining module, configured to obtain a reference abnormal image, wherein the reference abnormal image is obtained by collecting an image of a first preset object, and the first preset object has a first abnormality degree; A second obtaining module is used to obtain an initialized embedding vector, where the embedding vector is a numerical vector representing image features in a multidimensional space; a third obtaining module, configured to obtain a reference normal image, wherein the reference normal image includes a second preset object, the second preset object and the first preset object are of the same object type, and the second preset object has a second abnormality degree lower than the first abnormality degree; A first processing module is configured to input the reference abnormal image and the initialized embedding vector into an image generation model, wherein the image generation model is configured to update the initialized embedding vector based on the reference abnormal image to obtain an updated embedding vector; The second processing module is used to input the reference normal image and the updated embedding vector into the image generation model, and the image generation model is used to apply the updated embedding vector to the reference normal image to generate an abnormal image.

13. A device for generating training samples for a detection model, characterized in that: Configured in an electronic device, the device includes: A first determining module is configured to determine a positive sample image, wherein the positive sample image is a reference normal image of a second preset object; a second determining module, configured to determine a generated abnormal image about the first preset object by the method according to any one of claims 1 to 9, to obtain a negative sample image; The first preset object and the second preset object belong to the same object type, the first preset object has a first abnormality degree, and the second preset object has a second abnormality degree lower than the first abnormality degree.

14. A detection device, characterized in that: Configured in an electronic device, the device includes: The detection module is used to detect the object to be tested using the trained detection model to obtain the detection result; Wherein, the detection result includes whether the type of the object to be detected is the first preset object or the second preset object. When the detection result shows that the object to be detected is the first preset object, the detection result also includes a positioning result of the abnormal area; The first preset object and the second preset object belong to the same object type, the first preset object has a first abnormality degree, the second preset object has a second abnormality degree lower than the first abnormality degree, and the detection model is obtained by sample training determined according to claim 10.

15. An electronic device comprising a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the abnormal image generation method as described in any one of claims 1 to 9 above, or to implement the training sample generation method of the detection model as described in claim 10 above, or to implement the detection method as described in claim 11 above.

16. A computer-readable storage medium, characterized in that For storing computer programs; The computer program enables the computer to execute the abnormal image generation method as described in any one of claims 1 to 9 above, or execute the training sample generation method of the detection model as described in claim 10 above, or execute the detection method as described in claim 11 above.

17. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the abnormal image generation method described in any one of claims 1 to 9, or execute the training sample generation method of the detection model described in claim 10, or execute the detection method described in claim 11.

Images