Image defect identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image defect identification method, a related device and a medium, and the method comprises the steps: obtaining a plurality of pieces of first variance data, the plurality of pieces of first variance data are respectively extracted in different time steps of a diffusion model, the diffusion model is set to generate a to-be-identified image through a plurality of time steps, the first variance data are variances among a plurality of de-noising features, and the plurality of de-noising features are obtained by multiple times of sampling based on random noise in a time step corresponding to the first variance data; fusing the plurality of pieces of first variance data to obtain second variance data; performing feature extraction based on the second variance data to obtain image defect features; and performing defect identification on the to-be-identified image based on the image defect features to obtain a defect identification result of the to-be-identified image. According to the invention, whether the generated image has defects can be accurately identified. The method and the device can be applied to scenes such as image generation, image quality scoring and training of an image generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to an image defect recognition method, device, electronic device, and storage medium. Background Art

[0002] The diffusion model is a generative model that generates the corresponding image by making noise predictions at multiple time steps and gradually removing the noise predicted at each time step from the Gaussian noise. In the process of image generation using the diffusion model, when denoising at each time step, a certain amount of random noise is often added to enrich the diversity of the image generation results and improve the generalization ability of the diffusion model. However, random noise is uncontrollable. The introduction of random noise may cause some unreasonable defective areas to appear in the final generated image, causing the diffusion model to generate images of poor quality. In related technologies, the quality of images generated by the diffusion model can be evaluated through defect recognition. However, defect recognition is generally performed based on basic features such as color and texture, and the accuracy of defect recognition needs to be improved. Summary of the Invention

[0003] The present disclosure provides an image defect recognition method, device, electronic device, and storage medium, which can improve the accuracy of defect recognition.

[0004] According to one aspect of the present disclosure, a method for identifying image defects is provided, the method comprising:

[0005] Acquiring a plurality of first variance data, the plurality of first variance data being extracted at different time steps of a diffusion model, the diffusion model being configured to generate an image to be recognized over the plurality of the time steps, the first variance data being variances between a plurality of denoising features, the plurality of denoising features being obtained by performing multiple samplings based on random noise in the time steps corresponding to the first variance data;

[0006] Fusing the plurality of first variance data to obtain second variance data;

[0007] Perform feature extraction based on the second variance data to obtain image defect features;

[0008] Defect recognition is performed on the image to be recognized based on the image defect feature to obtain a defect recognition result of the image to be recognized, and the defect recognition result is used to indicate whether there is a defect in the image to be recognized.

[0009] According to another aspect of the present disclosure, there is provided an image defect recognition device, the device comprising:

[0010] a first variance acquisition module, configured to acquire a plurality of first variance data, the plurality of first variance data being extracted in different time steps of a diffusion model, the diffusion model being configured to generate an image to be recognized over the plurality of time steps, the first variance data being variances between a plurality of denoising features, the plurality of denoising features being obtained by multiple samplings based on random noise in the time steps corresponding to the first variance data;

[0011] A second variance acquisition module, configured to fuse a plurality of the first variance data to obtain second variance data;

[0012] a feature extraction module, configured to perform feature extraction based on the second variance data to obtain image defect features;

[0013] The recognition module is used to perform defect recognition on the image to be recognized based on the image defect feature to obtain a defect recognition result of the image to be recognized, wherein the defect recognition result is used to indicate whether there is a defect in the image to be recognized.

[0014] Optionally, the feature extraction module is specifically used to:

[0015] performing convolution on the second variance data to obtain image blur features;

[0016] Acquire the image to be identified, and perform semantic coding on the image to be identified to obtain image semantic features of the image to be identified;

[0017] The image blur feature and the image semantic feature are fused to obtain a fused feature, and the fused feature is mapped to obtain an image defect feature.

[0018] Optionally, the size of the image blur feature along the preset dimension is at least twice the size of the image semantic feature along the preset dimension, and the feature extraction module is specifically configured to:

[0019] Segmenting the image fuzzy feature along a preset dimension to obtain a first fuzzy sub-feature and a second fuzzy sub-feature, wherein the size of the first fuzzy sub-feature along each dimension is consistent with the size of the image semantic feature along each dimension;

[0020] Multiplying the first fuzzy sub-feature and the image semantic feature bit by bit to obtain a fused sub-feature;

[0021] The fusion sub-feature and the second fuzzy sub-feature are concatenated to obtain a fusion feature.

[0022] Optionally, the feature extraction module is specifically used to:

[0023] Performing global pooling on the fused sub-features to obtain channel semantic features;

[0024] Mapping the channel semantic features through a multi-layer perceptron to obtain channel attention weight data;

[0025] The second fuzzy sub-feature is weighted based on the channel attention weight data and then spliced with the fusion sub-feature to obtain a fusion feature.

[0026] Optionally, the feature extraction module is specifically used to:

[0027] Inputting the fused features into a trained defect feature extraction model, wherein the defect feature extraction model includes at least one first convolutional layer;

[0028] The fused features are convolved by the first convolution layer, and the image defect features are determined based on the convolution processing result of the first convolution layer.

[0029] Optionally, the defect feature extraction model further includes at least one second convolution layer, a convolution kernel size of the first convolution layer and a convolution kernel size of each of the second convolution layers are different, and the feature extraction module is specifically configured to:

[0030] Performing convolution processing on the fused features through the first convolution layer to obtain a first convolution feature;

[0031] Inputting the first convolutional features into each of the second convolutional layers respectively to obtain second convolutional features corresponding to each of the second convolutional layers;

[0032] Concatenate the first convolutional features and each of the second convolutional features, and pool the concatenated results to obtain a pooled feature vector;

[0033] Perform feature fusion on the pooled feature vector to obtain the image defect feature.

[0034] Optionally, the image fuzzy feature is obtained using a fuzzy feature extraction model, the image defect feature is obtained using a defect feature extraction model, and the defect recognition result is obtained using a defect recognition model. The device further includes a training module, and the training module is specifically configured to:

[0035] Acquire a plurality of training samples, wherein each of the training samples includes a sample text and a reference label, wherein the reference label is used to indicate whether there is a defect in an image associated with the sample text;

[0036] Denoising the sample text using the diffusion model to obtain a sample image, obtaining sample variance data corresponding to each of the time steps of the diffusion model, and fusing multiple sample variance data;

[0037] Extracting sample fuzzy features based on the fused plurality of sample variance data using the fuzzy feature extraction model;

[0038] Extracting sample defect features based on the sample fuzzy features using the defect feature extraction model;

[0039] Performing defect recognition based on the sample defect features using the defect recognition model to obtain a sample recognition result of the sample image;

[0040] A loss value is determined based on a difference between the sample recognition result and the reference label, and parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model are optimized based on the loss value.

[0041] Optionally, the training module is specifically used to:

[0042] Performing back propagation based on the loss value and a preset learning rate to optimize parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model;

[0043] In the process of training the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model through a plurality of the training samples, the learning rate decays according to a preset decay step size.

[0044] Optionally, the variance acquisition module is specifically used to:

[0045] Obtaining the noise characteristics predicted by the diffusion model at each of the time steps;

[0046] For any of the time steps, obtaining a denoising coefficient corresponding to the time step, performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple denoising features, and determining a characteristic element variance of each pixel point and a first mean value of each pixel point based on the multiple denoising features, wherein the denoising feature includes a characteristic element of each pixel point, the characteristic element variance is the variance between the characteristic elements obtained by multiple sampling at the same pixel point, and the first mean value is the mean value between the characteristic elements obtained by multiple sampling at the same pixel point;

[0047] First variance data is obtained based on a quotient between the feature element variance and the first mean value of the same pixel point.

[0048] Optionally, the variance acquisition module is specifically used to:

[0049] Performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple first sampling results of the noise feature and multiple second sampling results of the random noise;

[0050] The original image feature predicted at the time step is obtained, and for each weighted sampling, the original image feature, the first sampling result, and the second sampling result are weightedly summed to obtain the denoising feature.

[0051] Optionally, the variance acquisition module is specifically used to:

[0052] Obtaining the noise characteristics predicted by the diffusion model at each of the time steps;

[0053] For any of the time steps, obtaining a denoising coefficient corresponding to the time step, setting the denoising coefficient to zero, and performing weighted sampling on the noise feature and random noise according to the denoising coefficient after being set to zero to obtain a third sampling result;

[0054] Setting the denoising coefficient to a preset value, performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient after being set to the preset value to obtain multiple reference features, and determining the difference between the maximum value and the minimum value of the multiple reference features;

[0055] First variance data is obtained based on a quotient between the difference value and the third sampling result.

[0056] Optionally, the variance fusion module is specifically used to:

[0057] determining a second mean value of the variance of the characteristic element of the same pixel point in the plurality of first variance data;

[0058] Second variance data is constructed based on the second mean value of each pixel point.

[0059] Optionally, the defect identification result includes a defect score, and the identification module is specifically configured to:

[0060] Inputting the image defect feature into a defect recognition model for performing defect recognition on the image to be recognized, and determining the similarity between the image defect feature and the support vector associated with each defect evaluation index based on a nonlinear kernel function in the defect recognition model;

[0061] The defect sub-scores of the image to be identified under each of the defect evaluation indicators are obtained according to the similarity conversion, and the defect score is obtained by weighted summation of each of the defect sub-scores.

[0062] Optionally, the defect identification result includes a defect score, and the identification module is specifically configured to:

[0063] Inputting the image defect features into a defect recognition model for performing defect recognition on the image to be recognized, and performing classification based on the image defect features to obtain a plurality of score categories;

[0064] A preset score associated with the score category is obtained, and the preset score is determined as the defect score.

[0065] Optionally, the variance acquisition module is used to:

[0066] Acquire first variance data extracted for a plurality of target time steps during a process in which a diffusion model generates an image to be recognized by performing a denoising operation; wherein the target time step is determined from the plurality of time steps by at least one of the following:

[0067] Determine the multiple time steps as the target time steps;

[0068] Alternatively, the time step whose corresponding step number is not greater than a first preset threshold is determined as the target time step;

[0069] Alternatively, the target time step is determined from the multiple time steps according to a preset interval, wherein the difference between the step numbers of two adjacent target time steps is the preset interval.

[0070] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the image defect recognition method as described above when executing the computer program.

[0071] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image defect recognition method as described above is implemented.

[0072] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of an electronic device, so that the electronic device performs the image defect recognition method as described above.

[0073] The image defect recognition method proposed in the embodiments of the present disclosure obtains multiple first variance data. Since the multiple first variance data are extracted at different time steps of a diffusion model, the diffusion model is configured to generate the image to be recognized over multiple time steps. The first variance data is the variance between multiple denoised features. The multiple denoised features are obtained by multiple samplings based on random noise in the time steps corresponding to the first variance data. Therefore, the first variance data can represent the differences between the multiple denoised features sampled by the diffusion model in a single time step and can reflect the magnitude of the influence of random noise when sampling at a single time step. The larger the first variance data, the higher the magnitude of the influence of random noise on the denoised features sampled at that time step. Then, by fusing the first variance data of multiple time steps, the second variance data obtained can reflect the magnitude of the influence of random noise on the overall process of the diffusion model generating the image to be recognized. Since the magnitude of the influence of random noise on each pixel of the image to be recognized during the generation process determines whether the values of these pixels are stable, the higher the magnitude of the influence of random noise, the more unstable the values of the corresponding pixels are, and the more likely these pixels are to cause defects in the image to be recognized. Based on this, by extracting image defect features from the second variance data and using the image defect features for defect identification, it is possible to more accurately identify whether there are defects in the generated image to be identified, thereby improving the accuracy of defect identification.

[0074] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0076] Figure 1 is a diagram illustrating an architecture of a system to which the image defect recognition method according to an embodiment of the present disclosure is applied;

[0077] Figure 2 A schematic diagram of a flow chart of an image defect recognition method provided by an embodiment of the present disclosure;

[0078] Figure 3 A schematic diagram of a process for performing multi-step denoising to generate an image using a diffusion model provided in an embodiment of the present disclosure;

[0079] Figure 4 A schematic diagram of the process of predicting noise characteristics using the diffusion model of the DiT architecture provided in an embodiment of the present disclosure;

[0080] Figure 5 A model architecture diagram of the image defect recognition method provided by an embodiment of the present disclosure;

[0081] Figure 6a A comparison diagram of the distribution diagram of the variance of characteristic elements of each pixel point extracted during the process of generating an image with defects and an image without defects using a diffusion model according to an embodiment of the present disclosure;

[0082] Figure 6b A distribution diagram of the variance of characteristic elements of each pixel point extracted during the process of generating a defect-free image using a diffusion model provided by an embodiment of the present disclosure;

[0083] Figure 6c A distribution diagram of the variance of characteristic elements of each pixel extracted during the process of generating a defective image using a diffusion model provided by an embodiment of the present disclosure;

[0084] Figure 7 Another schematic diagram of the architecture of a model for applying the image defect recognition method provided by an embodiment of the present disclosure;

[0085] Figure 8 Another schematic diagram of the architecture of a model for applying the image defect recognition method provided in an embodiment of the present disclosure

[0086] Figure 9 A schematic diagram of the architecture of a defect feature extraction model provided in an embodiment of the present disclosure;

[0087] Figure 10 A schematic diagram of the composition of training samples provided in an embodiment of the present disclosure;

[0088] Figure 11 A schematic diagram of a flow chart of applying the image defect recognition method of the present disclosure in an image generation scenario provided by an embodiment of the present disclosure;

[0089] Figure 12 A schematic flow chart of applying the image defect recognition method of the present disclosure in a diffusion model training scenario provided by an embodiment of the present disclosure;

[0090] Figure 13 A schematic diagram of the structure of an image defect recognition device provided by an embodiment of the present disclosure;

[0091] Figure 14 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0092] Figure 15 A partial structural block diagram of the server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0094] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object, such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data for the normal operation of the embodiment of the present disclosure will be obtained.

[0095] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0096] In the related art, in order to enrich the diversity of images generated using diffusion models, a certain amount of random noise is often introduced during the sampling process of each time step of the diffusion model. The uncontrollable nature of this random noise may cause the image ultimately generated by the diffusion model to have irrational defects such as missing torsos. The image quality assessment models in the related art are often trained based on natural images. The models generally focus on basic features such as color and texture, but these irrational defects are often not present in natural images. This makes it difficult for the image quality assessment models in the related art to learn how to determine whether there are obvious defects in the generated image. The related art lacks a method that can accurately identify whether there are defects in the generated image.

[0097] Based on this, the embodiments of the present disclosure provide an image defect recognition method, device, electronic device, and storage medium, which can accurately identify whether there are defects in a generated image.

[0098] Reference Figure 1 , Figure 1This is a schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, wherein the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network.

[0099] Exemplarily, the server 102 receives the text sent by the terminal 101 and generates an image to be identified based on the text using a diffusion model. The server 102 obtains multiple first variance data, and the multiple first variance data are extracted in different time steps of the diffusion model respectively. The diffusion model is set to generate the image to be identified after multiple time steps. The first variance data is the variance between multiple denoising features, and the multiple denoising features are obtained by multiple sampling based on random noise in the time steps corresponding to the first variance data; the multiple first variance data are fused to obtain second variance data; feature extraction is performed based on the second variance data to obtain image defect features; defect recognition is performed on the image to be identified based on the image defect features to obtain a defect recognition result of the image to be identified, and the defect recognition result is used to characterize whether there are defects in the image to be identified.

[0100] When server 102 uses the diffusion model to generate an image, it obtains multiple first variance data. Since these first variance data are extracted at different time steps of the diffusion model, and the diffusion model is configured to generate the image to be identified over multiple time steps, the first variance data represents the variance between multiple denoised features. These denoised features are obtained by multiple samplings based on random noise in the time steps corresponding to the first variance data. Therefore, this first variance data can represent the differences between the multiple denoised features sampled by the diffusion model in a single time step and can reflect the extent to which random noise affects the sampling at that time step. A larger first variance data value indicates a higher extent to which random noise affects the denoised features sampled at that time step. Then, by fusing the first variance data from multiple time steps, the second variance data obtained can reflect the extent to which random noise affects the overall process of generating the image to be identified using the diffusion model. Since the extent to which each pixel in the image to be identified is affected by random noise during its generation determines whether its values are stable, a higher extent to which random noise affects the corresponding pixel values indicates a more unstable value, and a greater likelihood that these pixels will cause defects in the image to be identified. Based on this, by extracting image defect features from the second variance data and using the image defect features for defect identification, it is possible to more accurately identify whether there are defects in the generated image to be identified, thereby improving the accuracy of defect identification.

[0101] Server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Furthermore, server 102 can be a node server in a blockchain network.

[0102] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 may be connected directly or indirectly via wired or wireless communication, which is not limited in the present embodiment.

[0103] Reference Figure 2 , Figure 2 This is an optional flow chart of the image defect recognition method provided by the embodiment of the present disclosure. The image defect recognition method can be executed by the terminal, or by the server, or by the terminal and the server in cooperation. The image defect recognition method includes but is not limited to the following steps S201 to S204.

[0104] Step S201: Acquire a plurality of first variance data.

[0105] Among them, multiple first variance data are extracted respectively in different time steps of the diffusion model, the diffusion model is set to generate the image to be identified through multiple time steps, the first variance data is the variance between multiple denoising features, and the multiple denoising features are obtained by multiple sampling based on random noise in the time steps corresponding to the first variance data.

[0106] The diffusion model is an image generation model that can receive guidance content such as text, voice, semantic labels, etc. input by the user to guide the diffusion model, and generate corresponding images based on the guidance content. Specifically, the diffusion model will predict the noise features to be removed in multiple time steps of performing the denoising operation, and denoise the original random noise based on the predicted noise features in each time step, thereby generating an image to be identified through multi-step denoising. In this embodiment, the diffusion model can be a diffusion model based on the U-net architecture, or a diffusion model based on the diffusion transformer (DiT) architecture. Of course, it can also be a diffusion model built based on other neural network architectures. The specific neural network architecture of the diffusion model is not limited in this embodiment. The image to be identified is an image generated by the diffusion model, rather than a natural image obtained by non-neural network models such as shooting and drawing.

[0107] The noise feature is a feature tensor that is generated by the diffusion model at a specific time step in the process of generating the image to be recognized. It is obtained by predicting the noise based on the diffusion model's input at that time step. The noise feature needs to be removed from the initial Gaussian noise at that time step. The method used by the diffusion model to predict the noise feature can vary depending on the neural network architecture of the diffusion model itself.

[0108] Specifically, the diffusion model generates an image to be identified by performing multiple time-step denoising based on the input guidance content. At each time step, the diffusion model predicts noise based on the original image features of the current time step, the guidance content, and the step number corresponding to the current time step, obtaining the noise features of the current time step. Furthermore, the model performs weighted sampling based on the original image features, noise features, and random noise of the current time step according to preset weight coefficients for the current time step to obtain the denoised features corresponding to the current time step. The original image features of the first time step are sampled from Gaussian noise, while the original image features of other time steps, excluding the first, are the denoised features generated in the previous time step. Thus, the diffusion model generates an image corresponding to the guidance content by performing multiple time-step denoising on the initial original image features based on the guidance content.

[0109] For example, referring to Figure 3 , Figure 3 This is a schematic diagram of the architecture of the diffusion model based on U-net. Under this diffusion model, first, during the training process, the basic image is encoded in the pixel space using an encoder to obtain the original image feature Z0 and perform multi-step forward noise addition to obtain Z composed of pure noise. T , and then Z T Perform denoising for multiple time steps and gradually restore Z T-1 To Z0. Of course, after the training is completed, it is only necessary to sample Z from the Gaussian noise T And it can be used as the original image feature of the first time step without obtaining Z by adding noise T Specifically, the diffusion model predicts noise through multiple encoders and multiple decoders, and the multiple encoders are cascaded in sequence, and the multiple decoders are also cascaded in sequence. There is a one-to-one correspondence between the multiple encoders and the multiple decoders, and the corresponding encoders and decoders are also connected through jump connections. In the first time step, the original image feature Z of the current time step T It will pass through multiple encoders in sequence, and each encoder will encode the input original image feature Z based on the guidance content input by the user (exemplarily shown in the figure as the embedding vector obtained based on the text) T After downsampling, the original image feature Z after multiple downsamplingT It will be passed to multiple decoders cascaded in sequence and upsampled multiple times through multiple decoders in sequence. Specifically, in one possible embodiment, each encoder and decoder is equipped with a cross attention mechanism, and the original image feature Z of the current time step is T It will be downsampled by each encoder in turn, and each encoder will receive the original image feature Z T The cross attention calculation is performed on the guidance content input by the user to obtain the downsampling result of the encoder at the current time step. After that, the downsampling result of each encoder is passed to the next encoder for the next downsampling until all encoders have the original image feature Z T After the downsampling is completed, the original image feature Z after multiple downsampling T It will be passed to multiple decoders cascaded in sequence, and upsampled by multiple decoders in sequence, so as to predict the noise characteristics corresponding to the current time step, and then obtain the denoising feature Z of the current time step based on the noise characteristics. T-1 , and the denoising feature Z T-1 As the original image feature of the next time step, the denoising process of the next time step is performed, and this cycle is repeated until the denoising feature Z0 of the last time step is obtained. The decoder decodes Z0 to obtain the image to be recognized. Specifically, each encoder receives the original image feature Z T The cross-attention calculation is performed based on the guidance content input by the user, and the input received from the previous decoder and the input received from the corresponding encoder to obtain the upsampling result of the decoder at the current time step. In addition, in another possible embodiment, each encoder is composed of several convolutional layers, and each decoder is composed of several deconvolutional layers. At this time, the original image feature Z can be convolved. T Perform multiple downsampling and then use the deconvolution layer to transform the original image features Z after multiple downsampling. T Perform multiple upsampling. In this way, the diffusion model can perform multiple downsampling through multiple encoders and multiple upsampling through multiple decoders, thereby predicting the noise characteristics of each time step based on the guidance content input by the user at each time step.

[0110] For example, refer to Figure 4The diffusion model is a diffusion model based on the diffusion transformer (DiT) architecture. In this diffusion model, there are some pre-processing modules, such as patching modules and embedding modules, followed by multiple cascaded diffusion transformers, and an adaptive normalization layer and a linear transformation layer after the last diffusion transformer to predict noise features and covariance. Each diffusion transformer is equipped with a multi-head attention mechanism, a feedforward neural network, a multi-layer perceptron, and some linear transformation layers for dimensionality scaling and bias processing. Under the diffusion model of this architecture, from the purely random noise feature Z T Start by performing denoising for multiple time steps in sequence, and gradually generate Z t 、Z t-1 , until the denoising result Z0 of the last time step is obtained. In the time step with step number t, the original image feature Z of the current time step is first patched by the patching module. t The image is divided into multiple patches of specific sizes, which constitute the input of the first diffusion converter. At the same time, the embedding module is used to convert the user input guidance content and the step number t of the current time step into corresponding embedding vectors and input them into the multi-layer perceptron of each diffusion converter. Afterwards, in each diffusion converter, the multi-layer perceptron uses the embedding vector output by the embedding module as the context condition, and calculates multiple dimensional scaling parameters α1, α2, β1, β2 and bias parameters γ1, γ2 according to the context condition. At the same time, the multi-head attention mechanism is used to perform multi-head attention calculation on the input of the diffusion converter. It can be understood that except for the first diffusion converter, The input of other diffusion converters outside the converter is the output result of the previous diffusion converter; then, the feedforward neural network is used to perform nonlinear mapping on the calculation results of the multi-head attention mechanism. In this process, some layer normalization is also used to perform layer normalization on the inputs of the multi-head attention mechanism and the feedforward neural network, and some linear transformation layers are used based on the dimension scaling parameters and bias parameters calculated by the multi-layer perceptron to perform dimension scaling and bias operations on the input and output of the multi-head attention mechanism and the input and output of the feedforward neural network, thereby injecting the guidance content and the step number information of the current time step into the noise prediction process to obtain the output of the diffusion converter. After obtaining the output result of the last diffusion converter, the output result of the last diffusion converter is decoded and reshaped by the adaptive normalization layer and the linear transformation layer to obtain the noise features predicted by the diffusion model at each time step, and then the original image feature Z of the time step is calculated based on the predicted noise features. t Perform denoising to obtain denoising feature Z t-1 , and then the denoising feature Z t-1 Update the original image features of the next time step to continue the iterative process.

[0111] The denoised feature is the output of the diffusion model at each time step, obtained after denoising the predicted noise features at that time step. Each denoised feature represents an intermediate generation result in the process of using the diffusion model to generate the image to be identified. Specifically, the denoised feature is actually a tensor composed of multiple feature elements, each of which is used to represent the characteristics of a pixel in a certain dimension in the generated image to be identified, such as the color value, grayscale value, or other characteristics of a pixel in the image. Specifically, at each time step, the diffusion model performs weighted sampling based on the noise features predicted at that time step and random noise to obtain the denoised feature corresponding to that time step. While the noise features predicted in a single time step are fixed, to enrich the diversity of the generated results, the random noise is not fixed during each weighted sampling. Therefore, in a single time step, if multiple weighted sampling is performed based on the noise features predicted in that time step and the random noise, the denoising features obtained by each weighted sampling are different. In related technologies, the diffusion model generally only samples once during inference, and a denoising feature is obtained for image generation.

[0112] It is understood that the tensor shapes of the multiple denoised features obtained by multiple weighted sampling based on noise features and random noise in a single time step are consistent. For example, each denoised feature is a matrix with an M*N shape. At the same time, in different denoised features, feature elements with the same position information correspond to features of the same dimension at the same pixel in the image to be identified. For example, in a single time step, 10 matrices can be sampled as denoised features based on the noise features predicted by the diffusion model and random noise. Then, the feature elements in the i-th row and j-th row of these 10 denoised features all correspond to features of the same dimension at the same pixel in the image to be identified. Accordingly, the first variance data includes the variance of the feature elements of each pixel. The feature element variance is the variance between the feature elements obtained by multiple sampling at the same pixel. Assuming 10 samplings, each sampling will produce a denoised feature. Each denoised feature includes the feature elements of each pixel. In this case, for any pixel, there are feature elements of the denoised features sampled 10 times. Therefore, the feature element variance corresponding to this pixel is the variance between these 10 feature elements. Therefore, in this embodiment, multiple sampling can be performed in a single time step based on the noise characteristics and random noise predicted in the time step to obtain multiple denoising features of the time step, and then the variance between the feature elements with the same position information (i.e., the same pixel point) in the multiple denoising features is calculated by position, thereby obtaining the variance between the feature elements in the multiple denoising features sampled multiple times by each pixel point in a single time step, thereby determining the first variance data of the time step.

[0113] Specifically, refer to Figure 5 , the related description of the diffusion model to generate the image to be identified refers to the above Figure 3 , which will not be elaborated here. The first variance data is extracted at each time step during the diffusion model's denoising process to generate the image to be identified. It can be used to characterize the differences between the multiple denoised features that may be obtained when the diffusion model performs multiple weighted sampling based on the noise features predicted at the current time step and random noise when executing the corresponding time step. It can be understood that since the noise features predicted by the diffusion model in a single time step are fixed, the differences between the multiple denoised features obtained by performing multiple weighted sampling based on the noise features and random noise in a single time step are caused by random noise. Therefore, the magnitude of the first variance data can reflect the extent to which the denoised features sampled in the corresponding time step are affected by random noise. It is not difficult to understand that random noise within a certain amplitude can effectively improve the diversity and generalization ability of the diffusion model when generating the image to be identified. However, during the image generation process, if the denoised features sampled by the diffusion model at each time step are excessively affected by random noise, the image generation process will become uncontrolled, resulting in obvious local defects in the image ultimately generated by the diffusion model. Based on this, in this embodiment, by obtaining the first variance data generated in the process of generating the image to be identified using the diffusion model, the probability of image defects in the image to be identified is judged based on the first variance data of multiple time steps.

[0114] Specifically, refer to Figures 6a to 6c ,in Figure 6a The figure shows a comparison of the distribution of the variance of the characteristic elements of each pixel point extracted in the process of generating an image with defects and an image without defects by the diffusion model. Figure 6b The figure shows the distribution of the variance of the characteristic elements of each pixel point extracted in the process of generating a defect-free image using the diffusion model. Figure 6c The figure shows the distribution of the variance of the characteristic elements of each pixel extracted during the process of generating defective images using the diffusion model. Figures 6a to 6c It is not difficult to find that the value distribution of the elements in the first variance data extracted in the process of the diffusion model generating an image with defects is significantly different from the value distribution of the elements in the first variance data extracted in the process of the diffusion model generating an image without defects. Figure 6bAs shown in FIG, for an image without defects, the frequency of occurrence of a significantly larger variance (such as a variance with an absolute value greater than 8) in the corresponding first variance data is relatively low, while as shown in FIG c, for an image with defects, the frequency of occurrence of a larger variance (such as a variance with an absolute value greater than 8) in the corresponding first variance data is relatively high. Therefore, in the present disclosure, in the process of generating an image to be identified by a diffusion model, the variance data between the feature elements in a plurality of denoising features sampled by each pixel point in a single time step can be extracted as a parameter for determining whether there is a defect in the image to be identified.

[0115] In one possible embodiment, again referring to Figure 5 , the corresponding first variance data may be extracted at each time step in the process of the diffusion model generating the image to be recognized. That is, the first variance data is extracted not only at time step t, but also from time step t-1 to time step 0.

[0116] In another possible embodiment, since the first variance data is extracted in each time step, the corresponding amount of calculation will be relatively large, the efficiency of extracting the first variance data will be relatively low, and at the same time, a high computational overhead will be caused to the terminal or server used to execute the image defect recognition method disclosed herein. Based on this, the corresponding first variance data can also be extracted from a specific portion of the time steps according to certain preset rules. Specifically, for a trained diffusion model, the number of time steps executed in the image generation process is fixed, and each time step is respectively provided with a corresponding step number. The step numbers corresponding to multiple time steps are often continuous and decreasing in sequence. The diffusion model will first execute the time step with the largest step number, and after completing the denoising process of the current time step, the step number will be reduced by 1 to execute the next time step, until the step number decreases to 0. Based on this, in a possible implementation, each time step is respectively provided with a corresponding step number, and step S201 includes: obtaining first variance data extracted from a plurality of target time steps in the process of generating an image to be identified by performing a denoising operation on the diffusion model; wherein the target time step is determined from the plurality of time steps by at least one of the following: determining the plurality of time steps as target time steps; or determining the time step whose corresponding step number is not greater than a first preset threshold as the target time step; or determining the target time step from the plurality of time steps according to a preset interval, wherein the difference between the step numbers of two adjacent target time steps is the preset interval.

[0117] Based on this, the target time step from which the first variance data should be extracted is determined according to a preset rule from the multiple time steps required for the diffusion model to be executed. This allows for flexible selection of whether to extract the first variance data at all time steps or only at some time steps, while ensuring that the first variance data extracted from the multiple target time steps still accurately represents the extent to which the diffusion model is affected by random noise during the process of generating the image to be recognized.

[0118] The target time step is the time step for which the corresponding first variance data is to be extracted during the process of the diffusion model generating the image to be recognized. It can be a portion of the multiple time steps executed by the diffusion model when generating the image to be recognized, or it can be all the time steps executed by the diffusion model. In this embodiment, a rule for the target time step can be preset, and the time step whose corresponding step number meets the preset rule is determined as the target time step.

[0119] Specifically, in one possible embodiment, all time steps executed during the diffusion model's generation of the image to be identified can be determined as target time steps. In this case, corresponding first variance data is extracted for each time step. Subsequently, the presence of defects in the image to be identified can be determined based on the first variance data for each time step. This more comprehensively considers the extent to which random noise affects the denoised features sampled at each time step during the generation of the image to be identified, thereby more accurately identifying whether defects exist in the image to be identified.

[0120] In another possible embodiment, when extracting first variance data at each time step, the computational complexity of extracting the first variance data and fusing multiple first variance data into second variance data is relatively large. However, when the diffusion model predicts the corresponding noise feature at each time step, it combines the denoising features from the previous time step to perform the noise prediction. It is not difficult to understand that when predicting the noise feature at a later time step, the feature elements of the pixels whose denoising features were significantly affected by random noise, sampled in the previous time step, are corrected to a certain extent. This allows the first variance data extracted at the later time step to better reflect whether defects will occur in the generated image to be identified compared to the first variance data extracted at the previous time step. When the diffusion model generates an image, the step numbers of the multiple time steps required are sequentially decreasing, i.e., the diffusion model starts with the step with the largest number and then sequentially executes the time steps with smaller step numbers. Based on this, in one possible embodiment, a first preset threshold can be preset, and the time step with a step number less than the first preset threshold is determined as the target time step. In this way, first variance data can be extracted only from multiple time steps executed later. This reduces the computational effort required to extract the first variance data and fuse it into the second variance data, while still ensuring that the extracted first variance data can more accurately reflect the magnitude of the random noise impact on each pixel in the image to be identified, thereby determining whether there are defects in the image to be identified. For example, if the diffusion model needs to execute 50 time steps during image generation and the first preset threshold is 40, only time steps with step numbers no greater than 40 are identified as target time steps, while the 10 time steps with step numbers 41 to 50 are not identified as target time steps. Subsequently, since the first time step executed by the diffusion model corresponds to step number 50, the second time step corresponds to step number 49, and so on, first variance data is subsequently extracted only from the 40 time steps executed later by the diffusion model, eliminating the need to extract first variance data from the first 10 time steps.

[0121] In another possible embodiment, a target time step may be determined from multiple time steps at a preset interval, such that the difference between the step numbers of two adjacent target time steps is the preset interval. In this way, the multiple target time steps for which first variance data is to be extracted are evenly distributed throughout the process of generating the image to be identified using the diffusion model. This allows for better consideration of the extent to which random noise affects the denoised features sampled at each stage of the diffusion model's generation of the image to be identified. This reduces the computational complexity of extracting the first variance data and fusing the first variance data into the second variance data, while still ensuring that the extracted multiple first variance data accurately reflect the extent to which random noise affects each pixel in the image to be identified. For example, if the diffusion model needs to execute 50 time steps during image generation, with a preset interval of 10, the first time step, corresponding to step number 50, is first determined as the target time step. Subsequently, at preset intervals, time steps numbered 40, 30, 20, and 10 are determined as target time steps, while the remaining time steps are not determined as target time steps.

[0122] It can be understood that, in this embodiment, after multiple target time steps are determined, the first variance data is extracted only in each target time step, and for other time steps that are not determined as target time steps, the first variance data will not be extracted in these time steps.

[0123] In one possible implementation, obtaining a plurality of first variance data includes: obtaining noise features predicted by a diffusion model at each time step; for any time step, obtaining a denoising coefficient corresponding to the time step, performing multiple weighted sampling on the noise features and random noise according to the denoising coefficient to obtain a plurality of denoising features, determining a characteristic element variance of each pixel point and a first mean of each pixel point according to the multiple denoising features, where the first mean is the mean of characteristic elements obtained by multiple samplings at the same pixel point; and obtaining the first variance data based on the quotient between the characteristic element variance and the first mean of the same pixel point.

[0124] Based on this, multiple denoising features are obtained by performing multiple samplings based on the predicted noise features and random noise in each time step. Since the variance of multiple feature elements of the same pixel point will be affected by the mean of these feature elements, in this embodiment, the variance and first mean of the feature elements of each pixel point in the multiple denoising features are calculated, and the first variance data is determined based on the quotient of the variance of the feature elements of each pixel point and the first mean, thereby eliminating the problem of inconsistent dimension of the variance caused by the difference in the mean of the same pixel point in a single time step. Similarly, the first mean is the mean between the feature elements obtained by multiple samplings at the same pixel point. Assuming that the sampling is performed 10 times, a denoising feature will be obtained each time, and each denoising feature includes the feature elements of each pixel point. At this time, for any pixel point, there are feature elements of the denoising features sampled 10 times, and the first mean corresponding to this pixel point is the mean between these 10 feature elements.

[0125] The denoising coefficient is a coefficient used to control the sampling ratio of noise features and random noise when weighted sampling is performed based on the predicted noise features and random noise in a single time step. In this embodiment, the denoising coefficient is a coefficient used to weight the random noise when sampling the denoising features; at this point, it is not difficult to understand that the weight coefficient when sampling the noise features can be the difference between a certain value and the denoising coefficient. For example, the denoising coefficient can be recorded as σ. Of course, when sampling the denoising features, in addition to the noise features and random noise, it is often necessary to combine the image features of the diffusion model before denoising at the corresponding time step. Therefore, the sum of the weights sampled in the noise features and random noise is often not 1. In this embodiment, the sum of the weights sampled in the noise features and random noise is recorded as w a , then the sampling ratios of noise characteristics and random noise can be respectively and σ.

[0126] Thus, the noise feature and random noise can be weighted sampled based on the denoising coefficient to obtain the denoised feature of the time step. Specifically, in one possible implementation, the noise feature and random noise are weighted sampled multiple times according to the denoising coefficient to obtain multiple denoised features, including: performing multiple weighted sampling of the noise feature and random noise according to the denoising coefficient to obtain multiple first sampling results of the noise feature and multiple second sampling results of the random noise; obtaining the original image feature predicted at the time step, and performing weighted summation of the original image feature, the first sampling result, and the second sampling result for each weighted sampling to obtain the denoised feature.

[0127] Based on this, in each time step, a weighted sum is performed based on the original image features input into the diffusion model at that time step, the noise features predicted by the diffusion model at that time step, and the random noise, so that the original image features are denoised according to the noise features and the random noise by means of weighted summation to obtain the denoised features of the corresponding time step.

[0128] The original image features are the image features before the diffusion model performs denoising at the corresponding time step. Referring to the relevant descriptions in the above embodiment, it can be seen that the diffusion model generates the image to be identified by gradually denoising a pure noise feature sampled from Gaussian noise. In the process of generating the image to be identified, the diffusion model performs denoising based on the predicted noise features at each time step to obtain the denoised features for that time step. The denoised features sampled at each time step serve as an intermediate result in the process of generating the image to be identified and are used for denoising at the next time step, thereby achieving the generation of the image to be identified using a stepwise denoising method. Based on this, in this embodiment, the original image features of the first time step executed by the diffusion model in the process of generating the image to be identified are a feature tensor composed of pure noise sampled from Gaussian noise. For time steps other than the first time step, the corresponding original image features are the denoised features of the previous time step.

[0129] After obtaining the original image features at each time step, a weighted summation can be performed on the original image features, the first sampling result, and the second sampling result, thereby removing the first sampling result sampled based on the noise feature and the second sampling result sampled based on the random noise from the original image features. Specifically, the weighted sampling process can be referred to the following formula:

[0130]

[0131] Where t is the time step number of the current execution of the diffusion model; x t-1 represents the denoising feature sampled in the time step with step number t; x t is the original image feature of the time step with step number t; is the noise characteristic predicted by the diffusion model at the time step number t; ∈ t is the data sampled from random noise in the time step with step number t; a t and σ t are all preset coefficients corresponding to the time step with step number t, where σ t is the denoising coefficient; a t-1 It is a preset coefficient corresponding to the time step with step number t-1.

[0132] It is worth noting that in this embodiment, multiple denoising features are sampled at each time step in order to calculate the first variance data for each time step. This process does not affect the process of the diffusion model generating the image to be recognized. The output result of the diffusion model at each time step can be one of the multiple denoising features obtained by weighted sampling based on the predicted noise features and random noise in the corresponding time step. For example, in a time step, 10 denoising features can be sampled based on the noise features predicted by the diffusion model and random noise to calculate the first variance data for that time step, and the first sampled denoising feature can be used as the output result of the diffusion model at that time step. The other 9 denoising features are only used to calculate the first variance data and are not used in the process of the diffusion model generating the image to be recognized. In one embodiment, the first variance data may be extracted from the diffusion model at each time step in the process of generating the image to be identified; in another embodiment, the first variance data may also be the diffusion model at each time step in the process of generating the image to be identified, and the predicted noise features are output to an additional neural network branch, which is then calculated by the additional neural network branch based on the noise features output by the diffusion model. This can avoid the impact of additional multiple sampling on the efficiency of the diffusion model in generating the image to be identified.

[0133] It can be understood that after sampling multiple denoising features, since the first variance data is used to characterize the magnitude of the influence of random noise on the denoising features sampled at each pixel point in a single time step, when the same pixel point is affected by the same magnitude of random noise, that is, when the variances of the multiple feature elements corresponding to the multiple denoising features of the pixel point in a single time step are the same, the larger the mean of the multiple feature elements, the smaller the magnitude of the actual influence of random noise on the pixel point. Conversely, the smaller the mean of the multiple feature elements, the larger the magnitude of the actual influence of random noise on the pixel point. Similarly, when the magnitude of the influence of random noise on the same pixel point is the same, the larger the mean of the multiple feature elements in the multiple denoising features of the pixel point in a single time step, the larger the variance of the pixel point in the time step, the smaller the mean of the multiple feature elements, and the smaller the corresponding variance. For example, the amplitude of the influence of random noise on multiple feature elements of a certain pixel point in time step A and the feature elements in time step B is 10%, and the mean of the multiple feature elements of the pixel point in time step A is 100, and the mean of the multiple feature elements in time step B is 10. At this time, the variance of the pixel point in time step A is 10, and the variance in time step B is 1.

[0134] It can be understood that the denoised features at each time step are obtained by weighted sampling of the noise features predicted at that time step and random noise. Random noise itself often follows a Gaussian distribution. In this case, the mean of multiple feature elements of a single pixel at a single time step often depends on the size of the element corresponding to that pixel in the noise feature. However, because the diffusion model predicts different noise features at different time steps, the means of multiple feature elements at the same pixel at different time steps will be different, resulting in inconsistent dimensionality of the variances of the feature elements at the same pixel at different time steps. The first variance data extracted at multiple time steps is subsequently fused into second variance data. If the first variance data is constructed directly based on the absolute size of the variances between the feature elements in the multiple denoised features sampled at each pixel in a single time step, the inconsistent dimensionality of the variances at the same pixel at different time steps will result in a certain degree of distortion in the second variance data obtained by fusion of the first variance data. Based on this, in this embodiment, after performing multiple weighted sampling on the noise features and random noise according to the denoising coefficient to obtain multiple denoising features, not only the variance of the characteristic elements of each pixel point in the multiple denoising features is calculated, but also the first mean of the characteristic elements of each pixel point in the multiple denoising features is calculated. Then, for each pixel point, the quotient of the corresponding variance and the first mean is calculated, thereby eliminating the dimensional difference in the variance caused by the mean difference. In this way, the distortion of the second variance data caused by the dimensional difference in the first variance data at different time steps is avoided, so that the second variance data obtained by fusing the various first variance data can more accurately reflect the amplitude of the influence of random noise on each pixel point in the overall process of generating the image to be identified.

[0135] Specifically, in this embodiment, weighted sampling is performed based on noise features and random noise in a single time step to obtain multiple denoising features. Then, for the multiple denoising features, the mean and variance are calculated based on multiple feature elements with the same position information in the multiple denoising features to obtain the variance and first mean of the multiple feature elements at that position.

[0136] Specifically, when the first variance data is a two-dimensional matrix, each variance in the first variance data can be calculated using the following formula:

[0137]

[0138] in, represents the variance of the i-th row and j-th column in the first variance data at the t-th time step; M represents the number of denoising features sampled at the t-th time step; Represents the feature element in the i-th row and j-th column of the m-th denoised feature sampled at the t-th time step.

[0139] It is understood that directly sampling multiple denoising features at each time step and then calculating the variance and mean between the feature elements of each pixel based on the multiple denoising features is computationally intensive. Therefore, in one embodiment, the variance and mean of the feature elements in the multiple denoising features sampled at each pixel at a single time step can also be determined by estimation. Specifically, in one possible implementation, obtaining multiple first variance data during the process of generating the image to be recognized by the diffusion model by performing a denoising operation includes: obtaining noise features predicted by the diffusion model at each time step; for any time step, obtaining a denoising coefficient corresponding to the time step, setting the denoising coefficient to zero, and performing weighted sampling of the noise feature and random noise based on the zeroed denoising coefficient to obtain a third sampling result; setting the denoising coefficient to a preset value, performing multiple weighted sampling of the noise feature and random noise based on the denoising coefficient set to the preset value to obtain multiple reference features, and determining the difference between the maximum and minimum values of the multiple reference features; and obtaining the first variance data based on the quotient between the difference and the third sampling result.

[0140] Based on this, in a single time step, the denoising coefficient used to adjust the random noise sampling ratio is first set to zero, and the noise feature and random noise are weighted sampled based on the denoising coefficient after being set to zero. When the denoising coefficient is set to zero and weighted sampling is performed based on this, a third sampling result is obtained, thereby estimating the mean of multiple denoised features obtained by performing multiple weighted sampling on the noise feature and random noise in this time step. Thereafter, the denoising coefficient is set to a preset value, and then the noise feature and random noise are weighted sampled based on the denoising coefficient set to the preset value to obtain multiple reference features. Then, according to the multiple reference features, the difference between the maximum and minimum values of the feature elements of each pixel in each reference feature is calculated, thereby estimating the variance of the feature elements in the multiple denoised features sampled by the pixel in this time step.

[0141] It can be understood that, referring to the relevant description of the above embodiment, random noise is noise that obeys a normal distribution with a mean of 0 and a variance of 1. When weighted sampling of noise features and random noise is performed based on the denoising coefficient, the mean of the multiple denoised features sampled is often determined by the noise features. Therefore, in this embodiment, by setting the denoising coefficient to zero, the random noise portion is completely ignored when sampling the denoised features. At this time, the third sampling result sampled from the noise features and random noise based on the denoising coefficient after setting to zero is determined by the noise features predicted at that time step. At this time, each characteristic element in the third sampling result can actually be regarded as the mean of the pixel points corresponding to each characteristic element. In this way, the mean of the characteristic elements in the multiple denoised features sampled by each pixel point in a single time step can be estimated more accurately, while effectively reducing the amount of calculation required to calculate the first variance data. Specifically, the details of sampling the denoised features based on the denoising coefficient after setting to zero can refer to the relevant description of weighted sampling of noise features and random noise based on the denoising coefficient in the above embodiment. The difference is that the denoising coefficient σ in this embodiment t is set to zero.

[0142] In this embodiment, the preset value may be 1. After the denoising coefficient is set to 1, referring to the formula for sampling denoising features in the above embodiment, it can be seen that the proportion of sampling from the noise feature becomes 0, which is equivalent to determining the denoising feature only by combining the original image features with the random noise. Afterwards, multiple weighted sampling is performed from the noise feature and the random noise based on the denoising coefficient set to the preset value to obtain multiple reference features. The details of this process can be referred to the relevant description of weighted sampling of noise features and random noise based on the denoising coefficient in the above embodiment. The difference is that the denoising coefficient σ in this embodiment is t It is set to 1.

[0143] After obtaining multiple reference features, the variance of the feature element of each pixel is determined based on the maximum and minimum values of the multiple reference features. Specifically, for each pixel, the maximum and minimum values of the feature element of the pixel in each reference feature are determined, and the variance of the feature element corresponding to the pixel is determined based on the difference between the two. This process can be referred to as the following formula:

[0144]

[0145] in, Represents the variance of the feature element of the pixel corresponding to the feature element in the i-th row and j-th column of multiple reference features, Represents the maximum value among the feature elements in the i-th row and j-th column of multiple reference features, represents the minimum value among the feature elements in the i-th row and j-th column of the multiple reference features. Therefore, in this embodiment, by calculating the square of the difference between the maximum and minimum values of the feature elements of each pixel in the multiple reference features, the feature element variance of the feature elements in the multiple denoised features sampled by the pixel at a single time step is estimated. This significantly reduces the amount of computation required to calculate the first variance data while ensuring that the calculated feature element variance can reflect the magnitude of random noise affecting the feature element values of the corresponding pixel in the denoised features.

[0146] After estimating the difference corresponding to each pixel point by the above method and obtaining the third sampling result, the quotient of the variance and mean of the characteristic element of each pixel point can be calculated based on the difference of each pixel point and the third sampling result, thereby normalizing the dimension of the variance of the characteristic element of each pixel point to obtain the first variance data after dimension normalization.

[0147] Step S202, fusing multiple first variance data to obtain second variance data;

[0148] The second variance data is determined based on the first variance data at multiple time steps. The second variance data can be used to characterize the extent to which the overall process of generating the image to be recognized by the diffusion model is affected by random noise. When fusing multiple first variance data sets, the variances of characteristic elements of the same pixel in the multiple first variance data sets can be fused.

[0149] It can be understood that the diffusion model generates the image to be recognized by performing gradual denoising over multiple time steps. The denoised features sampled at other time steps, except for the denoised features at the last time step, are actually an intermediate image in the process of generating the image to be recognized. At the same time, in each time step, the diffusion model predicts the noise features of the current time step based on the denoised features sampled at the previous time step, and then removes the noise features predicted at the current time step from the denoised features of the previous time step. In other words, the diffusion model gradually generates the image to be recognized by denoising the denoised features sampled at the previous time step in each time step. It is not difficult to understand that under this image generation mode, even if some feature elements in the denoised features sampled at the first time step are highly affected by random noise, resulting in defects in the intermediate image corresponding to the denoised features sampled at this time step, the diffusion model may gradually correct these feature elements that are highly affected by random noise during noise prediction and denoising in multiple subsequent time steps, thereby avoiding defects in the image to be recognized caused by the denoised features sampled at a single time step. Therefore, it is difficult to determine whether there are defects in the final image to be identified based on the first variance data of a single time step. Based on this, in this embodiment, it is necessary to fuse multiple first variance data extracted from multiple time steps to obtain second variance data that can characterize the magnitude of the impact of random noise on the overall process of generating the image to be identified by the diffusion model, so as to subsequently determine whether there are defects in the image to be identified based on the second variance data.

[0150] In a possible implementation, step S202 includes: determining a second mean value of the variance of the characteristic element of the same pixel point in a plurality of first variance data; and constructing second variance data according to the second mean values of the respective pixel points.

[0151] Based on this, by calculating the mean of the first variance data extracted in each time step by pixel point, the second mean of the variance of the characteristic elements corresponding to each pixel point in multiple time steps is obtained, and the second variance data is constructed based on this. In this way, the second variance data can characterize the amplitude of the random noise influence of each pixel point in the overall process of generating the image to be identified. In this way, whether there are defects in the image to be identified can be more accurately identified based on the second variance data.

[0152] Specifically, in one embodiment, referring to the above embodiment, it can be seen that the first variance data extracted in each time step includes the variance between the feature elements corresponding to each pixel in the image to be identified. It is not difficult to understand that the first variance data extracted in each time step is actually a tensor with the same shape as the denoised feature, and the elements with the same position information in the first variance data extracted in different time steps are respectively used to represent the magnitude of the influence of random noise on the feature elements of the same pixel in different time steps. Exemplarily, in the process of generating the image to be identified, the corresponding first variance data are extracted in 50 time steps. At this time, in the 50 extracted first variance data, the elements in the i-th row and j-th column are the magnitudes of the influence of random noise on the sampling results of the same dimension of the same pixel in these 50 time steps. Based on this, in this embodiment, the elements with the same position information in each first variance data are averaged to obtain the second mean of the variance of each pixel in the different first variance data. At this time, it is not difficult to understand that the second mean can be used to reflect the magnitude of the influence of random noise on the corresponding pixel in the overall process of generating the image to be identified.

[0153] In one embodiment, the second variance data may be calculated using the following formula:

[0154]

[0155] Among them, V i (x) represents the t-th first variance data. It is not difficult to understand that the first variance data V extracted at each time step i (x) is a tensor of shape m*n*k, N is the number of time steps from which the first variance data needs to be extracted, that is, the number of first variance data extracted in the process of generating the image to be recognized, and V(x) represents the second variance data.

[0156] In another possible embodiment, referring to the relevant description above, it can be seen that when the sampling results of the diffusion model in the first executed time step are excessively affected by random noise, a certain degree of correction may be made when the noise prediction is performed in the subsequent executed time step. Therefore, the degree to which the denoised features sampled in the first executed time step are affected by random noise will be corrected to a certain extent when the noise features are predicted in the subsequent time step. However, the denoised features sampled in the subsequent executed time step are more likely to be affected by random noise, which will cause defects in the final generated image to be recognized. Based on this, in this embodiment, a corresponding weight can be preset for each time step for which first variance data needs to be extracted, and then the first variance data extracted in multiple time steps are fused into second variance data through weighted summation. Specifically, since each time step executed by the diffusion model is respectively provided with a corresponding step number, a corresponding weight coefficient can be set for each step number. Since the step numbers of each time step executed by the diffusion model are successively decreasing, in this embodiment, the size of the weight coefficient and the size of the step number are negatively correlated. In this way, it can be ensured that the first variance data extracted in the first executed time step will be fused into the second variance data with a smaller weight coefficient, and the first variance data extracted in the later executed time step will be fused into the second variance data with a larger weight coefficient, so that the second variance data can better characterize the amplitude of the random noise influence of the final generated image to be identified. It is not difficult to understand that the sum of the weight coefficients corresponding to all time steps for extracting the first variance data is 1. After determining the weight coefficient corresponding to each time step, each first variance data can be weighted and summed by position according to the weight coefficient, thereby fusion of the variance of the feature elements of the same pixel point of multiple first variance data to obtain the second variance data. At this time, the second variance data can be expressed by the following formula:

[0157]

[0158] Among them, V t (x) represents the t-th first variance data. It is not difficult to understand that the first variance data V extracted at each time step t (x) is a tensor of shape m*n*k; N is the number of time steps for extracting the first variance data, that is, the number of first variance data extracted in the process of generating the image to be recognized; w i represents the preset weight coefficient corresponding to the time step of extracting the t-th first variance data; V(x) represents the second variance data.

[0159] Of course, in some possible embodiments, in addition to the mean and weighted sum, other statistical representations of multiple first variance data can also be used as the second variance data. However, it should be noted that since the defects that may exist in the image to be identified are often caused by extreme situations that are greatly affected by random noise in a few time steps, when using other statistical representations of multiple first variance data as the second variance data, it is necessary to ensure that the statistical representation of the second variance data can reflect both the overall situation of the multiple first variance data and the influence of some extreme values in the multiple first variance data on the second variance data. For example, the second variance data can also be determined by using a combination of the median and extreme value of the characteristic elements of each pixel point in the multiple first variance data.

[0160] Step S203: performing feature extraction based on the second variance data to obtain image defect features;

[0161] In one embodiment, after determining the second variance data, convolution can be performed on the second variance data to extract features from the second variance data and obtain image defect features. By convolution of the second variance data, features at a higher level of abstraction can be extracted from the second variance data, allowing for more accurate identification of defects in the image to be identified based on the image defect features.

[0162] It is understandable that, considering that the content of some images to be identified may contain areas that need to be blurred or defocused, for example, when the image to be identified needs to present a hazy image style, or the image to be identified to be generated is an image of a rainy and foggy day, or when there are image elements such as gauze in the image to be identified, some details in the image to be identified need to present a certain degree of blur effect, and the pixels in these areas that need to present a blur effect often exhibit a high degree of randomness. This will cause the variance between the multiple feature elements sampled by these pixels in a single time step to be large, making the first variance data and the second variance data determined based on the first variance data overall large, thereby causing the subsequent defect recognition results obtained based on the second variance data to be distorted. Based on this, in a possible implementation, step S202 includes: convolving the second variance data to obtain image blur features; obtaining the image to be identified, and semantically encoding the image to be identified to obtain image semantic features of the image to be identified; feature fusion of the image blur features and the image semantic features to obtain fusion features, and mapping the fusion features to obtain image defect features.

[0163] Based on this, the image blur features are extracted by convolving the second variance data. Semantic encoding is then performed on the image to be identified, extracting its semantic features. The image semantic features and image blur features are then fused, and the fusion results are mapped to image defect features. In this way, the semantic information of the image to be identified is injected into the image defect features. Subsequent defect identification based on the image defect features considers not only the second variance data obtained during the diffusion model generation of the image to be identified, but also the semantics of the image to be identified itself. Image semantic features reflect information such as image elements and image style in the image to be identified. This avoids the situation where the second variance data is too large due to the required blur regions in the image to be identified, leading to the misidentification of some images to be identified that require blur regions as defective.

[0164] The image blur feature is a feature tensor extracted from the second variance data. Specifically, in one embodiment, referring to Figure 7 , the second variance data can be convolved by a fuzzy feature extraction model composed of one or more convolutional layers, thereby extracting a feature tensor of the second variance data as an image fuzzy feature.

[0165] Image semantic features are feature tensors used to represent the semantic information of the generated image to be identified. They can characterize the image content of the image to be identified. For example, they can characterize the image style of the image to be identified, the objects it contains, and the color, size, shape, and other attributes of these objects. By semantically encoding the image to be identified, the semantic information of the image to be identified is extracted. This information can then be combined with the semantic information of the image to be identified for subsequent defect identification, avoiding the misidentification of images that should contain fuzzy areas as images with defects due to the characteristics of the image content itself.

[0166] Specifically, in this embodiment, after the diffusion model completes the denoising of all time steps and generates the image to be recognized, refer to Figure 7 , the image to be identified can be input into a trained semantic encoder, and the semantic encoder is used to semantically encode and normalize the image to be identified to obtain the image semantic features.

[0167] In one possible embodiment, an image encoder trained by a contrastive learning method may be used to semantically encode the image to be recognized. Specifically, a text encoder and an image encoder may be initialized respectively, and then the text encoder and the image encoder are trained using corresponding image-text pairs. These image-text pairs include a set of corresponding images and texts, wherein the text is used to describe the image. Then, the text and image in each image-text pair are encoded using the text encoder and the image encoder to obtain a text feature vector and an image feature vector. Then, the similarity between each text feature vector and each image feature vector is calculated and the predicted matching relationship between the image and the text is predicted based on the similarity. Then, cross entropy calculation is performed based on the predicted matching relationship and the actual matching relationship between the image and the text to obtain a corresponding loss value. Back propagation is performed based on the loss value to optimize the parameters of the text encoder and the parameters of the image encoder, thereby maximizing the similarity between the text feature vector and the image feature vector corresponding to the text and image in the same image-text pair, and obtaining a trained image encoder as a semantic encoder.

[0168] In addition, since the image to be identified is generated by the diffusion model based on the guidance content such as text and audio input by the user, the content in the image to be identified is determined by the guidance content input by the user. In one possible embodiment, the guidance content of the user input diffusion model can also be semantically encoded to obtain image semantic features. It is understandable that at this time, according to the modality of the guidance content input by the user, the guidance content can be encoded using an encoder of the text, audio and other modalities trained by the contrastive learning method to obtain image semantic features. For the details of the training process of these encoders of the text, audio and other modalities, reference can be made to the process of training the image encoder in the above embodiment, which will not be repeated here.

[0169] After obtaining the image blur feature and the image semantic feature, the image blur feature and the image semantic feature can be fused to obtain a fused feature, and then the image defect feature used for subsequent defect recognition is determined based on the fused feature. In this way, when performing defect recognition based on the image defect feature, the extent to which the process of generating the image to be recognized is affected by random noise and the characteristics of the content of the image to be recognized itself can be fully considered to avoid misrecognition caused by the image style or image elements of the image to be recognized itself. Specifically, the image blur feature and the image semantic feature can be spliced, and the splicing result can be used as the fused feature; or, the image blur feature and the image semantic feature can be cross-attention calculated, and the result of the cross-attention calculation can be used as the fused feature, which is not limited in this embodiment. Afterwards, the fused feature can be linearly mapped to map the fused feature into a tensor of a specific shape to obtain the image defect feature. For example, the fused feature is mapped into a vector of a specific length through a fully connected layer, and the vector is used as the image defect feature.

[0170] In one possible implementation, the size of the image blur feature along a preset dimension is at least twice the size of the image semantic feature along the preset dimension, and the image blur feature and the image semantic feature are fused to obtain a fused feature, including: segmenting the image blur feature along the preset dimension to obtain a first blur sub-feature and a second blur sub-feature, wherein the size of the first blur sub-feature along each dimension is consistent with the size of the image semantic feature along each dimension; bitwise multiplying the first blur sub-feature and the image semantic feature to obtain a fused sub-feature; and splicing the fused sub-feature and the second fuzzy sub-feature to obtain a fused feature.

[0171] Based on this, by segmenting the image fuzzy features, and then bitwise multiplying the first fuzzy sub-features obtained by segmentation and the image defect features, the image semantic features and the first fuzzy sub-features are fused to obtain fused sub-features, and then the fused sub-features and the second fuzzy sub-features obtained by segmenting the image fuzzy features are spliced as fused features. In this way, the semantic information and fuzzy features of the image to be identified can be fused, and the original image fuzzy features can be retained to a certain extent. In this way, when performing defect identification subsequently, it is possible to fully consider whether the randomness of the pixel points is caused by the semantics of the image to be identified itself, and it is also possible to avoid ignoring the fact that the randomness of each pixel point itself far exceeds the blur amplitude required by the image when the semantic information of the image to be identified itself represents that the image to be identified contains an image area that needs to be blurred, and thus identifying it as a defect.

[0172] Specifically, in this embodiment, the image blur feature can be obtained by using two or more convolution layers to convolve the second variance data in sequence, wherein the output dimension of the first convolution layer is consistent with the dimension of the image semantic feature, and the subsequent multiple convolution layers can be provided with multiple convolution channels for widening the output of the first convolution layer to twice or more along the preset dimension, thereby obtaining the image blur feature, wherein the preset dimension is one of the multiple tensor dimensions of the image blur feature, and the size of the image blur feature along the preset dimension refers to the size of one of the tensor dimensions of the image blur feature, and the preset dimension can be the dimension that stores the most information (i.e., the largest size) in the image blur feature. Exemplarily, the second variance data can be represented as a 16*16*1024 tensor. It is not difficult to understand that because the second variance data includes the variance corresponding to each pixel in the image to be identified, the image semantic feature obtained by semantic encoding of the image to be identified can also be represented as a tensor with a shape of 16*16*1024; then the second variance data is convolved in sequence through two convolutional layers with a convolution kernel size of 1*1, where the number of channels of the first convolutional layer is 1, and the number of channels of the second convolutional layer is 2. After the second variance data is convolved by the first convolution layer, a tensor with a shape of 16*16*1024 is obtained. Then, the output of the first convolution layer is convolved by the second convolution layer, and the output result of the first convolution layer is widened to twice along the preset dimension to obtain a tensor with a shape of 16*16*2048. Thus, the second variance data is convolved in sequence through two convolution layers to obtain image blur features. At this time, the size of the image blur feature along the lowest dimension is twice the size of the lowest dimension of the image semantic feature. It is not difficult to understand that, in this embodiment, the first blur sub-feature and the second blur sub-feature obtained by segmenting the image blur feature are actually the outputs of different channels of a convolution layer, and both include elements corresponding to each pixel point in the image to be identified, that is, the first blur sub-feature and the second blur sub-feature both contain full data corresponding to all pixel points in the image to be identified.

[0173] After obtaining the image blur feature, the image blur feature is segmented. Since the size of the image blur feature along a specific dimension is at least twice the size of the image semantic feature along the specific dimension, after segmenting the image blur feature along the preset dimension, a first blur sub-feature and a second blur sub-feature can be obtained, wherein the shape of the first blur sub-feature and the shape of the image semantic feature are exactly the same. For example, after sequentially convolving the second variance data through two convolutional layers, the image blur feature obtained is a tensor with a shape of 16*16*2048, and the image semantic feature is a tensor with a shape of 16*16*1024. At this time, the image blur feature can be segmented into two sub-features, both of which have a shape of 16*16*1024, of which the first is the first blur sub-feature and the second is the second blur sub-feature. Of course, in a possible embodiment, the second sub-feature obtained by segmentation can also be used as the first blur sub-feature, and the first sub-feature obtained by segmentation can be used as the second blur sub-feature. That is, in this embodiment, any part of the image blur feature that has the same shape as the image semantic feature can be used as the first blur sub-feature, and the remaining part can be determined as the second blur sub-feature. This depends on the processing paradigm learned by the various convolutional layers used to extract the image blur features and the model used to identify defects based on image defect features during the training process, and is not limited in this embodiment.

[0174] It can be understood that, since the size of the first fuzzy sub-feature along each dimension is the same as the size of the image semantic feature along each dimension, for each tensor element in the image semantic feature, an element with the same position information can be found in the first fuzzy sub-feature. Based on this, in this embodiment, referring to Figure 8 , the image semantic feature and the first fuzzy sub-feature can be fused by bitwise multiplication of the first fuzzy sub-feature and the image semantic feature to obtain a fused sub-feature. Specifically, the fused sub-feature can be determined by referring to the following formula:

[0175]

[0176] in, is the element with coordinates (i, j, k) in the fused sub-feature, is the element with coordinates (i, j, k) in the first fuzzy sub-feature, It is the element with coordinates (i, j, k) in the image semantic feature.

[0177] After obtaining the fusion sub-features, refer to Figure 8, the fusion sub-feature and the second fuzzy sub-feature are spliced together to obtain the fusion feature. Specifically, the fusion sub-feature and the second fuzzy sub-feature can be spliced together according to the positional relationship between the first sub-fuzzy feature and the second sub-fuzzy feature to obtain the fusion feature. In this way, the fusion feature includes both the joint representation part of the second variance data and the semantic information of the image to be identified, and the part of the second variance data that is purely used to represent each pixel. Subsequently, when the image defect feature is determined based on the fusion feature and the defect is identified based on the image defect feature, it can fully consider whether the blurring of some pixels in the image to be identified is caused by the semantic information of the image to be identified itself, and it can also avoid ignoring the fact that the blurring of some pixels themselves far exceeds the blurring amplitude required by the image when the semantic information of the image to be identified itself represents that the image to be identified contains the image area that needs to be blurred, which will still cause defects in the image to be identified.

[0178] In one possible implementation, the fusion sub-feature and the second fuzzy sub-feature are spliced together to obtain a fusion feature, including: globally pooling the fusion sub-feature to obtain a channel semantic feature; mapping the channel semantic feature through a multi-layer perceptron to obtain channel attention weight data; and weighting the second fuzzy sub-feature based on the channel attention weight data and splicing it with the fusion sub-feature to obtain a fusion feature.

[0179] Based on this, we globally pool the fused sub-features and then use a multi-layer perceptron to map the channel semantic features obtained from global pooling to obtain multiple channel attention weights. These weights are then used to weight the second fuzzy sub-features, thereby highlighting defect-sensitive data in the second fuzzy sub-features and suppressing defect-insensitive data in the second fuzzy sub-features. This allows us to more accurately identify defects in the image after the fused features are mapped into image defect features.

[0180] Specifically, it can be understood that the image blur feature is obtained by convolving the second variance data. During the convolution process, different features of the second variance data are actually extracted from multiple different feature dimensions, such as color, texture, and objects in the image. Each feature dimension can be regarded as a feature map channel. It is not difficult to understand that the second blur sub-feature contains the features of each different channel.

[0181] In this embodiment, global average pooling is performed on the fused sub-features to compress the spatial information of each channel into a single statistic, resulting in a channel semantic feature. This channel semantic feature reflects the importance of different channel features to defect identification. For example, if the activation value of the data of a certain channel in the defect area is significantly higher than that in other areas, the pooling result will reflect that the data of this channel is more important to the defect.

[0182] After extracting the semantic features of each channel, the channel semantic features are nonlinearly mapped through the Multilayer Perceptron (MLP). During the training process, the MLP will further learn to generate complex correlations between different channels. After the nonlinear mapping of the MLP, the channel attention weight data can be obtained. The attention weight data further reflects the importance of the data of each channel to the defect recognition results. The channel attention weight data contains the attention weights corresponding to each channel.

[0183] Afterwards, the second fuzzy sub-feature is weighted by the channel attention weight data, that is, the elements of each channel in the second fuzzy sub-feature are multiplied by the attention weight of the corresponding channel. For example, the attention weight of a channel A is w L At this time, the values of each element corresponding to channel A in the second fuzzy sub-feature are respectively combined with the weight w L Multiplying the weighted second fuzzy sub-features, the features of the channels that are more important for defect detection are emphasized, while the features of the channels that are less important are suppressed. The weighted second fuzzy sub-features are then concatenated with the fusion sub-features to form a fused feature. This allows for more accurate identification of defects in the image after subsequent defect feature extraction based on the fused feature.

[0184] In one possible implementation, mapping the fused features to obtain image defect features includes: inputting the fused features into a trained defect feature extraction model, wherein the defect feature extraction model includes at least one first convolutional layer; performing convolution processing on the fused features through the first convolutional layer, and determining the image defect features based on the convolution processing result of the first convolutional layer.

[0185] Based on this, the fusion features are convolved by using the convolution layer in the defect feature extraction model to extract the spatial features composed of each pixel point in the fusion features, and the fusion features with a lower level of abstraction are mapped into defect representations with a higher level of abstraction to determine the image defect features, so that the subsequent identification of whether there are defects in the image to be identified can be more accurately based on the image defect features.

[0186] Specifically, the defect feature extraction model is a trained model that can be used to extract more abstract defect representations from fused features. The defect feature extraction model includes at least one first convolutional layer. The training process of the defect feature extraction model will be described later and is not detailed here.

[0187] By convolving the fused features using the first convolutional layer in the defect feature extraction model, we can further extract features at a higher level of abstraction from the fused features, so that the defect recognition model can subsequently better utilize these high-level features to identify whether there are defects in the image to be identified.

[0188] In one possible implementation, the defect feature extraction model also includes at least one second convolution layer, the convolution kernel size of the first convolution layer is different from the convolution kernel size of each second convolution layer, the fusion feature is convolved through the first convolution layer, and the image defect feature is determined based on the convolution processing result of the first convolution layer, including: convolution processing of the fusion feature through the first convolution layer to obtain a first convolution feature; inputting the first convolution feature into each second convolution layer respectively to obtain a second convolution feature corresponding to each second convolution layer; splicing the first convolution feature and each second convolution feature, pooling the splicing result to obtain a pooled feature vector; and performing feature fusion on the pooled feature vector to obtain an image defect feature.

[0189] Based on this, the first convolutional layer features are convolved separately using multiple second convolutional layers with different convolution kernel sizes. Smaller convolution kernels are used to capture local detail features of the image to be identified from the fused features, while larger convolution kernels are used to extract regional features of a wider range. The convolution features extracted by different convolution kernels are then spliced together, the spliced results are pooled, and the pooled feature vectors obtained after pooling are fused to obtain image defect features. In this way, the image defect features can reflect whether image defects will form in various regions of the image to be identified at different sizes. Subsequently, the presence of defects in the image to be identified can be more accurately identified based on the image defect features.

[0190] Specifically, in this embodiment, the convolution sum size of the first convolution layer can be 1*1, and the convolution kernel sizes of each second convolution layer can be 3*3, 5*5, 7*7, .... It can be understood that the receptive field sizes of the convolution layers formed by convolution kernels of different sizes are different. The different receptive fields indicate that the second convolution features obtained after convolution can be used to characterize the defect characteristics exhibited in different local area sizes. For example, for a 3*3 convolution kernel, during convolution, the local weighted sum of a 3*3 matrix in the first convolution feature is calculated each time. The summation result reflects the magnitude of the overall impact of random noise on the image area composed of 9 pixels in a 3*3 matrix in the image to be identified. For a 5*5 convolution kernel, during convolution, the local weighted sum of a 5*5 matrix in the first convolution feature is calculated each time. The summation result reflects the magnitude of the overall impact of random noise on the image area composed of 25 pixels in a 5*5 matrix in the image to be identified, and so on. The first convolution feature is convolved again using convolution kernels of different sizes to obtain the second convolution feature corresponding to each second convolution layer, so as to better capture the local features of the image to be recognized under different receptive field sizes.

[0191] It is understandable that the first convolutional feature and each second convolutional feature represent local features of the image to be identified at different receptive field sizes. When performing defect identification, the size of the image area occupied by a possible defect in the image to be identified is not fixed. Therefore, these local features at different receptive field sizes need to be considered. Based on this, in this embodiment, the first convolutional feature and each second convolutional feature are spliced together.

[0192] It can be understood that in this embodiment, the first convolution feature and each second convolution feature are two-dimensional or higher-dimensional feature tensors. After the first convolution feature and each second convolution feature are spliced together, a high-dimensional and large-scale tensor will be obtained. This high-dimensional large-scale tensor is not suitable for direct use in subsequent defect identification. Based on this, in this embodiment, after the first convolution feature and each second convolution feature are spliced together, the splicing result is pooled to reduce the dimensionality of the splicing result into a one-dimensional feature vector, which is more suitable for defect identification. For example, referring to Figure 9The defect feature extraction model has two convolutional layers: a first convolutional layer and two second convolutional layers. The first convolutional feature and each second convolutional feature are both 3D tensors with a shape of 16*16*1024. After concatenation, a 3D tensor with a shape of 16*16*3072 is obtained as the concatenated result. The concatenated result is then pooled through a pooling layer with a pooling window size of 16*16*1, resulting in a pooled feature vector of 1*1*3072.

[0193] After obtaining the pooled feature vector, since the length of the pooled feature vector is affected by the number of second convolutional layers in the defect feature extraction model, when the number of second convolutional layers is different, the length of the pooled feature vector obtained by splicing the first convolutional features and each second convolutional feature and pooling is also different, and the model used for defect recognition usually needs to receive a feature vector with a specific shape to perform defect recognition. Based on this, in this embodiment, it is necessary to perform feature fusion processing on the pooled feature vector through a feature fusion layer to fuse the local features extracted under different receptive field sizes so that the pooled feature vector can be converted into an image defect feature with a specific shape. Specifically, the feature fusion layer can be a fully connected layer, and the output dimension size of the fully connected layer is fixed.

[0194] Exemplarily, the output dimension of the fully connected layer is fixed at 1*1*1024, which ensures that the shape of the final generated image defect feature is fixed. Exemplarily, if the defect feature extraction model is set with two second convolutional layers, the first convolutional feature and each second convolutional feature are a three-dimensional tensor with a shape of 16*16*1024. After splicing and pooling them, the shape of the pooled feature vector obtained is 1*1*3072. After mapping through the fully connected layer, an image defect feature with a shape of 1*1*1024 will be output.

[0195] Step S204 , performing defect recognition on the image to be recognized based on the image defect features to obtain a defect recognition result of the image to be recognized.

[0196] Specifically, the defect recognition result is used to characterize whether there are defects in the image to be recognized, and specifically, it can be used to characterize the probability of the presence of defects in the image to be recognized. For example, the defect recognition result can be text such as "defect exists" or "defect does not exist". Of course, in one possible embodiment, the defect recognition result can also be a parameter used to indirectly characterize the presence or absence of defects in the image to be recognized. For example, it can be a defect score, where the defect score can be a score used to indicate the probability of the presence of defects in the image to be recognized. A higher defect score indicates a higher probability of the presence of defects in the image to be recognized. Conversely, a lower defect score indicates a lower probability of the presence of defects in the image to be recognized. Alternatively, the defect score can also be a score used to indicate the image quality of the image to be recognized. A higher defect score indicates a higher quality of the image to be recognized and a lower probability of the presence of defects in the image to be recognized. Conversely, a lower defect score indicates a lower quality of the image to be recognized and a higher probability of the presence of defects in the image to be recognized. The specific physical meaning of the defect score can be set according to actual scenario requirements. It should be noted that, in this embodiment, defects refer to image areas that may exist in the image to be identified that are inconsistent with the image semantics and obviously unreasonable, such as obvious misalignment of objects, or unclear textures of some objects, obviously wrong colors, or obviously wrong outlines of some objects, such as twisted, repeated or missing torsos of organisms, square or conical balls, etc.

[0197] Specifically, in this embodiment, defect recognition can be performed based on image defect features using a trained defect recognition model.

[0198] In one possible implementation, the defect recognition result includes a defect score, and defect recognition is performed on the image to be recognized based on the image defect features to obtain the defect recognition result of the image to be recognized, including: inputting the image defect features into a defect recognition model used to perform defect recognition on the image to be recognized, and determining the similarity between the image defect features and the support vectors associated with each defect evaluation index based on the nonlinear kernel function in the defect recognition model; obtaining the defect sub-score of the image to be recognized under each defect evaluation index based on the similarity conversion, and obtaining the defect score by weighted summing up each defect sub-score.

[0199] Based on this, by calculating the similarity between the image defect features and the support vectors associated with each defect evaluation index, the defect sub-score of the image to be identified under each defect evaluation index is obtained, thereby determining the probability that the image to be identified may have defects under each defect evaluation index; then, the defect sub-scores are summed up to obtain the defect score. In this way, the defect score can represent the comprehensive probability of the existence of defects in the image to be identified under multiple defect evaluation indicators, thereby more comprehensively and accurately identifying whether there are defects in the image to be identified.

[0200] Specifically, the defect evaluation index is a preset index used to determine whether there are defects in the image to be identified. It can be the composition, image texture, color, etc. of the image to be identified. Each defect evaluation index is associated with at least one support vector. These support vectors can be determined in the process of training the defect recognition model. Specifically, in the process of training the defect recognition model, multiple sample images can be first encoded into multiple sample vectors, and in the training process, the sample vectors that particularly meet a certain evaluation result of a preset evaluation index are used to determine the support vectors. In this way, each support vector corresponds to a certain evaluation result of a certain defect evaluation index. For example, a support vector may represent that the composition of the image is reasonable, while another support vector may represent that the texture of the image is clear, and so on.

[0201] A nonlinear kernel function is a preset function used to calculate the similarity between a vector input to the defect recognition model and each support vector. After inputting the image defect features into the defect recognition model, the relationship between image quality and image similarity is often nonlinear. In this embodiment, after inputting the image defect features into the defect recognition model, the nonlinear kernel function is used to calculate the similarity between the image defect features and each support vector. Specifically, in one embodiment, the nonlinear kernel function can be a Gaussian kernel function. The formula for calculating the similarity between the image defect features and each support vector using the nonlinear kernel function is as follows:

[0202]

[0203] Among them, x i , represents the image defect feature, x j represents a support vector, K(x i ,x j ) represents the similarity between the image defect feature and a certain support vector; γ is a preset hyperparameter.

[0204] After determining the similarity between the image defect features and each support vector, the defect sub-score for each defect evaluation metric for the image to be identified can be determined. Different defect evaluation metrics have varying degrees of importance in determining whether a defect exists. For example, the composition of an image to be identified is often more useful than color in determining whether a defect exists. Therefore, the defect sub-score for each defect evaluation metric can be weighted differently. In this embodiment, the defect score for the image to be identified is obtained by taking a weighted sum of the defect sub-scores.

[0205] In one possible implementation, the defect recognition result includes a defect score, and defect recognition is performed on the image to be recognized based on the image defect features to obtain the defect recognition result of the image to be recognized. It may also include: inputting the image defect features into a defect recognition model used to perform defect recognition on the image to be recognized, and classifying based on the image defect features to obtain multiple score categories; obtaining a preset score associated with the score category, and determining the preset score as the defect score.

[0206] Based on this, by using the defect recognition model to perform multi-classification based on the image defect features, the image to be identified is classified into the image of the corresponding score category, and the defect score of the image to be identified is determined based on the preset score associated with the score category to which the image to be identified is assigned. In this way, the defect recognition problem of the image to be identified is converted into a multi-classification problem, reducing the difficulty of defect recognition.

[0207] Specifically, in this embodiment, the defect recognition model is composed of multiple defect recognition sub-models, each corresponding to a preset candidate score category. Each defect recognition sub-model can be a binary classifier, and each binary classifier is used to determine the matching probability between the candidate score categories corresponding to the binary classifier of the image to be recognized. After obtaining the image defect features, the image defect features are simultaneously input into each defect recognition sub-model. The multiple defect recognition sub-models then predict the matching probability between the image to be recognized and the preset candidate score categories. The candidate score category with the highest matching probability is then determined as the score category of the image to be recognized.

[0208] In one possible embodiment, each preset candidate score category can be directly a score. For example, seven candidate score categories can be set, ranging from 1 to 7. Based on the score category assigned to the image to be identified, the associated preset score can be directly determined to obtain a defect score. It is not difficult to understand that the defect score can be used to represent the probability of defects in the image to be identified. For example, in one embodiment, the higher the defect score, the higher the probability of defects in the image to be identified. Of course, in another embodiment, the defect score can also be used to represent the image quality of the image to be identified. In this case, the higher the defect score, the higher the image quality of the image to be identified and the lower the probability of defects in the image to be identified.

[0209] In another embodiment, the preset candidate score categories may also be text labels used to represent the probability of defects in the image to be identified, such as high, medium-high, medium, medium-low, and low. In this case, a corresponding preset score needs to be set for each score category. After determining the score category of the image to be identified, the defect score of the image to be identified is determined based on the mapping relationship between the candidate score categories and the preset scores.

[0210] In one possible implementation, the image fuzzy feature is obtained using a fuzzy feature extraction model, the image defect feature is obtained using a defect feature extraction model, and the defect recognition result is obtained using a defect recognition model. Before obtaining multiple first variance data, the method also includes: obtaining multiple training samples, wherein each training sample includes a sample text and a reference label, wherein the reference label is used to indicate whether there is a defect in the image associated with the sample text; using a diffusion model to perform denoising based on the sample text to obtain a sample image, obtaining sample variance data corresponding to the diffusion model at each time step, and fusing multiple sample variance data; using a fuzzy feature extraction model to extract sample fuzzy features based on the fused multiple sample variance data; using a defect feature extraction model to extract sample defect features based on the sample fuzzy features; using a defect feature extraction model to perform defect recognition based on the sample defect features to obtain a sample recognition result of the sample image; determining a loss value based on the difference between the sample recognition result and the reference label, and optimizing the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model based on the loss value.

[0211] Based on this, the fuzzy feature extraction model, defect feature extraction model and defect recognition model are jointly trained by using training samples. During the training process, the diffusion model is frozen and then the diffusion model is used to generate sample images based on sample texts. Sample variance data is extracted from each time step in the process of generating sample images from the diffusion model, instead of using fixed sample variance data to train the fuzzy feature extraction model, defect feature extraction model and defect recognition model. In this way, the fuzzy feature extraction model, defect feature extraction model and defect recognition model can better learn how to extract image fuzzy features from the second variance data, how to generate and extract image features based on image fuzzy features, and how to identify defects based on image defect features, thereby improving the generalization ability of the fuzzy feature extraction model, defect feature extraction model and defect recognition model.

[0212] The training sample is sample data used to jointly train the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model. Each training sample includes at least sample text and a reference label of the image associated with the sample text. The sample text is a guide text for inputting into the diffusion model for image generation. The sample text describes the content of the image to be generated by the diffusion model. The reference label can also be a defect score representing the image associated with the sample text. In one possible embodiment, referring to Figure 10 , each training sample can also include a label image associated with the sample text, and the reference label is the score of the label image, for example, Figure 10In the example, the sample text of a training sample is "a picture of the ocean with clouds and birds, with a low depth of field effect during the daytime", and the reference label of the corresponding image is 5 points. In addition, in a possible embodiment, the reference label also includes reference sub-labels of multiple dimensions, such as image-text semantic consistency, image quality score, etc. The reference label is determined by integrating multiple reference sub-labels, for example, Figure 10 The image-text alignment score of the image associated with the sample text in the example is 4 (represented by 4 stars), while the image quality score is 6. The reference label is the average of the two, that is, 5. In addition, in a single training sample, the sample text can be associated with multiple ordered images at the same time, and different images can be assigned different reference labels. In this case, when calculating the loss value later, the reference label closest to the predicted sample recognition result can be selected to calculate the loss value.

[0213] In one embodiment, the training sample may be sample data in an open source dataset.

[0214] Afterwards, the sample text is input into the diffusion model, and the diffusion model is used to generate an image based on the sample text. It should be noted that in this embodiment, the diffusion model can be a pre-trained diffusion model. During the training process of this embodiment, the diffusion model parameters are frozen. That is, during training, only the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model are optimized. The details of using the diffusion model to generate a sample image based on the sample text can be found in the process of using the diffusion model to generate an image to be recognized in the above embodiment, and will not be repeated here.

[0215] In this embodiment, the sample variance data of each time step in the process of the diffusion model generating the image to be identified can be obtained respectively. The details of extracting the sample variance data can refer to the method of extracting the first variance data in the process of the diffusion model generating the image to be identified in the above embodiment, and will not be repeated here.

[0216] After extracting the sample variance data of multiple time steps, the mean or weighted sum of the multiple sample variance data is calculated to fuse the multiple sample variance data; then, the fusion result of the multiple sample variance data is convolved using the fuzzy feature extraction model to extract the sample fuzzy features.

[0217] After obtaining the sample fuzzy features, the defect feature extraction model is used to further extract the sample defect features from the sample fuzzy features.

[0218] In another embodiment, the sample image can also be semantically encoded to obtain sample semantic features, and then the sample semantic features and sample fuzzy features are fused to obtain sample fusion features, and the sample fusion features are mapped into a vector with a specific shape as a sample defect feature.

[0219] In another embodiment, the sample fusion features can also be convolved through multiple convolution layers with different convolution kernel sizes, and then the output results of the multiple convolution layers are spliced, and the spliced results are pooled, and then the pooled results are mapped into a vector with a specific shape through a fully connected layer as a sample defect feature.

[0220] After obtaining the sample defect features, the defect recognition model is used to identify the defects based on the sample defect features to obtain a sample recognition result. This sample recognition result can reflect whether there are defects in the sample image. In one embodiment, the sample recognition result can be a score used to represent the probability of the presence of a defect in the sample image. By calculating the difference between the sample recognition result and the corresponding reference label, a corresponding loss value can be obtained. Specifically, in one embodiment, the mean square error between the sample recognition result and the reference label can be calculated using the following formula to obtain the corresponding loss value:

[0221]

[0222] Among them, loss is the loss value, n is the number of training samples, y i represents the reference label of the i-th training sample, Represents the sample recognition result for the i-th training sample. After calculating the loss value, backpropagation is performed based on this loss value to calculate the gradients of each neural network layer in the fuzzy feature extraction model, defect feature extraction model, and defect recognition model. The parameters of the fuzzy feature extraction model, defect feature extraction model, and defect recognition model are then optimized using gradient descent. This continues until a preset training end condition is met, such as when the loss value falls below a preset loss threshold or the number of training times for the fuzzy feature extraction model, defect feature extraction model, and defect recognition model reaches a preset training threshold.

[0223] In one possible implementation, the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model are optimized based on the loss value, including: back propagation based on the loss value and a preset learning rate to optimize the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model; wherein, in the process of training the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model through multiple training samples, the learning rate decays according to a preset decay step size.

[0224] Based on this, in the process of training the fuzzy feature extraction model, defect feature extraction model and defect recognition model, the step size of updating the parameters of the fuzzy feature extraction model, defect feature extraction model and defect recognition model during back propagation is controlled based on the learning rate, and the learning rate when the model updates the parameters is gradually reduced during the training process. In this way, the parameters of the fuzzy feature extraction model, defect feature extraction model and defect recognition model are updated with a high learning rate in the initial stage of training to approach the optimal solution. At the same time, the learning rate is gradually reduced to more finely optimize the parameters of the fuzzy feature extraction model, defect feature extraction model and defect recognition model in the later stage of training, thereby balancing the convergence speed and stability of the training process.

[0225] The learning rate is a coefficient used to control the parameter update stride of the neural network model. It is understandable that when training a neural network model, after calculating the loss value, the gradient of each neural network layer in the model is calculated based on the loss value through the backpropagation algorithm, and then the model parameters are updated by gradient descent. The learning rate is used to control the gradient scaling when updating the parameters. Specifically, after performing backpropagation and calculating the gradients of each neural network layer in the model, the learning rate can be used to control the stride of the model parameter update according to the following formula:

[0226]

[0227] Among them, θ old represents the parameters before the model is updated; θ new represents the updated parameters of the model; η is the learning rate; Represents the gradient of each neural network layer of the model calculated by the back-propagation process.

[0228] It is understandable that the model training process often requires multiple rounds of iterations of the model. In this embodiment, multiple rounds of iterations are also required for the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model using multiple training samples. At this time, the learning rate can be dynamically adjusted during different rounds of iterations to better balance the convergence speed of the model and the stability of the training process. Specifically, in this embodiment, training the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model once using all training samples is set as one round of iteration. Every several rounds of iterations, the learning rate is reduced according to a preset decay step size. For example, the initial learning rate is set to 0.0004, the attenuation step is set to 0.1 times, and the learning rate is attenuated according to the attenuation step every 5 rounds of iteration. At this time, when all samples are used to iterate the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model for the first 5 rounds, the learning rate of 0.0004 is used to control the update step of the model parameters in the back propagation process; in the 6th round of iteration, the learning rate is attenuated to 0.00004. After that, when the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model are iterated for the 6th to 10th rounds, the learning rate of 0.00004 is used to control the update step of the model parameters in the back propagation process, and in the 11th round of iteration, the learning rate is attenuated to 0.000004, and so on, until the training is completed.

[0229] The image defect recognition method proposed in the embodiment of the present disclosure can be used in multiple scenarios. For example, it can be used in the scenario of generating an image. Figure 11 When using the diffusion model for image generation, after the generation system 1101 obtains the text input by the user from the front-end interface 1102, it can simultaneously use multiple diffusion models such as diffusion model 1 to diffusion model N to generate images according to the text input by the user to obtain multiple generated images. At the same time, by using the image defect recognition method disclosed in the present invention in the filtering system 1103, the probability of defects in the images generated by each diffusion model is identified, and based on the probability of defects in each image, part of the images are filtered out, and only images with a low probability of defects are retained. The generated images are sorted from low to high according to the probability of defects and returned to the front-end interface 1102 for display, and the generated images are displayed to the user to improve the user experience.

[0230] For another example, the method of the embodiment of the present disclosure can be applied in the training scenario of the diffusion model, referring to Figure 12In step 1201, sample data is first obtained, and then step 1202 is executed to train the diffusion model. In the process of training the diffusion model through the sample data, step 1203 is executed to identify defects. The image defect recognition method proposed in the present disclosure is used to identify defects in the image generated by the diffusion model during the training process. Then, step 1204 is executed to determine whether to stop training. Specifically, it is determined whether there are defects in the sample images generated by the diffusion model during the training process, and whether the training process of the diffusion model is stable and whether the diffusion model has learned how to generate defect-free images during the training process. The process then determines whether to stop training the diffusion model. If the defect recognition result indicates that training cannot be stopped, step 1205 is executed to determine whether to continue training the diffusion model for the next round. Specifically, the round number of the current round is recorded during each round of diffusion model training. When executing step 1205, the round number of the current round is incremented by one to determine whether the incremented round number reaches a preset round threshold. If the incremented round number does not reach the preset round threshold, the process continues training the diffusion model for the next round, i.e., skipping to step 1202. Otherwise, the process skips to step 1201 to check whether the sample data meets the requirements. If the defect recognition result indicates that training can be stopped, step 1206 is executed to complete the training, obtaining a trained diffusion model to further optimize the diffusion model training process.

[0231] Reference Figure 13 The present disclosure also provides an image defect recognition device 1300, which includes:

[0232] A variance acquisition module 1310 is configured to acquire a plurality of first variance data, the plurality of first variance data being extracted at different time steps of a diffusion model, the diffusion model being configured to generate an image to be recognized over the plurality of time steps, the first variance data being variances between a plurality of denoised features, the plurality of denoised features being obtained by multiple samplings based on random noise in the time steps corresponding to the first variance data;

[0233] A variance fusion module 1320 is configured to fuse multiple first variance data to obtain second variance data;

[0234] A feature extraction module 1330 is configured to perform feature extraction based on the second variance data to obtain image defect features;

[0235] The recognition module 1340 is used to perform defect recognition on the image to be recognized based on the image defect feature, and obtain a defect recognition result of the image to be recognized. The defect recognition result is used to indicate whether there is a defect in the image to be recognized.

[0236] Optionally, the feature extraction module 1330 is specifically configured to:

[0237] Convolve the second variance data to obtain image blur features;

[0238] Obtaining an image to be identified, and performing semantic encoding on the image to be identified to obtain image semantic features of the image to be identified;

[0239] The image blur feature and the image semantic feature are fused to obtain the fused feature, and the fused feature is mapped to obtain the image defect feature.

[0240] Optionally, the size of the image blur feature along the preset dimension is at least twice the size of the image semantic feature along the preset dimension. The feature extraction module 1330 is specifically configured to:

[0241] Segmenting the image fuzzy feature along a preset dimension to obtain a first fuzzy sub-feature and a second fuzzy sub-feature, wherein the size of the first fuzzy sub-feature along each dimension is consistent with the size of the image semantic feature along each dimension;

[0242] Multiply the first fuzzy sub-feature and the image semantic feature bit by bit to obtain the fused sub-feature;

[0243] The fusion sub-feature and the second fuzzy sub-feature are concatenated to obtain the fusion feature.

[0244] Optionally, the feature extraction module 1330 is specifically configured to:

[0245] Perform global pooling on the fused sub-features to obtain channel semantic features;

[0246] The channel semantic features are mapped through a multi-layer perceptron to obtain the channel attention weight data;

[0247] The second fuzzy sub-feature is weighted based on the channel attention weight data and then concatenated with the fusion sub-feature to obtain the fusion feature.

[0248] Optionally, the feature extraction module 1330 is specifically configured to:

[0249] Inputting the fused features into a trained defect feature extraction model, wherein the defect feature extraction model includes at least one first convolutional layer;

[0250] The fused features are convolved through the first convolutional layer, and the image defect features are determined based on the convolution processing results of the first convolutional layer.

[0251] Optionally, the defect feature extraction model further includes at least one second convolution layer, and the convolution kernel size of the first convolution layer is different from the convolution kernel size of each second convolution layer. The feature extraction module 1330 is specifically configured to:

[0252] The fused features are convolved through the first convolution layer to obtain the first convolution features;

[0253] Input the first convolution features into each second convolution layer respectively to obtain the second convolution features corresponding to each second convolution layer;

[0254] Concatenate the first convolution feature and each second convolution feature, and pool the concatenated result to obtain a pooled feature vector;

[0255] The pooled feature vector is fused to obtain the image defect features.

[0256] Optionally, the image fuzzy features are obtained using a fuzzy feature extraction model, the image defect features are obtained using a defect feature extraction model, and the defect recognition result is obtained using a defect recognition model. The device further includes a training module, which is specifically used to:

[0257] Acquire multiple training samples, wherein each training sample includes a sample text and a reference label, wherein the reference label is used to indicate whether there is a defect in an image associated with the sample text;

[0258] Use the diffusion model to denoise the sample text to obtain a sample image, obtain the sample variance data corresponding to the diffusion model at each time step, and fuse multiple sample variance data;

[0259] The fuzzy feature extraction model is used to extract the sample fuzzy features based on the fused multiple sample variance data;

[0260] The defect feature extraction model is used to extract the sample defect features based on the sample fuzzy features;

[0261] Defect recognition is performed based on sample defect features using a defect recognition model to obtain a sample recognition result of the sample image;

[0262] A loss value is determined based on the difference between the sample recognition result and the reference label, and the parameters of the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model are optimized based on the loss value.

[0263] Optionally, the training module is specifically used to:

[0264] Backpropagation is performed based on the loss value and the preset learning rate to optimize the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model;

[0265] In the process of training the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model through multiple training samples, the learning rate decays according to a preset decay step size.

[0266] Optionally, the variance acquisition module 1310 is specifically configured to:

[0267] Obtain the noise characteristics predicted by the diffusion model at each time step;

[0268] For any time step, obtain a denoising coefficient corresponding to the time step, perform multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple denoising features, determine the characteristic element variance of each pixel point and the first mean value of each pixel point based on the multiple denoising features, the denoising feature includes the characteristic element of each pixel point, the characteristic element variance is the variance between the characteristic elements obtained by multiple samplings at the same pixel point, and the first mean value is the mean between the characteristic elements obtained by multiple samplings at the same pixel point;

[0269] First variance data is obtained based on a quotient between the variance of the characteristic element and the first mean value of the same pixel point.

[0270] Optionally, the variance acquisition module 1310 is specifically configured to:

[0271] Performing multiple weighted sampling on the noise feature and the random noise according to the denoising coefficient to obtain multiple first sampling results of the noise feature and multiple second sampling results of the random noise;

[0272] The original image features predicted at the time step are obtained, and for each weighted sampling, the original image features, the first sampling result, and the second sampling result are weightedly summed to obtain the denoised features.

[0273] Optionally, the variance acquisition module 1310 is specifically configured to:

[0274] Obtain the noise characteristics predicted by the diffusion model at each time step;

[0275] For any time step, obtain the denoising coefficient corresponding to the time step, set the denoising coefficient to zero, perform weighted sampling on the noise feature and random noise according to the denoising coefficient after setting to zero, and obtain a third sampling result;

[0276] Setting a denoising coefficient to a preset value, performing multiple weighted sampling on the noise feature and the random noise according to the denoising coefficient after being set to the preset value to obtain multiple reference features, and determining the difference between the maximum value and the minimum value of the multiple reference features;

[0277] Based on a quotient between the difference value and the third sampling result, first variance data is obtained.

[0278] Optionally, the variance fusion module 1320 is specifically configured to:

[0279] Determining a second mean of the variance of the characteristic element of the same pixel point in the plurality of first variance data;

[0280] The second variance data is constructed based on the second mean value of each pixel point.

[0281] Optionally, the defect identification result includes a defect score, and the identification module 1340 is specifically configured to:

[0282] The image defect features are input into a defect recognition model for defect recognition of the image to be recognized, and the similarity between the image defect features and the support vectors associated with each defect evaluation index is determined based on a nonlinear kernel function in the defect recognition model;

[0283] The defect sub-scores of the image to be identified under each defect evaluation index are obtained according to the similarity conversion, and the defect score is obtained by weighted summation of each defect sub-score.

[0284] Optionally, the defect identification result includes a defect score, and the identification module 1340 is specifically configured to:

[0285] Inputting the image defect features into a defect recognition model for defect recognition of the image to be recognized, and performing classification based on the image defect features to obtain multiple score categories;

[0286] A preset score associated with the score category is obtained, and the preset score is determined as the defect score.

[0287] Optionally, the variance acquisition module 1310 is used to:

[0288] Acquire first variance data extracted for a plurality of target time steps during a process in which the diffusion model generates an image to be recognized by performing a denoising operation; wherein the target time step is determined from the plurality of time steps by at least one of the following:

[0289] Determine multiple time steps as target time steps;

[0290] Alternatively, a time step whose corresponding step number is not greater than a first preset threshold is determined as the target time step;

[0291] Alternatively, the target time step is determined from a plurality of time steps according to a preset interval, wherein the difference between the step numbers of two adjacent target time steps is the preset interval.

[0292] The electronic device for executing the above-mentioned image defect recognition method provided by the embodiment of the present disclosure may be a terminal, referring to Figure 14 , Figure 14 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure. The terminal includes: a camera assembly 1410, a first memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a first processor 1480, and a first power supply 1490. Those skilled in the art will understand that Figure 14 The terminal structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0293] The camera assembly 1410 can be used to capture images or videos. Optionally, the camera assembly 1410 includes a front camera and a rear camera. Typically, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0294] The first memory 1420 may be used to store software programs and modules. The first processor 1480 executes the software programs and modules stored in the first memory 1420 to execute various functional applications and data processing of the terminal.

[0295] The input unit 1430 may be configured to receive input digital or character information and generate key signal input related to terminal settings and function control. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432 .

[0296] The display unit 1440 may be configured to display input information or provided information and various menus of the terminal. The display unit 1440 may include a display panel 1441 .

[0297] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 may provide an audio interface.

[0298] The first power source 1490 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0299] The number of sensors 1450 can be one or more, and the one or more sensors 1450 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0300] The accelerometer can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the accelerometer can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1480 can control the display unit 1440 to display the user interface in a horizontal or vertical view based on the gravity acceleration signal collected by the accelerometer. The accelerometer can also be used to collect game or user motion data.

[0301] The gyroscope sensor can detect the terminal's body orientation and rotation angle. It can also work with the accelerometer to collect 3D motions of the user on the terminal. Based on the data collected by the gyroscope sensor, the first processor 1480 can implement the following functions: motion sensing (such as changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0302] The pressure sensor can be set on the side frame of the terminal and / or the lower layer of the display unit 1440. When the pressure sensor is set on the side frame of the terminal, it can detect the user's grip signal of the terminal, and the first processor 1480 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor. When the pressure sensor is set on the lower layer of the display unit 1440, the first processor 1480 controls the operability controls on the UI interface based on the user's pressure operation on the display unit 1440. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0303] The optical sensor is used to collect ambient light intensity. In one embodiment, the first processor 1480 can control the display brightness of the display unit 1540 based on the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1440 is increased; when the ambient light intensity is low, the display brightness of the display unit 1440 is decreased. In another embodiment, the first processor 1480 can also dynamically adjust the shooting parameters of the camera assembly 1410 based on the ambient light intensity collected by the optical sensor.

[0304] In this embodiment, the first processor 1480 included in the terminal can execute the image defect recognition method of the previous embodiment.

[0305] The electronic device for executing the above-mentioned image defect recognition method provided by the embodiment of the present disclosure may also be a server, referring to Figure 15 , Figure 15 This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1510 and a second memory 1530, and one or more storage media 1540 (for example, one or more massive storage devices) storing application programs 1543 or data 1542. Among them, the second memory 1530 and the storage medium 1540 can be short-term storage or persistent storage. The program stored in the storage medium 1540 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1510 can be configured to communicate with the storage medium 1540 to execute a series of instruction operations in the storage medium 1540 on the server.

[0306] The server may also include one or more second power supplies 1520, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1560, and / or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0307] The second processor 1510 in the server may be configured to execute the image defect recognition method.

[0308] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program. The computer program is used to execute the image defect recognition method of each of the aforementioned embodiments.

[0309] The present disclosure also provides a computer program product, including a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to implement the above-described image defect recognition method.

[0310] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe embodiments of the present disclosure, e.g., capable of being implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0311] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0312] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0313] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0314] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0315] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0316] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0317] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0318] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above implementation. Technical personnel familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for identifying image defects, characterized in that: The method comprises: Acquiring a plurality of first variance data, the plurality of first variance data being extracted at different time steps of a diffusion model, the diffusion model being configured to generate an image to be recognized over the plurality of the time steps, the first variance data being variances between a plurality of denoising features, the plurality of denoising features being obtained by performing multiple samplings based on random noise in the time steps corresponding to the first variance data; Fusing the plurality of first variance data to obtain second variance data; Perform feature extraction based on the second variance data to obtain image defect features; Defect recognition is performed on the image to be recognized based on the image defect feature to obtain a defect recognition result of the image to be recognized, and the defect recognition result is used to indicate whether there is a defect in the image to be recognized.

2. The image defect recognition method according to claim 1, characterized in that: The extracting features based on the second variance data to obtain image defect features includes: performing convolution on the second variance data to obtain image blur features; Acquire the image to be identified, and perform semantic coding on the image to be identified to obtain image semantic features of the image to be identified; The image blur feature and the image semantic feature are fused to obtain a fused feature, and the fused feature is mapped to obtain an image defect feature.

3. The image defect recognition method according to claim 2, characterized in that: The size of the image blur feature along the preset dimension is at least twice the size of the image semantic feature along the preset dimension, and fusing the image blur feature and the image semantic feature to obtain the fused feature includes: Segmenting the image fuzzy feature along the preset dimension to obtain a first fuzzy sub-feature and a second fuzzy sub-feature, wherein the size of the first fuzzy sub-feature along each dimension is consistent with the size of the image semantic feature along each dimension; Multiplying the first fuzzy sub-feature and the image semantic feature bit by bit to obtain a fused sub-feature; The fusion sub-feature and the second fuzzy sub-feature are concatenated to obtain a fusion feature.

4. The image defect recognition method according to claim 3, characterized in that: The step of splicing the fusion sub-feature and the second fuzzy sub-feature to obtain a fusion feature includes: Performing global pooling on the fused sub-features to obtain channel semantic features; Mapping the channel semantic features through a multi-layer perceptron to obtain channel attention weight data; The second fuzzy sub-feature is weighted based on the channel attention weight data and then spliced with the fusion sub-feature to obtain a fusion feature.

5. The image defect recognition method according to claim 2, wherein: Mapping the fused features to obtain image defect features includes: Inputting the fused features into a trained defect feature extraction model, wherein the defect feature extraction model includes at least one first convolutional layer; The fused features are convolved by the first convolution layer, and the image defect features are determined based on the convolution processing result of the first convolution layer.

6. The image defect recognition method according to claim 5, characterized in that: The defect feature extraction model further includes at least one second convolution layer, the convolution kernel size of the first convolution layer is different from the convolution kernel size of each second convolution layer, and the convolution processing of the fusion feature by the first convolution layer is performed, and the image defect feature is determined based on the convolution processing result of the first convolution layer, including: Performing convolution processing on the fused features through the first convolution layer to obtain a first convolution feature; Inputting the first convolutional features into each of the second convolutional layers respectively to obtain second convolutional features corresponding to each of the second convolutional layers; Concatenate the first convolutional features and each of the second convolutional features, and pool the concatenated results to obtain a pooled feature vector; Perform feature fusion on the pooled feature vector to obtain the image defect feature.

7. The image defect recognition method according to any one of claims 2 to 6, characterized in that: The image fuzzy feature is obtained using a fuzzy feature extraction model, the image defect feature is obtained using a defect feature extraction model, and the defect recognition result is obtained using a defect recognition model. Before obtaining the plurality of first variance data, the method further includes: Acquire a plurality of training samples, wherein each of the training samples includes a sample text and a reference label, wherein the reference label is used to indicate whether there is a defect in an image associated with the sample text; Denoising the sample text using the diffusion model to obtain a sample image, obtaining sample variance data corresponding to each of the time steps of the diffusion model, and fusing multiple sample variance data; Extracting sample fuzzy features based on the fused plurality of sample variance data using the fuzzy feature extraction model; Extracting sample defect features based on the sample fuzzy features using the defect feature extraction model; Using the defect recognition model to perform defect recognition based on the sample defect features, to obtain a sample recognition result of the sample image; A loss value is determined based on a difference between the sample recognition result and the reference label, and parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model are optimized based on the loss value.

8. The image defect recognition method according to claim 7, characterized in that: The optimizing the parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model based on the loss value includes: Performing back propagation based on the loss value and a preset learning rate to optimize parameters of the fuzzy feature extraction model, the defect feature extraction model, and the defect recognition model; In the process of training the fuzzy feature extraction model, the defect feature extraction model and the defect recognition model through a plurality of the training samples, the learning rate decays according to a preset decay step size.

9. The image defect recognition method according to any one of claims 1 to 8, characterized in that: The obtaining of a plurality of first variance data includes: Obtaining noise characteristics predicted by the diffusion model at each of the time steps; For any of the time steps, obtaining a denoising coefficient corresponding to the time step, performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple denoising features, and determining a characteristic element variance of each pixel point and a first mean value of each pixel point based on the multiple denoising features, wherein the denoising feature includes a characteristic element of each pixel point, the characteristic element variance is the variance between the characteristic elements obtained by multiple sampling at the same pixel point, and the first mean value is the mean value between the characteristic elements obtained by multiple sampling at the same pixel point; First variance data is obtained based on a quotient between the feature element variance and the first mean value of the same pixel point.

10. The image defect recognition method according to claim 9, characterized in that: The step of performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple denoising features includes: Performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient to obtain multiple first sampling results of the noise feature and multiple second sampling results of the random noise; The original image feature predicted at the time step is obtained, and for each weighted sampling, the original image feature, the first sampling result, and the second sampling result are weightedly summed to obtain the denoising feature.

11. The generated image defect recognition method according to any one of claims 1 to 8, characterized in that: The obtaining of a plurality of first variance data includes: Obtaining noise characteristics predicted by the diffusion model at each of the time steps; For any of the time steps, obtaining a denoising coefficient corresponding to the time step, setting the denoising coefficient to zero, and performing weighted sampling on the noise feature and random noise according to the denoising coefficient after being set to zero to obtain a third sampling result; Setting the denoising coefficient to a preset value, performing multiple weighted sampling on the noise feature and random noise according to the denoising coefficient after being set to the preset value to obtain multiple reference features, and determining the difference between the maximum value and the minimum value of the multiple reference features; First variance data is obtained based on a quotient between the difference value and the third sampling result.

12. The image defect recognition method according to any one of claims 1 to 11, characterized in that: The fusing of the plurality of first variance data to obtain second variance data includes: Determine a second mean value of the variance of the characteristic element of the same pixel point in the plurality of first variance data; Second variance data is constructed based on the second mean value of each pixel point.

13. The image defect recognition method according to any one of claims 1 to 12, characterized in that: The defect recognition result includes a defect score. The defect recognition of the image to be recognized based on the image defect feature is performed to obtain the defect recognition result of the image to be recognized, including: Inputting the image defect feature into a defect recognition model for performing defect recognition on the image to be recognized, and determining the similarity between the image defect feature and the support vector associated with each defect evaluation index based on a nonlinear kernel function in the defect recognition model; The defect sub-scores of the image to be identified under each of the defect evaluation indicators are obtained according to the similarity conversion, and the defect score is obtained by weighted summation of each of the defect sub-scores.

14. The image defect recognition method according to any one of claims 1 to 12, characterized in that: The defect recognition result includes a defect score. The defect recognition of the image to be recognized based on the image defect feature is performed to obtain the defect recognition result of the image to be recognized, including: Inputting the image defect features into a defect recognition model for performing defect recognition on the image to be recognized, and performing classification based on the image defect features to obtain a plurality of score categories; A preset score associated with the score category is obtained, and the preset score is determined as the defect score.

15. An image defect recognition device, characterized in that: The device comprises: a first variance acquisition module, configured to acquire a plurality of first variance data, the plurality of first variance data being extracted in different time steps of a diffusion model, the diffusion model being configured to generate an image to be recognized over the plurality of time steps, the first variance data being variances between a plurality of denoising features, the plurality of denoising features being obtained by multiple samplings based on random noise in the time steps corresponding to the first variance data; A second variance acquisition module, configured to fuse a plurality of the first variance data to obtain second variance data; a feature extraction module, configured to perform feature extraction based on the second variance data to obtain image defect features; The recognition module is used to perform defect recognition on the image to be recognized based on the image defect feature to obtain a defect recognition result of the image to be recognized, wherein the defect recognition result is used to indicate whether there is a defect in the image to be recognized.

16. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the image defect recognition method according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image defect recognition method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image defect recognition method according to any one of claims 1 to 14 is implemented.

Images