Method and apparatus for processing image, device, storage medium, and computer program product

Abstract

The present application relates to the technical field of artificial intelligence, and discloses a method and apparatus for processing an image, a device, a storage medium, and a computer program product. The method comprises: performing feature extraction on an image to obtain an extracted image feature; encoding and decoding the extracted image feature to obtain a reconstructed image feature; inputting a difference value between the reconstructed image feature and the extracted image feature into a discriminator to obtain first recognition information outputted by the discriminator, the first recognition information comprising first information elements, and the first information elements being used for indicating the probability that a corresponding pixel is anomalous; and on the basis of the extracted image feature, the reconstructed image feature, and the first recognition information, determining an anomaly recognition result of the image. According to the described solution, anomaly conditions of the pixels in an image are comprehensively determined by combining image features before and after reconstruction along with a discrimination result determined on the basis of the difference between the image features before and after reconstruction, so that pixel-level image anomaly detection can be achieved.

Description

Image processing method, device, equipment, storage medium and computer program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 8, 2023, with application number 2023116921640 and application name “Image processing method, device, equipment, storage medium and computer program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to image processing technology. Background Art

[0003] Image anomaly detection is a technology that detects whether there are abnormal or defective pixels in an image. Image anomaly detection can be achieved through artificial intelligence.

[0004] In related technologies, an unsupervised learning approach can be used in advance to train a machine learning model based on normal images, so that the machine learning model can produce a larger reconstruction error on abnormal images, thereby realizing abnormality detection.

[0005] However, in the above-mentioned related technologies, the machine learning model trained based on unsupervised learning has low accuracy in detecting image anomalies.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image processing method, apparatus, device, storage medium, and computer program product that can improve the accuracy of image anomaly detection. The technical solution is as follows:

[0008] In one aspect, an image processing method is provided, which is executed by a computer device, and the method comprises:

[0009] Performing feature extraction on the image to obtain image extraction features of the image;

[0010] Encoding and decoding the image extraction features to obtain image reconstruction features;

[0011] Inputting the difference between the image reconstructed feature and the image extracted feature into a discriminator to obtain first identification information output by the discriminator, wherein the first identification information includes a first information element corresponding to each pixel in the image, and the first information element is used to indicate the probability that the corresponding pixel is abnormal; the discriminator is a machine learning model trained based on synthetic image samples, and the synthetic image samples are image samples with abnormalities synthesized from image samples without abnormalities;

[0012] An abnormality recognition result of the image is determined based on the image extraction feature, the image reconstruction feature and the first recognition information.

[0013] In one aspect, an image processing method is provided, the method comprising:

[0014] Based on a first image sample, first mask information and a second image sample are obtained; the first mask information is used to indicate pixels without abnormalities in the first image sample; the second image sample is an image sample synthesized based on the first image sample and the second mask information, and the second mask information is used to indicate pixels with abnormalities that need to be added to the first image sample;

[0015] Inputting the first image sample into a feature extractor in an image anomaly detection model to obtain a first image extracted feature sample output by the feature extractor; inputting the first image extracted feature sample into a codec in the image anomaly detection model to obtain a first image reconstructed feature sample output by the codec; inputting a difference between the first image reconstructed feature sample and the first image extracted feature sample into a discriminator in the image anomaly detection model to obtain a first identification information sample output by the discriminator;

[0016] Inputting the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; inputting the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; inputting the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator;

[0017] determining a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample;

[0018] Updating the parameters of the image anomaly detection model using the loss function value;

[0019] Wherein, the image anomaly detection model is used to implement the image processing method in the first aspect above.

[0020] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0021] A feature extraction module, used to extract features from an image and obtain image extraction features of the image;

[0022] A coding and decoding module, configured to encode and decode the image extraction features to obtain image reconstruction features;

[0023] a reconstruction module, configured to input a difference between the image reconstructed feature and the image extracted feature into a discriminator to obtain first identification information output by the discriminator, wherein the first identification information includes a first information element corresponding to each pixel in the image, the first information element being used to indicate a probability that an abnormality exists in the corresponding pixel; the discriminator is a machine learning model trained based on synthetic image samples, wherein the synthetic image samples are image samples containing abnormalities synthesized from image samples without abnormalities;

[0024] A recognition module is used to determine an abnormality recognition result of the image based on the image extraction feature, the image reconstruction feature and the first recognition information.

[0025] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0026] a sample processing module configured to obtain first mask information and a second image sample based on a first image sample; the first mask information being used to indicate pixels without abnormalities in the first image sample; and the second image sample being an image sample synthesized based on the first image sample and the second mask information, and the second mask information being used to indicate pixels with abnormalities that need to be added to the first image sample;

[0027] a model processing module, configured to input the first image sample into a feature extractor in an image anomaly detection model to obtain a first image extracted feature sample output by the feature extractor; input the first image extracted feature sample into a codec in the image anomaly detection model to obtain a first image reconstructed feature sample output by the codec; input the difference between the first image reconstructed feature sample and the first image extracted feature sample into a discriminator in the image anomaly detection model to obtain a first identification information sample output by the discriminator; input the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; input the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; input the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator;

[0028] a loss acquisition module, configured to determine a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample;

[0029] A parameter updating module, configured to update the parameters of the image anomaly detection model using the loss function value;

[0030] Wherein, the image anomaly detection model is used to implement the image processing method in the first aspect above.

[0031] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image processing method as described in the above-mentioned embodiment of the present application.

[0032] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image processing method as described in the above-mentioned embodiment of the present application.

[0033] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described in the above embodiments.

[0034] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0035] The solution shown in the embodiment of the present application extracts image extraction features from the image, reconstructs the image extraction features through encoding and decoding to obtain image reconstruction features, and further uses a discriminator to perform abnormality discrimination based on the difference between the image reconstruction features and the image extraction features to obtain first recognition information indicating the probability that an abnormality exists in each pixel in the image. Finally, the image features before and after reconstruction and the first recognition information are combined to obtain a comprehensive abnormality recognition result of the image. In addition to using the image features before and after reconstruction to identify the abnormalities of pixels in the image, the above-mentioned scheme also uses a discriminator to discriminate the abnormalities of pixels in the image based on the difference between the image features before and after reconstruction. On the one hand, by combining the image features before and after reconstruction and the discrimination results determined based on the difference between the image features before and after reconstruction, the abnormalities of each pixel in the image are comprehensively determined, which can achieve pixel-level image anomaly detection while ensuring the accuracy of image anomaly detection. On the other hand, the above-mentioned discriminator is trained using synthetic image samples synthesized based on normal image samples. Therefore, the training process of the discriminator does not require manual collection and labeling of abnormal images, and can be trained through unsupervised learning, thereby expanding the number of samples available for discriminator training, improving the effect of training the discriminator through synthetic image samples, narrowing the difference between the effect of training the discriminator through real abnormal images, ensuring the accuracy of the discriminator, and then ensuring the accuracy of abnormality detection of pixels in the image through the discriminator. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] FIG1 is a system configuration diagram of an image processing system involved in the present application;

[0038] FIG2 is a schematic flow chart showing an image processing method according to an exemplary embodiment;

[0039] FIG3 is a schematic flow chart showing an image processing method according to an exemplary embodiment;

[0040] FIG4 is a schematic flow chart showing an image processing method according to an exemplary embodiment;

[0041] FIG5 is a schematic flow chart showing an image processing method according to an exemplary embodiment;

[0042] FIG6 is a diagram of the unsupervised + discriminant reconstruction network architecture involved in this application;

[0043] FIG7 is a comparison diagram of test results involved in this application;

[0044] FIG8 is a schematic diagram of anomaly identification comparison involved in this application;

[0045] FIG9 is a structural block diagram of an image processing device provided by an exemplary embodiment of the present application;

[0046] FIG10 is a structural block diagram of an image processing device provided by an exemplary embodiment of the present application;

[0047] FIG11 is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0049] Please refer to Figure 1, which shows a system structure diagram of an image processing system involved in various embodiments of the present application. As shown in Figure 1, the system includes an image acquisition device 120, a terminal 140, and a server 160; optionally, the system may also include a database 180.

[0050] The image acquisition device 120 may be a device for acquiring images and may have a built-in or external camera component.

[0051] The image acquisition device 120 may include an image output interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, or an Ethernet interface, etc.; or, the above-mentioned image output interface may also be a wireless interface, such as a Wireless Local Area Network (WLAN) interface, a Bluetooth interface, etc.

[0052] Correspondingly, depending on the type of the above-mentioned image output interface, there may be multiple ways to export the image. For example, the image may be exported to the terminal 140 via a wired or short-distance wireless method, or the image may be exported to the terminal 140 or the server 160 via a local area network or the Internet.

[0053] The terminal 140 may be a terminal device with certain processing capabilities and interface display functions. For example, the terminal 140 may be a mobile phone, a tablet computer, an e-book reader, smart glasses, a laptop computer, a desktop computer, and the like.

[0054] The terminal 140 may be a terminal used by a user or a terminal used by a developer.

[0055] When the terminal 140 is a terminal used by developers, the developers can develop a machine learning model for image processing through the terminal 140 and deploy the machine learning model to the server 160 or the terminal used by the user.

[0056] When the terminal 140 is a terminal used by the user, an application program for acquiring images, performing abnormality recognition processing on the images, and presenting recognition results may be installed in the terminal 140. The application program may have built-in or call the above-mentioned machine learning model for image processing. After the terminal 140 acquires the image captured by the image acquisition device 120, it may acquire the image through the above-mentioned application program, perform abnormality recognition processing on the image, obtain the corresponding abnormality recognition result, and present the abnormality recognition result for the user's reference.

[0057] In the system shown in Figure 1, terminal 140 and image acquisition device 120 are physically separate devices. Alternatively, in another possible implementation, when terminal 140 is a user terminal, terminal 140 and image acquisition device 120 may be integrated into a single device; for example, terminal 140 may be a smartphone with a built-in camera.

[0058] Among them, server 160 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0059] Among them, the above-mentioned server 160 can be a server that provides background services for the application installed in the terminal 140. The background server can be used for version management of the application, background processing of images obtained by the application and returning processing results, background training of machine learning models developed by developers, etc.

[0060] The database 180 may be a Redis database, or may be another type of database. The database 180 is used to store various types of data.

[0061] Optionally, the terminal 140 and the server 160 may be connected via a communication network. Optionally, the image acquisition device 120 and the server 160 may also be connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0062] Optionally, the system may further include a management device (not shown in FIG1 ), which is connected to the server 160 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0063] FIG2 is a flow chart illustrating an image processing method according to an exemplary embodiment. The method can be performed by a computer device, for example, a server, or a terminal, or a combination of a server and a terminal. The server can be server 160 in the embodiment shown in FIG1 , and the terminal can be terminal 140 used by a user in the embodiment shown in FIG1 . The computer device can be implemented as a model application device for image anomaly identification. As shown in FIG2 , the image processing method can include the following steps.

[0064] Step 210: Perform feature extraction on the image to obtain image extraction features of the image.

[0065] In an embodiment of the present application, the computer device may perform convolution, pooling, and other processing on the image to extract image extraction features from the image.

[0066] Among them, when the computer device extracts features from an image, it can extract image features of multiple different scales (that is, the number of feature dimensions), and fuse the image features of multiple different scales to obtain image extraction features.

[0067] For example, a computer device can perform multiple levels of feature extraction on an image. When performing feature extraction on the first level, convolution and other operations are performed on the input image to obtain image features on the first level. When performing feature extraction on subsequent levels, image features on the previous one or more levels are performed on the image features to obtain image features on subsequent levels. The dimension of the image features at the previous level can be greater than or equal to the dimension of the image features at the subsequent level.

[0068] Optionally, if the dimension of the image features at the previous level is larger than the dimension of the image features at the subsequent level, the computer device can first unify the dimensions of the image features at multiple levels when fusing image features of multiple different scales. For example, unify them into the dimensions of the image features at the last level to obtain multiple image features of the same dimension, and then splice the multiple image features of the same dimension to obtain image extraction features.

[0069] Optionally, when fusing multiple image features of the same dimension, the computer device may also average or weighted average the features of each dimension of the multiple image features of the same dimension to obtain the image extraction feature.

[0070] Step 220: Encode and decode the image extraction features to obtain image reconstruction features.

[0071] In the embodiment of the present application, the above-mentioned process of encoding and decoding the image extraction features may refer to the process of crossing and reconstructing the features of each dimension in the image extraction features.

[0072] For example, computer equipment can perform multi-level processing on features of various dimensions in image extraction features. In each level of processing, weighting, convolution, pooling and other processing based on the attention mechanism are performed on the image features, and the processed image features are output to the next level to finally obtain image reconstruction features.

[0073] Step 230: Input the difference between the image reconstructed feature and the image extracted feature into the discriminator to obtain first identification information output by the discriminator. The first identification information includes a first information element corresponding to each pixel in the image. The first information element is used to indicate the probability that the corresponding pixel is abnormal.

[0074] The discriminator is a machine learning model trained based on synthetic image samples, where the synthetic image samples are image samples with abnormalities synthesized based on image samples without abnormalities.

[0075] In an embodiment of the present application, an image with abnormalities can be synthesized in advance from an image without abnormalities, and then a discriminator can be trained based on the image with abnormalities. The input of the discriminator is the difference between the image reconstructed features and the image extracted features, and the output of the discriminator is information indicating the probability of the presence of abnormalities in each pixel in the image.

[0076] For example, the image in step 210 may include H×W pixels, and the first identification information includes H×W first information elements, where the H×W first information elements correspond one-to-one to the H×W pixels.

[0077] The first information element may be a probability value indicating the probability that the corresponding pixel is abnormal, or an anomaly score indicating the probability that the corresponding pixel is abnormal.

[0078] For example, taking the above-mentioned first information element as an abnormality score as an example, the above-mentioned first identification information can be as shown in the following Table 1.

[0079] Table 1

[0080] As shown in Table 1 above, each space represents a first information element, and each first information element is expressed as an anomaly score. The value range of the anomaly score is [0, 100]. The higher the anomaly score, the higher the probability that the corresponding pixel has an anomaly.

[0081] Step 240: Determine an abnormality recognition result of the image based on the image extraction features, the image reconstruction features, and the first recognition information.

[0082] In an embodiment of the present application, the computer device can determine the probability of an abnormality in each pixel in the image based on the difference between the image extraction features and the image reconstruction features, and then combine the first identification information to comprehensively obtain an abnormality identification result of the image.

[0083] To sum up, the solution shown in the embodiment of the present application extracts image extraction features from the image, reconstructs the image extraction features through encoding and decoding to obtain image reconstruction features, and further uses a discriminator to perform abnormality discrimination based on the difference between the image reconstruction features and the image extraction features to obtain first identification information indicating the probability that an abnormality exists in each pixel in the image. Finally, the image features before and after reconstruction and the first identification information are combined to obtain a comprehensive abnormality recognition result of the image. In addition to using the image features before and after reconstruction to identify the abnormalities of pixels in the image, the above-mentioned scheme also uses a discriminator to discriminate the abnormalities of pixels in the image based on the difference between the image features before and after reconstruction. On the one hand, by combining the image features before and after reconstruction and the discrimination results determined based on the difference between the image features before and after reconstruction, the abnormalities of each pixel in the image are comprehensively determined, which can achieve pixel-level image anomaly detection while ensuring the accuracy of image anomaly detection. On the other hand, the above-mentioned discriminator is trained using synthetic image samples synthesized based on normal image samples. Therefore, the training process of the discriminator does not require manual collection and labeling of abnormal images, and can be trained through unsupervised learning, thereby expanding the number of samples available for discriminator training, improving the effect of training the discriminator through synthetic image samples, narrowing the difference between the effect of training the discriminator through real abnormal images, ensuring the accuracy of the discriminator, and then ensuring the accuracy of abnormality detection of pixels in the image through the discriminator.

[0084] Based on the embodiment shown in Figure 2, please refer to Figure 3, which is a flowchart of an image processing method according to an exemplary embodiment. As shown in Figure 3, the above step 240 can be implemented as step 240a and step 240b.

[0085] Step 240a: Determine second identification information based on the difference between the image extraction feature and the image reconstruction feature; the second identification information includes a second information element corresponding to each pixel in the image, and the second information element is used to indicate the probability that the corresponding pixel is abnormal.

[0086] For example, the second identification information includes H×W second information elements, which correspond one-to-one to the H×W pixels included in the image in step 210. The second information elements are used to indicate the probability that the corresponding pixels are abnormal.

[0087] The second information element may be a probability value indicating the probability that the corresponding pixel is abnormal, or an anomaly score indicating a positive correlation or negative correlation with the probability that the corresponding pixel is abnormal.

[0088] In some embodiments, the dimension of the difference between the above-mentioned image extraction feature and the image reconstruction feature is smaller than the number of pixels of the target image; the above-mentioned determination of the second identification information based on the difference between the image extraction feature and the image reconstruction feature may include: taking the L2 norm of the difference between the image extraction feature and the image reconstruction feature, and upsampling the L2 norm to obtain the second identification information.

[0089] In an embodiment of the present application, the computer device can subtract the image extraction feature and the image reconstruction feature element by element to obtain the difference between the image extraction feature and the image reconstruction feature, and take the L2 norm of the difference between the image extraction feature and the image reconstruction feature and upsample it to obtain the second identification information.

[0090] For example, the dimensions of the above-mentioned image extraction features and image reconstruction features are both h×w, where h×w is less than H×W (i.e., the number of pixels included in the image), for example, h=H / 16, w=W / 16; wherein, the computer device subtracts the features on the same dimension in the image extraction features and the image reconstruction features, calculates the L2 norm, and the dimension of the L2 norm is also h×w. Then, the calculation result (i.e., the above-mentioned L2 norm) is upsampled (for example, upsampling is performed through bilinear interpolation processing) to obtain the second identification information that also contains H×W elements (i.e., the above-mentioned second information element).

[0091] The above-mentioned subtraction of features on the same dimension in the image extraction feature and the image reconstruction feature refers to subtracting the feature element located in the i-th row and j-th column in the image extraction feature from the feature element located in the i-th row and j-th column in the image reconstruction feature, where i includes an integer greater than or equal to 1 and less than or equal to h, and j includes an integer greater than or equal to 1 and less than or equal to w, and ultimately h×w subtraction results will be obtained.

[0092] In an embodiment of the present application, after the computer device interacts and reconstructs the image extraction features, it detects whether there are abnormalities in each pixel in the image by calculating the L2 norm of the features before and after the interactive reconstruction, thereby ensuring the feasibility of detecting image abnormalities based on the reconstruction error of the image features.

[0093] Step 240b: Based on the first identification information and the second identification information, determine segmentation information in the abnormality identification result; the segmentation information includes a third information element corresponding to each pixel in the image, and the third information element is used to indicate the probability that the corresponding pixel has an abnormality.

[0094] For example, the segmentation information includes H×W third information elements, which correspond one-to-one to H×W pixels in the image in step 210 . The third information elements are used to indicate the probability that the corresponding pixels have an abnormality.

[0095] The third information element may be a probability value indicating the probability that the corresponding pixel is abnormal, or an anomaly score indicating a positive correlation or negative correlation with the probability that the corresponding pixel is abnormal.

[0096] In an embodiment of the present application, the dimensions of the first identification information and the second identification information are the same, both being H×W. When determining the segmentation information in the abnormal identification result, the third information element corresponding to each dimension can be determined to obtain segmentation information that also has H×W elements.

[0097] For example, the above-mentioned determination of the third information element corresponding to each dimension can be to combine the first information element located at the mth row and nth column in the first identification information and the second information element located at the mth row and nth column in the second identification information to determine the third information element located at the mth row and nth column in the segmentation information, wherein m includes an integer greater than or equal to 1 and less than or equal to H, and n includes an integer greater than or equal to 1 and less than or equal to W, and ultimately H×W third information elements in the segmentation information will be obtained.

[0098] Among them, the computer equipment combines the reconstruction error of the image features and the difference between the image features before and after reconstruction based on the discriminator to judge the abnormality of the pixels in the image, comprehensively determine the abnormality of each pixel in the image, and ensure the accuracy of pixel-level abnormality detection.

[0099] In one possible implementation, determining segmentation information in the abnormality recognition result based on the first identification information and the second identification information includes:

[0100] Perform weighted summation or weighted average on the first identification information and the second identification information to obtain segmentation information.

[0101] For example, the computer device can perform weighted summation or weighted average on the information elements of the same dimension in the first identification information and the second identification information to obtain H×W weighted summation or weighted average results. The H×W weighted summation or weighted average results are the third information element, which constitutes the above-mentioned segmentation information.

[0102] For example, the above-mentioned weighted summation or weighted averaging of the information elements on the same dimension in the first identification information and the second identification information can be a weighted summation or weighted averaging of the first information element located in the mth row and nth column of the first identification information and the second information element located in the mth row and nth column of the second identification information to obtain the third information element located in the mth row and nth column in the segmentation information, wherein m includes an integer greater than or equal to 1 and less than or equal to H, and n includes an integer greater than or equal to 1 and less than or equal to W, and ultimately H×W third information elements in the segmentation information will be obtained.

[0103] In an embodiment of the present application, the computer device performs weighted summation or weighted averaging on information elements of the same dimension in the first identification information and the second identification information, and can flexibly adjust the proportion of the first identification information and the second identification information by setting weights, thereby ensuring the flexibility and accuracy of the segmented information obtained by integrating the first identification information and the second identification information.

[0104] In a possible implementation, the method further includes:

[0105] Based on the segmentation information, an image classification result in the abnormality recognition result is determined, and the image classification result is used to indicate the probability that the image is an abnormal image.

[0106] The image classification result may be a probability value indicating the probability that the image in step 210 is an abnormal image. Alternatively, the image classification result may be an abnormality score, which may be positively correlated or negatively correlated with the probability that the image in step 210 is an abnormal image.

[0107] In an embodiment of the present application, the above-mentioned segmentation information is a pixel-level anomaly recognition result, that is, the segmentation information is used to indicate whether there is an anomaly in each pixel in the image; on this basis, the computer device can also obtain an image-level anomaly recognition result based on the segmentation information, that is, the above-mentioned image classification result is used to indicate whether the image is an abnormal image as a whole, thereby improving the diversity of image anomaly recognition results and further expanding the applicable scenarios of image anomaly recognition.

[0108] In one possible implementation, determining the image classification result in the abnormality recognition result based on the segmentation information includes:

[0109] The maximum value or standard deviation of each third information element in the segmentation information is taken to obtain the image classification result.

[0110] In an embodiment of the present application, a computer device takes the maximum value or standard deviation of each third information element in the segmentation information as the image-level anomaly detection result, providing a feasible solution for determining the image-level anomaly recognition result through the pixel-level anomaly recognition result, thereby ensuring the accuracy of the image-level anomaly recognition result.

[0111] Based on the embodiment shown in Figure 2 or Figure 3, please refer to Figure 4, which is a flowchart of an image processing method according to an exemplary embodiment. As shown in Figure 4, the above steps 210, 220, and 230 can be implemented as steps 210a, 220a, and 230a, respectively.

[0112] Step 210a: Input the image into the feature extractor in the image anomaly detection model to obtain image extraction features output by the feature extractor.

[0113] For example, the feature extractor receives image data of dimensions 3×H×W at its input, corresponding to H×W pixels in the image, where each pixel corresponds to data from three channels (e.g., R, G, and B values). The feature extractor processes the 3×H×W input image data and outputs extracted image features of dimensions h×w.

[0114] Step 220a: Input the image extraction features into the codec in the image anomaly detection model to obtain the image reconstruction features output by the codec. The size of the image reconstruction features is the same as the size of the image extraction features.

[0115] For example, the dimension of the image extraction features received by the input end of the above codec is h×w. After processing the image extraction features with the dimension of h×w, the codec can output image reconstruction features with the dimension of h×w.

[0116] Step 230a: Input the difference between the image reconstructed features and the image extracted features into the discriminator in the image anomaly detection model to obtain first recognition information output by the discriminator.

[0117] For example, the dimension of the difference between the image reconstructed features and the image extracted features received at the input end of the above-mentioned discriminator is h×w. The discriminator processes the difference between the image features before and after reconstruction with the dimension of h×w, and can output the first recognition information with the dimension of H×W.

[0118] The above-mentioned image anomaly detection model is a machine learning model obtained by unsupervised training based on the first image sample; the first image sample is an image sample without any anomalies.

[0119] In an embodiment of the present application, a computer device uses a pre-trained image anomaly detection model to identify anomalies in an image, thereby ensuring the efficiency and accuracy of image anomaly identification.

[0120] Based on the embodiment shown in FIG4 , please refer to FIG5 , which is a flow chart of an image processing method according to an exemplary embodiment. The method can be performed by a computer device, for example, the computer device can be a server, or the computer device can be a terminal, or the computer device can include a server and a terminal, wherein the server can be the server 160 in the embodiment shown in FIG1 above, and the terminal can be the terminal 140 used by the developer in the embodiment shown in FIG1 above. The computer device can be implemented as a model training device for model training. As shown in FIG5 , before the above-mentioned step 210, the method can include the following steps.

[0121] Step 510: Based on the first image sample, obtain first mask information and a second image sample; the first mask information is used to indicate that there are no abnormal pixels in the first image sample, and the second image sample is an image sample synthesized based on the first image sample and the second mask information, and the second mask information is used to indicate abnormal pixels that need to be added to the first image sample.

[0122] Since the first image sample is an image sample without anomalies, the first mask information can directly indicate that no anomalies exist in each pixel in the first image sample. For example, assuming the dimensions of the first image sample are H×W, the first mask information can be a two-dimensional matrix with dimensions H×W, where each dimension of the two-dimensional matrix corresponds to a pixel in the first image sample, and the value of each dimension in the two-dimensional matrix can be 0, for example, to indicate that no anomalies exist in each pixel in the first image sample.

[0123] The computer device can obtain the second mask information pre-set by the developer, or randomly generate the second mask information to indicate the abnormal pixels that need to be deployed based on the first image sample. For example, assuming that the dimension of the first image sample is H×W, the second mask information can be a two-dimensional matrix with a dimension of H×W, where each dimension of the two-dimensional matrix corresponds to a pixel in the first image sample, and the value of each dimension in the two-dimensional matrix is ​​used to indicate whether the corresponding pixel should be an abnormal pixel. For example, if the value of a dimension in the two-dimensional matrix is ​​1, it indicates that the corresponding pixel in the first image sample should be abnormal. If the value of the dimension is 0, it indicates that the corresponding pixel in the first image sample should be normal.

[0124] In addition, the computer device can also synthesize a second image sample based on the above-mentioned second mask information and the first image sample. For example, the computer device can modify the values ​​of corresponding pixels in the first image sample based on the abnormal pixels indicated by the second mask information. For example, the numerical values ​​of the pixels in the first image sample indicated by the second mask information where the abnormalities exist are randomly replaced to obtain the above-mentioned second image sample.

[0125] Step 520: Input the first image sample into the feature extractor to obtain the first image extracted feature sample output by the feature extractor; input the first image extracted feature sample into the codec to obtain the first image reconstructed feature sample output by the codec; input the difference between the first image reconstructed feature sample and the first image extracted feature sample into the discriminator to obtain the first identification information sample output by the discriminator; input the second image sample into the feature extractor to obtain the second image extracted feature sample output by the feature extractor; input the second image extracted feature sample into the codec to obtain the second image reconstructed feature sample output by the codec; input the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain the second identification information sample output by the discriminator.

[0126] Among them, the process in which each branch part in the image anomaly detection model (i.e., the above-mentioned feature extractor, codec, and discriminator) processes the first image sample and the second image sample to obtain the identification information sample is similar to the process in which each branch part in the image anomaly detection model processes the image to obtain the first identification information, and will not be repeated here.

[0127] Step 530: Determine a loss function value based on the first image extracted feature samples, the first image reconstructed feature samples, the first identification information samples, the second image extracted feature samples, the second image reconstructed feature samples, and the second identification information samples.

[0128] In an embodiment of the present application, during the model training process, the above-mentioned first mask information and second mask information are equivalent to the annotation information corresponding to the first image sample and the second image sample respectively. Based on the annotation information and the intermediate data and output results obtained by the image anomaly detection model in the process of processing the image sample, the loss function can be calculated.

[0129] In one possible implementation, determining the loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample includes:

[0130] determining a first loss function value based on a difference between the first image extracted feature sample and the first image reconstructed feature sample;

[0131] determining a second loss function value based on a difference between the first identification information sample and the first mask information;

[0132] determining a third loss function value based on a difference between the second image extracted feature sample and the second image reconstructed feature sample;

[0133] Based on the difference between the second identification information sample and the second mask information, a fourth loss function value is determined.

[0134] Among them, in the loss function calculation process, the above-mentioned first loss function value and the third loss function value are the losses generated in the process of the image anomaly detection model reconstructing the features of the image sample, and the above-mentioned second loss function value and the fourth loss function value are the losses generated in the process of the image anomaly detection model predicting the abnormal pixels in the input image sample.

[0135] At the same time, combining the losses generated in the reconstruction process and the losses generated in the abnormal pixel prediction process, training the various branches in the image anomaly detection model can simultaneously improve the image feature reconstruction performance and anomaly prediction performance of the image anomaly detection model, which is conducive to making the image anomaly detection model produce more accurate detection results in the application stage.

[0136] Step 540: Update the parameters of the image anomaly detection model using the loss function value.

[0137] For example, the computer device can update the parameters of the feature extractor and the codec through the first loss function value and the third loss function value, and update the parameters of the feature extractor, the codec and the discriminator through the second loss function value and the fourth loss function value.

[0138] This application proposes a model training method that combines unsupervised and discriminative reconstruction to train an image anomaly detection model. Specifically, this method proposes a pixel-level discriminator to further enhance the distinction between normal and abnormal samples and refine the reconstruction error. During the inference phase, the results of the unsupervised reconstruction and the pixel-level discriminator are combined to significantly improve pixel-level anomaly segmentation.

[0139] Please refer to Figure 6, which shows the unsupervised + discriminant reconstruction network architecture diagram involved in this application. As shown in Figure 6, the network architecture mainly includes three modules, namely: feature extraction part 610, unsupervised reconstruction part 620 and pixel-level discriminant reconstruction part 630.

[0140] The following three parts of the above network architecture are introduced in turn:

[0141] 1) Feature Extraction: Following existing anomaly detection work, we use a pre-trained model on ImageNet to extract image features. Given an input image For example, this application uses a pre-trained convolutional network (e.g. EfficientNet-b4) extracts multi-scale feature maps from stages 1 to 4 Right now:

[0142] in Here c i is the channel number, h i ×w i is the i-th spatial size level feature map. Considering that the feature map at the bottom layer has weak semantics but high resolution, the performance improvement is limited but the computational cost is expensive. In order to combine features from different hierarchical structures All feature maps are resized to the same size (h4×w4), which is the size of the smallest feature map. Then, these resized feature maps are concatenated in the channel dimension. The whole process can be expressed as:

[0143] in, h=h4,w=w4.

[0144] The feature extraction part 610 may use any pre-trained model based on a convolutional neural network or a Transformer architecture.

[0145] 2) Unsupervised reconstruction: Unsupervised reconstruction follows the standard Transformer architecture, which consists of two parts: encoding and decoding. First, a linear projection layer with position embedding is used to reduce the dimensionality of the multi-level features F extracted from the pre-trained model. Then, the encoding structure and decoding structure composed of a series of Transformer blocks are used to perform global interaction and reconstruction of the features. Here, each Transformer block consists of multi-head attention and a fully connected feedforward network. Finally, the output of the decoding structure is further dimensionalized using a linear projection layer to restore the original input feature dimension. The output is recorded as The reconstruction loss function calculates the mean square error (MSE) between the reconstructed features and the original features, that is:

[0146] Among them, the specific network structure of the above-mentioned unsupervised reconstruction part can adopt Transformer, or convolutional neural network, or fully connected neural network, etc.

[0147] 3) Discriminative reconstruction: The performance of anomaly segmentation using only unsupervised reconstruction is still poor. Because it is only trained on non-anomaly training data, this may lead to "weak discriminant lines" between normal samples and abnormal samples in the feature space, that is, when the difference between abnormal samples and normal samples is not obvious enough, the trained model has difficulty in identifying abnormal samples. Therefore, the discriminator should be trained to enhance the discrimination between normal and abnormal samples, and the simplest implementation method is to train a binary classifier based on normal images and defective images. However, for unsupervised anomaly detection tasks, usually only normal images can be obtained as training samples. In fact, this is also a setting for actual industrial detection applications, because defective products are always rare and difficult to collect on a large scale. In order to train the discriminator, normal images can be used to synthesize defective images. This application can synthesize defective images at the pixel level, and then train the discriminator based on the synthesized images. In addition, this application designs a lightweight discriminator to refine the reconstruction error of the dual-mask automatic encoder and decoder.

[0148] Given a normal training image I n and the corresponding anomaly mask Y n , the defect image and abnormal mask synthesized by the two are expressed as I s and Y s Then, the normal image and the synthetic defect image {I t |t=n,s} is input into the multi-level feature extractor, and its multi-level features are exported as {F t |t=n,s}. Then, the unsupervised reconstruction network is used to reconstruct {F t |t=n,s}, and the corresponding features are expressed as Here, this application uses element-wise subtraction of the original features and the reconstructed features to measure their differences, namely:

[0149] The above discriminator is designed with multiple convolution blocks to learn features, followed by a 1×1 convolution layer to perform pixel segmentation. Here, each convolution block consists of a 3×3 convolution, BatchNorm, ReLU, and a 2×2 deconvolution. The absolute reconstruction error {E t |t=n,s} is input into the designed discriminator and the estimated anomaly map is obtained To calculate and the true value Y t The loss between Resize to Y t Considering that abnormal pixels usually account for a minority in anomaly detection, the Dice loss can be used to calculate the loss function, which is very effective for learning from extremely unbalanced data, namely:

[0150] Where (i, j) represents Y t or spatial location.

[0151] The abnormal reasoning process is as follows:

[0152] Pixel-level anomaly segmentation: The result of anomaly segmentation is an anomaly score map, which assigns an anomaly score to each pixel. For unsupervised reconstruction, the anomaly score map is calculated as the upsampled result of the L2 norm of the reconstruction error, as follows:

[0153] For the discriminative reconstruction, the anomaly score map is predicted as Finally, S res and Combined together, as the final anomaly segmentation map, that is:

[0154] Among them, ω∈[0,1] is the weight.

[0155] Image-level anomaly classification: Anomaly classification aims to detect whether an image contains abnormal regions. It converts the pixel-level prediction error S into an image-level anomaly score by finding the maximum value or standard deviation in the spatial dimension.

[0156] Experimental Setup: EfficientNet-b4, pre-trained on ImageNet, was selected as the base network for feature extraction. The input image size was 224×224. Three stage features were extracted and rescaled to a uniform size of 14×14. Finally, the channel-wise concatenation was performed to form a raw feature size of 272×14×14. Both the encoder and decoder used Transformer blocks.

[0157] The main experimental conclusions are as follows:

[0158] 1) This solution can solve the problem of inaccurate positioning and segmentation of defective pixels in unsupervised reconstruction methods.

[0159] Figure 7 is a comparison chart of test results involved in this application. As shown in Figure 7, it shows a comparative diagram of the training loss and test indicators of the unsupervised reconstruction network and the unsupervised + discriminant reconstruction encoding and decoding network proposed in this application on the MVTec dataset.

[0160] The results show that ordinary unsupervised reconstruction networks only perform well in image-level defect detection. The unsupervised + discriminative reconstruction encoding and decoding network proposed in this application has a significant performance improvement in pixel-level defect segmentation.

[0161] 2) This solution significantly improves the performance of unified anomaly detection on a real industrial anomaly detection dataset, especially for pixel-level anomaly segmentation. Please refer to Figure 8, which shows a comparative diagram of anomaly recognition involved in this application. As shown in Figure 8, the anomaly detection boundaries of the proposed method (column 3) are more refined and accurate compared to the unsupervised anomaly detection method (column 2) on 15 MVTec categories. This is close to the manually annotated ground truth (column 4).

[0162] FIG9 is a block diagram of an image processing apparatus provided by an exemplary embodiment of the present application. As shown in FIG9 , the apparatus includes the following parts:

[0163] A feature extraction module 901 is used to extract features from an image and obtain image extraction features of the image;

[0164] The encoding and decoding module 902 is used to encode and decode the image extraction features to obtain image reconstruction features;

[0165] Reconstruction module 903 is configured to input the difference between the image reconstructed feature and the image extracted feature into a discriminator to obtain first identification information output by the discriminator, wherein the first identification information includes a first information element corresponding to each pixel in the image, and the first information element is used to indicate the probability that the corresponding pixel is abnormal. The discriminator is a machine learning model trained based on synthetic image samples, wherein the synthetic image samples are image samples containing abnormalities synthesized from image samples without abnormalities.

[0166] The recognition module 904 is configured to determine an abnormality recognition result of the image based on the image extraction features, the image reconstruction features, and the first recognition information.

[0167] In a possible implementation, the identification module 904 is configured to:

[0168] determining second identification information based on a difference between the image extraction feature and the image reconstruction feature; wherein the second identification information includes a second information element corresponding to each pixel in the image, the second information element being used to indicate a probability that the corresponding pixel is abnormal;

[0169] Based on the first identification information and the second identification information, segmentation information in the identification result is determined; the segmentation information includes a third information element corresponding to each pixel in the image, and the third information element is used to indicate the probability that the corresponding pixel has an abnormality.

[0170] In one possible implementation, the dimension of the difference between the image extraction feature and the image reconstruction feature is smaller than the number of pixels of the image; the recognition module 904 is used to take the L2 norm of the difference between the image extraction feature and the image reconstruction feature, and upsample the L2 norm to obtain the second recognition information.

[0171] In a possible implementation, the identification module 904 is configured to perform a weighted sum or weighted average on the first identification information and the second identification information to obtain the segmentation information.

[0172] In a possible implementation, the identification module 904 is further configured to determine an image classification result in the abnormality identification result based on the segmentation information, where the image classification result is used to indicate a probability that the image is an abnormal image.

[0173] In a possible implementation, the identification module 904 is configured to take a maximum value or a standard deviation of each third information element in the segmentation information to obtain the image classification result.

[0174] In a possible implementation, the feature extraction module 901 is configured to input the image into a feature extractor in an image anomaly detection model to obtain the image extraction features output by the feature extractor;

[0175] The codec module 902 is configured to input the image extraction features into a codec in the image anomaly detection model to obtain the image reconstruction features output by the codec, where the size of the image reconstruction features is the same as the size of the image extraction features;

[0176] The reconstruction module 903 is configured to input the difference between the image reconstructed feature and the image extracted feature into a discriminator in the image anomaly detection model to obtain the first recognition information output by the discriminator;

[0177] The image anomaly detection model is a machine learning model obtained by unsupervised training based on a first image sample; the first image sample is an image sample without anomalies.

[0178] In a possible implementation, the apparatus further includes:

[0179] a sample processing module configured to obtain, based on the first image sample, first mask information and a second image sample before the feature extraction module performs feature extraction on the image to obtain image extraction features of the image; the first mask information is configured to indicate pixels without abnormalities in the first image sample; the second image sample is an image sample synthesized based on the first image sample and the second mask information; the second mask information is configured to indicate pixels with abnormalities that need to be added to the first image sample;

[0180] a model processing module, configured to input the first image sample into the feature extractor to obtain a first image extracted feature sample output by the feature extractor; input the first image extracted feature sample into the codec to obtain a first image reconstructed feature sample output by the codec; input the difference between the first image reconstructed feature sample and the first image extracted feature sample into the discriminator to obtain a first identification information sample output by the discriminator; input the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; input the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; input the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator;

[0181] a loss acquisition module, configured to determine a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample;

[0182] A parameter updating module is used to update the parameters of the image anomaly detection model through the loss function value.

[0183] In a possible implementation, the loss acquisition module is configured to:

[0184] determining a first loss function value based on a difference between the first image extracted feature sample and the first image reconstructed feature sample;

[0185] determining a second loss function value based on a difference between the first identification information sample and the first mask information;

[0186] determining a third loss function value based on a difference between the second image extracted feature sample and the second image reconstructed feature sample;

[0187] A fourth loss function value is determined based on a difference between the second identification information sample and the second mask information.

[0188] To sum up, the solution shown in the embodiment of the present application extracts image extraction features from the image, reconstructs the image extraction features through encoding and decoding to obtain image reconstruction features, and further uses a discriminator to perform abnormality discrimination based on the difference between the image reconstruction features and the image extraction features to obtain first identification information indicating the probability that an abnormality exists in each pixel in the image. Finally, the image features before and after reconstruction and the first identification information are combined to obtain a comprehensive abnormality recognition result of the image. In addition to using the image features before and after reconstruction to identify the abnormalities of pixels in the image, the above-mentioned scheme also uses a discriminator to discriminate the abnormalities of pixels in the image based on the difference between the image features before and after reconstruction. On the one hand, by combining the image features before and after reconstruction and the discrimination results determined based on the difference between the image features before and after reconstruction, the abnormalities of each pixel in the image are comprehensively determined, which can achieve pixel-level image anomaly detection while ensuring the accuracy of image anomaly detection. On the other hand, the above-mentioned discriminator is trained using synthetic image samples synthesized based on normal image samples. Therefore, the training process of the discriminator does not require manual collection and labeling of abnormal images, and can be trained through unsupervised learning, thereby expanding the number of samples available for discriminator training, improving the effect of training the discriminator through synthetic image samples, narrowing the difference between the effect of training the discriminator through real abnormal images, ensuring the accuracy of the discriminator, and then ensuring the accuracy of abnormality detection of pixels in the image through the discriminator.

[0189] FIG10 is a block diagram of an image processing apparatus provided by an exemplary embodiment of the present application. As shown in FIG9 , the apparatus includes the following parts:

[0190] The sample processing module 1001 is configured to obtain first mask information and a second image sample based on a first image sample; the first mask information is used to indicate pixels in the first image sample that do not contain abnormalities; the second image sample is an image sample synthesized based on the first image sample and the second mask information; the second mask information is used to indicate pixels in the first image sample that contain abnormalities and need to be added;

[0191] The model processing module 1002 is configured to input the first image sample into a feature extractor in an image anomaly detection model to obtain a first image extracted feature sample output by the feature extractor; input the first image extracted feature sample into a codec in the image anomaly detection model to obtain a first image reconstructed feature sample output by the codec; input the difference between the first image reconstructed feature sample and the first image extracted feature sample into a discriminator in the image anomaly detection model to obtain a first identification information sample output by the discriminator; input the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; input the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; input the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator;

[0192] a loss acquisition module 1003, configured to determine a loss function value based on the first image extraction feature sample, the first image reconstruction feature sample, the first identification information sample, the second image extraction feature sample, the second image reconstruction feature sample, and the second identification information sample;

[0193] A parameter updating module 1004 is configured to update the parameters of the image anomaly detection model using the loss function value;

[0194] The image anomaly detection model is used to implement the image processing method shown in Figures 2 to 4 above.

[0195] In a possible implementation, the loss acquisition module is configured to:

[0196] determining a first loss function value based on a difference between the first image extracted feature sample and the first image reconstructed feature sample;

[0197] determining a second loss function value based on a difference between the first identification information sample and the first mask information;

[0198] determining a third loss function value based on a difference between the second image extracted feature sample and the second image reconstructed feature sample;

[0199] A fourth loss function value is determined based on a difference between the second identification information sample and the second mask information.

[0200] It should be noted that the image processing device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0201] FIG11 shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. Specifically:

[0202] The server 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. The server 1100 also includes a mass storage device 1106 for storing an operating system 1113, application programs 1114, and other program modules 1115.

[0203] The mass storage device 1106 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1106 and its associated computer-readable media provide non-volatile storage for the server 1100. In other words, the mass storage device 1106 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0204] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1104 and mass storage device 1106 can be collectively referred to as memory.

[0205] According to various embodiments of the present application, the server 1100 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 1100 may be connected to the network 1112 via the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 may be used to connect to other types of networks or remote computer systems (not shown).

[0206] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0207] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the image processing method provided by the above-mentioned method embodiments.

[0208] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image processing method provided by the above-mentioned method embodiments.

[0209] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described in any of the above embodiments.

[0210] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0211] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0212] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, executed by a computer device, comprising: Performing feature extraction on the image to obtain image extraction features of the image; Encoding and decoding the image extraction features to obtain image reconstruction features; Inputting the difference between the image reconstructed feature and the image extracted feature into a discriminator to obtain first identification information output by the discriminator, wherein the first identification information includes a first information element corresponding to each pixel in the image, and the first information element is used to indicate the probability that the corresponding pixel is abnormal; the discriminator is a machine learning model trained based on synthetic image samples, and the synthetic image samples are image samples with abnormalities synthesized from image samples without abnormalities; An abnormality recognition result of the image is determined based on the image extraction feature, the image reconstruction feature and the first recognition information.

2. The method according to claim 1, wherein determining the abnormality recognition result of the image based on the image extraction feature, the image reconstruction feature, and the first recognition information comprises: determining second identification information based on a difference between the image extraction feature and the image reconstruction feature; The second identification information includes a second information element corresponding to each pixel in the image, and the second information element is used to indicate the probability that the corresponding pixel is abnormal; Based on the first identification information and the second identification information, segmentation information in the abnormality identification result is determined; the segmentation information includes a third information element corresponding to each pixel in the image, and the third information element is used to indicate the probability that the corresponding pixel has an abnormality.

3. The method according to claim 2, wherein the dimension of the difference between the image extraction feature and the image reconstruction feature is smaller than the number of pixels of the image; The determining of the second identification information based on the difference between the image extraction feature and the image reconstruction feature includes: An L2 norm is taken for a difference between the image extraction feature and the image reconstruction feature, and the L2 norm is up-sampled to obtain the second identification information.

4. The method according to claim 2 or 3, wherein determining segmentation information in the abnormality recognition result based on the first recognition information and the second recognition information comprises: Performing a weighted sum or weighted average on the first identification information and the second identification information to obtain the segmentation information.

5. The method according to any one of claims 2 to 4, further comprising: An image classification result in the abnormality recognition result is determined based on the segmentation information, where the image classification result is used to indicate a probability that the image is an abnormal image.

6. The method according to claim 5, wherein determining the image classification result in the abnormality recognition result based on the segmentation information comprises: The maximum value or standard deviation of each of the third information elements in the segmentation information is taken to obtain the image classification result.

7. The method according to any one of claims 1 to 6, wherein extracting features from an image to obtain image extraction features of the image comprises: Inputting the image into a feature extractor in an image anomaly detection model to obtain the image extraction features output by the feature extractor; The encoding and decoding processing of the image extraction features to obtain image reconstruction features includes: Inputting the image extraction feature into a codec in the image anomaly detection model to obtain the image reconstruction feature output by the codec, where the size of the image reconstruction feature is the same as the size of the image extraction feature; The step of inputting the difference between the image reconstruction feature and the image extraction feature into a discriminator to obtain first identification information output by the discriminator includes: Inputting the difference between the image reconstruction feature and the image extraction feature into the discriminator in the image anomaly detection model to obtain the first recognition information output by the discriminator; The image anomaly detection model is a machine learning model obtained by unsupervised training based on a first image sample; the first image sample is an image sample without anomalies.

8. The method according to claim 7, wherein the image anomaly detection model is trained by: Based on the first image sample, first mask information and a second image sample are obtained; the first mask information is used to indicate pixels without abnormalities in the first image sample; the second image sample is an image sample synthesized based on the first image sample and the second mask information, and the second mask information is used to indicate pixels with abnormalities that need to be added to the first image sample; Inputting the first image sample into the feature extractor to obtain a first image extracted feature sample output by the feature extractor; inputting the first image extracted feature sample into the codec to obtain a first image reconstructed feature sample output by the codec; inputting the difference between the first image reconstructed feature sample and the first image extracted feature sample into the discriminator to obtain a first identification information sample output by the discriminator; Inputting the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; inputting the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; inputting the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator; determining a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample; The parameters of the image anomaly detection model are updated according to the loss function value.

9. The method according to claim 8, wherein determining a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample comprises: determining a first loss function value based on a difference between the first image extracted feature sample and the first image reconstructed feature sample; determining a second loss function value based on a difference between the first identification information sample and the first mask information; determining a third loss function value based on a difference between the second image extracted feature sample and the second image reconstructed feature sample; A fourth loss function value is determined based on a difference between the second identification information sample and the second mask information.

10. An image processing method, executed by a computer device, comprising: Based on the first image sample, obtaining first mask information and a second image sample; The first mask information is used to indicate pixels without abnormalities in the first image sample, the second image sample is an image sample synthesized based on the first image sample and the second mask information, and the second mask information is used to indicate pixels with abnormalities that need to be added to the first image sample; Inputting the first image sample into a feature extractor in an image anomaly detection model to obtain a first image extracted feature sample output by the feature extractor; inputting the first image extracted feature sample into a codec in the image anomaly detection model to obtain a first image reconstructed feature sample output by the codec; inputting a difference between the first image reconstructed feature sample and the first image extracted feature sample into a discriminator in the image anomaly detection model to obtain a first identification information sample output by the discriminator; Inputting the second image sample into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; inputting the second image extracted feature sample into the codec to obtain a second image reconstructed feature sample output by the codec; inputting the difference between the second image reconstructed feature sample and the second image extracted feature sample into the discriminator to obtain a second identification information sample output by the discriminator; determining a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample; Updating the parameters of the image anomaly detection model using the loss function value; Wherein, the image anomaly detection model is used to implement the image processing method described in any one of claims 1 to 9.

11. An image processing device, comprising: A feature extraction module, used to extract features from an image and obtain image extraction features of the image; A coding and decoding module, configured to encode and decode the image extraction features to obtain image reconstruction features; a reconstruction module, configured to input a difference between the image reconstructed feature and the image extracted feature into a discriminator to obtain first identification information output by the discriminator, wherein the first identification information includes a first information element corresponding to each pixel in the image, the first information element being used to indicate a probability that an abnormality exists in the corresponding pixel; the discriminator is a machine learning model trained based on synthetic image samples, wherein the synthetic image samples are image samples containing abnormalities synthesized from image samples without abnormalities; An identification module is used to determine an abnormality identification result of the image based on the image extraction feature, the image reconstruction feature and the first identification information.

12. An image processing device, comprising: A sample processing module, configured to obtain first mask information and a second image sample based on the first image sample; The first mask information is used to indicate pixels without abnormalities in the first image sample, the second image sample is an image sample synthesized based on the first image sample and the second mask information, and the second mask information is used to indicate pixels with abnormalities that need to be added to the first image sample; a model processing module, configured to input the first image sample into a feature extractor in an image anomaly detection model to obtain a first image extracted feature sample output by the feature extractor; and input the first image extracted feature sample into a codec in the image anomaly detection model to obtain a first image reconstructed feature sample output by the codec; The The difference between the first image reconstructed feature sample and the first image extracted feature sample is input into the discriminator in the image anomaly detection model to obtain a first identification information sample output by the discriminator; the second image sample is input into the feature extractor to obtain a second image extracted feature sample output by the feature extractor; the second image extracted feature sample is input into the codec to obtain a second image reconstructed feature sample output by the codec; the difference between the second image reconstructed feature sample and the second image extracted feature sample is input into the discriminator to obtain a second identification information sample output by the discriminator; a loss acquisition module, configured to determine a loss function value based on the first image extracted feature sample, the first image reconstructed feature sample, the first identification information sample, the second image extracted feature sample, the second image reconstructed feature sample, and the second identification information sample; A parameter updating module, configured to update the parameters of the image anomaly detection model using the loss function value; Wherein, the image anomaly detection model is used to implement the image processing method described in any one of claims 1 to 9.

13. A computer device comprising a processor and a memory, wherein the memory stores at least one program, and the processor loads and executes the at least one program to implement the image processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 10.

15. A computer program product comprising computer instructions, wherein when the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images