Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: performing anomaly detection on an object in an input image to obtain a first anomaly score which is used for indicating the possibility that an abnormal object exists in the input image; when the first abnormal score is greater than or equal to a first threshold value, obtaining a style type of the input image; performing anomaly detection on an object in the input image through a classification model matched with the style type to obtain a second anomaly score, the second anomaly score being used for indicating the possibility that an abnormal object exists in the input image; and when the second anomaly score is greater than or equal to a second threshold value, determining the input image as an abnormal image with an abnormal object. On the basis of the first anomaly detection, the input image is subjected to anomaly detection again through the classification model matched with the style type of the input image, so that the detection accuracy of the abnormal image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the development of Internet technology, image processing technology has become increasingly mature. In order to efficiently generate and process digital image works with personalized characteristics, a large number of artificial intelligence (AI) technologies have emerged.

[0003] For example, users can use AI image generation technology to generate AI images that match the content described in the descriptive text based on the descriptive text. However, the objects in the AI ​​images are prone to abnormalities such as deformity, blur, and incoordination, such as deformed faces, blurred faces, incoordination of facial features, multiple hands, multiple feet, etc. Such images can be called abnormal images.

[0004] The related technology performs anomaly detection of AI images through a single classification model to prevent abnormal images from being put into use. However, the single classification model has problems of missed detection and inaccurate detection, resulting in low detection accuracy of abnormal images. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device and storage medium. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, there is provided an image processing method, the method comprising:

[0007] Performing anomaly detection on an object in an input image to obtain a first anomaly score, wherein the first anomaly score is used to indicate the possibility that an abnormal object exists in the input image;

[0008] When the first anomaly score is greater than or equal to a first threshold, obtaining a style type of the input image;

[0009] Performing anomaly detection on the object in the input image by using a classification model matching the style type to obtain a second anomaly score, wherein the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image;

[0010] When the second anomaly score is greater than or equal to a second threshold, the input image is determined to be an abnormal image in which an abnormal object exists.

[0011] According to one aspect of an embodiment of the present application, there is provided an image processing device, the device comprising:

[0012] A first score acquisition module, configured to perform anomaly detection on an object in an input image to obtain a first anomaly score, wherein the first anomaly score is used to indicate the possibility of an abnormal object existing in the input image;

[0013] a style type acquisition module, configured to acquire the style type of the input image when the first anomaly score is greater than or equal to a first threshold;

[0014] a second score acquisition module, configured to perform anomaly detection on the object in the input image by using a classification model matching the style type to obtain a second anomaly score, wherein the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image;

[0015] The abnormal image determination module is used to determine the input image as an abnormal image with an abnormal object when the second abnormal score is greater than or equal to a second threshold.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned image processing method.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0018] According to one aspect of the embodiments of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned image processing method.

[0019] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0020] On the basis of filtering out the input image according to the first anomaly detection, the classification model matching the style type of the input image is used to perform anomaly detection on the object in the input image again, so as to further determine whether the input image is an abnormal image based on the second anomaly check. Compared with using only a single classification model, it is beneficial to improve the detection accuracy of abnormal images. At the same time, since images of different style types contain different features and details, the classification model matching the style type is used to perform anomaly detection again, so that the classification model is consistent with the input image, thereby reducing the impact of the features and details under the style type on the prediction results of the classification model, which is beneficial to further improve the detection accuracy of abnormal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of a computer system provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a method for determining an abnormal image provided by an embodiment of the present application;

[0025] Figure 4 is a flowchart of an image processing method provided by another embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a method for repairing an abnormal image provided by an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a repaired abnormal image provided by an embodiment of the present application;

[0028] Figure 7 is a flowchart of an image processing method provided by another embodiment of the present application;

[0029] Figure 8 is a block diagram of an image processing device provided by an embodiment of the present application;

[0030] Fig. 9 is a block diagram of an image processing device provided by another embodiment of the present application;

[0031] Fig.10It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0033] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided by an embodiment of the present application. The computer system can be implemented as an architecture of an image processing system, and the image processing system is used to process images, such as image detection, image restoration, etc. The computer system may include: a terminal device 10 and a server 20.

[0034] The terminal device 10 may be an electronic device such as a mobile phone, a tablet computer, a multimedia player, a PC (Personal Computer), an intelligent robot, a vehicle-mounted terminal device, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a wearable device, etc. The terminal device 10 may be installed with a client of a target application, and the target application may be at least one of the following applications: an image generation application, an image restoration application, an image detection application, a live broadcast application, a music playback application, a game application, a social entertainment application, a simulation learning application, a shopping application, and an education and training application.

[0035] The above application can be an online application. For example, the client of the above application can be installed and run in different terminal devices for use by different users. Taking a music player application as an example, the music player application can be installed and run in different terminal devices in the form of a mobile client, a PC client, a web page, a mini-program, etc. The above application can be an offline application, which is not limited in the embodiments of the present application.

[0036] In the embodiment of the present application, the above-mentioned target application has at least one of the following functions: image detection function, image repair function. Among them, the image detection function may refer to the function of detecting whether an image is abnormal, such as the function of detecting whether an object in an image is abnormal. The image repair function may refer to the function of repairing an abnormality in an image, such as the function of repairing an abnormal part of an abnormal object in an image, which is not limited in the embodiment of the present application.

[0037] The server 20 may be used to provide background services for the terminal device 10. For example, the server 20 may be a background server of the target application described above, and may be used to provide background services for a client of the target application installed and running in the terminal device 10. The server 20 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing service center.

[0038] The terminal device 10 and the server 20 can communicate with each other via the network 30 .

[0039] The technical solution provided by the embodiment of the present application is applicable to any image processing scenario, such as image (such as personal portrait) generation scenario, image detection scenario, video generation scenario, animation generation scenario, image restoration scenario, etc. The technical solution provided by the embodiment of the present application can improve the detection accuracy of abnormal images and the restoration efficiency of abnormal images.

[0040] Exemplarily, the client of the above-mentioned target application supports AI image generation. When obtaining the descriptive text input by the user, the client uses AI image generation technology to automatically generate an AI image based on the descriptive text. For example, if the descriptive text describes the content of "beautiful sunset at sea", the AI ​​image generation technology generates an AI image to present the "beautiful sunset at sea".

[0041] The client performs anomaly detection on the object in the AI ​​image to obtain a first anomaly score, which is used to indicate the possibility of the presence of an abnormal object in the AI ​​image, such as detecting whether the sun in the AI ​​image is an abnormal object. In the case where the presence of an abnormal object in the AI ​​image is preliminarily determined based on the first anomaly score, the client obtains the style type of the AI ​​image, and performs anomaly detection on the object in the AI ​​image again through a classification model matching the style type to obtain a second anomaly score, which is also used to indicate the possibility of the presence of an abnormal object in the AI ​​image.

[0042] If it is determined based on the second anomaly score that the AI ​​image does have an abnormal object, the client repairs the abnormal object in the AI ​​image to obtain a repaired AI image. If it is detected that there is no abnormal object in the repaired AI image, the client provides the repaired AI image to the user.

[0043] Optionally, the above process can also be performed by the background server of the target application. For example, when the client obtains the descriptive text input by the user, the client sends the descriptive text to the background server, and the background server performs processes such as AI image generation, anomaly detection and repair, thereby obtaining a repaired AI image without abnormal objects. The background server sends the repaired AI image to the client to provide it to the user through the client. The above process can also be completed collaboratively by the client of the target application and the background server of the target application, such as part of the process is completed by the client and the other part of the process is completed by the background server, which is not limited to this in the embodiments of the present application.

[0044] The technical solution provided by the embodiment of the present application will be described below through a method embodiment. For matters not described in the method embodiment, reference can be made to the above embodiment.

[0045] Please refer to Figure 2 , which shows a flow chart of an image processing method provided by an embodiment of the present application. The method can be applied to the terminal device 10 or the server 20 introduced above. In the embodiment of the present application, the client in the terminal device 10 is taken as the execution subject as an example, and the method may include the following steps (201-204):

[0046] Step 201 : performing anomaly detection on an object in an input image to obtain a first anomaly score, where the first anomaly score is used to indicate the possibility of an abnormal object existing in the input image.

[0047] The input image may be an image to be detected for anomalies. The embodiment of the present application does not limit the source of the input image. Exemplarily, the input image may be an image generated by the client itself, an image uploaded to the client by a user, or an image collected online by the client. For example, the input image may be an AI image generated by the client based on descriptive text.

[0048] The above-mentioned object may refer to a target for performing anomaly detection. The embodiment of the present application does not limit the type of the object. Exemplarily, the type of the above-mentioned object may be a "person" type, an "animal" type, or a "plant" type, and any visible target in the image may be an object. For example, anomaly detection is performed on a "person" object in an input image to determine whether the "person" object has an abnormality.

[0049] The above-mentioned abnormality detection may refer to the process of detecting whether there are abnormal objects in the image, and the abnormal objects may refer to objects with abnormalities. The judgment criteria of abnormal objects can be set and adjusted according to actual use requirements, and the embodiments of the present application are not limited to this. Exemplarily, objects that do not meet the user's expectations can be determined as abnormal objects, such as objects that do not meet the description of the descriptive text entered by the user, and deformed objects can also be determined as abnormal objects, such as "character" objects with multiple hands and multiple feet. Objects with uncoordinated expressions, unclear faces, etc. can also be determined as abnormal objects.

[0050] In one example, according to the abnormality degree of the abnormal object, the abnormal object can be divided into at least three abnormality levels: severe abnormality, moderate abnormality, and mild abnormality.

[0051] Exemplarily, objects that do not conform to the basic structure of the object may be classified as severely abnormal, such as "human" objects with multiple hands, multiple heads, and multiple ears; "animal" objects with multiple hands, multiple heads, and multiple ears; "plant" objects with multiple hands, multiple heads, and multiple ears; and objects with characteristics of other types of objects.

[0052] For objects that are basically consistent with the basic structure of the object, they can be classified as moderately abnormal, such as "human objects" with multiple fingers and multiple toes, and "animal objects" with abnormal hair.

[0053] Objects that match the basic structure of the object but lack details can be classified as slightly abnormal, such as "human objects" with inconsistent facial expressions, "human objects" with unclear facial features, objects with inaccurate skin color, objects with inaccurate posture, and objects that do not meet user expectations.

[0054] In an embodiment of the present application, an anomaly score is used to indicate the possibility of an abnormal object in an image, and the above-mentioned first anomaly score can be used to refer to the anomaly score of the input image when it is first determined. The anomaly score can be a probability value or an integer value, which is not limited in the embodiment of the present application. Exemplarily, the range of the anomaly score can be 0 to 1 to indicate the possibility of an abnormal object in the image, and the larger the value of the anomaly score, the greater the possibility of an abnormal object in the image. For example, 0.7 represents a greater possibility of an abnormal object in the image, and 0.3 indicates that the image is relatively normal.

[0055] In one example, anomaly detection may be performed on an object in an input image using a general classification model to obtain a first anomaly score; wherein the general classification model is trained based on images of at least two different styles.

[0056] Optionally, in the case where anomaly detection is performed on an object in an input image by a general classification model, the first anomaly score may be used to refer to an anomaly score of the input image determined for the first time by the general classification model.

[0057] The above-mentioned general classification model can be a machine learning model for performing anomaly detection on an image. Exemplarily, the above-mentioned general classification model can be used to extract features from an input image to obtain a feature map of the input image, and then predict the probability value of an object in the input image being an abnormal object based on the feature map, thereby realizing anomaly detection. Among them, the probability value predicted by the general classification model is the first anomaly score. In a feasible example, the general classification model can also be used to perform binary classification on the input image, such as output 1 represents that the object in the input image is abnormal, and output 0 represents that the object in the input image is not abnormal.

[0058] The embodiment of the present application does not limit the number of objects in the input image. When there are multiple objects in the input image, the general classification model can obtain the anomaly scores of the multiple objects respectively, and determine the highest anomaly score as the first anomaly score.

[0059] Optionally, the general classification model can be constructed based on at least one of the following: CNN (Convolutional Neural Networks), DNN (Deep Neural Networks), DenseNet (Dense Convolutional Network), ResNET (Residual Network), VGGNet (Visual Geometry Group Net), StyleGAN2 (a neural network for acquiring latent features of an image in a latent space), ViT (Vision Transformer, a deep learning model for processing computer vision tasks), and EfficientNet (an efficient convolutional neural network).

[0060] After constructing the general classification model, at least two images of different styles can be used to iteratively train the general classification model to obtain a trained general classification model. Exemplarily, the at least two different style types include at least two of the following: realistic style, simulation style, and fictional style. Among them, a realistic-style image can refer to a real image captured by a shooting device, or it can be an image generated in a realistic style. A simulation-style image can be an image obtained by simulating a real image, which is relatively similar to the real image. A fictional-style image can be a completely fictional generated image, or it can be a semi-simulation and semi-fictional generated image, such as an AI image, an animation image, a cartoon image, etc.

[0061] For example, multiple realistic style images and multiple fictional style images can be mixed, and the parameters of the general classification model can be iteratively adjusted with the parameter gradient descent of the general classification model as the goal to obtain a trained general classification model. The trained general classification model can be used to capture various possible abnormal features of abnormal objects to achieve universal judgment of input images and widely identify abnormal images of various styles, which is conducive to avoiding the omission of abnormal images and thus improving the recall rate of abnormal images.

[0062] For example, refer to Figure 3 A trained general classification model 301 is deployed in the client. After acquiring the input image, the client calls the general classification model 301 to perform anomaly detection on the object in the input image to obtain a first anomaly score.

[0063] Optionally, before inputting the input image into the general classification model 301, the input image may be preprocessed so that the size of the input image is the same as the size of the image used to train the general classification model 301, and the preprocessing includes at least one of the following: standardization and resizing.

[0064] Optionally, the generalization ability of the general classification model can also be improved through data enhancement (such as rotation, cropping, color adjustment, etc.) and transfer learning, which is not limited to the embodiments of the present application.

[0065] Step 202: When the first anomaly score is greater than or equal to the first threshold, the style type of the input image is obtained.

[0066] The above-mentioned first threshold is used to determine whether there is an abnormal object in the input image. It can be set and adjusted according to actual usage requirements, and the embodiment of the present application does not limit this.

[0067] For example, when the first anomaly score is a probability value, the first threshold value may be set to 0.7. When the first anomaly score is greater than or equal to 0.7, the client may determine that an abnormal object may exist in the input image. When the first anomaly score is less than 0.7, the client may determine that there is no abnormal object in the input image, that is, the client may determine that the input image is a normal image and can be put into use normally.

[0068] In one example, the input image may be classified by a style classification model to obtain the style type of the input image, wherein the style classification model may be a machine learning model for performing style classification on an image.

[0069] Exemplarily, the above-mentioned style classification model can be used to extract features of the input image to obtain a feature map of the input image, and then predict the probability values ​​of the input image under different style types based on the feature map, and determine the style type with the largest probability value as the style type of the input image.

[0070] Optionally, the style classification model can be constructed based on at least one of the following: CNN, DNN, DenseNet, ResNET, VGGNet, StyleGAN2, ViT, EfficientNet. After the style classification model is constructed, the style classification model can be iteratively trained using at least two images of different style types to obtain a trained style classification model.

[0071] For example, refer to Figure 3 , a trained style classification model 302 is also deployed in the client. When the first anomaly score is greater than or equal to 0.7, the client determines that there may be an abnormal object in the input image, and then calls the style classification model 302 to classify the input image to obtain the style type of the input image. For example, when the style classification model 302 is used to determine whether the image is realistic or fictional, if the probability value of the input image under the realistic style is greater than the probability value of the input image under the fictional style, the client can determine the style type of the input image as realistic.

[0072] Step 203: Perform anomaly detection on the object in the input image by using a classification model that matches the style type to obtain a second anomaly score, where the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image.

[0073] At least two classification models can be deployed in the client, and each classification model can be used to perform anomaly detection on objects in the input image, but different classification models can be trained based on images of different styles. For example, a classification model that matches a realistic style can be trained based on images of a realistic style, so that the classification model focuses on abnormal features that are unique to the realistic style. Abnormal features can be features that indicate abnormalities, such as abnormal features corresponding to multiple hands and multiple heads that can be used to characterize multiple hands and multiple heads. For another example, a classification model that matches a fictional style can be trained based on images of a fictional style, so that the classification model focuses on abnormal features that are unique to the fictional style, which is conducive to improving the accuracy of determining abnormal images.

[0074] Optionally, each classification model has the same architecture but different parameters to adapt to processing images of different style types. The classification model can be constructed based on at least one of the following: CNN, DNN, DenseNet, ResNET, VGGNet, StyleGAN2, ViT, EfficientNet, Transformer (a deep learning model based on self-attention mechanism). After the classification model is constructed, the classification model can be iteratively trained using images of a single style type to obtain a trained classification model. Exemplarily, the classification model can also be constructed using a lightweight CNN or a smaller Transformer (i.e., a model with a small number of parameters).

[0075] After obtaining the style type of the input image, the client can match the style type of the input image with the style type corresponding to the classification model (i.e., the style type of the image used to train the classification model) to determine the classification model that can be used to process the input image. The second anomaly score may refer to the anomaly score output by the classification model.

[0076] Exemplarily, the above classification model can be used to extract features from the input image, obtain a feature map of the input image, and then predict the probability value of the object in the input image being an abnormal object based on the feature map, thereby realizing anomaly detection. Among them, the probability value predicted by the classification model is the second anomaly score. In a feasible example, the classification model can also be used to perform binary classification on the input image, such as output 1 represents that the object in the input image is abnormal, and output 0 represents that the object in the input image is not abnormal. Optionally, when there are multiple objects in the input image, the classification model can obtain the anomaly scores of the multiple objects respectively, and determine the highest anomaly score as the second anomaly score.

[0077] For example, refer to Figure 3, a plurality of trained classification models 303 are deployed in the client, each classification model 303 corresponds to a different style type. After obtaining the style type of the input image, the client can select a classification model 303 that matches the style type of the input image from the plurality of classification models 303, and perform anomaly detection on the object in the input image through the classification model 303 to obtain a second anomaly score.

[0078] In a feasible example, at least two classification models are used to perform anomaly detection on objects in the input image to obtain an anomaly score set, and a weighted sum is performed on at least two anomaly scores in the anomaly score set to obtain a second anomaly score.

[0079] Among them, the weight parameter of the anomaly score corresponding to the classification model matching the style type is greater than 0.5. In this way, while the anomaly score corresponding to the classification model matching the style type is dominant, the anomaly scores corresponding to the classification models matching other style types can be taken into account, thereby improving the comprehensiveness of the second anomaly score, which is beneficial to improving the accuracy of the second anomaly score.

[0080] Step 204: When the second abnormality score is greater than or equal to the second threshold, the input image is determined to be an abnormal image with an abnormal object.

[0081] The second threshold is also used to determine whether there is an abnormal object in the input image, which can be set and adjusted according to actual use requirements, and the embodiment of the present application does not limit this. The abnormal image may refer to an image with an abnormal object.

[0082] For example, when the second anomaly score is a probability value, the second threshold value may be set to 0.8. When the second anomaly score is greater than or equal to 0.8, the client may determine that an abnormal object may exist in the input image. When the second anomaly score is less than 0.8, the client may determine that there is no abnormal object in the input image, that is, the client may determine that the input image is a normal image and can be put into use normally.

[0083] In one example, different classification models may correspond to different second thresholds. Since different style types have different tolerances to abnormality, different second thresholds may be set for different style types, which is conducive to further improving the accuracy of determining abnormal images.

[0084] For example, since the tolerance of the fictional style to the degree of abnormality is greater than that of the realistic style, the second threshold corresponding to the classification model for processing realistic style images can be set to be greater than the second threshold corresponding to the classification model for processing fictional style images. For example, the second threshold corresponding to the classification model for processing realistic style images can be set to 0.8, and the second threshold corresponding to the classification model for processing fictional style images can be set to 0.6.

[0085] Optionally, when the second anomaly score is greater than or equal to the second threshold corresponding to the classification model, the client determines the input image as an abnormal image with an abnormal object. For example, when the style type of the input image is realistic, if the second anomaly score is greater than or equal to 0.8, the client determines the input image as an abnormal image with an abnormal object. When the style type of the input image is fictional, if the second anomaly score is greater than or equal to 0.6, the client determines the input image as an abnormal image with an abnormal object.

[0086] When the second anomaly score is less than the second threshold corresponding to the classification model, the client determines the input image as a normal image, that is, it can be put into use.

[0087] In summary, the technical solution provided by the embodiment of the present application, based on the input image screened out according to the first anomaly detection, performs anomaly detection on the object in the input image again through the classification model matching the style type of the input image, so as to further determine whether the input image is an abnormal image based on the second anomaly check, which is conducive to improving the detection accuracy of abnormal images compared to using only a single classification model. At the same time, since images of different style types contain different features and details, the classification model matching the style type is used to perform anomaly detection again, so that the classification model is consistent with the input image, thereby reducing the impact of the features and details under the style type on the prediction results of the classification model, which is conducive to further improving the detection accuracy of abnormal images.

[0088] In some embodiments, after detecting an abnormal image, the present application embodiment also supports repairing the abnormal image, referring to Figure 4 , the embodiment of the present application may also include the following steps:

[0089] Step S1, identifying an abnormal region in an abnormal image, where the abnormal region is used to indicate the area occupied by an abnormal part of an abnormal object in the abnormal image.

[0090] An abnormal image may refer to an image in which an abnormal object exists. The abnormal image and the abnormal object are the same as those described in the above embodiment. The contents not described in the embodiments of the present application may refer to the above embodiment and will not be repeated here.

[0091] The abnormal part may refer to the abnormal part of the abnormal object. For example, if the head of the abnormal object is abnormal, the area occupied by the head of the abnormal object in the abnormal image may be determined as the abnormal area. Optionally, there may be multiple abnormal areas in the abnormal image, such as multiple abnormal areas corresponding to the same abnormal object or different abnormal areas corresponding to different abnormal objects. Different abnormal areas are used to indicate different abnormal parts.

[0092] Optionally, the abnormal region can be referred to by the coordinates of the abnormal region in the abnormal image, such as the coordinates of the center point of the abnormal region, and the width and height of the abnormal region, or the coordinates of the upper left corner of the abnormal region, and the coordinates of the lower right corner of the abnormal region. The embodiment of the present application does not limit this. The embodiment of the present application does not limit the shape of the abnormal region, such as the abnormal region can be in a square style or a circular style.

[0093] In one example, image segmentation technology or image detection technology can be used to identify abnormal regions in abnormal images. Exemplarily, image segmentation technology or image detection technology can be used to build a region recognition model, and then the trained region recognition model can be used to identify abnormal regions in abnormal images. The region recognition model can refer to a machine learning model for identifying abnormal parts in an image.

[0094] Among them, in the case of using image segmentation technology to build a region recognition model, the machine learning model that can be used to build the region recognition model includes at least one of the following: CNN, DNN, U-Net (U-type network, a deep learning model for image segmentation tasks), SegNet (a deep learning model for image semantic segmentation), TCNN (Tweaked CNN, adjusted CNN), DAN (Deep Alignment Networks), DeepLab (a deep learning model using void convolution). In the case of using image detection technology to build a region recognition model, the machine learning model that can be used to build the region recognition model includes at least one of the following: Mask R-CNN (Mask Region-based CNN, a deep learning model).

[0095] Among them, U-Net is an architecture widely used in medical image segmentation, suitable for pixel-level prediction of object areas. U-Net retains low-level detail information through skip connections, which helps to improve segmentation accuracy. Mask R-CNN is based on the instance segmentation method of Faster R-CNN. It not only performs target detection (locating objects), but also segments each target area, which is suitable for accurate detection of target areas. DeepLab uses dilated convolutions to enhance the receptive field of the model and can capture more contextual information, which is suitable for segmentation tasks.

[0096] After the region recognition model is constructed, the parameters of the region recognition model are iteratively adjusted using the training data to obtain the trained region recognition model. The training data includes normal images and abnormal images, and each abnormal image corresponds to the position of the abnormal region marked. For image segmentation technology, it is performed by marking the segmentation mask map; for image detection technology, it is performed by marking the bounding box. Both the segmentation mask map and the bounding box are used to indicate the area occupied by the abnormal part in the abnormal region.

[0097] Optionally, the loss function used to train the region recognition model may include at least one of the following: a cross-entropy loss function (Cross-Entropy Loss), a Dice coefficient loss function, and a smooth L1 loss function (Smooth L1Loss). Among them, the cross-entropy loss function is suitable for pixel-level segmentation tasks, especially binary classification tasks (such as abnormal areas and non-abnormal areas). The Dice coefficient loss function can be used to optimize overlapping areas in image segmentation tasks, especially for unbalanced categories (such as abnormal areas account for a small proportion of the entire abnormal image). The smooth L1 loss function is suitable for regression tasks, such as bounding box regression. The loss function used to train the region recognition model can be set and adjusted according to actual use requirements, and the embodiments of the present application are not limited to this.

[0098] Optionally, the training optimization method of the region recognition model includes at least one of the following: Adam (Adaptive Moment) optimizer (an efficient optimization algorithm suitable for parameter update in deep learning) and SGD (Stochastic Gradient Descent) optimizer. The performance of the region recognition model can be improved by adjusting hyperparameters such as learning rate and batch size.

[0099] Optionally, data enhancement (such as rotation, scaling, flipping, etc.) may be performed on the images included in the training data to improve the generalization ability of the region recognition model.

[0100] In one example, before the abnormal region in the abnormal image is identified by the region recognition model, the abnormal image can also be preprocessed. The preprocessed abnormal image is consistent with the image included in the training data in terms of size, number of channels and other formats, so that the abnormal image is suitable for reasoning of the region recognition model.

[0101] Exemplarily, the preprocessing includes at least one of the following:

[0102] 1. Size standardization: Adjust the abnormal images to a uniform size to meet the input requirements of the region recognition model.

[0103] 2. Normalization: Normalize the pixel values ​​of the abnormal image to the interval [0, 1] or [-1, 1], which can be set and adjusted according to actual usage requirements.

[0104] In one example, when a region recognition model is constructed using image segmentation technology, the region recognition model can identify a segmentation mask image of the same size as the abnormal image, and each pixel in the segmentation mask image has a value (such as between 0 and 1) to indicate whether the location belongs to the abnormal area.

[0105] For example, taking the construction of a region recognition model based on U-Net as an example, the region recognition model extracts the features of the abnormal image through an encoder (such as a convolution layer and a pooling layer), and then restores the spatial resolution of the abnormal image through a decoder (up-sampling layer), thereby achieving pixel-level segmentation to obtain a segmentation mask map. The region in the segmentation mask map where the pixel value is greater than a threshold (such as about 0.5) can be determined as an abnormal region, and the boundary of the abnormal region can also be obtained by extracting the contour, which is not limited in the embodiments of the present application.

[0106] When the region recognition model is constructed using image detection technology, the region recognition model can identify a bounding box, and the area corresponding to the bounding box is the abnormal area. For example, taking the construction of a region recognition model based on Mask R-CNN as an example, the region recognition model will simultaneously output the bounding box and segmentation mask map of the detected abnormal part, and then determine the abnormal area based on the bounding box and the segmentation mask map, such as determining the coordinates of the abnormal area by parsing the coordinates of the bounding box, such as the coordinates of the upper left corner and the lower right corner of the bounding box.

[0107] refer to Figure 5 After obtaining the abnormal image, the client can identify the abnormal area in the abnormal image through the area recognition model 501.

[0108] Step S2: Generate prompt information based on the abnormal area, where the prompt information is used to guide the repair of the abnormal part in the abnormal area.

[0109] In image generation and image restoration tasks, prompt information is a text description provided to the image generation model to guide the generation of the image or the restoration of the image. In an embodiment of the present application, the prompt information can be used to describe how to repair the abnormal area. Exemplarily, the prompt information can be a text describing how to repair the abnormal area, such as "repair eye deformity", "restore facial proportions", "repair facial asymmetry", etc. The image generation model generates or redraws the abnormal area based on the prompt information to ensure that the repaired abnormal part meets the expected repair standard.

[0110] The embodiments of the present application do not limit the architecture of the image generation model, such as diffusion models (such as SD (Stable Diffusion) Model), Transformer, CLIP (Contrastive Language-Image Pre-training, a text-image pre-training model based on contrastive learning), and any machine learning model that can generate images based on prompt information.

[0111] In one example, the prompt information can be generated by analyzing the features in the abnormal area and combining with the predefined template. Exemplarily, the prompt information can be used to indicate at least three types of information: the repair method of the abnormal image, the repair amplitude of the abnormal image, and the repair area of ​​the abnormal image. Among them, the repair method is used to indicate the method of repairing the abnormal part, the repair amplitude is used to indicate the amplitude of adjusting the abnormal part, and the repair area is used to indicate the abnormal part. For example, for the prompt information "fine-tune the facial proportion", the repair area is "face", the repair method is "adjust the proportion", and the repair amplitude is "fine-tune".

[0112] The above abnormal area can be used to determine the repair area of ​​the abnormal image, for example, the abnormal area can be directly determined as the repair area. The repair area is used to indicate the area to be repaired in the abnormal image. The client can determine the repair method and repair range by analyzing the features in the abnormal area.

[0113] Exemplarily, the embodiments of the present application may also include the following contents:

[0114] 1. Classify the abnormal image to obtain a classification result, which is used to indicate the type of abnormal object in the abnormal image.

[0115] Optionally, the objects in the abnormal image are classified by an object classification model to obtain a classification result. The object classification model can be constructed based on at least one of the following: CNN, ResNet, Inception (a deep learning model based on convolutional neural network), EfficientNet. The object classification model can be trained by multiple classification methods so that it can output the probability value of each type, and the type with the largest probability value is the classification result.

[0116] For example, the object classification model can simultaneously predict the probability values ​​corresponding to the three types of "person", "animal" and "plant". If the probability value of the "person" type is the largest, the classification result of the abnormal image can be determined as the "person" type, that is, the type of the abnormal object is the "person" type.

[0117] The classification results can be used to determine the target area for subsequent feature point extraction. For example, for the "person" type, the client can use the feature point extraction model used to extract the feature points of the "person", for the "animal" type, the client can use the feature point extraction model used to extract the feature points of the "animal", and for the "plant" type, the client can use the feature point extraction model used to extract the feature points of the "plant". Different types use feature point extraction models that match them to extract feature points, which is beneficial to improving the accuracy of feature point extraction, and thus helps to improve the accuracy of determining abnormality types and abnormality levels.

[0118] refer to Figure 5 After obtaining the abnormal image, the client can classify the abnormal image through the object classification model 504 to obtain the classification result.

[0119] 2. Extract feature points of the abnormal image through a feature point extraction model that matches the classification result to obtain a feature point set, which includes feature points of the abnormal object.

[0120] At least two feature point extraction models can be deployed in the client, and each feature point extraction model can be used to extract feature points in an image, but different feature point extraction models can be trained based on different types of images. For example, a feature point extraction model that matches the "person" type can be trained based on an image of the "person" type (i.e., the object in the image is a "person"), so that the feature point extraction model focuses on feature points that are specific to the "person" type. For another example, a feature point extraction model that matches the "animal" type can be trained based on an image of the "animal" type (i.e., the object in the image is an "animal"), so that the feature point extraction model focuses on feature points that are specific to the "animal" type, which is conducive to improving the accuracy of feature point extraction.

[0121] Optionally, each feature point extraction model has the same architecture but different parameters to adapt to processing different types of images. The feature point extraction model can be constructed based on at least one of the following: CNN, DNN, DenseNet, ResNET, VGGNet, StyleGAN2, ViT, EfficientNet, Transformer (a deep learning model based on self-attention mechanism).

[0122] Optionally, the architecture and parameters of each feature point extraction model are different to adapt to processing different types of images. Exemplarily, for the "person" type, a CNN-based human posture estimation model can be used to build a feature point extraction model, such as OpenPose (a multi-level neural network structure that can be used to extract the positions of key points of the human body), HRNet (High-Resolution Network, a deep learning model for visual tasks), etc. For the "animal" type, a deep learning-based animal posture estimation model can be used to build a feature point extraction model. For the "plant" type, a deep learning-based plant posture estimation model can be used to build a feature point extraction model.

[0123] After the feature point extraction model is constructed, a single type of image may be used to iteratively train the feature point extraction model to obtain a trained feature point extraction model.

[0124] Feature points can refer to key points of an object, such as joints of a person, endpoints of an animal's limbs, main parts of a plant (such as roots, trunks, branches and leaves), etc. Feature points can be represented by the coordinates of the key points in the abnormal image.

[0125] The feature point extraction model that matches the classification result may refer to a feature point extraction model trained using an image that matches the classification result (i.e., type). After obtaining the classification result of the abnormal image, the client may match the classification result of the abnormal image with the type corresponding to the feature point extraction model to determine a feature point extraction model that can be used to process the abnormal image.

[0126] refer to Figure 5 , a plurality of trained feature point extraction models 502 are deployed in the client, each feature point extraction model 502 is used to extract feature points of different types of objects. After obtaining the classification result of the abnormal image (i.e., the type of the abnormal object), the client can select a feature point extraction model 502 that matches the classification result of the abnormal image from the plurality of feature point extraction models 502, and extract feature points of the abnormal image through the feature point extraction model 502 to obtain a feature point set, which includes the feature points of the abnormal object.

[0127] 3. Based on the feature point set, a determination result is obtained, and the determination result is used to indicate the abnormal type of the abnormal object.

[0128] Optionally, based on the feature points in the feature point set, the client can determine whether there is an abnormality in the posture or form of the object in the abnormal image, and determine the abnormal type of the abnormal object based on the posture or form of the abnormal object. Exemplarily, the abnormal type may include at least one of the following: object deformity (such as multiple hands and feet), limb disharmony (such as facial disharmony), loss of details (such as lack of content described by text), inconsistent with user expectations, and object generation error.

[0129] In one example, the abnormal object can be subjected to posture determination according to the feature point set to determine the abnormal type of the abnormal object and obtain a determination result. Exemplarily, posture determination can be performed based on coordinate relationships and geometric constraints. For example, the rationality of the posture of the abnormal object is determined by calculating the angle, distance or proportional relationship between the joints based on the key points to determine whether the limbs are not coordinated. It is also possible to determine whether the object is deformed (such as multiple hands and feet) by counting the number and relative position of the joint points according to the key points. It is also possible to combine a deep learning model, such as using a trained posture detection network (such as OpenPose) combined with rules to analyze the spatial distribution of feature points to determine the abnormal type of the abnormal object. The present application embodiment does not limit the method for determining the abnormal type.

[0130] 4. Determine the abnormality level based on the judgment result. The abnormality level is used to indicate the degree of abnormality of the abnormal object.

[0131] Optionally, the abnormality degree of the abnormal object is determined according to the determination result, and then the abnormality level is determined according to the corresponding relationship between the abnormality degree and the abnormality level. The abnormality level includes at least: severe abnormality, moderate abnormality and mild abnormality.

[0132] Exemplarily, the abnormality level can be determined by a set of predefined rules. For example, if additional limbs are found, the degree of abnormality can be determined based on the number and position of the additional limbs, and the abnormality level can be determined based on the degree of abnormality. Different abnormality types can also be trained by a classifier, and the trained classifier can output an abnormality level based on the specific situation of the image (such as the number and position of the limbs). It is also possible to combine a regression model to output a specific score based on the abnormality level of the abnormal object, and determine the abnormality level based on the score.

[0133] Optionally, the abnormality level may provide a final decision for an application layer (eg, an alarm system), such as determining whether the abnormal image needs to be repaired and the extent of the repair of the abnormal image.

[0134] In one example, after obtaining the judgment result, abnormality level and abnormal area, the client can determine the repair method of the abnormal image based on the judgment result; determine the repair extent of the abnormal image based on the abnormality level; determine the repair area of ​​the abnormal image based on the abnormal area; and generate prompt information based on the repair method, repair extent and repair area.

[0135] Among them, the abnormal type of the abnormal object can be determined based on the judgment result, and then the repair method of the abnormal image can be determined. Exemplarily, if the abnormal type is object deformity, the repair method can be to repair the deformity. For example, if the abnormal object has multiple hands, the repair method is to reduce the number of hands. The degree of abnormality of the abnormal object can be determined based on the abnormality level, and then the repair range of the abnormal image can be determined. Exemplarily, if the limbs of the abnormal object are seriously uncoordinated, the repair range is to significantly adjust the proportions between the limbs. Based on the abnormal area, the target to be repaired (i.e., the abnormal part) and the position of the target can be determined.

[0136] For example, the preset template of the prompt information may be: using a repair method to repair the repair area according to the repair range. Figure 5 ,The client generates prompt information based on the determination result, abnormal level and abnormal ,area of ​​abnormal images.

[0137] The embodiment of the present application helps to improve the accuracy of repairing abnormal images by accurately generating prompt information based on the abnormality type, abnormality level and abnormal area.

[0138] Step S3, based on the prompt information and the abnormal area, a local image is generated by an image generation model, and the local image includes the repaired abnormal part.

[0139] Optionally, based on the prompt information and the abnormal area, the image generation model can be guided to perform local repair only in the abnormal area without repairing the entire abnormal image, thereby facilitating improving the repair efficiency of the abnormal image.

[0140] The size of the local image may be the same as the size of the abnormal area. It may be obtained by redrawing the abnormal part in the abnormal area, or by repairing the abnormal part in the abnormal area. This embodiment of the present application does not limit this.

[0141] For example, taking the construction of an image generation model based on the SD model as an example, the SD model is a generation model based on a diffusion process. The coded data of the prompt information and the coded data of the abnormal area are input into the SD model. The SD model performs a series of operations on a random noise image, and generates a local image related to the prompt information under the cross-attention of the prompt information and the abnormal area. The local image corresponds to the abnormal area. The SD model will generate a natural restoration result, i.e., a local image, in the abnormal area based on the prompt information, the surrounding background information of the abnormal area, and the local features.

[0142] The SD model includes a diffusion process and a reverse diffusion process. Diffusion process: The SD model first adds noise (Gaussian noise) to a real image (such as an abnormal area in an abnormal image), gradually destroying the content of the image (such as an abnormal area in an abnormal image) until the image completely becomes noise and obtains a noise image. This process helps the SD model learn the characteristics of the image. Reverse diffusion: The SD model starts with a noise image and gradually restores the details of the image through reverse diffusion steps. At each step, the SD model adjusts according to the noise image and the prompt information to generate a local image that matches the prompt information.

[0143] The encoded data of the prompt information is combined with the noisy image in the latent space to guide the local image generation. The SD model does not operate directly in the pixel space, but processes through the latent space. The latent space is a high-dimensional representation of the image, and the image generation process is carried out in the latent space to reduce the computational complexity and improve the quality of image generation.

[0144] For example, refer to Figure 5 Based on the prompt information and the abnormal area, a local image corresponding to the abnormal image is generated through the image generation model 503.

[0145] Step S4, merging the local image into the abnormal region of the abnormal image to obtain a repaired abnormal image.

[0146] Optionally, the merging position of the local image is determined according to the coordinates of the abnormal area, and the local image is merged with the abnormal image at the merging position to obtain a repaired abnormal image.

[0147] For example, refer to Figure 6 The expression of the abnormal object 601 in the abnormal image is not coordinated. The client can generate a local image through an image generation model based on the prompt information of "slightly adjust the facial expression" and the abnormal area used to indicate the facial area. The local image includes the facial area with the expression repaired. The local image is merged with the abnormal image to obtain the abnormal object 602 with the expression repaired, that is, the repaired abnormal image.

[0148] Optionally, after obtaining the repaired abnormal image, the client may also adjust the pixels in the edge area between the local image and the abnormal image to obtain an optimized abnormal image, so as to improve the fusion effect between the local image and the abnormal image, thereby ensuring the consistency between the local image and the abnormal image.

[0149] Among them, the boundary between the local image and the abnormal image is taken as the inner boundary, and the outer boundary is obtained by extending outward by a set length. The area enclosed by the inner boundary and the outer boundary is the edge area, and the set length can be set and adjusted according to actual usage requirements.

[0150] Exemplarily, a refinement technique may be used to adjust pixels in an edge region. The refinement technique may include at least one of the following: edge smoothing, color adjustment. Edge smoothing may refer to a process of reducing the severity of brightness changes in an edge region so that the brightness changes naturally, such as interpolating based on the brightness of pixels on an inner boundary and the brightness of pixels on an outer boundary to obtain new brightness for each pixel in the edge region. Color adjustment may refer to a process of reducing the severity of color changes in an edge region so that the color changes naturally, such as interpolating based on the color of pixels on an inner boundary and the color of pixels on an outer boundary to obtain new colors for each pixel in the edge region.

[0151] In one example, the processing of the input image (such as abnormality detection and repair) is a cyclic process, and the end condition of the cyclic process is: until the input image is detected as a normal image, the processing of the input image is stopped.

[0152] Exemplarily, after obtaining the repaired and optimized abnormal image, the client continues to perform abnormality detection on the objects in the optimized abnormal image. If there are still abnormal objects in the optimized abnormal image, the client continues to repair the optimized abnormal image. If there are no abnormal objects in the optimized abnormal image, the client stops processing the optimized abnormal image and outputs the optimized abnormal image.

[0153] For example, the above general classification model and the above classification model can continue to be used to implement abnormality detection of optimized abnormal images, and the embodiment of the present application can also include the following content:

[0154] 1. Perform anomaly detection on the object in the optimized abnormal image to obtain a third anomaly score, where the third anomaly score is used to indicate the possibility that an abnormal object exists in the optimized abnormal image.

[0155] Optionally, anomaly detection is performed on the object in the optimized abnormal image by using a general classification model to obtain a third anomaly score.

[0156] 2. When the third anomaly score is less than the first threshold, stop processing the optimized abnormal image.

[0157] When the third anomaly score is less than the first threshold, the client may determine that there is no abnormal object in the optimized abnormal image, that is, the abnormal object has been successfully repaired, and the optimized abnormal image may be put into use.

[0158] 3. When the third anomaly score is greater than or equal to the first threshold, the object anomaly in the optimized abnormal image is detected by the classification model to obtain a fourth anomaly score, which is used to indicate the possibility of the presence of an abnormal object in the optimized abnormal image.

[0159] When the third anomaly score is greater than or equal to the first threshold, the client may determine that there may still be an abnormal object in the optimized abnormal image, and further judgment is required through a classification model, wherein the classification model is a classification model that matches the style type of the input image.

[0160] 4. When the fourth abnormality score is less than the second threshold corresponding to the classification model, stop processing the optimized abnormal image.

[0161] When the fourth anomaly score is less than the second threshold corresponding to the classification model, the client can determine that there is no abnormal object in the optimized abnormal image, that is, the abnormal object has been successfully repaired, and the optimized abnormal image can be put into use.

[0162] When the fourth anomaly score is greater than or equal to the second threshold corresponding to the classification model, the client may determine that there are still abnormal objects in the optimized abnormal image, and needs to continue repairing the optimized abnormal image until there are no abnormal objects in the abnormal image.

[0163] The embodiment of the present application reuses the general classification model and the classification model to perform anomaly detection on the optimized abnormal image again until the abnormal image is repaired, which is beneficial to ensuring the repair effect of the abnormal image.

[0164] To sum up, in the technical solution provided in the embodiment of the present application, based on the prompt information and the abnormal area, the image generation model can be guided to perform local repair only in the abnormal area without the need to repair the entire abnormal image, which is beneficial to improving the repair efficiency of the abnormal image.

[0165] In addition, the embodiment of the present application can realize automatic repair of abnormal images without manually selecting from multiple redrawn images, saving manpower and material resources, and helping to reduce the cost of repairing abnormal images. At the same time, compared with the problem of redrawing the input image and causing the entire image effect to be changed, the embodiment of the present application only performs local repair on the input image, which can retain the entire image effect of the input image, thereby helping to improve the repair effect of abnormal images.

[0166] In some embodiments, reference Figure 7 The technical solution provided in the embodiments of the present application may include the following contents:

[0167] 1. The client obtains the input image.

[0168] 2. The client performs anomaly detection on objects in the input image through a general classification model to determine whether there are abnormal objects in the input image, such as objects with multiple hands, multiple feet, or residual limbs.

[0169] 3. When it is detected that an abnormal object may exist in the input image, the client obtains the style type of the input image.

[0170] If the anomaly score output by the general classification model is greater than or equal to 0.7, it can be determined that there may be an abnormal object in the input image. If the anomaly score output by the general classification model is less than 0.7, it can be determined that there is no abnormal object in the input image.

[0171] Optionally, when it is detected that no abnormal object exists in the input image, the client stops processing the input image.

[0172] 4. The client performs anomaly detection on the objects in the input image again through the classification model that matches the style type to determine whether there are actually abnormal objects in the input image.

[0173] If the style type of the input image is realistic, the client will perform anomaly detection on the objects in the input image again through the classification model that matches the realistic style. If the style type of the input image is fictional, the client will perform anomaly detection on the objects in the input image again through the classification model that matches the fictional style.

[0174] 5. When it is detected that there may still be abnormal objects in the input image, the client obtains the abnormal area of ​​the input image and generates prompt information. The abnormal area is used to indicate the area occupied by the abnormal part of the abnormal object in the input image, and the prompt information is used to guide the repair of the abnormal part in the abnormal area.

[0175] If the anomaly score output by the classification model that matches the realistic style is greater than or equal to 0.8, it can be determined that there are indeed abnormal objects in the input image. If the anomaly score output by the classification model that matches the realistic style is less than 0.8, it can be determined that there are no abnormal objects in the input image.

[0176] For example, if the anomaly score output by the classification model matching the fictional style is greater than or equal to 0.6, it can be determined that there are indeed abnormal objects in the input image. If the anomaly score output by the classification model matching the fictional style is less than 0.6, it can be determined that there are no abnormal objects in the input image.

[0177] Optionally, prompt information may be generated based on the abnormal type, abnormal level and abnormal area of ​​the abnormal image.

[0178] Optionally, when it is detected that no abnormal object exists in the input image, the client stops processing the input image.

[0179] 6. Based on the prompt information and the abnormal area, the client redraws the abnormal area through the image generation model to obtain a local image, which includes the repaired abnormal part.

[0180] 7. The client merges the local image into the abnormal area of ​​the input image to obtain the repaired input image.

[0181] After obtaining the repaired input image, the client uses the repaired input image as the input image and re-executes the anomaly detection process to determine whether the input image has been repaired. If the input image has not been repaired, the client continues to repair the input image until the input image has been repaired.

[0182] In summary, the technical solution provided by the embodiment of the present application, based on the input image screened out according to the first anomaly detection, performs anomaly detection on the object in the input image again through the classification model matching the style type of the input image, so as to further determine whether the input image is an abnormal image based on the second anomaly check, which is conducive to improving the detection accuracy of abnormal images compared to using only a single classification model. At the same time, since images of different style types contain different features and details, the classification model matching the style type is used to perform anomaly detection again, so that the classification model is consistent with the input image, thereby reducing the impact of the features and details under the style type on the prediction results of the classification model, which is conducive to further improving the detection accuracy of abnormal images.

[0183] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0184] Please refer to Figure 8 , which shows a block diagram of a video generation device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned video generation method example, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device introduced above, and can also be set in a computer device. The device 800 may include: a first score acquisition module 801, a style type acquisition module 802, a second score acquisition module 803 and an abnormal image determination module 804.

[0185] The first score acquisition module 801 is used to perform anomaly detection on an object in an input image to obtain a first anomaly score, where the first anomaly score is used to indicate the possibility of an abnormal object existing in the input image.

[0186] The style type acquisition module 802 is used to acquire the style type of the input image when the first anomaly score is greater than or equal to a first threshold.

[0187] The second score acquisition module 803 is used to perform anomaly detection on the object in the input image by using a classification model matching the style type to obtain a second anomaly score, where the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image.

[0188] The abnormal image determination module 804 is configured to determine the input image as an abnormal image containing an abnormal object when the second abnormality score is greater than or equal to a second threshold.

[0189] In some embodiments, the abnormal image determination module 804 is used to determine the input image as the abnormal image containing an abnormal object when the second abnormality score is greater than or equal to a second threshold corresponding to the classification model; wherein different classification models correspond to different second thresholds.

[0190] In some embodiments, the first score acquisition module 801 is used to perform anomaly detection on the object in the input image through a general classification model to obtain the first anomaly score; wherein the general classification model is trained based on images of at least two different style types.

[0191] In some embodiments, Fig. 9 As shown, the device 800 further includes: an abnormal region identification module 805 , a prompt information generation module 806 , a local image generation module 807 and an abnormal image restoration module 808 .

[0192] The abnormal region identification module 805 is used to identify the abnormal region in the abnormal image, where the abnormal region is used to indicate the area occupied by the abnormal part of the abnormal object in the abnormal image.

[0193] The prompt information generating module 806 is used to generate prompt information based on the abnormal area, and the prompt information is used to guide the repair of the abnormal part in the abnormal area.

[0194] The local image generation module 807 is used to generate a local image based on the prompt information and the abnormal area through an image generation model, wherein the local image includes the restored abnormal part.

[0195] The abnormal image repair module 808 is used to merge the local image into the abnormal area of ​​the abnormal image to obtain a repaired abnormal image.

[0196] In some embodiments, Fig. 9 As shown, the device 800 further includes: an abnormal image optimization module 809 , a third score acquisition module 810 , and an image stop repair module 811 .

[0197] The abnormal image optimization module 809 is used to adjust the pixels in the edge area between the local image and the abnormal image to obtain an optimized abnormal image.

[0198] The third score acquisition module 810 is used to perform anomaly detection on the object in the optimized abnormal image to obtain a third anomaly score, where the third anomaly score is used to indicate the possibility that an abnormal object exists in the optimized abnormal image.

[0199] The image stop repair module 811 is used to stop processing the optimized abnormal image when the third abnormality score is less than the first threshold.

[0200] In some embodiments, Fig. 9 As shown, the device 800 further includes: a fourth score acquisition module 812.

[0201] The fourth score acquisition module 812 is used to detect the object abnormality in the optimized abnormal image through the classification model when the third abnormality score is greater than or equal to the first threshold, so as to obtain a fourth abnormality score, wherein the fourth abnormality score is used to indicate the possibility of the existence of an abnormal object in the optimized abnormal image.

[0202] The image stop repair module 811 is further configured to stop processing the optimized abnormal image when the fourth abnormality score is less than a second threshold corresponding to the classification model.

[0203] In some embodiments, Fig. 9 As shown, the device 800 further includes: a classification result acquisition module 813 , a feature point extraction module 814 , a determination result acquisition module 815 and an abnormality level determination module 816 .

[0204] The classification result acquisition module 813 is used to classify the abnormal image to obtain a classification result, and the classification result is used to indicate the type of abnormal object in the abnormal image.

[0205] The feature point extraction module 814 is used to extract feature points of the abnormal image by using a feature point extraction model that matches the classification result to obtain a feature point set, wherein the feature point set includes feature points of the abnormal object.

[0206] The determination result acquisition module 815 is used to obtain a determination result according to the feature point set, and the determination result is used to indicate the abnormal type of the abnormal object.

[0207] The abnormality level determination module 816 is used to determine the abnormality level according to the determination result, and the abnormality level is used to indicate the abnormal degree of the abnormal object.

[0208] In some embodiments, the prompt information generating module 806 is further used to:

[0209] Determining a repair method for the abnormal image based on the determination result;

[0210] Determining a repair extent of the abnormal image based on the abnormality level;

[0211] Based on the abnormal area, determining a repair area of ​​the abnormal image;

[0212] The prompt information is generated based on the repair method, the repair range and the repair area.

[0213] In summary, in the technical solution provided by the embodiment of the present application, on the basis of screening out the input image according to the first anomaly detection, the object in the input image is again subjected to anomaly detection by a classification model matching the style type of the input image, so as to further determine whether the input image is an abnormal image based on the second anomaly check, which is conducive to improving the detection accuracy of abnormal images compared to using only a single classification model. At the same time, since images of different style types contain different features and details, anomaly detection is performed again using a classification model matching the style type, so that the classification model is consistent with the input image, thereby reducing the impact of the features and details under the style type on the prediction results of the classification model, which is conducive to further improving the detection accuracy of abnormal images.

[0214] It should be noted that the device provided in the above embodiment only uses the division of the above functional modules as an example to implement its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0215] Please refer to Fig.10 , which shows a schematic diagram of the structure of a computer device provided by an embodiment of the present application. The computer device can be any electronic device with data calculation, processing and storage functions, and the computer device can be implemented as Figure 1The terminal device 10 / server 20 in the implementation environment of the solution shown may specifically include the following contents.

[0216] The computer device 1000 includes a central processing unit (such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and an FPGA (Field Programmable Gate Array)) 1001, a system memory 1004 including a RAM (Random-Access Memory) 1002 and a ROM (Read-Only Memory) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 also includes a basic input / output system (Input / Output System, I / O system) 1006 for helping various devices in the server to transmit information, and a large-capacity storage device 1007 for storing an operating system 1013, application programs 1014 and other program modules 1015.

[0217] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 such as a mouse and a keyboard for user inputting information. The display 1008 and the input device 1009 are connected to the central processing unit 1001 through an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, a printer, or other types of output devices.

[0218] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer readable medium provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0219] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0220] According to the embodiment of the present application, the computer device 1000 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1000 can be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0221] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned image processing method.

[0222] In some embodiments, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0223] Optionally, the computer readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0224] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned image processing method.

[0225] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0226] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing anomaly detection on an object in an input image to obtain a first anomaly score, wherein the first anomaly score is used to indicate the possibility that an abnormal object exists in the input image; When the first anomaly score is greater than or equal to a first threshold, obtaining a style type of the input image; Performing anomaly detection on the object in the input image by using a classification model matching the style type to obtain a second anomaly score, wherein the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image; When the second anomaly score is greater than or equal to a second threshold, the input image is determined to be an abnormal image in which an abnormal object exists.

2. The method according to claim 1, characterized in that The step of determining the input image as an abnormal image with an abnormal object when the second abnormality score is greater than or equal to a second threshold value comprises: When the second anomaly score is greater than or equal to a second threshold corresponding to the classification model, determining the input image as the abnormal image containing an abnormal object; Among them, different classification models correspond to different second thresholds.

3. The method according to claim 1, characterized in that The performing anomaly detection on the object in the input image to obtain a first anomaly score includes: Performing anomaly detection on the object in the input image by a general classification model to obtain the first anomaly score; The general classification model is obtained by training based on at least two images of different styles.

4. The method according to any one of claims 1 to 3, characterized in that: After determining the input image as an abnormal image with an abnormal object, the method further includes: Identify and obtain an abnormal region in the abnormal image, where the abnormal region is used to indicate an area occupied by an abnormal part of the abnormal object in the abnormal image; Based on the abnormal area, generating prompt information, wherein the prompt information is used to guide the repair of the abnormal part in the abnormal area; Based on the prompt information and the abnormal area, a local image is generated by an image generation model, wherein the local image includes the repaired abnormal part; The local image is merged into the abnormal region of the abnormal image to obtain a repaired abnormal image.

5. The method according to any one of claims 1 to 3, characterized in that: After merging the local image into the abnormal region of the abnormal image to obtain a repaired abnormal image, the method further includes: Adjusting pixels in an edge region between the local image and the abnormal image to obtain an optimized abnormal image; Performing anomaly detection on the object in the optimized abnormal image to obtain a third anomaly score, wherein the third anomaly score is used to indicate the possibility that an abnormal object exists in the optimized abnormal image; When the third abnormality score is less than the first threshold, the processing of the optimized abnormal image is stopped.

6. The method according to claim 5, characterized in that The method further comprises: When the third anomaly score is greater than or equal to the first threshold, detecting anomalies of objects in the optimized abnormal image by the classification model to obtain a fourth anomaly score, wherein the fourth anomaly score is used to indicate the possibility that an abnormal object exists in the optimized abnormal image; When the fourth abnormality score is less than the second threshold corresponding to the classification model, the processing of the optimized abnormal image is stopped.

7. The method according to claim 4, characterized in that Before generating prompt information based on the abnormal area, the method further includes: Classifying the abnormal image to obtain a classification result, wherein the classification result is used to indicate the type of abnormal object in the abnormal image; Extracting feature points of the abnormal image by a feature point extraction model matching the classification result to obtain a feature point set, wherein the feature point set includes feature points of the abnormal object; Obtaining a determination result according to the feature point set, wherein the determination result is used to indicate an abnormal type of the abnormal object; According to the determination result, an abnormality level is determined, where the abnormality level is used to indicate the degree of abnormality of the abnormal object.

8. The method according to claim 7, characterized in that The generating prompt information based on the abnormal area includes: Determining a repair method for the abnormal image based on the determination result; Determining a repair extent of the abnormal image based on the abnormality level; Based on the abnormal area, determining a repair area of ​​the abnormal image; The prompt information is generated based on the repair method, the repair range and the repair area.

9. An image processing device, characterized in that: The device comprises: A first score acquisition module, configured to perform anomaly detection on an object in an input image to obtain a first anomaly score, wherein the first anomaly score is used to indicate the possibility of an abnormal object existing in the input image; a style type acquisition module, configured to acquire the style type of the input image when the first anomaly score is greater than or equal to a first threshold; A second score acquisition module, configured to perform anomaly detection on the object in the input image by using a classification model matching the style type to obtain a second anomaly score, wherein the second anomaly score is used to indicate the possibility of the presence of an abnormal object in the input image; The abnormal image determination module is used to determine the input image as an abnormal image with an abnormal object when the second abnormal score is greater than or equal to a second threshold.

10. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program, the computer program is stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the image processing method according to any one of claims 1 to 8.

Images