Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: segmenting a to-be-processed image into a figure image and a background image; identifying a first type of figure image and a second type of figure image in the figure image, the figure definition in the first type of figure image being greater than the figure definition in the second type of figure image; performing definition enhancement processing on the first type of figure image by using a first definition enhancement mode to obtain a first figure enhanced image; and synthesizing the first character enhanced image, the second type of character image and the background image to obtain a target image. In the embodiment of the invention, the definition enhancement processing process is more targeted, the main body figure can be emphasized, and the non-main body figure and the background are prevented from being excessively optimized, so that the image processing effect can be improved, and the image processing accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art

[0002] In image processing technology, it is a common requirement to process the image clarity to make the image clearer. By processing the image clarity, it can help improve the image details, reduce blur, and increase contrast and overall clarity.

[0003] In current image clarity processing technology, the clarity of the image is usually processed evenly to enhance the clarity of the entire image. However, when processing images containing multiple people, this method will result in the characters that need to be highlighted not being highlighted, the characters that do not need to be highlighted or the background being over-optimized, resulting in poor image quality after processing. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an image processing method, device, electronic device and storage medium to improve the effect and accuracy of image definition processing. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, an image processing method is provided, comprising:

[0006] Segment the image to be processed into a person image and a background image;

[0007] Identifying a first type of person image and a second type of person image in the person images, wherein the clarity of the person in the first type of person image is greater than the clarity of the person in the second type of person image;

[0008] Using a first definition enhancement method, performing definition enhancement processing on the first type of person image to obtain a first person enhanced image;

[0009] The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image.

[0010] Optionally, before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes:

[0011] Using a second definition enhancement method, the background image is subjected to definition enhancement processing to obtain a background enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the second definition enhancement method;

[0012] The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image, including: the first person enhanced image, the second type person image and the background enhanced image are synthesized to obtain a target image.

[0013] Optionally, before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes:

[0014] Using a third definition enhancement method, performing definition enhancement processing on the second type of person image to obtain a second person enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the third definition enhancement method;

[0015] The first character enhanced image, the second type character image and the background image are synthesized to obtain a target image, including: when the background enhanced image does not exist, the first character enhanced image, the second character enhanced image and the background image are synthesized to obtain the target image; or when the background enhanced image exists, the first character enhanced image, the second character enhanced image and the background enhanced image are synthesized to obtain the target image.

[0016] Optionally, identifying the first type of person image and the second type of person image in the person images includes:

[0017] Performing face recognition on the person image to obtain a face region image in the person image;

[0018] Obtaining a clarity score of the face region image;

[0019] The person images to which the face area images having a clarity score greater than a preset score belong are taken as the first type of person images, and the remaining other person images are taken as the second type of person images.

[0020] Optionally, obtaining the clarity score of the face area image includes: using an image quality scoring algorithm to score the face area image to obtain the clarity score of the face area image.

[0021] Optionally, after performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes:

[0022] Identifying the gender of the person in the first type of person image;

[0023] A first facial adjustment parameter corresponding to the gender of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the first facial adjustment parameter.

[0024] Optionally, after performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes:

[0025] Identifying the age of the person in the first type of person image;

[0026] A second facial adjustment parameter corresponding to the age of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the second facial adjustment parameter.

[0027] In a second aspect of the present invention, there is also provided an image processing device, comprising:

[0028] A segmentation module, used for segmenting the image to be processed into a person image and a background image;

[0029] A recognition module, used for recognizing a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image;

[0030] A first enhancement module, configured to perform a definition enhancement process on the first type of person image using a first definition enhancement method to obtain a first person enhanced image;

[0031] A synthesis module is used to synthesize the first person enhanced image, the second type person image and the background image to obtain a target image.

[0032] Optionally, the device further comprises:

[0033] A second enhancement module, used to perform a definition enhancement process on the background image using a second definition enhancement method to obtain a background enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the second definition enhancement method;

[0034] The synthesis module is specifically used to synthesize the first person enhanced image, the second type person image and the background enhanced image to obtain a target image.

[0035] Optionally, the device further comprises:

[0036] a third enhancement module, configured to perform a definition enhancement process on the second type of person image by using a third definition enhancement method to obtain a second person enhanced image; wherein the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the third definition enhancement method;

[0037] The synthesis module is specifically used to synthesize the first person enhanced image, the second person enhanced image and the background image to obtain a target image when the background enhanced image does not exist; or to synthesize the first person enhanced image, the second person enhanced image and the background enhanced image to obtain a target image when the background enhanced image exists.

[0038] Optionally, the identification module includes:

[0039] A face recognition unit, used to perform face recognition on the person image to obtain a face region image in the person image;

[0040] A scoring unit, used to obtain a clarity score of the face region image;

[0041] The determination unit is used to take the person image to which the face area image having a clarity score greater than a preset score belongs as the first type of person image, and take the remaining other person images as the second type of person images.

[0042] Optionally, the scoring unit is specifically configured to score the face area image using an image quality scoring algorithm to obtain a clarity score of the face area image.

[0043] Optionally, the device also includes: a first processing module, used to identify the gender of a person in the first type of person image; obtain a first facial adjustment parameter corresponding to the gender of the person, and perform facial adjustment processing on the first person enhanced image according to the first facial adjustment parameter.

[0044] Optionally, the device also includes: a second processing module, used to identify the age of the person in the first type of person image; obtain a second facial adjustment parameter corresponding to the age of the person, and perform facial adjustment processing on the first person enhanced image according to the second facial adjustment parameter.

[0045] In another aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-mentioned image processing methods when executing the program stored in the memory.

[0046] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed, the computer implements any of the above-mentioned image processing methods.

[0047] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to implement any of the above-mentioned image processing methods.

[0048] The image processing method, device, electronic device and storage medium provided by the embodiments of the present invention divide the image to be processed into a character image and a background image; identify a first type of character image and a second type of character image in the character image, wherein the character clarity in the first type of character image is greater than the character clarity in the second type of character image; perform clarity enhancement processing on the first type of character image using a first clarity enhancement method to obtain a first character enhanced image; and synthesize the first character enhanced image, the second type of character image and the background image to obtain a target image. It can be seen that in the embodiments of the present invention, the first type of character image and the second type of character image are divided based on the character clarity, and the first type of character image with higher character clarity is subjected to clarity enhancement processing, so that the clarity enhancement processing process is more targeted, and can focus on highlighting the main character, and avoid over-optimization of non-main character and background, thereby improving the effect of image processing and improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0050] Figure 1 The figure is a flowchart of the steps of an image processing method in an embodiment of the present invention.

[0051] Figure 2 The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0052] Figure 3 The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0053] Figure 4 The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0054] Figure 5 Schematic diagram of an image processing process in an embodiment of the present invention.

[0055] Figure 6 4 is a structural block diagram of an image processing device in an embodiment of the present invention.

[0056] Figure 7 The figure is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0058] Figure 1 The figure is a flowchart of the steps of an image processing method in an embodiment of the present invention.

[0059] like Figure 1 As shown, the image processing method may include the following steps:

[0060] Step 101, segmenting the image to be processed into a person image and a background image.

[0061] In the embodiment of the present invention, the image to be processed refers to an image containing multiple characters.

[0062] In the embodiment of the present invention, any applicable image segmentation method may be used to process the image to be processed, thereby segmenting the image to be processed into a person image and a background image.

[0063] Exemplarily, an image segmentation model for segmenting an image into a person image and a background image may be pre-trained.

[0064] The training process of an image segmentation model may include:

[0065] First, build the image segmentation model to be trained.

[0066] The image segmentation model to be trained can be any neural network model with image segmentation function. For example, the image segmentation model to be trained can include but is not limited to: R-CNN (Region Convolutional Neural Network) series models, U-Net (U-net) models, DeepLab (deep experiment) series models, etc.

[0067] Then, get the sample data.

[0068] The sample data may include a sample image and label information of the sample image, where the label information of the sample image is used to indicate the actual type of pixels in the sample image, where the type may include person pixels and background pixels.

[0069] Next, the image segmentation model to be trained is trained using the sample data, and the trained model is used as the image segmentation model.

[0070] Specifically, the sample image is used as the input of the image segmentation model to be trained, and the output of the image segmentation model to be trained is the predicted type of the pixel in the sample image; based on the output of the image segmentation model to be trained and the label information of the sample image, the loss function of the model is calculated; when the loss function is less than a preset loss threshold, the training is determined to be completed. Exemplarily, the loss function may include but is not limited to: a cross entropy loss function, an absolute value loss function, a square loss function, and the like.

[0071] Exemplarily, after the image segmentation model is trained, the image segmentation model can be used to segment the image to be processed into a person image and a background image.

[0072] Specifically, the image to be processed is used as the input of the image segmentation model to obtain the output of the image segmentation model. The output of the image segmentation model is the predicted type of pixels in the image to be processed. Since the types of pixels are divided into character pixels and background pixels, the character image and background image in the image to be processed can be known based on the output of the image segmentation model.

[0073] Of course, other image segmentation methods may also be used to segment the image to be processed into a person image and a background image, such as an edge-based segmentation method, a region-based segmentation method, etc., which is not limited in this embodiment.

[0074] Step 102 , identifying a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image.

[0075] In the embodiment of the present invention, since the image to be processed contains multiple persons, the segmented person images may be multiple. In order to distinguish the main person and the non-main person in the image to be processed, the first type of person image and the second type of person image in the person image may be further identified based on the person definition. The person definition in the first type of person image is greater than the person definition in the second type of person image. Therefore, it can be considered that the person in the first type of person image is the main person, and the person in the second type of person image is the non-main person.

[0076] In an optional implementation, the process of identifying the first type of person image and the second type of person image in the person image may include the following steps A1 to A3:

[0077] Step A1, performing face recognition on the person image to obtain a face area image in the person image.

[0078] In the embodiment of the present invention, face recognition can be performed for each person image to obtain a face region image in each person image. For any person image, face recognition can be performed for the person image to obtain a face frame in the person image, so that the face frame can be intercepted as the face region image in the person image. Of course, feature data and other information of the face in the person image can also be obtained through face recognition.

[0079] In the embodiment of the present invention, any applicable face recognition method can be used to perform face recognition on a person image, thereby obtaining a face region image in the person image.

[0080] Exemplarily, the face recognition method used in the embodiments of the present invention may include but is not limited to: RetinaFace algorithm, CNN (Convolutional Neural Network) algorithm, FaceNet algorithm, and the like.

[0081] Step A2, obtaining a clarity score of the face region image.

[0082] In an optional implementation, considering that the subject person in the foreground of the image has a higher shooting quality when shooting an image, the subject person has a higher definition, so the difference in image quality can be used to distinguish the subject person from the non-subject person. Therefore, the image quality scoring algorithm can be used to score the face area image to obtain the definition score of the face area image. Specifically, the face area image is used as the input of the image quality scoring algorithm to obtain the definition score of the face area image output by the image quality scoring algorithm after calculation.

[0083] In the embodiment of the present invention, any applicable image quality scoring algorithm may be used to score the face region image, thereby obtaining a clarity score of the face region image.

[0084] Exemplarily, the image quality scoring algorithm used in the embodiment of the present invention may include but is not limited to: Laplacian variance quality scoring algorithm, EOG (Energy of Gradient) algorithm, Tenengrad gradient algorithm, Brenner gradient algorithm, and the like.

[0085] Take the Laplacian variance quality scoring algorithm as an example. The Laplacian variance quality scoring algorithm is an image quality assessment method based on image processing. It combines the concepts of Laplacian operator and variance to evaluate the clarity and contrast of an image.

[0086] The Laplacian operator is a second-order differential operator used to calculate the second-order derivative of an image, thereby highlighting the edge and contour information in the image. In image processing, the Laplacian operator is often used for edge detection and image sharpening. When the image is clear and has rich edge information, the image processed by the Laplacian operator will show obvious edge features, with pixel values ​​close to extreme values ​​at the edges and relatively smooth in non-edge areas.

[0087] Variance is a statistic that measures the degree of discreteness of data. In image processing, it is often used to evaluate the clarity and contrast of an image. For grayscale images, the larger the variance, the greater the discreteness of the pixel values ​​in the image, that is, the wider the grayscale distribution in the image, which usually means that the image contains more details and edge information, so the image is clearer.

[0088] The implementation steps of the Laplacian variance quality scoring algorithm are as follows:

[0089] (1) Image preprocessing: First, the image is converted into a grayscale image for subsequent processing.

[0090] (2) Applying the Laplacian operator: Applying the Laplacian operator to the grayscale image to obtain a processed image. Specifically, the Laplacian function in an image processing library (such as OpenCV) can be used to implement this.

[0091] (3) Calculate variance: Calculate the variance of the image after Laplacian processing. The variance can be obtained by traversing the image pixel values ​​and calculating the average of the squares of the differences between them and the mean.

[0092] (4) Scoring: Score the image quality based on the calculated variance value. Generally speaking, the larger the variance value, the clearer the image and the higher the quality score. The specific scoring criteria can be determined based on the actual application scenario.

[0093] Step A3, taking the person images to which the face area images having a clarity score greater than a preset score belong as the first type of person images, and taking the remaining other person images as the second type of person images.

[0094] Since the higher the clarity score of the face region image, the more important the person in the face region image is, the person images to which the face region images with a clarity score greater than the preset score belong can be taken as the first type of person images, and the remaining other person images can be taken as the second type of person images. The specific value of the preset score can be set according to actual needs, and this embodiment does not impose any limitation on this.

[0095] Step 103: Using a first definition enhancement method, perform definition enhancement processing on the first type of person image to obtain a first person enhanced image.

[0096] Since the definition of the person in the first type of person image is relatively high, it can be considered as the main person. Therefore, in order to highlight the main person, the first definition enhancement method can be used to perform definition enhancement processing on the first type of person image to obtain a first person enhanced image. Among them, the first definition enhancement method can select a face definition enhancement method with a higher definition enhancement strength, so as to perform more optimized definition processing on the main person.

[0097] In the embodiment of the present invention, any applicable definition enhancement method may be used to perform definition enhancement processing on the first type of person image, thereby obtaining a first person enhanced image.

[0098] Exemplarily, the first clarity enhancement method may include, but is not limited to: CodeFormer model, GFP-GAN (Generative Facial Prior Generative Adversarial Networks) model, and the like.

[0099] Take the CodeFormer model as an example. The CodeFormer model can receive blurred, low-resolution or damaged face images as input and generate clearer, high-resolution images. CodeFormer combines the ideas of codebook and Transformer. The model transforms the face super-resolution task into predicting a discrete codebook, encoding the image features into a finite discrete space, greatly reducing uncertainty and reducing dependence on low-resolution images. The model introduces the Transformer module to model global relationships to eliminate the restoration problems that may be caused by low-quality image features in the presence of diverse degradation. Through the modeling capabilities of the Transformer, the global information in the image can be better captured and the restoration quality can be improved. When the CodeFormer model is used, the first type of person image is used as the input of the CodeFormer model. After internal processing of the CodeFormer model, the first person enhanced image can be output.

[0100] In an optional embodiment, in order to further optimize the person in the first person image, after obtaining the first person enhanced image, the gender of the person in the first type of person image can be identified, and the first facial adjustment parameters corresponding to the gender of the person can be obtained, and the first person enhanced image can be subjected to facial adjustment processing according to the first facial adjustment parameters.

[0101] Exemplarily, the first type of person image can be processed by using the person gender recognition model to identify the person gender in the first type of person image. Specifically, the first type of person image is used as the input of the person gender recognition model to obtain the person gender in the first type of person image output by the person gender recognition model.

[0102] The specific form of the person gender recognition model can be selected according to actual needs, and this embodiment does not limit this.

[0103] For example, in a person gender recognition model, the following process can be performed:

[0104] Face detection: The process of quickly locating and extracting the face area in an image. For example, a CNN algorithm based on deep learning can be used.

[0105] Feature extraction: Extract gender-related features from facial images. These features may include the shape, skin color, texture, etc. For example, feature extraction methods such as PCA (Principal Component Analysis) and LDA (Linear Discriminant Analysis) can be used.

[0106] Gender classification: Gender classification is performed using the extracted features. For example, classifiers such as SVM (Support Vector Machines) and neural networks can be used.

[0107] In an embodiment of the present invention, a correspondence between a character's gender and a first facial adjustment parameter may be preset, and the first facial adjustment parameter is used to indicate what kind of adjustment processing is performed on the face. The first facial adjustment parameter may include, but is not limited to, a first facial contour adjustment parameter, a first facial skin adjustment parameter, and the like. The first facial contour adjustment parameter may include, but is not limited to, highlighting the softness of the facial contour, highlighting the facial contour lines, and the like. The first facial skin adjustment parameter may include, but is not limited to, highlighting the smoothness of the facial skin, highlighting the facial skin texture, and the like. For example, a female face highlights the softness of the facial contour and the smoothness of the facial skin, a male face highlights the facial contour lines and the facial skin texture, and the like.

[0108] After identifying the gender of the person, the first facial adjustment parameters corresponding to the gender of the person can be obtained through the above-mentioned correspondence, and the image editing tool can be called to transmit the first facial adjustment parameters corresponding to the gender of the person to the image editing tool, so that the image editing tool performs facial adjustment processing on the first person enhanced image according to the first facial adjustment parameters corresponding to the gender of the person.

[0109] In an optional embodiment, in order to further optimize the person in the first person image, after obtaining the first person enhanced image, the age of the person in the first type of person image can be identified, and the second facial adjustment parameter corresponding to the age of the person can be obtained, and the first person enhanced image can be subjected to facial adjustment processing according to the second facial adjustment parameter.

[0110] Exemplarily, the character age recognition model can be used to perform recognition processing on the first type of character image, so as to recognize the age of the character in the first type of character image. Specifically, the first type of character image is used as the input of the character age recognition model, and the age of the character in the first type of character image output by the character age recognition model is obtained.

[0111] The specific form of the character age recognition model can be selected according to actual needs, and this embodiment does not limit this.

[0112] For example, in the character gender and age model, the following process can be performed:

[0113] Face detection: The process of quickly locating and extracting the face area in an image. For example, a CNN algorithm based on deep learning can be used.

[0114] Feature extraction: Extract age-related features from facial images. These features may include facial contours, the shape and position of facial features, skin texture, etc. For example, feature extraction methods such as PCA (Principal Component Analysis) and LDA (Linear Discriminant Analysis) can be used.

[0115] Age prediction: Use the extracted features to predict age. For example, classifiers such as SVM (Support Vector Machines) and neural networks can be used.

[0116] In the embodiment of the present invention, the correspondence between the character's age and the second facial adjustment parameter can be preset, and the second facial adjustment parameter is used to indicate what kind of adjustment processing is performed on the face. The second facial adjustment parameter may include but is not limited to the second facial skin adjustment parameter. The second facial skin adjustment parameter may include but is not limited to highlighting the smoothness of the facial skin, highlighting the facial skin defects, etc. For example, young people of low and middle age highlight the smoothness and cleanliness of the facial skin, and the elderly of high age highlight the facial skin defects such as spots and wrinkles, etc.

[0117] After identifying the age of the character, the second facial adjustment parameters corresponding to the age of the character can be obtained through the above-mentioned correspondence, and the image editing tool can be called to transmit the second facial adjustment parameters corresponding to the age of the character to the image editing tool, so that the image editing tool performs facial adjustment processing on the first character enhanced image according to the second facial adjustment parameters corresponding to the age of the character.

[0118] Step 104 , synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image.

[0119] In the embodiment of the present invention, any applicable image synthesis method can be used to synthesize the first person enhanced image, the second type person image and the background image to obtain the target image. For example, image editing software, online image synthesis tools, etc. can be used to perform image synthesis processing, which is not limited in this embodiment.

[0120] The specific process of image synthesis can be processed based on actual experience, and this embodiment will not be discussed in detail here.

[0121] In an embodiment of the present invention, first-type character images and second-type character images are divided based on character clarity, and clarity enhancement processing is performed on the first-type character images with higher character clarity. Therefore, the clarity enhancement processing process is more targeted, and can highlight the main character and avoid over-optimization of non-main characters and backgrounds, thereby improving the effect of image processing and improving the accuracy of image processing.

[0122] Figure 2 The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0123] like Figure 2 As shown, the image processing method may include the following steps:

[0124] Step 201, segmenting the image to be processed into a person image and a background image.

[0125] Step 202: identifying a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image.

[0126] Step 203: Perform definition enhancement processing on the first type of person image using a first definition enhancement method to obtain a first person enhanced image.

[0127] Step 204, using a second clarity enhancement method, performing clarity enhancement processing on the background image to obtain a background enhanced image; the clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the second clarity enhancement method.

[0128] In the embodiment of the present invention, considering that the background image in the image to be processed may be blurred, the second definition enhancement method can be used to perform definition enhancement processing on the background image to obtain a background enhanced image.

[0129] The clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the second clarity enhancement method, that is, the background image is slightly enhanced to avoid over-optimization of the background, thereby affecting the highlighting effect of the main character.

[0130] In the embodiment of the present invention, any applicable definition enhancement method may be used to perform definition enhancement processing on the background image, thereby obtaining a background enhanced image.

[0131] Exemplarily, the second clarity enhancement method may include but is not limited to: SwinIR (Swin Transformer for Image Restoration, image restoration based on shift window transform) model, SRCNN (Super-Resolution Convolutional Neural Network, super-resolution convolutional neural network) model, SRGAN (Super-Resolution Generative Adversarial Networks, super-resolution generative adversarial network) model, and the like.

[0132] Taking the SwinIR model as an example, the SwinIR model consists of three parts: shallow feature extraction, deep feature extraction and high-quality image reconstruction module. Among them, the deep feature extraction module consists of several residual Swin Transformer blocks (RSTBs), each of which contains multiple Swin Transformer layers and residual connections, which can effectively capture long-distance dependencies in the image, thereby achieving high-quality image restoration. When the SwinIR model is used, the background image is used as the input of the SwinIR model, and after internal processing of the SwinIR model, the background enhanced image can be output.

[0133] Step 205 , synthesize the first person enhanced image, the second type person image and the background enhanced image to obtain a target image.

[0134] Figure 3The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0135] like Figure 3 As shown, the image processing method may include the following steps:

[0136] Step 301, segmenting the image to be processed into a person image and a background image.

[0137] Step 302: identifying a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image.

[0138] Step 303: Perform definition enhancement processing on the first type of person image using a first definition enhancement method to obtain a first person enhanced image.

[0139] Step 304, using a third clarity enhancement method, performing clarity enhancement processing on the second type of person image to obtain a second person enhanced image; the clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the third clarity enhancement method.

[0140] In the embodiment of the present invention, considering that the second type of person image in the image to be processed may be blurred, the third clarity enhancement method can be used to perform clarity enhancement processing on the second type of person image to obtain a second person enhanced image.

[0141] The clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the third clarity enhancement method. That is, a slight face clarity enhancement method is performed on the second type of person image to avoid over-optimization of the person in the second type of person image, thereby affecting the highlighting effect of the main person.

[0142] In the embodiment of the present invention, any applicable definition enhancement method may be used to perform definition enhancement processing on the second type of person image, thereby obtaining a second person enhanced image.

[0143] Exemplarily, the third clarity enhancement method may include but is not limited to: SwinIR model, SRCNN model, SRGAN model, and the like.

[0144] Step 305 , synthesize the first person enhanced image, the second person enhanced image and the background image to obtain a target image.

[0145] Figure 4 The figure is a flowchart of another image processing method in an embodiment of the present invention.

[0146] like Figure 4 As shown, the image processing method may include the following steps:

[0147] Step 401, segmenting the image to be processed into a person image and a background image.

[0148] Step 402 , identifying a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image.

[0149] Step 403: Perform definition enhancement processing on the first type of person image using a first definition enhancement method to obtain a first person enhanced image.

[0150] Step 404, using a second clarity enhancement method, performing clarity enhancement processing on the background image to obtain a background enhanced image; the clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the second clarity enhancement method.

[0151] Step 405, using a third clarity enhancement method, performing clarity enhancement processing on the second type of person image to obtain a second person enhanced image; the clarity enhancement strength of the first clarity enhancement method is greater than the clarity enhancement strength of the third clarity enhancement method.

[0152] Step 406: synthesize the first person enhanced image, the second person enhanced image and the background enhanced image to obtain a target image.

[0153] for Figure 4 For the specific processing process of each step, please refer to the above Figures 1 to 3 The relevant description in , this embodiment will not be discussed in detail here.

[0154] In an embodiment of the present invention, by performing clarity enhancement processing of different intensities on the first type of character image, the second type of character image, and the background image in the processed image, it is possible to enhance the clarity of the overall image while further highlighting the main character, and maintain the naturalness of non-main characters and the background, thereby avoiding elements other than the main character being too abrupt.

[0155] Figure 5 Schematic diagram of an image processing process in an embodiment of the present invention.

[0156] like Figure 5As shown, the image processing process may include: inputting an image; performing image segmentation to obtain a background image and a character image; using the Laplacian variance quality scoring algorithm to score the clarity of the character image to obtain a main character image (i.e., a first type of character image) and a non-main character image (i.e., a second type of character image); using the CodeFormer model to perform face super-resolution processing on the main character image; using the SwinIR model to perform image super-resolution processing on the background image; and synthesizing the image after face super-resolution processing, the image after image super-resolution processing, and the non-main character image to obtain a target image.

[0157] The image processing method in the embodiment of the present invention can be used in scenarios such as advertising material production, film and television poster production, old drama poster renovation, drama frame extraction image optimization, UGC (User Generated Content) interactive image uploading, etc., to improve the visual quality and clarity of user images, and has wide application value in social media, advertising, personal photography and other industries.

[0158] Figure 6 4 is a structural block diagram of an image processing device in an embodiment of the present invention.

[0159] like Figure 6 As shown, the image processing device may include the following modules:

[0160] The segmentation module 601 is used to segment the image to be processed into a person image and a background image;

[0161] An identification module 602 is used to identify a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image;

[0162] A first enhancement module 603 is used to perform a definition enhancement process on the first type of person image using a first definition enhancement method to obtain a first person enhanced image;

[0163] The synthesis module 604 is used to synthesize the first person enhanced image, the second type person image and the background image to obtain a target image.

[0164] Optionally, the device further comprises:

[0165] A second enhancement module, used to perform a definition enhancement process on the background image using a second definition enhancement method to obtain a background enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the second definition enhancement method;

[0166] The synthesis module 604 is specifically used to synthesize the first person enhanced image, the second type person image and the background enhanced image to obtain a target image.

[0167] Optionally, the device further comprises:

[0168] a third enhancement module, configured to perform a definition enhancement process on the second type of person image by using a third definition enhancement method to obtain a second person enhanced image; wherein the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the third definition enhancement method;

[0169] The synthesis module 604 is specifically used to synthesize the first person enhanced image, the second person enhanced image and the background image to obtain a target image when the background enhanced image does not exist; or to synthesize the first person enhanced image, the second person enhanced image and the background enhanced image to obtain a target image when the background enhanced image exists.

[0170] Optionally, the identification module 602 includes:

[0171] A face recognition unit, used to perform face recognition on the person image to obtain a face region image in the person image;

[0172] A scoring unit, used to obtain a clarity score of the face region image;

[0173] The determination unit is used to take the person image to which the face area image having a clarity score greater than a preset score belongs as the first type of person image, and take the remaining other person images as the second type of person images.

[0174] Optionally, the scoring unit is specifically configured to score the face area image using an image quality scoring algorithm to obtain a clarity score of the face area image.

[0175] Optionally, the device also includes: a first processing module, used to identify the gender of a person in the first type of person image; obtain a first facial adjustment parameter corresponding to the gender of the person, and perform facial adjustment processing on the first person enhanced image according to the first facial adjustment parameter.

[0176] Optionally, the device also includes: a second processing module, used to identify the age of the person in the first type of person image; obtain a second facial adjustment parameter corresponding to the age of the person, and perform facial adjustment processing on the first person enhanced image according to the second facial adjustment parameter.

[0177] In an embodiment of the present invention, first-type character images and second-type character images are divided based on character clarity, and clarity enhancement processing is performed on the first-type character images with higher character clarity. Therefore, the clarity enhancement processing process is more targeted, and can highlight the main character and avoid over-optimization of non-main characters and backgrounds, thereby improving the effect of image processing and improving the accuracy of image processing.

[0178] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701 , a communication interface 702 , a memory 703 and a communication bus 704 , wherein the processor 701 , the communication interface 702 , and the memory 703 communicate with each other via the communication bus 704 .

[0179] Memory 703, used for storing computer programs;

[0180] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0181] Segment the image to be processed into a person image and a background image;

[0182] Identifying a first type of person image and a second type of person image in the person images, wherein the clarity of the person in the first type of person image is greater than the clarity of the person in the second type of person image;

[0183] Using a first definition enhancement method, performing definition enhancement processing on the first type of person image to obtain a first person enhanced image;

[0184] The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image.

[0185] Optionally, before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes:

[0186] Using a second definition enhancement method, the background image is subjected to definition enhancement processing to obtain a background enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the second definition enhancement method;

[0187] The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image, including: the first person enhanced image, the second type person image and the background enhanced image are synthesized to obtain a target image.

[0188] Optionally, before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes:

[0189] Using a third definition enhancement method, performing definition enhancement processing on the second type of person image to obtain a second person enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the third definition enhancement method;

[0190] The first character enhanced image, the second type character image and the background image are synthesized to obtain a target image, including: when the background enhanced image does not exist, the first character enhanced image, the second character enhanced image and the background image are synthesized to obtain the target image; or when the background enhanced image exists, the first character enhanced image, the second character enhanced image and the background enhanced image are synthesized to obtain the target image.

[0191] Optionally, identifying the first type of person image and the second type of person image in the person images includes:

[0192] Performing face recognition on the person image to obtain a face region image in the person image;

[0193] Obtaining a clarity score of the face region image;

[0194] The person images to which the face area images having a clarity score greater than a preset score belong are taken as the first type of person images, and the remaining other person images are taken as the second type of person images.

[0195] Optionally, obtaining the clarity score of the face area image includes: using an image quality scoring algorithm to score the face area image to obtain the clarity score of the face area image.

[0196] Optionally, after performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes:

[0197] Identifying the gender of the person in the first type of person image;

[0198] A first facial adjustment parameter corresponding to the gender of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the first facial adjustment parameter.

[0199] Optionally, after performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes:

[0200] Identifying the age of the person in the first type of person image;

[0201] A second facial adjustment parameter corresponding to the age of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the second facial adjustment parameter.

[0202] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0203] The communication interface is used for communication between the above terminal and other devices.

[0204] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0205] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0206] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer implements the image processing method described in any one of the above embodiments.

[0207] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer implements the image processing method described in any one of the above embodiments.

[0208] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0209] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0210] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0211] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Segment the image to be processed into a person image and a background image; Identifying a first type of person image and a second type of person image in the person images, wherein the clarity of the person in the first type of person image is greater than the clarity of the person in the second type of person image; Using a first definition enhancement method, performing definition enhancement processing on the first type of person image to obtain a first person enhanced image; The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image.

2. The method according to claim 1, characterized in that Before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes: Using a second definition enhancement method, the background image is subjected to definition enhancement processing to obtain a background enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the second definition enhancement method; The first person enhanced image, the second type person image and the background image are synthesized to obtain a target image, including: the first person enhanced image, the second type person image and the background enhanced image are synthesized to obtain a target image.

3. The method according to claim 1 or 2, characterized in that: Before synthesizing the first person enhanced image, the second type person image and the background image to obtain a target image, the method further includes: Using a third definition enhancement method, performing definition enhancement processing on the second type of person image to obtain a second person enhanced image; the definition enhancement strength of the first definition enhancement method is greater than the definition enhancement strength of the third definition enhancement method; The first character enhanced image, the second type character image and the background image are synthesized to obtain a target image, including: when the background enhanced image does not exist, the first character enhanced image, the second character enhanced image and the background image are synthesized to obtain the target image; or when the background enhanced image exists, the first character enhanced image, the second character enhanced image and the background enhanced image are synthesized to obtain the target image.

4. The method according to claim 1, characterized in that: Identifying a first type of person image and a second type of person image in the person images includes: Performing face recognition on the person image to obtain a face region image in the person image; Obtaining a clarity score of the face region image; The person images to which the face area images having a clarity score greater than a preset score belong are taken as the first type of person images, and the remaining other person images are taken as the second type of person images.

5. The method according to claim 4, characterized in that Obtaining a clarity score of the face region image, including: The face region image is scored using an image quality scoring algorithm to obtain a clarity score for the face region image.

6. The method according to claim 1, characterized in that After performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes: Identifying the gender of the person in the first type of person image; A first facial adjustment parameter corresponding to the gender of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the first facial adjustment parameter.

7. The method according to claim 1, characterized in that After performing definition enhancement processing on the first type of person image by using the first definition enhancement method to obtain a first person enhanced image, the method further includes: Identifying the age of the person in the first type of person image; A second facial adjustment parameter corresponding to the age of the person is obtained, and facial adjustment processing is performed on the first person enhanced image according to the second facial adjustment parameter.

8. An image processing device, characterized in that: include: A segmentation module, used for segmenting the image to be processed into a person image and a background image; A recognition module, used for recognizing a first type of person image and a second type of person image in the person images, wherein the person definition in the first type of person image is greater than the person definition in the second type of person image; A first enhancement module, configured to perform a definition enhancement process on the first type of person image using a first definition enhancement method to obtain a first person enhanced image; A synthesis module is used to synthesize the first person enhanced image, the second type person image and the background image to obtain a target image.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images