Image processing method, device and computer readable storage medium
Through the combination of target site detection and image repair model, the blur and distortion problems of the diffusion model in the detailed area processing are solved, and the image generation quality and processing efficiency are improved.
Patent Information
- Application Number
- CN202311641190.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-01
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-12-01
AI Technical Summary
The existing image processing methods based on diffusion model are prone to blur and distortion in smaller detail areas such as the face, resulting in poor image generation quality and thus reducing image processing efficiency.
Through target site detection and style conversion, the target area image is extracted and input into the image repair model is marked, and the target area is extracted using the repair control model to extract feature information for target area repair processing, and the repaired area is fused with the original image to improve the image generation quality.
Effectively repair small areas of the image with poor effects after style conversion, improving image generation quality and processing efficiency.
Smart Images

Figure CN117670737B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to an image processing method, device and computer-readable storage medium. Background Art
[0002] With the rapid development of Internet technology, image processing methods have also developed rapidly. For example, artificial intelligence (AI) technology is applied to image generation. Artificial intelligence models such as diffusion models can be used to generate images of a specific style based on images input by users, thereby improving image processing efficiency.
[0003] In the process of research and practice of existing technologies, it was found that in the images generated by the existing diffusion model-based image processing methods, blurring, distortion and other problems are very likely to occur in smaller detail areas such as faces, resulting in poor image generation quality and low image processing efficiency. Summary of the invention
[0004] The embodiments of the present application provide an image processing method, device, and computer-readable storage medium, which can repair small areas with poor effects in an image after style conversion, improve image generation quality, and thereby improve image processing efficiency.
[0005] The present application provides an image processing method, including:
[0006] Acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame;
[0007] Performing style conversion on the object image to obtain a first object image in a target style;
[0008] Extracting a target area image from the first object image based on the target detection frame and inputting it into an image restoration model, marking a target area where the target part is located in the target area image, and obtaining target area information and inputting it into the image restoration model;
[0009] Extracting feature information of the target part through a repair control model, and obtaining control features to input into the image repair model;
[0010] Performing a target area repair process on the target area in the target area image based on the control feature, the target area image and the target area information by the image repair model to obtain a repaired target area image;
[0011] The restored target area image and the first object image are fused to obtain a restored target object image.
[0012] Accordingly, an embodiment of the present application provides an image processing device, including:
[0013] An information acquisition unit, configured to acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame;
[0014] A style conversion unit, configured to perform style conversion on the object image to obtain a first object image in a target style;
[0015] An image extraction unit, configured to extract a target area image from the first object image based on the target detection frame and input the image into an image restoration model, mark a target area where the target part is located in the target area image, and obtain target area information and input the information into the image restoration model;
[0016] A feature extraction unit, used to extract feature information of the target part through a repair control model, and obtain control features to input into the image repair model;
[0017] An image restoration unit, configured to perform a restoration process on a target area in the target area image based on the control feature, the target area image and the target area information through the image restoration model to obtain a restored target area image;
[0018] The image fusion unit is used to fuse the restored target area image with the first object image to obtain a restored target object image.
[0019] In one embodiment, the target part is a face part, the target part feature information includes facial key point information and a facial image, and the feature extraction unit includes:
[0020] A first feature extraction subunit is used to extract features from the facial image to obtain facial image features and input them into a repair control model;
[0021] The second feature extraction subunit is used to extract features of the facial key point information and the facial image features through the restoration control model.
[0022] In one embodiment, the second feature extraction subunit includes:
[0023] An image acquisition module, used for acquiring a preset image of a target image size;
[0024] A key point drawing module, used for drawing key points in the preset image based on the facial key point information, and obtaining a key point image indicating the position distribution of facial key points in the target part;
[0025] The feature extraction module is used to extract features of the key point image and the facial image by repairing the control model.
[0026] In one embodiment, the image extraction unit includes:
[0027] A cropping subunit, configured to crop an initial region image corresponding to the target part in the first object image based on the target detection frame;
[0028] The image scaling subunit is used to obtain a preset target image size, perform image scaling processing on the initial area image according to the target image size to obtain a target area image, and input the target area image into the image restoration model.
[0029] In one embodiment, the cutting subunit is used to:
[0030] Determining a first target area where the target part is located in the first object image based on the target detection frame;
[0031] Acquire an expansion parameter corresponding to the first target region, and perform region expansion on the first target region in the first object image according to the expansion parameter to obtain an initial region;
[0032] Cropping the initial region in the first object image to obtain an initial region image;
[0033] The key point drawing module is used for:
[0034] Key points are drawn in the preset image based on the facial key point information, the expansion parameters, and the target image size.
[0035] In one embodiment, the image fusion unit is used to:
[0036] Scaling the restored target region image based on the size information of the initial region image to obtain a target region restoration image;
[0037] According to the position information of the initial region image in the first object image, the target part repair image is pasted to the first object image to obtain a repaired target object image.
[0038] In one embodiment, the target part feature information further includes object attribute information, and the image restoration unit is used to:
[0039] Obtaining style description information corresponding to the target style;
[0040] Merging the style description information and the object attribute information to obtain target description information;
[0041] The image restoration model is used to perform restoration processing on the target area in the target area image based on the control feature, the target area image, the target area information and the target description information.
[0042] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any image processing method provided in the embodiment of the present application.
[0043] In addition, an embodiment of the present application also provides a computer device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the image processing method provided in the embodiment of the present application.
[0044] The embodiment of the present application also provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the image processing method provided in the embodiment of the present application.
[0045] The embodiment of the present application acquires an object image, performs target part detection on the object image, obtains a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; performs style conversion on the object image to obtain a first object image under the target style; extracts a target area image from the first object image based on the target detection frame and inputs it into an image restoration model, marks the target area where the target part is located in the target area image, obtains target area information and inputs it into the image restoration model; extracts features of the target part feature information through a restoration control model to obtain control features and inputs them into the image restoration model; performs restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model to obtain a restored target area image; performs image fusion on the restored target area image and the first object image to obtain a restored target object image. In this way, by identifying the target detection frame where the target part is located and the target part feature information in the object image, and extracting the target area image and the target area information indicating the target area to be repaired in the first object image obtained based on the object image style conversion according to the target detection frame, the control feature is extracted from the target part feature information through the repair control model, and at the same time, the image repair model is used to repair the target part of the target area in the target area image based on the control feature, the target area image and the target area information, so that the repaired target area image and the first object image are fused to obtain the repaired target object image, thereby realizing the repair of small areas with poor effect in the image after style conversion, improving the image generation quality, and then improving the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1 It is a schematic diagram of an implementation scenario of an image processing method provided in an embodiment of the present application;
[0048] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;
[0049] Figure 3a It is a schematic diagram of the overall architecture of an image processing method provided in an embodiment of the present application;
[0050] Figure 3bIt is a schematic diagram of a region expansion of an image processing method provided in an embodiment of the present application;
[0051] Figure 4 This is a schematic diagram of an image restoration effect of an image processing method provided in an embodiment of the present application;
[0052] Figure 5 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;
[0053] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0055] The embodiments of the present application provide an image processing method, device and computer-readable storage medium. The image processing device can be integrated in a computer device, which can be a server or a terminal.
[0056] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Terminals may include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. Terminals and servers can be directly or indirectly connected via wired or wireless communications, and this application does not limit this.
[0057] See also Figure 1 , taking the image processing device integrated into a computer device as an example, Figure 1A schematic diagram of an implementation scenario of the image processing method provided in an embodiment of the present application, wherein the computer device can be a server or a terminal, and the computer device can obtain an object image, perform target part detection on the object image, obtain a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; perform style conversion on the object image to obtain a first object image under the target style; extract a target area image from the first object image based on the target detection frame and input it into an image restoration model, mark the target area where the target part is located in the target area image, obtain target area information and input it into the image restoration model; perform feature extraction on the target part feature information through a restoration control model to obtain control features and input them into the image restoration model; perform target part restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model to obtain a restored target area image; perform image fusion on the restored target area image and the first object image to obtain a restored target object image.
[0058] It should be noted that Figure 1 The implementation environment scenario diagram of the image processing method shown is only an example. The implementation environment scenario of the image processing method described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of data processing and the emergence of new business scenarios, the technical solution provided by the present application is also applicable to similar technical problems.
[0059] The solutions provided by the embodiments of the present application are specifically described by the following embodiments. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0060] This embodiment will be described from the perspective of an image processing apparatus. The image processing apparatus may be integrated into a computer device, which may be a server. This application does not limit this.
[0061] See also Figure 2 , Figure 2 : is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method comprises:
[0062] In step 101, an object image is acquired, and target part detection is performed on the object image to obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame.
[0063] Among them, the object image can be an image containing an object, or an image input by a user for style conversion. The object can be a target object in the object image, for example, a person, a real person or a virtual character, an animal, a plant, or an object such as a building or an object. For example, a user can convert the object image into an image in the style of Chinese style animation by inputting object images such as selfies and landscape photos. The target part can be a part of the object in the object image, which is used for image restoration, for example, it can be a part such as a face, a hand, or hair. If the target part accounts for a small proportion in the object image, when the object image is subjected to image generation processing such as style conversion, the area where the target part is located in the generated image is very likely to be blurred, distorted, or distorted, so that the image generation effect is poor. For this reason, the target part in the object image can be image restored to obtain an image with better quality. The target detection box can be a bounding box (BoundingBox, referred to as Bbox) containing the target part detected in the object image. The bounding box can be a directed rectangular box used to represent the position and size of the object in target detection.
[0064] The target part feature information may be information characterizing the features of the target part in the object image. The features of the target part may include information such as the distribution of key points in the target part, image features and attributes of the target part. For example, the target part feature information may include key point information, image features and object attribute information of the target part in the object image. The key point information may be information describing the position distribution of key points of the target part in the object image in the target detection frame. The image features may be information characterizing the image of the area where the target part is located in the object image. The object attribute information may be information describing the properties and relationships of the object in the object image, for example, may include information such as color, gender, age, newness, season, etc. Optionally, the object attribute information may be attribute information of an object identified based on the target part in the object image. For example, object attribute recognition may be performed on the target part in the object image according to the target detection frame to obtain attribute information such as age, gender, newness, etc. of the object in the object image. In one embodiment, when the target part is a human face, the key point information may be information describing the distribution of key points of the human face in the object image, the key points may include key feature points such as eyes, mouth, nose, etc., the image features may be facial features, and the attribute information may be object attribute information detected based on the face in the object image, which may describe the attribute information of the person corresponding to the current face in the object image, for example, may include information such as the gender and age of the object.
[0065] There are many ways to perform target part detection on an object image, obtain a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame. For example, a target detection algorithm can be used to identify a detection frame where the target part is located in the object image to obtain the target detection frame, so that key point detection can be performed on the image contained in the target detection frame to obtain key point information corresponding to the target part, object attribute information of the target part can be identified in the target detection frame, and feature extraction can be performed on the image contained in the target detection frame to obtain image features corresponding to the target part, so that the key point information, object attribute information and image features can be determined as target part feature information.
[0066] In one embodiment, please refer to Figure 3, Figure 3a It is a schematic diagram of the overall architecture of an image processing method provided by an embodiment of the present application, wherein the target part can be a face part, and a face detection module (FaceModule) can be used to perform target part detection on the object image to obtain a face detection frame (Bbox, i.e., a target detection frame), key point information of the face, which can include 5 key points (5key points) such as eyes, nose, and mouth, face features (face embedding), and object attribute information such as the age and gender of the person corresponding to the face.
[0067] In step 102, style conversion is performed on the object image to obtain a first object image in a target style.
[0068] Among them, the target style can be an image style designed according to needs, for example, it can include artistic styles such as cartoon style, character silhouette style, abstract painting, ink painting, and styles designed according to actual needs, etc., which can be set according to actual needs. Style conversion, that is, style migration, can migrate the style of an object image to a target style. For example, a real-life photo input by a user can be converted into a corresponding cartoon image. The first object image can be an object image obtained by converting the object image into a target style. For example, assuming that the user inputs an object image containing a real person a, and the target style is a Chinese-style cartoon style, by performing style conversion on the object image, a first object image containing a Chinese-style cartoon character b obtained by style conversion based on real person a can be obtained.
[0069] There are many ways to perform style conversion on the object image to obtain the first object image under the target style. For example, a style conversion model can be used to perform style conversion on the object image to obtain the first object image under the target style. The style conversion model can be a model for style conversion, and can be a diffusion model, for example, a diffusion model such as Stable Diffusion (SD model, AI animation generation tool). Optional, please continue to refer to Figure 3a , the image-to-image model (SD-img2img) in the SD model can be used to perform style conversion on the object image, etc., and the specific settings can be made according to actual conditions, and the embodiments of the present application are not limited here.
[0070] In step 103, a target area image is extracted from the first object image based on the target detection frame and input into the image restoration model, and a target area where the target part is located in the target area image is marked to obtain target area information and input into the image restoration model.
[0071] Among them, the target area image can be an image containing the target part in the first object image, and can be an image containing an area that needs to be repaired (target part) and a part of the area that does not need to be repaired (surrounding area of the target part), which can facilitate image repair. The image repair model can be a model for repairing the area where the target part is located in the first object image, and can be a model for controlling the image repair model to repair the target area in the target area image. The image repair model can be a model for repairing the image. Optionally, the image repair model can be a diffusion model, for example, it can be an SD model (SD-inpainting) based on the image completion function. The image repair model can use the encoder (Transformer Encoder) of the transformer model as the basic structure of the model. The target area can be the area where the target part is located in the target area image, and the target area information can characterize the position of the target area in the target area image, the area boundary and other information, and the target area where the target part is located can be determined in the target area image based on the target area information. The target area information can be used to indicate the area that needs to be repaired in the target area image by the image repair model.
[0072] There are many ways to extract the target area image from the first object image based on the target detection frame and input it into the image restoration model. For example, an initial area image corresponding to the target part can be cropped out from the first object image based on the target detection frame, a preset target image size can be obtained, and the initial area image can be scaled according to the target image size to obtain the target area image, and the target area image can be input into the image restoration model.
[0073] Among them, the initial area image can be an image containing the target part cropped from the first object image, and the target image size can be the size of the target area image for image restoration. Since the target part in the object image and the first object image occupies a small area, it is necessary to enlarge the cropped initial area image, so that image restoration and generation are performed based on the enlarged initial area image, and a restored image with more delicate details can be obtained, thereby improving the restoration effect. The target image size can be set according to the design situation, and the embodiment of the present application does not limit it here.
[0074] There may be multiple ways to crop an initial area image corresponding to a target part in the first object image based on a target detection frame. For example, a first target area where the target part is located may be determined in the first object image based on the target detection frame, an expansion parameter corresponding to the first target area may be obtained, the first target area may be expanded in the first object image according to the expansion parameter to obtain an initial area, and the initial area in the first object image may be cropped to obtain an initial area image.
[0075] The first target area may be an area determined in the first object image according to the target detection frame, and the image sizes of the first object image and the object image may be the same, so that the first target area may be determined in the first object image according to the position of the target detection frame in the object image. The expansion parameter may be a parameter for expanding the first target area in the first object image. Since the target area image for image restoration needs to include a part of the area that does not need to be restored, that is, the external area of the area where the target part is located, this part of the area does not change during image restoration. Therefore, after image restoration is performed on the area corresponding to the target part, the restored target area image can be easily fused with the first object image based on the unchanged area to obtain the final restored first object image, thereby improving the efficiency of image restoration. The initial area may be an area determined after the first target area is expanded in the first object image.
[0076] Optionally, the expansion parameter may be an offset distance of each boundary of the first target area offset outward in a corresponding vertical direction. For example, please refer to Figure 3b , Figure 3bIt is a schematic diagram of the area expansion of an image processing method provided in an embodiment of the present application. Assuming that the offset distance is d, each boundary of the first target area can be moved outward by a distance d in the vertical direction of the boundary to obtain the initial area. Among them, the offset distance of each boundary can be different or the same, and can be set according to actual needs. Optionally, the first target area can also be expanded in the direction from the center of the first target area to each vertex of the first target area, and other methods can also be used to expand the first target area, which can be set according to actual conditions, and the embodiment of the present application is not limited here.
[0077] After extracting the target area image from the first object image based on the target detection frame and inputting it into the image restoration model, the target area where the target part is located in the target area image can also be marked to obtain the target area information. There are many ways to mark the target area where the target part is located in the target area image. For example, the target area can be the area where the target part is located, that is, the area corresponding to the target detection frame. In this way, the corresponding area of the target detection frame in the target area image can be marked in the target area image to obtain the target area information. For example, please continue to refer to Figure 3a , an image with the same size as the target area image can be obtained, and the pixel value of each pixel in the image is set to 0, that is, black, and the pixel value of each pixel in the target area where the target part is located in the image is set to 1, that is, white, so as to mark the target area, and the image can be used as the target area information.
[0078] In step 104, feature information of the target part is extracted through the restoration control model, and the obtained control features are input into the image restoration model.
[0079] Among them, the control feature can be a feature extracted by the repair control model for controlling the image repair model to perform image repair on the target area image based on the key point information and image features of the target part. Based on the control feature, the image repair model can obtain the relevant feature information of the target part in the object image, so that the target area image can be repaired more accurately and more in line with the object image, thereby improving the image repair effect and image generation quality, and thus improving the image processing efficiency.
[0080] Optionally, the target part feature information may include key point information and image features of the target part in the object image. Therefore, the control feature may be a feature extracted from the key point information and image features, and may include information such as the position distribution of the key points of the target part in the object image and the shape features corresponding to each key point. Thus, based on the control feature, the image restoration model may obtain the position distribution of the key points of the target part in the target area of the target area image to be restored, and may also obtain the shape features corresponding to each key point, such as the curvature of the mouth, the line of sight of the eyes and other information. Therefore, based on the relevant feature information of the target part in these object images, the target area image can be restored more accurately and more closely in line with the object image, thereby improving the image restoration effect.
[0081] Among them, there can be many ways to extract feature information of the target part through the repair control model. For example, the target part can be a face part, and the target part feature information includes facial key point information and facial image. Feature extraction can be performed on the facial image, and the facial image features are input into the repair control model. Feature extraction of facial key point information and facial image features can be performed through the repair control model.
[0082] The facial key point information may be information describing the position distribution of facial key points, the facial image may be an image containing a human face, and the facial image feature may be an image feature of a part of a human face.
[0083] There are many ways to extract facial key point information and facial image features through the repair control model. For example, a preset image of the target image size can be obtained, and key points can be drawn in the preset image based on the facial key point information to obtain a key point image indicating the position distribution of facial key points in the target part. The key point image and facial image features can be extracted through the repair control model.
[0084] The preset image may be a pre-set image, for example, please continue to refer to Figure 3a , the preset image can be an image with a pixel value of 0, that is, a black image, or other types of images, which can be set specifically according to actual needs. The key point image can be an image indicating the position distribution of facial key points in the target part. Optionally, in order to correspond between feature information, the image size of the key point image should be the same as the image size of the target area image. Accordingly, the area where the key points in the key point image are distributed should correspond to the target area where the target part is located in the target area image. Therefore, the area where the corresponding target part is located in the preset image can be determined according to the area where the target part is located in the target area image, so that key points can be drawn in the area where the target part is located in the preset image according to the facial key point information to obtain a key point image.
[0085] Optionally, after expanding the first target area to obtain the target area image, when drawing key points in the preset image based on the facial key point information, the key points can be drawn in the preset image based on the facial key point information, the expansion parameters and the target image size.
[0086] There are many ways to draw key points in a preset image based on facial key point information, expansion parameters and target image size. For example, the image size of the initial area image can be obtained to obtain a first image size, and image scaling parameters can be determined according to the first image size and the target image size. The shrinkage parameters are determined according to the expansion parameters, so that key points can be drawn in the preset image according to the shrinkage parameters and the image scaling parameters, as well as the facial key point information.
[0087] Among them, the image scaling parameter can be a parameter for scaling the initial area image to the target area image, for example, it can be a magnification factor for enlarging the initial area image to the target area image, and the shrinkage parameter can be a parameter for shrinking the area in a preset image, wherein the offset direction of the shrinkage parameter is opposite to the offset direction of the expansion parameter, and the offset distance value of the shrinkage parameter is the same as the offset distance value of the expansion parameter.
[0088] Among them, there can be multiple ways to draw key points in a preset image based on the indentation parameters, image scaling parameters, and facial key point information. For example, the offset distance value of the indentation parameters can be weighted according to the image scaling parameters to obtain a target offset distance value, and the facial key point information can be adjusted according to the image scaling parameters to obtain target facial key point information that describes the position distribution of the facial key points in the enlarged target detection frame. Therefore, regional indentation can be performed in the preset image according to the target offset distance value, and the area corresponding to the enlarged target detection frame can be determined in the preset image. Therefore, key points can be drawn in the preset image according to the position distribution of the target facial key point information in the enlarged target detection frame to obtain a key point image.
[0089] In step 105, the target area in the target area image is repaired by the image repair model based on the control feature, the target area image and the target area information to obtain a repaired target area image.
[0090] The repaired target region image may be a target region image obtained by performing a repair process on a target part of the target region in the target region image.
[0091] Among them, there can be multiple ways to repair the target part of the target area in the target area image through the image repair model based on the control features, the target area image and the target area information. For example, the target part feature information can also include object attribute information, so as to obtain the style description information corresponding to the target style, merge the style description information and the object attribute information to obtain the target description information, and repair the target part of the target area in the target area image through the image repair model based on the control features, the target area image, the target area information and the target description information.
[0092] Among them, the style description information can be description information (prompt) used for style conversion of the target style, and the target description information can be text information obtained by merging the style description information and the object attribute information. Optionally, the style description information and the object attribute information can be spliced to obtain the target description information, etc.
[0093] In one embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of the image restoration effect of an image processing method provided in an embodiment of the present application, including a target area image and a restored target area image. It can be seen that the image processing method provided in an embodiment of the present application can be used to repair target parts with a relatively small proportion, such as distorted, blurred or distorted faces in the first object image, thereby obtaining an image with a better effect, thereby effectively improving the image generation effect and further improving the image processing efficiency.
[0094] In step 106, the restored target area image and the first object image are fused to obtain a restored target object image.
[0095] The target object image may be a first object image in which a target area where a target part is located is repaired.
[0096] There are many ways to perform image fusion of the repaired target area image and the first object image to obtain the repaired target object image. For example, the repaired target area image can be scaled based on the size information of the initial area image to obtain the target part repair image. According to the position information of the initial area image in the first object image, the target part repair image can be pasted to the first object image to obtain the repaired target object image.
[0097] The target part repair image may be a repaired target region image adapted to the image size of the first object image. In order to smoothly fit the repaired target region image to the first object image, the repaired target region image that is enlarged based on the first object image needs to be reduced accordingly so that the image size of the repaired target region image is the same as the size of the initial region image, so that the unrepaired region in the repaired target region image can be smoothly fitted with the first object image, so that the repaired target object image can be obtained, thereby improving the image generation quality.
[0098] From the above, it can be seen that the embodiment of the present application acquires an object image, performs target part detection on the object image, obtains a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; performs style conversion on the object image to obtain a first object image under the target style; extracts a target area image from the first object image based on the target detection frame and inputs it into an image restoration model, marks the target area where the target part is located in the target area image, obtains target area information and inputs it into the image restoration model; extracts features of the target part feature information through a restoration control model to obtain control features and inputs them into the image restoration model; performs restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model to obtain a restored target area image; performs image fusion on the restored target area image and the first object image to obtain a restored target object image. In this way, by identifying the target detection frame where the target part is located and the target part feature information in the object image, and extracting the target area image and the target area information indicating the target area to be repaired in the first object image obtained based on the object image style conversion according to the target detection frame, the control feature is extracted from the target part feature information through the repair control model, and at the same time, the image repair model is used to repair the target part of the target area in the target area image based on the control feature, the target area image and the target area information, so that the repaired target area image and the first object image are fused to obtain the repaired target object image, thereby realizing the repair of small areas with poor effect in the image after style conversion, improving the image generation quality, and then improving the image processing efficiency.
[0099] In order to better implement the above method, an embodiment of the present invention further provides an image processing apparatus, which can be integrated in a computer device, and the computer device can be a server.
[0100] For example, Figure 5As shown, it is a schematic diagram of the structure of the image processing device provided in an embodiment of the present application. The image processing device may include an information acquisition unit 201, a style conversion unit 202, an image extraction unit 203, a feature extraction unit 204, an image restoration unit 205 and an image fusion unit 206, as follows:
[0101] The information acquisition unit 201 is used to acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame;
[0102] A style conversion unit 202, configured to perform style conversion on the object image to obtain a first object image in a target style;
[0103] An image extraction unit 203 is used to extract a target area image from the first object image based on the target detection frame and input it into the image restoration model, mark the target area where the target part is located in the target area image, and obtain target area information and input it into the image restoration model;
[0104] A feature extraction unit 204 is used to extract feature information of the target part through the restoration control model, and obtain control features to input into the image restoration model;
[0105] An image restoration unit 205 is used to perform a restoration process on a target area in the target area image based on the control feature, the target area image and the target area information through an image restoration model to obtain a restored target area image;
[0106] The image fusion unit 206 is used to fuse the restored target area image with the first object image to obtain a restored target object image.
[0107] In one embodiment, the target part is a face part, and the feature information of the target part includes facial key point information and a facial image. The feature extraction unit 204 includes:
[0108] The first feature extraction subunit is used to extract features from the facial image, and obtain facial image features to input into the repair control model;
[0109] The second feature extraction subunit is used to extract features of facial key point information and facial image features by repairing the control model.
[0110] In one embodiment, the second feature extraction subunit includes:
[0111] An image acquisition module, used for acquiring a preset image of a target image size;
[0112] A key point drawing module, used to draw key points in a preset image based on facial key point information, and obtain a key point image indicating the position distribution of facial key points in a target part;
[0113] The feature extraction module is used to extract the key point image and facial image features by repairing the control model.
[0114] In one embodiment, the image extraction unit 203 includes:
[0115] A cropping subunit, configured to crop an initial region image corresponding to a target part in the first object image based on the target detection frame;
[0116] The image scaling subunit is used to obtain a preset target image size, perform image scaling processing on the initial area image according to the target image size, obtain the target area image, and input the target area image into the image restoration model.
[0117] In one embodiment, the cutting subunit is used to:
[0118] Determine a first target area where the target part is located in the first object image based on the target detection frame;
[0119] Acquire an expansion parameter corresponding to the first target area, and expand the first target area in the first object image according to the expansion parameter to obtain an initial area;
[0120] Cropping an initial region in the first object image to obtain an initial region image;
[0121] Key point drawing module, used for:
[0122] Key points are drawn in a preset image based on facial key point information, expansion parameters, and target image size.
[0123] In one embodiment, the image fusion unit 206 is used to:
[0124] Scaling the restored target area image based on the size information of the initial area image to obtain a target area restoration image;
[0125] According to the position information of the initial region image in the first object image, the target part repair image is pasted to the first object image to obtain a repaired target object image.
[0126] In one embodiment, the target part feature information also includes object attribute information, and the image restoration unit 205 is used to:
[0127] Get the style description information corresponding to the target style;
[0128] Merge the style description information and the object attribute information to obtain the target description information;
[0129] The target area in the target area image is repaired by the image repair model based on the control features, the target area image, the target area information and the target description information.
[0130] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0131] As can be seen from the above, the embodiment of the present application obtains an object image through the information acquisition unit 201, performs target part detection on the object image, obtains a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; the style conversion unit 202 performs style conversion on the object image to obtain a first object image under the target style; the image extraction unit 203 extracts a target area image from the first object image based on the target detection frame and inputs it into the image restoration model, marks the target area where the target part is located in the target area image, obtains target area information and inputs it into the image restoration model; the feature extraction unit 204 extracts feature information of the target part through the restoration control model, obtains control features and inputs them into the image restoration model; the image restoration unit 205 performs target part restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model to obtain a restored target area image; the image fusion unit 206 performs image fusion on the restored target area image and the first object image to obtain a restored target object image. In this way, by identifying the target detection frame where the target part is located and the target part feature information in the object image, and extracting the target area image and the target area information indicating the target area to be repaired in the first object image obtained based on the object image style conversion according to the target detection frame, the control feature is extracted from the target part feature information through the repair control model, and at the same time, the image repair model is used to repair the target part of the target area in the target area image based on the control feature, the target area image and the target area information, so that the repaired target area image and the first object image are fused to obtain the repaired target object image, thereby realizing the repair of small areas with poor effect in the image after style conversion, improving the image generation quality, and then improving the image processing efficiency.
[0132] The present application also provides a computer device, such as Figure 6 As shown, it shows a schematic diagram of the structure of a computer device involved in an embodiment of the present application, and the computer device may be a server, specifically:
[0133] The computer device may include one or more processing core processors 301, one or more computer readable storage media memories 302, a power supply 303, an input unit 304 and other components. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0134] The processor 301 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. It executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0135] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0136] The computer device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0137] The computer device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0138] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:
[0139] Acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; perform style conversion on the object image to obtain a first object image under the target style; extract a target area image from the first object image based on the target detection frame and input it into an image restoration model, mark the target area where the target part is located in the target area image, obtain target area information and input it into the image restoration model; perform feature extraction on the target part feature information through a restoration control model, obtain control features and input them into the image restoration model; perform target part restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model, and obtain a restored target area image; perform image fusion on the restored target area image and the first object image to obtain a restored target object image.
[0140] The specific implementation of each of the above operations can be found in the previous embodiments, which will not be described in detail here. It should be noted that the computer device provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above method embodiment, which will not be described in detail here.
[0141] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0142] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image processing methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0143] Acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part is located in the object image, and target part feature information identified in the target part based on the target detection frame; perform style conversion on the object image to obtain a first object image under the target style; extract a target area image from the first object image based on the target detection frame and input it into an image restoration model, mark the target area where the target part is located in the target area image, obtain target area information and input it into the image restoration model; perform feature extraction on the target part feature information through a restoration control model, obtain control features and input them into the image restoration model; perform target part restoration processing on the target area in the target area image based on the control features, the target area image and the target area information through the image restoration model, and obtain a restored target area image; perform image fusion on the restored target area image and the first object image to obtain a restored target object image.
[0144] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0145] Since the instructions stored in the computer-readable storage medium can execute the steps in any image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0146] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations provided in the above embodiments.
[0147] The above is a detailed introduction to an image processing method, device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An image processing method, characterized in that: include: Acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame; Performing style conversion on the object image to obtain a first object image in a target style; Extracting a target area image from the first object image based on the target detection frame and inputting it into an image restoration model, marking a target area where the target part is located in the target area image, and obtaining target area information and inputting it into the image restoration model; Extracting feature information of the target part through a repair control model to obtain control features, which are input into the image repair model. The control features are features extracted by the repair control model and used to control the image repair model to perform image repair on the target area image based on key point information and image features of the target part. Based on the control features, the image repair model acquires relevant feature information of the target part in the object image. Performing a target area repair process on the target area in the target area image based on the control feature, the target area image and the target area information by the image repair model to obtain a repaired target area image; The restored target area image and the first object image are fused to obtain a restored target object image.
2. The image processing method according to claim 1, characterized in that: The target part is a face part, the feature information of the target part includes facial key point information and a facial image, and the feature extraction of the feature information of the target part by using the repair control model includes: Extracting features from the facial image to obtain facial image features and inputting them into a repair control model; Feature extraction is performed on the facial key point information and the facial image features through the restoration control model.
3. The image processing method according to claim 2, characterized in that: The step of extracting features of the facial key point information and the facial image features by repairing the control model includes: Get the preset image of the target image size; Based on the facial key point information, key point drawing is performed in the preset image to obtain a key point image indicating the position distribution of facial key points in the target part; The key point image and the facial image features are extracted by repairing the control model.
4. The image processing method according to claim 3, characterized in that: The step of extracting a target area image from the first object image based on the target detection frame and inputting the image restoration model comprises: Based on the target detection frame, cropping an initial region image corresponding to the target part in the first object image; A preset target image size is obtained, and image scaling processing is performed on the initial region image according to the target image size to obtain a target region image, and the target region image is input into the image restoration model.
5. The image processing method according to claim 4, characterized in that: The step of cutting out an initial region image corresponding to the target part in the first object image based on the target detection frame includes: Determining a first target area where the target part is located in the first object image based on the target detection frame; Acquire an expansion parameter corresponding to the first target region, and perform region expansion on the first target region in the first object image according to the expansion parameter to obtain an initial region; Cropping the initial region in the first object image to obtain an initial region image; The step of drawing key points in the preset image based on the facial key point information includes: Key points are drawn in the preset image based on the facial key point information, the expansion parameters, and the target image size.
6. The image processing method according to claim 4, characterized in that: The step of fusing the restored target area image with the first object image to obtain a restored target object image includes: Scaling the restored target region image based on the size information of the initial region image to obtain a target region restoration image; According to the position information of the initial region image in the first object image, the target part repair image is pasted to the first object image to obtain a repaired target object image.
7. The image processing method according to any one of claims 2 to 6, characterized in that: The target part feature information also includes object attribute information, and the image restoration model performs a target part restoration process on the target area in the target area image based on the control feature, the target area image and the target area information, including: Obtaining style description information corresponding to the target style; Merging the style description information and the object attribute information to obtain target description information; The image restoration model is used to perform restoration processing on the target area in the target area image based on the control feature, the target area image, the target area information and the target description information.
8. An image processing device, characterized in that: include: An information acquisition unit, configured to acquire an object image, perform target part detection on the object image, obtain a target detection frame where the target part in the object image is located, and target part feature information identified in the target part based on the target detection frame; A style conversion unit, configured to perform style conversion on the object image to obtain a first object image in a target style; An image extraction unit, configured to extract a target area image from the first object image based on the target detection frame and input the image into an image restoration model, mark a target area where the target part is located in the target area image, and obtain target area information and input the information into the image restoration model; A feature extraction unit is used to extract feature information of the target part through a repair control model, and obtain control features to input into the image repair model, wherein the control features are features extracted by the repair control model and used to control the image repair model to perform image repair on the target area image based on key point information and image features of the target part, and based on the control features, the image repair model obtains relevant feature information of the target part in the object image; An image restoration unit, configured to perform a restoration process on a target area in the target area image based on the control feature, the target area image and the target area information through the image restoration model to obtain a restored target area image; The image fusion unit is used to fuse the restored target area image with the first object image to obtain a restored target object image.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 7.
10. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method according to any one of claims 1 to 7 when executing the computer program.
