Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, the method comprising: obtaining a special effect image corresponding to an original image; the special effect image is an image generated after performing special effect processing on a set object in the original image; deforming a target type region of the set object in the special effect image to obtain a deformed object region; the shape corresponding to the target type region in the deformed object region matches the shape corresponding to the target type region in the original image; the target type region is an image region corresponding to at least one part type of the set object; replacing the region of the set object in the original image with the deformed object region to obtain a processed image. The present disclosure can improve the image inverse pasting effect.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Image back pasting refers to back pasting certain areas of the special effect image generated by the image generation model back to the source image, such as back pasting the background, clothes, neck and other areas in the special effect image back to the source image.

[0003] However, when the style of the special effects image is very different from that of the source image, some areas of the source image (such as the neck area) that are pasted back often cannot match other areas in the source image (such as the clothing area); this makes the pasted image more inconsistent, less realistic, and has a poor image pasting effect.

[0004] Therefore, there is a problem in the conventional technology that the image reverse pasting effect is not high. Summary of the invention

[0005] The present disclosure provides an image processing method, device, electronic device and storage medium to at least solve the problem of low image reverse pasting effect in the related art. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0007] Acquire a special effect image corresponding to the original image; the special effect image is an image generated after performing special effect processing on a set object in the original image;

[0008] Performing deformation processing on the target type region of the set object in the special effect image to obtain a deformed object region; the target type region in the deformed object region matches the shape of the target type region in the original image;

[0009] The region of the set object in the original image is replaced with the deformed object region to obtain a processed image.

[0010] In a possible implementation, the step of deforming the target type region of the set object in the special effect image to obtain a deformed object region includes:

[0011] Acquire a shape corresponding to a target type region in the original image to obtain a first region shape;

[0012] According to the first region shape, a deformation process is performed on the target type region of the set object in the special effect image to obtain the deformed object region.

[0013] In a possible implementation, the step of deforming the target type region of the set object in the special effect image according to the shape of the first region to obtain the deformed object region includes:

[0014] Acquire a shape corresponding to a target type region in the special effect image to obtain a second region shape;

[0015] Based on the difference between the second region shape and the first region shape, a deformation process is performed on the target type region of the set object in the special effect image to obtain the deformed object region.

[0016] In a possible implementation, acquiring a shape corresponding to the target type region in the original image to obtain a first region shape includes:

[0017] Acquire first outer contour information corresponding to the target type area in the original image;

[0018] Based on the first outer contour information, a first shape key point is determined; the first shape key point is used to characterize the shape of the first region.

[0019] In a possible implementation, determining the first shape key point based on the first outer contour information includes:

[0020] Determine, according to the first outer contour information, vertex coordinates corresponding to the target type region in the original image and a corresponding region boundary;

[0021] The first shape key point is determined based on the vertex coordinates and the region boundary.

[0022] In a possible implementation, if the set object is a portrait and the target type region is a neck region of the portrait, obtaining first outer contour information corresponding to the target type region in the original image includes:

[0023] Inputting the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model;

[0024] The first outer contour information corresponding to the neck area in the original image is determined according to the contour mask corresponding to the neck area.

[0025] In a possible implementation, acquiring a shape corresponding to the target type region in the special effect image to obtain a second region shape includes:

[0026] Acquire second outer contour information corresponding to the target type area in the special effect image;

[0027] Based on the second outer contour information, second shape key points are determined; the second shape key points are used to characterize the shape of the second region.

[0028] In a possible implementation, determining the second shape key point based on the second outer contour information includes:

[0029] Determine, according to the second outer contour information, vertex coordinates corresponding to the target type region in the special effect image and a corresponding region boundary;

[0030] The second shape key point is determined based on the vertex coordinates and the region boundary.

[0031] In a possible implementation, the obtaining of a special effect image corresponding to the original image includes:

[0032] Inputting the original image into a pre-trained generative adversarial network to obtain the special effect image;

[0033] The pre-trained generative adversarial network is used to transform the set object in the original image from an original style to a target style to generate the special effect image.

[0034] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0035] An acquisition unit is configured to acquire a special effect image corresponding to an original image; the special effect image is an image generated after performing special effect processing on a set object in the original image;

[0036] a deformation unit configured to perform deformation processing on a target type region of the set object in the special effect image to obtain a deformed object region; a shape corresponding to the target type region in the deformed object region matches a shape corresponding to the target type region in the original image; the target type region is an image region corresponding to at least one part type of the set object;

[0037] The replacement unit is configured to replace the region of the set object in the original image with the deformed object region to obtain a processed image.

[0038] In one possible implementation, the deformation unit is configured to execute acquisition of a shape corresponding to a target type area in the original image to obtain a first area shape; and according to the first area shape, deform the target type area of ​​the set object in the special effect image to obtain the deformed object area.

[0039] In one possible implementation, the deformation unit is configured to execute acquisition of a shape corresponding to a target type area in the special effect image to obtain a second area shape; based on a difference between the second area shape and the first area shape, deform the target type area of ​​the set object in the special effect image to obtain the deformed object area.

[0040] In one possible implementation, the deformation unit is configured to obtain first outer contour information corresponding to the target type area in the original image; determine first shape key points based on the first outer contour information; and the first shape key points are used to characterize the shape of the first area.

[0041] In one possible implementation, the deformation unit is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the original image according to the first outer contour information; and determine the first shape key point based on the vertex coordinates and the area boundary.

[0042] In one possible implementation, if the set object is a portrait and the target type area is the neck area of ​​the portrait, the deformation unit is configured to input the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model; and based on the contour mask corresponding to the neck area, determine the first outer contour information corresponding to the neck area in the original image.

[0043] In one possible implementation, the deformation unit is configured to obtain second outer contour information corresponding to the target type area in the special effect image; determine second shape key points based on the second outer contour information; and the second shape key points are used to characterize the shape of the second area.

[0044] In one possible implementation, the deformation unit is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the special effect image according to the second outer contour information; and determine the second shape key point based on the vertex coordinates and the area boundary.

[0045] In one possible implementation, the acquisition unit is configured to input the original image into a pre-trained generative adversarial network to obtain the special effect image; wherein the pre-trained generative adversarial network is used to transform the set object in the original image from an original style to a target style to generate the special effect image.

[0046] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the image processing method as described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0047] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0048] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device performs the image processing method described in any possible implementation manner of the first aspect.

[0049] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: a special effects image is generated by obtaining a set object in an original image and performing special effects processing on the target type area of ​​the set object in the special effects image, and a deformed object area is obtained; wherein the shape of the target type area in the deformed object area matches that of the target type area in the original image; and then the area of ​​the set object in the original image is replaced with the deformed object area to obtain a processed image; in this way, the object area after appropriate deformation can be matched with areas other than the target type area in the original image, for example, the neck area of ​​the special effects image is pasted back to the source image, and the neck area can be well matched with the clothing area in the source image; thereby improving the authenticity of the pasted image and the pasting effect of the image.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0052] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0053] Figure 2a is a schematic diagram of an original image according to an exemplary embodiment.

[0054] Figure 2bis a schematic diagram of a special effect image according to an exemplary embodiment.

[0055] Figure 2c is a schematic diagram showing a processed image according to an exemplary embodiment.

[0056] Figure 2d It is a schematic diagram showing a direct reverse pasting image according to an exemplary embodiment.

[0057] Figure 3 is a schematic diagram showing a deformed target type area according to an exemplary embodiment.

[0058] Figure 4 It is a schematic diagram showing outer contour information according to an exemplary embodiment.

[0059] Figure 5 The figure is a schematic diagram of obtaining shape key points according to an exemplary embodiment.

[0060] Figure 6 The figure is a schematic diagram showing segmentation of a contour mask of a human body sub-region according to an exemplary embodiment.

[0061] Figure 7 is a flowchart of another image processing method according to an exemplary embodiment.

[0062] Figure 8 The figure is a flowchart of a practical application of an image processing method according to an exemplary embodiment.

[0063] Fig. 9 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0064] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0067] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0068] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method can be used in an electronic device, and includes the following steps.

[0069] In step S110, a special effect image corresponding to the original image is obtained.

[0070] Among them, the original image is input into the pre-trained generative adversarial network to obtain a special effect image; wherein the pre-trained generative adversarial network is used to transform the set object in the original image from the original style to the target style to generate a special effect image.

[0071] Among them, the special effect image is an image generated after performing GAN special effects processing on the set object in the original image.

[0072] Among them, GAN special effects can refer to image special effects obtained by processing images using pre-trained GAN (Generative Adversarial Networks).

[0073] In actual applications, GAN special effects have become increasingly common in current short video applications or camera applications, such as aging, becoming a child, comic faces, hand-painted and other special effects. Because the changes in facial attributes or styles are very natural and realistic, they are deeply loved by users.

[0074] The set object may be an animal, a person or other object.

[0075] In a specific implementation, the electronic device can obtain an original image and input the original image into a pre-trained special effects generation network. Through the pre-trained special effects generation network, the set object in the original image is converted from a first style to a second style (for example, the character A in the original image is converted from the original adult style to a child style) to obtain a special effects image.

[0076] In step S120, a deformation process is performed on the target type region of the set object in the special effect image to obtain a deformed object region.

[0077] The shape corresponding to the target type region in the deformed object region matches the shape corresponding to the target type region in the original image.

[0078] The target type region is an image region corresponding to at least one part type of the set object. For example, when the set object is a person, the target type region may be the neck region of the person.

[0079] In a specific implementation, after the electronic device obtains the special effect image corresponding to the original image, the electronic device can paste the target type area in the special effect image back to the original image. Specifically, the electronic device can identify the target type area corresponding to the set object in the special effect image. At the same time, the electronic device identifies the target type area corresponding to the set object in the original image (i.e., the set object of the first style). Then, the electronic device deforms the target type area corresponding to the set object in the special effect image (i.e., the set object of the second style) so that the deformed object area. The shape of the target type area in the deformed object area matches the shape of the target type area in the original image.

[0080] For example, when the set object is a portrait, the target type area is the neck area; the electronic device pastes the neck area in the special effect image back to the original image. Specifically, the electronic device can identify the neck area corresponding to the set object of the second style in the special effect image. At the same time, the neck area corresponding to the set object of the first style in the original image is identified. Then, the electronic device deforms the neck area corresponding to the set object of the second style to obtain the deformed object area (i.e., the portrait area corresponding to the set object of the second style). The shape of the neck area in the deformed object area matches the shape of the neck area corresponding to the set object of the first style in the original image.

[0081] In step S130, the region of the set object in the original image is replaced with the deformed object region to obtain a processed image.

[0082] In a specific implementation, after obtaining the deformed object area, the electronic device replaces the area of ​​the set object in the original image with the deformed object area, and pastes the deformed object area back into the original image to obtain a processed image.

[0083] Continuing with the above example, after the electronic device obtains the deformed object region, the electronic device replaces the portrait region in the original image with the deformed object region to obtain a processed image.

[0084] The electronic device can convert the character A in the original image from the original adult style to the child style. For the convenience of understanding by those skilled in the art, please refer to FIG. 2. Among them, Figure 2a is the original image, Figure 2b For special effects images, Figure 2c For the processed image, Figure 2d The direct reverse pasting image is obtained by directly replacing the human body area of ​​person A in the original image with the human body area of ​​person A in the special effect image. Figure 2c As shown, the shape of the neck region of the character A with a child style in the processed image is the same as the shape of the neck region of the character A with an adult style in the original image, and the texture of the neck region of the character A with a child style in the processed image is the same as the texture of the neck region of the character A with a child style in the special effect image. That is, the neck region of the character A with a child style in the processed image can be well connected with the clothing region in the processed image, and the authenticity of the processed image is higher than that of the directly reversed image.

[0085] In the above-mentioned image processing method, a special effect image is generated by obtaining a set object in the original image and performing special effect processing on the set object, and a target type area of ​​the set object in the special effect image is deformed to obtain a deformed object area; wherein the shape of the target type area in the deformed object area matches that of the target type area in the original image; then the area of ​​the set object in the original image is replaced with the deformed object area to obtain a processed image; in this way, the object area after appropriate deformation can be matched with areas other than the target type area in the original image, for example, the neck area of ​​the special effect image is pasted back to the source image, and the neck area can be well matched with the clothing area in the source image; thereby improving the authenticity of the pasted image and improving the pasting effect of the image.

[0086] In an exemplary embodiment, a target type area of ​​a set object in a special effect image is deformed to obtain a deformed object area, including: acquiring a shape corresponding to the target type area in the original image to obtain a first area shape; and deforming the target type area of ​​the set object in the special effect image according to the first area shape to obtain a deformed object area.

[0087] In a specific implementation, when the electronic device performs deformation processing on the target type area of ​​a set object in a special effect image to obtain the deformed object area, the electronic device can obtain the shape corresponding to the target type area in the original image to obtain a first area shape; then, the electronic device performs non-rigid deformation processing (for example, TPS thin plate spline transformation processing) on ​​the target type area of ​​the set object in the special effect image according to the first area shape to obtain the deformed object area.

[0088] The process in which the electronic device deforms the target type area of ​​the set object in the special effect image according to the first area shape to obtain the deformed object area includes: acquiring the shape corresponding to the target type area in the special effect image to obtain the second area shape; and based on the difference between the second area shape and the first area shape, deforming the target type area of ​​the set object in the special effect image to obtain the deformed object area.

[0089] In a specific implementation, the electronic device may obtain the shape corresponding to the target type area in the special effect image in the same manner as the shape corresponding to the target type area in the original image, and obtain the shape of the second area; then, the electronic device may deform the target type area of ​​the set object in the special effect image based on the difference between the second area shape and the first area shape, and obtain the deformed object area. Specifically, the electronic device may obtain N shape key points A for characterizing the shape of the first area, and N shape key points B for characterizing the shape of the second area. Then, the electronic device determines multiple groups of matching key point pairs based on the N shape key points A and the N shape key points B, and based on the multiple groups of matching key point pairs, uses the TPS thin plate spline transformation processing algorithm to deform the target type area of ​​the set object in the special effect image, and obtains the deformed target type area. Then, the electronic device replaces the target type area in the area corresponding to the set object in the special effect image with the deformed target type area, and obtains the deformed object area. To facilitate understanding by those skilled in the art, Figure 3 A schematic diagram of a deformed target type region is provided, wherein 310 is the deformed target type region.

[0090] The technical solution of this embodiment obtains the shape corresponding to the target type area in the original image to obtain the first area shape, and deforms the target type area of ​​the set object in the special effect image according to the first area shape, so that the shapes of the target type area in the deformed object area obtained by processing can be accurately matched with the target type area in the original image.

[0091] In an exemplary embodiment, a shape corresponding to a target type region in an original image is obtained to obtain a first region shape, including: obtaining first outer contour information corresponding to the target type region in the original image; determining first shape key points based on the first outer contour information; the first shape key points are used to characterize the first region shape.

[0092] In a specific implementation, when the electronic device acquires the shape corresponding to the target type area in the original image and obtains the shape of the first area, the electronic device can use a pre-trained target type area recognition model to identify the first outer contour information corresponding to the target type area in the original image. Specifically, the electronic device can input the original image into a pre-trained target type area recognition model, and output a target type area mask (mask image) corresponding to the original image through the pre-trained target type area recognition model. Then, the electronic device can extract the first outer contour information corresponding to the target type area based on the target type area mask. Finally, the electronic device generates shape key points for characterizing the shape of the first area based on the first outer contour information. To facilitate understanding by those skilled in the art, Figure 4 A schematic diagram of outer contour information is provided; wherein 410 is the original image, and 420 is the outer contour information corresponding to the target type region when the target type region is the neck region.

[0093] The technical solution of this embodiment obtains the first outer contour information corresponding to the target type area in the original image; based on the first outer contour information, the first shape key points that can be used to characterize the shape of the first area are accurately determined, thereby reducing the amount of data processing required by the electronic device to perform deformation processing on the image.

[0094] In an exemplary embodiment, determining the first shape key point based on the first outer contour information includes: determining the vertex coordinates corresponding to the target type area in the original image and the corresponding area boundary according to the first outer contour information; determining the first shape key point based on the vertex coordinates and the area boundary.

[0095] In a specific implementation, when the electronic device determines the first shape key point based on the first outer contour information, the electronic device may

[0096] In order to facilitate the understanding of those skilled in the art, Figure 5 A schematic diagram for obtaining shape key points is provided; see Figure 5The electronic device can determine at least four vertex coordinates 510 corresponding to the target type area in the original image and the area boundary 520 corresponding to the target type area based on the first outer contour information in the original image; then, the electronic device can divide the area boundary equidistantly based on the vertex coordinates to obtain four vertices and equidistantly divided key points as the final first shape key points 530.

[0097] The technical solution of this embodiment determines the vertex coordinates and the corresponding area boundary of the target type area in the original image according to the first outer contour information; based on the vertex coordinates and the area boundary, accurately determines the first shape key point used to characterize the shape of the first area.

[0098] In an exemplary embodiment, if the object is set to be a portrait and the target type area is the neck area of ​​the portrait, the first outer contour information corresponding to the target type area in the original image is obtained, including: inputting the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model; and determining the first outer contour information corresponding to the neck area in the original image based on the contour mask corresponding to the neck area.

[0099] Among them, the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model.

[0100] In a specific implementation, if the object is set to be a portrait, and the target type area is the neck area of ​​the portrait, when the electronic device obtains the first outer contour information corresponding to the target type area in the original image, the electronic device can input the original image into a pre-trained face parsing model (human body parsing model) to obtain the contour masks of each human body sub-area in the original image; then, in the contour masks of the human body sub-areas, determine the contour mask corresponding to the neck area. The electronic device can determine the first outer contour information corresponding to the neck area in the original image based on the contour mask corresponding to the neck area. To facilitate understanding by those skilled in the art, Figure 6 A schematic diagram of segmentation of a contour mask of a human body sub-region is provided.

[0101] The technical solution of this embodiment obtains a contour mask corresponding to the neck area in the original image by inputting the original image into a pre-trained face parsing model, so that the first outer contour information corresponding to the neck area in the original image can be accurately and quickly determined based on the contour mask corresponding to the neck area.

[0102] In an exemplary embodiment, obtaining the shape corresponding to the target type region in the special effect image to obtain the shape of the second region includes: obtaining the second outer contour information corresponding to the target type region in the special effect image; determining the second shape key points based on the second outer contour information; the second shape key points are used to characterize the shape of the second region. Wherein, determining the second shape key points based on the second outer contour information includes: determining the vertex coordinates corresponding to the target type region in the special effect image and the corresponding region boundary according to the second outer contour information; determining the second shape key points based on the vertex coordinates and the region boundary.

[0103] It should be noted that the method used in the step of obtaining the shape corresponding to the target type area in the special effect image and obtaining the shape of the second area is the same as the method used in the step of obtaining the shape corresponding to the target type area in the original image and obtaining the shape of the first area. Therefore, the specific definition of the step of obtaining the shape corresponding to the target type area in the special effect image and obtaining the shape of the second area can be referred to the specific definition of the step of obtaining the shape corresponding to the target type area in the original image and obtaining the shape of the first area, which will not be repeated here.

[0104] Figure 7 is a flowchart of another image processing method according to an exemplary embodiment. Figure 7 As shown, the method includes the following steps.

[0105] In step S710, a special effect image corresponding to the original image is obtained; the special effect image is an image generated after performing special effect processing on a set object in the original image.

[0106] In step S720, first outer contour information corresponding to the target type area in the original image is obtained.

[0107] In step S730, first shape key points are determined based on the first outer contour information; the first shape key points are used to characterize the shape of the first region.

[0108] In step S740, second outer contour information corresponding to the target type area in the special effect image is obtained.

[0109] In step S750, second shape key points are determined based on the second outer contour information; the second shape key points are used to characterize the shape of the second region.

[0110] In step S760, based on the difference between the shape of the second area and the shape of the first area, the target type area of ​​the set object in the special effect image is deformed to obtain a deformed object area; the shape of the target type area in the deformed object area matches the shape of the target type area in the original image.

[0111] In step S770, the region of the set object in the original image is replaced with the deformed object region to obtain a processed image.

[0112] It should be noted that the specific limitations of the above steps can be found in the specific limitations of an image processing method mentioned above, which will not be repeated here.

[0113] In order to facilitate the understanding of those skilled in the art, Figure 8 The exemplary embodiment shows a practical application flow chart of an image processing method; wherein, if the object is set to be a portrait, and the target type area is the neck area of ​​the portrait, the electronic device can input the original image (input image) into the human body parsing model to obtain the contour mask corresponding to the neck area in the original image. Then, the electronic device determines at least four vertex coordinates corresponding to the neck area in the original image, and the corresponding area boundary according to the contour mask corresponding to the neck (neck) area in the original image; then, the electronic device can divide the area boundary equidistantly based on the vertex coordinates to obtain four vertices and equally divided key points as the first shape key points. At the same time, the electronic device can input the special effect image (generated image) into the human body parsing model to obtain the contour mask corresponding to the neck area in the special effect image. Then, the electronic device determines at least four vertex coordinates corresponding to the neck area in the special effect image, and the corresponding area boundary according to the contour mask corresponding to the neck (neck) area in the special effect image; then, the electronic device can divide the area boundary equidistantly based on the vertex coordinates to obtain four vertices and equally divided key points as the second shape key points. Then, the electronic device can determine multiple sets of matching key point pairs from the N first shape key points and the N second shape key points, and based on the multiple sets of matching key point pairs, use the TPS thin plate spline transformation processing algorithm to deform the target type area of ​​the set object in the special effect image to obtain the deformed target type area. Then, the electronic device replaces the target type area in the area corresponding to the set object in the special effect image with the deformed target type area to obtain the deformed object area. The electronic device replaces the portrait area in the original image with the deformed object area to obtain the processed image.

[0114] It should be understood that although Figure 1 , Figure 7 and Figure 8The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 7 and Figure 8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0115] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0116] Fig. 9 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Fig. 9 , the device comprises:

[0117] The acquisition unit 910 is configured to acquire a special effect image corresponding to the original image; the special effect image is an image generated after performing special effect processing on a set object in the original image;

[0118] The deformation unit 920 is configured to perform deformation processing on the target type region of the set object in the special effect image to obtain a deformed object region; the target type region in the deformed object region matches the shape of the target type region in the original image;

[0119] The replacement unit 930 is configured to replace the region of the set object in the original image with the deformed object region to obtain a processed image.

[0120] In one possible implementation, the deformation unit 920 is configured to execute acquisition of a shape corresponding to a target type area in the original image to obtain a first area shape; and according to the first area shape, deform the target type area of ​​the set object in the special effect image to obtain the deformed object area.

[0121] In one possible implementation, the deformation unit 920 is configured to execute acquisition of a shape corresponding to a target type area in the special effect image to obtain a second area shape; based on a difference between the second area shape and the first area shape, deform the target type area of ​​the set object in the special effect image to obtain the deformed object area.

[0122] In one possible implementation, the deformation unit 920 is configured to obtain first outer contour information corresponding to the target type area in the original image; determine first shape key points based on the first outer contour information; and the first shape key points are used to characterize the shape of the first area.

[0123] In one possible implementation, the deformation unit 920 is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the original image according to the first outer contour information; and determine the first shape key point based on the vertex coordinates and the area boundary.

[0124] In one possible implementation, if the set object is a portrait and the target type area is the neck area of ​​the portrait, the deformation unit 920 is configured to execute the input of the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model; and according to the contour mask corresponding to the neck area, the first outer contour information corresponding to the neck area in the original image is determined.

[0125] In one possible implementation, the deformation unit 920 is configured to obtain second outer contour information corresponding to the target type area in the special effect image; determine second shape key points based on the second outer contour information; and the second shape key points are used to characterize the shape of the second area.

[0126] In one possible implementation, the deformation unit 920 is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the special effect image according to the second outer contour information; and determine the second shape key point based on the vertex coordinates and the area boundary.

[0127] In one possible implementation, the acquisition unit 910 is configured to input the original image into a pre-trained generative adversarial network to obtain the special effect image; wherein the pre-trained generative adversarial network is used to transform the set object in the original image from an original style to a target style to generate the special effect image.

[0128] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0129] Fig.10 1 is a block diagram of an electronic device 1000 for executing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0130] Reference Fig.10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0131] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0132] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, an optical disk, or a graphene memory.

[0133] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0134] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0135] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting an audio signal.

[0136] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0137] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect changes in the position of the electronic device 1000 or components of the electronic device 1000, the presence or absence of contact between the user and the electronic device 1000, the orientation or acceleration / deceleration of the device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0138] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0139] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0140] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the above instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0141] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method.

[0142] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0143] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0144] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Get the special effect image corresponding to the original image; The special effect image is an image generated by performing special effect processing on a set object in the original image; Performing deformation processing on the target type region of the set object in the special effect image to obtain a deformed object region; The shape corresponding to the target type region in the deformed object region matches the shape corresponding to the target type region in the original image; the target type region is an image region corresponding to at least one part type of the set object; Replacing the region of the set object in the original image with the deformed object region to obtain a processed image; The step of deforming the target type region of the set object in the special effect image to obtain a deformed object region includes: Acquire a shape corresponding to a target type region in the original image to obtain a first region shape; According to the first region shape, a deformation process is performed on the target type region of the set object in the special effect image to obtain the deformed object region.

2. The image processing method according to claim 1, characterized in that: The step of deforming the target type region of the set object in the special effect image according to the first region shape to obtain the deformed object region includes: Acquire a shape corresponding to a target type region in the special effect image to obtain a second region shape; Based on the difference between the second region shape and the first region shape, a deformation process is performed on the target type region of the set object in the special effect image to obtain the deformed object region.

3. The image processing method according to claim 1, characterized in that: The step of acquiring a shape corresponding to the target type region in the original image to obtain a first region shape includes: Acquire first outer contour information corresponding to the target type area in the original image; Based on the first outer contour information, a first shape key point is determined; the first shape key point is used to characterize the shape of the first region.

4. The image processing method according to claim 3, characterized in that: The determining of the first shape key point based on the first outer contour information includes: Determine, according to the first outer contour information, vertex coordinates corresponding to the target type region in the original image and a corresponding region boundary; The first shape key point is determined based on the vertex coordinates and the region boundary.

5. The image processing method according to claim 3, characterized in that: If the set object is a portrait and the target type region is the neck region of the portrait, obtaining first outer contour information corresponding to the target type region in the original image includes: Inputting the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model; The first outer contour information corresponding to the neck area in the original image is determined according to the contour mask corresponding to the neck area.

6. The image processing method according to claim 2, characterized in that: The step of acquiring a shape corresponding to the target type region in the special effect image to obtain a second region shape includes: Acquire second outer contour information corresponding to the target type area in the special effect image; Based on the second outer contour information, second shape key points are determined; the second shape key points are used to characterize the shape of the second region.

7. The image processing method according to claim 6, characterized in that: The determining of the second shape key points based on the second outer contour information includes: Determine, according to the second outer contour information, vertex coordinates corresponding to the target type region in the special effect image and a corresponding region boundary; The second shape key point is determined based on the vertex coordinates and the region boundary.

8. The image processing method according to any one of claims 1 to 7, characterized in that: The obtaining of a special effect image corresponding to the original image includes: Inputting the original image into a pre-trained generative adversarial network to obtain the special effect image; The pre-trained generative adversarial network is used to transform the set object in the original image from an original style to a target style to generate the special effect image.

9. An image processing device, characterized in that: include: An acquisition unit is configured to acquire a special effect image corresponding to the original image; The special effect image is an image generated by performing special effect processing on a set object in the original image; A deformation unit, configured to perform deformation processing on the target type area of ​​the set object in the special effect image to obtain a deformed object area; The shape corresponding to the target type region in the deformed object region matches the shape corresponding to the target type region in the original image; the target type region is an image region corresponding to at least one part type of the set object; a replacement unit, configured to replace a region of a set object in the original image with the deformed object region to obtain a processed image; The deformation unit is configured to execute acquisition of a shape corresponding to a target type region in the original image to obtain a first region shape; and to perform deformation processing on the target type region of the set object in the special effect image according to the first region shape to obtain the deformed object region.

10. The image processing device according to claim 9, characterized in that: The deformation unit is configured to execute acquisition of a shape corresponding to a target type area in the special effect image to obtain a second area shape; based on a difference between the second area shape and the first area shape, deform the target type area of ​​the set object in the special effect image to obtain the deformed object area.

11. The image processing device according to claim 9, characterized in that: The deformation unit is configured to execute acquisition of first outer contour information corresponding to the target type area in the original image; Based on the first outer contour information, determining a first shape key point; The first shape key points are used to characterize the shape of the first region.

12. The image processing device according to claim 11, characterized in that: The deformation unit is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the original image according to the first outer contour information; and determine the first shape key point based on the vertex coordinates and the area boundary.

13. The image processing device according to claim 11, characterized in that: If the set object is a portrait and the target type area is the neck area of ​​the portrait, the deformation unit is configured to input the original image into a pre-trained face parsing model to obtain a contour mask corresponding to the neck area in the original image; the contour mask is obtained by performing image semantic segmentation processing on the original image by the pre-trained face parsing model; and according to the contour mask corresponding to the neck area, determine the first outer contour information corresponding to the neck area in the original image.

14. The image processing device according to claim 10, characterized in that: The deformation unit is configured to execute acquisition of second outer contour information corresponding to the target type area in the special effect image; Determining a second shape key point based on the second outer contour information; The second shape key points are used to characterize the shape of the second region.

15. The image processing device according to claim 14, characterized in that: The deformation unit is configured to determine the vertex coordinates and the corresponding area boundary corresponding to the target type area in the special effect image according to the second outer contour information; and determine the second shape key point based on the vertex coordinates and the area boundary.

16. The image processing device according to any one of claims 9 to 15, characterized in that: The acquisition unit is configured to input the original image into a pre-trained generative adversarial network to obtain the special effect image; wherein the pre-trained generative adversarial network is used to transform the set object in the original image from an original style to a target style to generate the special effect image.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 8.

19. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as described in any one of claims 1 to 8.

Images