Image processing method, device, terminal device and readable storage medium

Abstract

The present application provides an image processing method, apparatus, terminal device and readable storage medium. The method comprises: obtaining an image to be processed and a reference image for the image to be processed, wherein the similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value; adjusting the image of a corresponding area in the image to be processed according to the image of a preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value; and using the adjusted image to be processed as a target image. Through the image processing method of the present application, a completely aligned image pair (target image and reference image) can be obtained, which can be used to train an image style transformation model with better transformation effect, thereby better meeting various image style transformation requirements and optimizing user experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image processing method, apparatus, terminal device and readable storage medium. Background Art

[0002] Image style transfer is an important research direction in computer vision tasks, which is mainly divided into two types of tasks: paired and unpaired. Unpaired image style transfer means that the training data is unpaired. When training the image style transfer model, the input image and the output image cannot be aligned one by one. Paired face style transfer means that the training data is paired, and each input image will have a corresponding output image, and the input image and the output image can be aligned one by one.

[0003] In the real world, it is difficult to obtain pairs of images that are completely aligned. In the face style transfer task, the obtained pairs of images are also not completely aligned. If such image data is directly used to train the image style transfer model, the model will not be able to accurately output the image corresponding to the input image, and the training effect will be poor. Therefore, in order to improve the training effect of the image style transfer model based on paired data, an image processing method is urgently needed in the related technology to obtain fully aligned paired image data. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, terminal device, and readable storage medium, which can obtain fully aligned paired image data. The image pairs can be used to train an image style transformation model with better transformation effect, thereby better meeting various image style transformation requirements. The specific technical solutions are as follows:

[0005] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0006] Obtaining an image to be processed and a reference image for the image to be processed, wherein a similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value;

[0007] Adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value;

[0008] The adjusted image to be processed is used as the target image.

[0009] Optionally, the method further comprises:

[0010] Taking the target image and the reference image as a sample image pair;

[0011] The sample image pairs are used to train an initial image style transformation model to obtain an image style transformation model, wherein the image style transformation model is used to output an image having a similarity with the input image greater than a second preset value after receiving the input image.

[0012] Optionally, adjusting the image of a corresponding area in the image to be processed according to the image of a preset area in the reference image includes:

[0013] Identify key points of the facial area of ​​the first target object, wherein the key points include facial features and facial contour points;

[0014] identifying key points of the facial area of ​​the second target object;

[0015] The facial features of the first target object are kept unchanged, and the facial contour points of the first target object are adjusted according to the facial contour points of the second target object.

[0016] Optionally, keeping the facial features of the first target object unchanged and adjusting the facial contour points of the first target object according to the facial contour points of the second target object includes:

[0017] Generating a first triangulation graph according to each key point of the facial region of the first target object, wherein the first triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0018] Generating a second triangulation graph according to each key point of the facial region of the second target object, wherein the second triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0019] The vertices of the triangle where the facial feature points are located in the first triangulation graph are kept fixed, and the positions of the vertices of the triangle where the facial contour points are located in the first triangulation graph are adjusted according to the positions of the vertices of the triangle where the facial contour points are located in the second triangulation graph.

[0020] Optionally, the area of ​​the first target object includes a facial area and a modified area of ​​the first target object, and the area of ​​the second target object includes a facial area and a modified area of ​​the second target object; the method further includes:

[0021] When the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value, obtaining a mask corresponding to the overlap area between the area of ​​the first target object and the area of ​​the second target object;

[0022] The image to be processed and the reference image are adjusted respectively using the mask corresponding to the overlapped area.

[0023] Optionally, obtaining a mask corresponding to an overlapping area between an area of ​​the first target object and an area of ​​the second target object includes:

[0024] Obtaining a mask corresponding to a region of a first target object in the image to be processed;

[0025] Obtaining a mask corresponding to a region of a second target object in the reference image;

[0026] The mask corresponding to the overlapping area is obtained according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object.

[0027] Optionally, the image to be processed includes a background area and an area of ​​the first target object, the reference image includes a background area and an area of ​​the second target object, and the method further includes:

[0028] The background areas of the image to be processed and the reference image are respectively replaced with the same background image.

[0029] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0030] A first obtaining module, used to obtain an image to be processed and a reference image for the image to be processed, wherein a similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value;

[0031] A first adjustment module, configured to adjust the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image, until the similarity between the image to be processed and the reference image is greater than a second preset value;

[0032] The first determining module is used to use the adjusted image to be processed as a target image.

[0033] Optionally, the device further comprises:

[0034] A second determination module, configured to use the target image and the reference image as a sample image pair;

[0035] A training module is used to train an initial image style transformation model using the sample image pairs to obtain an image style transformation model, wherein the image style transformation model is used to output an image having a similarity with the input image greater than a second preset value after receiving the input image.

[0036] Optionally, the first adjustment module includes:

[0037] A first recognition submodule, used to recognize key points of the facial area of ​​the first target object, wherein the key points include facial features and facial contour points;

[0038] A second recognition submodule, used to recognize key points of the facial area of ​​the second target object;

[0039] The first adjustment submodule is used to keep the facial features of the first target object unchanged and adjust the facial contour points of the first target object according to the facial contour points of the second target object.

[0040] Optionally, the first adjustment submodule includes:

[0041] A first generating submodule, configured to generate a first triangulation graph according to each key point of the facial region of the first target object, wherein the first triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0042] A second generating submodule, configured to generate a second triangulation graph according to each key point of the facial region of the second target object, wherein the second triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0043] The second adjustment submodule is used to keep the vertices of the triangle where the facial feature points in the first triangulation graph are located fixed, and adjust the positions of the vertices of the triangle where the facial contour points in the first triangulation graph are located according to the positions of the vertices of the triangle where the facial contour points in the second triangulation graph are located.

[0044] Optionally, the area of ​​the first target object includes a facial area and a modified area of ​​the first target object, and the area of ​​the second target object includes a facial area and a modified area of ​​the second target object; the device further includes:

[0045] A second obtaining module, configured to obtain a mask corresponding to an overlapping area between the area of ​​the first target object and the area of ​​the second target object when the overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value;

[0046] The second adjustment module is used to adjust the image to be processed and the reference image respectively by using the mask corresponding to the overlapped area.

[0047] Optionally, the second obtaining module includes:

[0048] A first obtaining submodule, used to obtain a mask corresponding to a region of a first target object in the image to be processed;

[0049] A second obtaining submodule, used to obtain a mask corresponding to a region of a second target object in the reference image;

[0050] The third obtaining submodule is used to obtain the mask corresponding to the overlapping area according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object.

[0051] Optionally, the image to be processed includes a background area and an area of ​​the first target object, the reference image includes a background area and an area of ​​the second target object, and the device further includes:

[0052] The replacement module is used to replace the background areas of the image to be processed and the reference image with the same background image.

[0053] In a third aspect of the embodiment of the present application, a terminal device is further provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0054] Memory, used to store computer programs;

[0055] The processor is used to implement the steps of the image processing method described in the first aspect of the embodiment of the present application when executing the program stored in the memory.

[0056] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the steps in any of the above-mentioned image processing methods.

[0057] Using the image processing method of the present application, firstly, an image to be processed and a reference image for the image to be processed are obtained, and the similarity between the first target object in the image to be processed and the second target object in the reference image is greater than a first preset value. Then, the image of the corresponding area in the image to be processed is adjusted according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value. Finally, the adjusted image to be processed is used as the target image. Through the image processing method of the present application, a completely aligned image pair (target image and reference image) can be obtained, and the image pair can be used to train an image style transformation model with better transformation effect, so as to better meet various image style transformation needs and optimize user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0059] Figure 1 is a flow chart of an image processing method shown in an embodiment of the present application;

[0060] Figure 2 is an image to be processed shown in an embodiment of the present application;

[0061] Figure 3 is a reference image shown in an embodiment of the present application;

[0062] Figure 4 is a schematic diagram of key points of a facial area shown in an embodiment of the present application;

[0063] Figure 5 is a schematic diagram of a target image shown in an embodiment of the present application;

[0064] Figure 6 is a schematic diagram of another reference image shown in an embodiment of the present application;

[0065] Figure 7 is a schematic diagram of a complete process of obtaining a target image shown in an embodiment of the present application;

[0066] Figure 8 is a structural block diagram of an image processing device shown in an embodiment of the present application;

[0067] Fig. 9 It is a structural diagram of a terminal device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0069] The image processing method in this application is mainly proposed based on paired image style transformation. In order to train an image style transformation model with better transformation effect in paired image style transformation tasks, this application proposes a method for obtaining fully aligned paired images, such as Figure 1 shown. Figure 1 1 is a flowchart of an image processing method shown in an embodiment of the present application. Figure 1 The image processing method of the present application may specifically include the following steps:

[0070] Step S11: obtaining an image to be processed and a reference image for the image to be processed, wherein the similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value.

[0071] In this embodiment, the goal of image processing for the image to be processed is to adjust the image to be processed to an image that is completely aligned with the reference image. In two frames of images, if the image of each position area in one frame of image completely overlaps with the image of the corresponding position area in the other frame of image, then the two frames of image are completely aligned. For example, the image of the upper left area of ​​image A overlaps with the image of the upper left area of ​​image B, the image of the upper right area of ​​image A overlaps with the image of the upper right area of ​​image B, the image of the lower left area of ​​image A overlaps with the image of the lower left area of ​​image B, and the image of the lower right area of ​​image A overlaps with the image of the lower right area of ​​image B, then image A and image B are a completely aligned image pair.

[0072] In this embodiment, the image size of the image to be processed and the reference image for the image to be processed are completely the same.

[0073] The target object may be a person object or a non-person object. The non-person object may be, for example, an animal, a vehicle, or the like. In this embodiment, the specific type of the target object is not limited. Generally, the first target object and the second target object refer to the same object, for example, the same person or the same vehicle.

[0074] In order to better illustrate the image processing method of the present application, it is particularly noted here that the target objects in the subsequent embodiments of the present application are all human objects as examples, such as Figure 2 and Figure 3 shown. Figure 2 This is an image to be processed shown in an embodiment of the present application. Figure 3 It is a reference image shown in an embodiment of the present application.

[0075] Figure 2 and Figure 3 contains character objects, Figure 3 Can be seen as Figure 2 The image after image style transformation. Figure 2 and Figure 3 are not completely aligned, so the goal of image processing in this application is to Figure 2 Adjust to Figure 3 Perfectly aligned images.

[0076] For a certain frame of image to be processed, not all other images can be used as its reference image, but they must meet certain similarity requirements. For the image to be processed, if there is another frame of image, the similarity between the target object in the image and the target object in the image to be processed is greater than a first preset value, then the image can be used as the reference image of the image to be processed. For example Figure 2 The characters and Figure 3 The similarity between the characters in is greater than the first preset value, therefore, Figure 3Can be used as Figure 2 Reference image.

[0077] Step S12: adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image, until the similarity between the image to be processed and the reference image is greater than a second preset value.

[0078] In this embodiment, when adjusting the image to be processed, the goal is to make the similarity between the image to be processed and the reference image greater than a second preset value, and the image of the corresponding area in the image to be processed is adjusted according to the image of the preset area in the reference image.

[0079] In this embodiment, the image area specifically includes the background area and the target object (eg Figure 2 or Figure 3 The target object region includes the target object's facial region (e.g. Figure 2 or Figure 3 The facial area of ​​the human subject in the image) and the modified area (e.g. Figure 2 or Figure 3 The modified area may be, for example, a hairstyle area, a hat area, a clothing area, etc. The modified area may be defined according to actual needs. The preset area may be any one or more of a background area, a facial area of ​​a target object, and a modified area of ​​a target object, and this embodiment does not specifically limit this.

[0080] For example, if the preset area is the face area of ​​the target object, then step S12 includes: adjusting the face area of ​​the target object in the image to be processed according to the face area of ​​the target object in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value. Figure 3 The image of the facial area of ​​the person in Figure 2 The facial area of ​​the person in the image is adjusted until Figure 2 and Figure 3 The similarity between them is greater than a second preset value.

[0081] For another example, if the preset area is the face area and the hairstyle area of ​​the target object, then step S12 includes: Figure 3 The image of the facial area of ​​the person in Figure 2 The facial area of ​​the character is adjusted according to Figure 3 The image of the hairstyle area of ​​the character in Figure 2 Adjust the image of the character's hairstyle area until Figure 2 and Figure 3 The similarity between them is greater than a second preset value.

[0082] In this embodiment, if the preset area includes both the facial area and the modified area of ​​the target object, the facial area of ​​the image to be processed may be adjusted first, and then the modified area of ​​the image to be processed may be adjusted. Since the modified area may be affected when the facial area of ​​the image to be processed is adjusted, the efficiency of image processing may be improved by adjusting the facial area first and then the modified area.

[0083] Step S13: taking the adjusted image to be processed as the target image.

[0084] In this embodiment, the similarity between the adjusted target image and the reference image is greater than the second preset value, so the target image and the reference image are completely aligned images.

[0085] Through the image processing method of this embodiment, firstly, an image to be processed and a reference image for the image to be processed are obtained, and the similarity between the first target object in the image to be processed and the second target object in the reference image is greater than a first preset value. Then, the image of the corresponding area in the image to be processed is adjusted according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value. Finally, the adjusted image to be processed is used as the target image. Through the image processing method of the present application, a completely aligned image pair (target image and reference image) can be obtained, which can be used to train an image style transformation model with better transformation effect, so as to better meet various image style transformation needs and optimize user experience.

[0086] In combination with the above embodiments, in one implementation, the image processing method of the present application may further include:

[0087] Taking the target image and the reference image as a sample image pair;

[0088] The sample image pairs are used to train an initial image style transformation model to obtain an image style transformation model, wherein the image style transformation model is used to output an image having a similarity with the input image greater than a second preset value after receiving the input image.

[0089] In this embodiment, after obtaining multiple fully aligned image pairs, these fully aligned image pairs can be used as sample image pairs, and the image style transformation model can be trained using the sample image pairs to achieve image style transformation, for example, the characters in the input image can be transformed into cartoon-style characters. For example, the input image is Figure 2 , the output image is Figure 3 Images in the style shown.

[0090] In this embodiment, the target image and the reference image are used as sample image pairs for training to obtain an image style transformation model with better transformation effect, which can meet various image style transformation requirements and optimize user experience.

[0091] In some implementation scenarios, the similarity between the image to be processed and the reference image is greater than a first preset value, but the overlap between the region of the first target object in the image to be processed and the region of the second target object in the reference image is low. The overlap is one of the factors that affect whether the image to be processed and the reference image are completely aligned (similarity greater than a second preset value), so if the image of the region of the first target object is adjusted when the overlap between the region of the first target object and the region of the second target object is low, the adjustment effect is poor.

[0092] Therefore, if the preset area is the area of ​​the first target object, then adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value may include:

[0093] If the preset area is an area of ​​a target object, when the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is greater than a preset overlap threshold, adjusting the image of the area of ​​the first target object in the image to be processed according to the image of the area of ​​the second target object in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value;

[0094] When the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a preset overlap threshold, the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is adjusted to be greater than the preset overlap, and then the image of the area of ​​the first target object in the image to be processed is adjusted according to the image of the area of ​​the second target object in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value.

[0095] In some implementation scenarios, the similarity between the image to be processed and the reference image is high, and an image aligned with the reference image can be obtained by simply adjusting the facial region of the image to be processed. This embodiment provides a method for adjusting the facial region of the image to be processed, which may specifically include:

[0096] Identify key points of the facial area of ​​the first target object, wherein the key points include facial features and facial contour points;

[0097] identifying key points of the facial area of ​​the second target object;

[0098] The facial features of the first target object are kept unchanged, and the facial contour points of the first target object are adjusted according to the facial contour points of the second target object.

[0099] In this embodiment, the preset area is mainly the facial area of ​​the target object. Adjusting the facial area of ​​the first target object mainly involves adjusting the facial contour of the first target object.

[0100] In this embodiment, the key points of the facial area include the facial features and facial contour points. The facial features mainly include the key points of the eyebrow area, the key points of the eye area, the key points of the nose area, and the key points of the mouth area. The facial contour points mainly refer to the points located at the edge of the face.

[0101] Figure 4 FIG. 1 is a schematic diagram of key points of a facial region shown in an embodiment of the present application. Figure 4 In the figure, all points represent key points of the facial area. The larger black points at the edge represent the facial features, and the smaller black points represent the facial contours. Figure 4 Just as an example, Figure 4 The individual points do not represent actual identified keypoints.

[0102] In actual implementation, the facial contour points of the forehead are not stretched. Therefore, when identifying the key points of the facial area, only the facial features and those below the eyebrows can be identified. There is no need to identify the facial contour points of the forehead above the eyebrows, thereby reducing the recognition amount and improving image recognition efficiency.

[0103] In a specific implementation, the key points of the facial area of ​​the first target object in the image to be processed and the key points of the facial area of ​​the second target object in the reference image are first identified. Then, the facial features of the first target object are kept unchanged, and the facial contour points of the first target object are adjusted according to the facial contour points of the second target object. Specifically, the method may include:

[0104] When the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is greater than a preset degree of overlap, the positions of the facial features of the first target object are kept unchanged, and the positions of the corresponding facial contour points of the first target object are adjusted according to the positions of the facial contour points of the second target object.

[0105] In this embodiment, in order to completely overlap the facial region of the first target object with the facial region of the second target object, it is necessary to finely adjust the positions of the corresponding facial contour points of the first target object according to the positions of the facial contour points of the second target object. During the adjustment process, the positions of the facial features of the first target object remain fixed, thereby ensuring that the facial features of the first target object do not change, so as to retain the identity characteristics of the first target object.

[0106] In this embodiment, the positions of the corresponding facial contour points of the first target object are finely adjusted according to the positions of the facial contour points of the second target object, so that the image of the facial area of ​​the first target object and the image of the facial area of ​​the second target object can be well aligned. During the adjustment process, the positions of the facial features of the first target object are kept unchanged, so that the identity characteristics of the first target object can be well preserved.

[0107] In combination with the above embodiments, in one implementation, this embodiment further provides a method for adjusting the facial contour of a first target object. Specifically, keeping the facial features of the first target object unchanged, adjusting the facial contour points of the first target object according to the facial contour points of the second target object may include the following steps:

[0108] Generating a first triangulation graph according to each key point of the facial region of the first target object, wherein the first triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0109] Generating a second triangulation graph according to each key point of the facial region of the second target object, wherein the second triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0110] The vertices of the triangle where the facial feature points are located in the first triangulation graph are kept fixed, and the positions of the vertices of the triangle where the facial contour points are located in the first triangulation graph are adjusted according to the positions of the vertices of the triangle where the facial contour points are located in the second triangulation graph.

[0111] In this embodiment, when adjusting the facial contour of the first target object according to the facial contour of the second target object, firstly, a first triangulation graph is generated according to each key point of the facial area of ​​the first target object, and a second triangulation graph is generated according to each key point of the facial area of ​​the second target object. The first triangulation graph and the second triangulation graph respectively include a plurality of triangles, and the vertices of each triangle are key points. Then, the positions of the vertices of the triangles where the facial features are located in the first triangulation graph are kept fixed, and the positions of the vertices of the triangles where the facial contour points are located in the second triangulation graph are finely adjusted according to the positions of the vertices of the triangles where the facial contour points are located in the first triangulation graph.

[0112] In this embodiment, a triangulation algorithm may be used to adjust the facial contour of the first target object according to the facial contour of the second target object, thereby achieving alignment of the facial area of ​​the first target object and the facial area of ​​the second target object while better preserving the identity features of the first target object.

[0113] In this embodiment, the key points of the facial areas of the image to be processed and the reference image can be identified, and then the facial contour points of the facial area of ​​the image to be processed are adjusted according to the facial contour points of the facial area of ​​the reference image, so that the overlap between the facial areas of the image to be processed and the reference image is higher, thereby avoiding ghosting around the facial area after style transformation through the model.

[0114] In some implementation scenarios, the similarity between the image to be processed and the reference image is average, and the facial area and the modified area of ​​the image to be processed need to be adjusted to obtain a target image aligned with the reference image. Therefore, for the case where the preset area is the area of ​​the target object, the present application also provides an image processing method, which may specifically include:

[0115] When the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value, obtaining a mask corresponding to the overlap area between the area of ​​the first target object and the area of ​​the second target object;

[0116] The image to be processed and the reference image are adjusted respectively using the mask corresponding to the overlapped area.

[0117] After adjusting the facial area of ​​the first target object, the modified area of ​​the first target object may change, for example, the hairstyle area may be deformed due to the facial contour adjustment. Therefore, after adjusting the facial contour points of the first target object according to the facial contour points of the second target object, it is necessary to align and adjust the area of ​​the first target object and the area of ​​the second target object again.

[0118] In this embodiment, in order to maximize the overlap between the area of ​​the first target object and the area of ​​the second target object, in addition to adjusting the processed image using the mask corresponding to the overlapping area between the area of ​​the first target object and the area of ​​the second target object, the reference image is also adjusted using the mask corresponding to the overlapping area.

[0119] The mask is a matrix containing the values ​​of each pixel, which is either 0 or 1. When the value is 1, it indicates that the pixel belongs to the overlapping area of ​​the first target object and the second target object, and when the value is 0, it indicates that the pixel does not belong to the overlapping area of ​​the first target object and the second target object.

[0120] In one implementation, obtaining a mask corresponding to an overlapping area between an area of ​​the first target object and an area of ​​the second target object may include:

[0121] Obtaining a mask corresponding to a region of a first target object in the image to be processed;

[0122] Obtaining a mask corresponding to a region of a second target object in the reference image;

[0123] The mask corresponding to the overlapping area is obtained according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object.

[0124] In this embodiment, the modified area can be a hairstyle area, a neck area, a clothing area, a hat area, etc. The modified area can be understood as other areas of the target object except the facial area. Figure 2 In the image, except for the facial area of ​​the character, the rest of the areas such as hairstyle, neck, and clothes are all modified areas.

[0125] When aligning and adjusting the region of the first target object and the region of the second target object, firstly, a mask corresponding to the region of the first target object and a mask corresponding to the region of the second target object are obtained.

[0126] For the mask corresponding to the area of ​​the first target object, when the value of a certain pixel point is 0, it means that the pixel point is not a pixel point in the area of ​​the first target object (that is, it may be a pixel point in the background area). When the value of a certain pixel point is 1, it means that the pixel point is a pixel point in the area of ​​the first target object. The mask corresponding to the area of ​​the first target object is multiplied with the image to be processed to obtain the image corresponding to the area of ​​the first target object.

[0127] For the mask corresponding to the area of ​​the second target object, when the value of a certain pixel is 0, it means that the pixel is not a pixel in the area of ​​the second target object (that is, it may be a pixel in the background area), and when the value of a certain pixel is 1, it means that the pixel is a pixel in the area of ​​the second target object. The mask corresponding to the area of ​​the second target object is multiplied with the reference image to obtain the image corresponding to the area of ​​the second target object.

[0128] Next, obtain the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object, that is, the mask corresponding to the overlapping area between the area of ​​the first target object and the area of ​​the second target object. When the value of a certain pixel point in the intersection is 1, it means that the pixel point belongs to both the area of ​​the first target object and the area of ​​the second target object. When the value of a certain pixel point in the intersection is 0, it means that the pixel point belongs to both the area of ​​the first target object and the area of ​​the second target object.

[0129] Using the mask corresponding to the overlapped area to adjust the image to be processed and the reference image respectively may specifically include:

[0130] Multiplying the mask corresponding to the overlapped area with the image to be processed to obtain an image of the area of ​​the first object;

[0131] The mask corresponding to the overlapped area is multiplied by the reference image to obtain an image of the area of ​​the second object.

[0132] In this embodiment, after the mask corresponding to the overlapped area is multiplied by the image to be processed and the reference image respectively, the image of the area of ​​the first object and the image of the area of ​​the second object can be completely overlapped.

[0133] Through this embodiment, the image of the area of ​​the first object in the image to be processed can be better overlapped with the image of the area of ​​the second object in the reference image, thereby ensuring the image alignment effect between the obtained target image and the reference image.

[0134] In this embodiment, the mask corresponding to the overlapping area can be obtained according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object. The mask corresponding to the overlapping area can be used to eliminate the modified area that has changed, thereby ensuring that there will be no obvious ghosting defects in the modified area after the style change (for example, ensuring that there will be no obvious ghosting around the hair outline before and after the style change).

[0135] In combination with the above embodiments, in one implementation, the image to be processed includes a background area and an area of ​​the first target object. The reference image includes a background area and an area of ​​the second target object. On this basis, the image processing method of the present application may further include:

[0136] The background areas of the image to be processed and the reference image are respectively replaced with the same background image.

[0137] In this embodiment, some image style transfer models require that the background of the image pair is the same. Therefore, when obtaining the aligned image pair, after performing the aforementioned facial region (facial contour) adjustment and the region of the target object (including the modified region), the background regions of the image to be processed and the reference image can be replaced with the same background image.

[0138] Figure 5 It is a schematic diagram of a target image shown in an embodiment of the present application. Figure 6 It is a schematic diagram of another reference image shown in an embodiment of the present application. Figure 2 After the image to be processed in is adjusted for the facial region, the region of the target object, and the background is replaced, the target image obtained is as follows: Figure 5 shown. Figure 3 After the reference image in the target object is adjusted and the background is replaced, the final target image is as follows: Figure 6 shown.

[0139] In this embodiment, images of the background areas of the image to be processed and the reference image can be identified (for example, a matting mask of the foreground can be obtained through a background matting model, and the background area and the foreground area can be distinguished through the matting mask, and the foreground area is the area of ​​the target object), and the background image of the image pair is replaced with the same background image, thereby ensuring that the background before and after the style change does not change.

[0140] Figure 7 This is a schematic diagram of a complete process of obtaining a target image according to an embodiment of the present application. Figure 7 , the image processing method of the present application is described in detail with a complete embodiment, which includes the following steps 1 to 5.

[0141] Step 1: Get the original paired image data, one frame is the image to be processed, such as Figure 2 As shown, one frame is the reference image, such as Figure 3 As shown in Figure 2, the original paired data is image data that is not completely aligned.

[0142] Step 2: Identify the key points of the facial area in the image to be processed and the key points of the facial area in the reference image (including facial features and facial contour points), keep the facial features of the first target object in the image to be processed unchanged, and adjust the facial contour points of the first target object according to the facial contour points of the second target object in the reference image, so that the facial area in the image to be processed and the facial area in the reference image can completely overlap.

[0143] In step 2, any algorithm may be used to identify key points of the facial area, and this embodiment does not impose any specific limitation on the identification algorithm.

[0144] Step 3: After step 2, detect whether the modified area (such as the hairstyle area, the neck area, etc.) in the image to be processed has changed. If it has changed, obtain the mask corresponding to the overlapping area between the area of ​​the first target object and the area of ​​the second target object, and use the mask corresponding to the overlapping area to adjust the image to be processed and the reference image respectively, so that the area of ​​the first target object and the area of ​​the second target object can completely overlap. If there is no change, go to step 4.

[0145] Step 4: For the image to be processed and the reference image obtained in step 3, the background area is identified respectively, and the images of the background areas of both images are replaced with a frame of the same background image.

[0146] In step 4, any algorithm may be used to identify the background area, and this embodiment does not impose any specific limitation on the algorithm for identifying the background area.

[0147] After step 4, the adjusted target image (such as Figure 5 ) and reference images (such as Figure 6 ), which are a completely aligned image pair.

[0148] Step 5: Repeat steps 1 to 4 above to obtain multiple fully aligned image pairs, use these image pairs as sample image pairs, train the initial image style transformation model, and obtain an image style transformation model. Figure 5 ), style transformation can be performed to obtain the style transformed image (such as Figure 6 ).

[0149] In the related art, after obtaining the original paired image data, directly using the original paired image data to train the image style transformation model will have the following problems:

[0150] 1. The facial deformation of the original paired image data is large, resulting in ghosting around the facial area after style transformation.

[0151] 2. The modified areas of the original paired image data are quite different, resulting in obvious ghosting defects in the modified areas (such as the hairstyle area) after the style transformation.

[0152] 3. The background areas of the original paired image data are quite different, which causes the background around the facial area to change after the style transformation.

[0153] In response to the first problem mentioned above, in the present application, the key points of the facial areas of the image to be processed and the reference image can be identified, and then the facial contour points of the facial area of ​​the image to be processed are adjusted according to the facial contour points of the facial area of ​​the reference image, so that the overlap between the images of the facial area of ​​the image to be processed and the reference image is higher, and then an image style transformation model with better transformation effect is trained to avoid ghosting around the facial area after the style transformation.

[0154] In response to the second problem mentioned above, in the present application, the mask corresponding to the overlapping area can be obtained according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object. The mask corresponding to the overlapping area can be used to eliminate the modified area that has changed, thereby ensuring that there will be no obvious ghosting defects in the modified area after the style change (for example, ensuring that there will be no obvious ghosting around the hair outline before and after the style change).

[0155] Regarding the third problem, since the original paired image data obtained cannot guarantee that the background is exactly the same, some image pairs have very different backgrounds. In this application, the images of the background areas of the image to be processed and the reference image can be identified, and the background images of the image pairs can be replaced with the same background images, thereby ensuring that the background does not change before and after the style transformation.

[0156] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0157] Based on the same inventive concept, an embodiment of the present invention provides an image processing device 800. Figure 8 , Figure 8 FIG. 1 is a structural block diagram of an image processing device shown in an embodiment of the present application. Figure 8 As shown, the device 800 includes:

[0158] A first obtaining module 801 is used to obtain an image to be processed and a reference image for the image to be processed, wherein the similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value;

[0159] A first adjustment module 802, configured to adjust the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image, until the similarity between the image to be processed and the reference image is greater than a second preset value;

[0160] The first determining module 803 is configured to use the adjusted image to be processed as a target image.

[0161] Optionally, the device 800 further includes:

[0162] A second determination module, configured to use the target image and the reference image as a sample image pair;

[0163] A training module is used to train an initial image style transformation model using the sample image pairs to obtain an image style transformation model, wherein the image style transformation model is used to output an image having a similarity with the input image greater than a second preset value after receiving the input image.

[0164] Optionally, the first adjustment module 802 includes:

[0165] A first recognition submodule, used to recognize key points of the facial area of ​​the first target object, wherein the key points include facial features and facial contour points;

[0166] A second recognition submodule, used to recognize key points of the facial area of ​​the second target object;

[0167] The first adjustment submodule is used to keep the facial features of the first target object unchanged and adjust the facial contour points of the first target object according to the facial contour points of the second target object.

[0168] Optionally, the first adjustment submodule includes:

[0169] A first generating submodule, configured to generate a first triangulation graph according to each key point of the facial region of the first target object, wherein the first triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0170] A second generating submodule, configured to generate a second triangulation graph according to each key point of the facial region of the second target object, wherein the second triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point;

[0171] The second adjustment submodule is used to keep the vertices of the triangle where the facial feature points in the first triangulation graph are located fixed, and adjust the positions of the vertices of the triangle where the facial contour points in the first triangulation graph are located according to the positions of the vertices of the triangle where the facial contour points in the second triangulation graph are located.

[0172] Optionally, the area of ​​the first target object includes a facial area and a modified area of ​​the first target object, and the area of ​​the second target object includes a facial area and a modified area of ​​the second target object; the apparatus 800 further includes:

[0173] A second obtaining module, configured to obtain a mask corresponding to an overlapping area between the area of ​​the first target object and the area of ​​the second target object when the overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value;

[0174] The second adjustment module is used to adjust the image to be processed and the reference image respectively by using the mask corresponding to the overlapped area.

[0175] Optionally, the second obtaining module includes:

[0176] A first obtaining submodule, used to obtain a mask corresponding to a region of a first target object in the image to be processed;

[0177] A second obtaining submodule, used to obtain a mask corresponding to a region of a second target object in the reference image;

[0178] The third obtaining submodule is used to obtain the mask corresponding to the overlapping area according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object.

[0179] Optionally, the image to be processed includes a background area and an area of ​​the first target object, the reference image includes a background area and an area of ​​the second target object, and the apparatus 800 further includes:

[0180] The replacement module is used to replace the background areas of the image to be processed and the reference image with the same background image.

[0181] The embodiment of the present invention also provides a terminal device, such as Fig. 9 shown. Fig. 9 is a schematic diagram of the structure of a terminal device shown in an embodiment of the present application. Fig. 9 The terminal device includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92 and the memory 93 communicate with each other via the communication bus 94;

[0182] Memory 93, used for storing computer programs;

[0183] The processor 91 is used to execute the program stored in the memory 93 to implement the following steps:

[0184] Obtaining an image to be processed and a reference image for the image to be processed, wherein a similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value;

[0185] Adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value;

[0186] The adjusted image to be processed is used as the target image.

[0187] Alternatively, when the processor 91 executes the program stored in the memory 93 , the steps in the above-mentioned other method embodiments are implemented.

[0188] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0189] The communication interface is used for communication between the above terminal and other devices.

[0190] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0191] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0192] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the web page display method described in any of the above embodiments.

[0193] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the web page display method described in any one of the above embodiments.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0195] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0196] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Obtaining an image to be processed and a reference image for the image to be processed, wherein a similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value; Adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image until the similarity between the image to be processed and the reference image is greater than a second preset value; The adjusted image to be processed is used as the target image; The step of adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image comprises: When the degree of overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value, obtaining a mask corresponding to the overlap area between the area of ​​the first target object and the area of ​​the second target object; The image to be processed and the reference image are adjusted respectively using the mask corresponding to the overlapped area.

2. The method according to claim 1, characterized in that The method further comprises: Taking the target image and the reference image as a sample image pair; The sample image pairs are used to train an initial image style transformation model to obtain an image style transformation model, wherein the image style transformation model is used to output an image having a similarity with the input image greater than a second preset value after receiving the input image.

3. The method according to claim 1, characterized in that Adjusting the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image includes: Identify key points of the facial area of ​​the first target object, wherein the key points include facial features and facial contour points; identifying key points of the facial area of ​​the second target object; The facial features of the first target object are kept unchanged, and the facial contour points of the first target object are adjusted according to the facial contour points of the second target object.

4. The method according to claim 3, characterized in that Keeping the facial features of the first target object unchanged, and adjusting the facial contour points of the first target object according to the facial contour points of the second target object, comprises: Generating a first triangulation graph according to each key point of the facial region of the first target object, wherein the first triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point; Generating a second triangulation graph according to each key point of the facial region of the second target object, wherein the second triangulation graph includes a plurality of triangles, and each vertex of the triangle is a key point; The vertices of the triangle where the facial feature points are located in the first triangulation graph are kept fixed, and the positions of the vertices of the triangle where the facial contour points are located in the first triangulation graph are adjusted according to the positions of the vertices of the triangle where the facial contour points are located in the second triangulation graph.

5. The method according to claim 3, characterized in that: The area of ​​the first target object includes a facial area and a modified area of ​​the first target object, and the area of ​​the second target object includes a facial area and a modified area of ​​the second target object.

6. The method according to claim 5, characterized in that Obtaining a mask corresponding to an overlapping area between an area of ​​the first target object and an area of ​​the second target object includes: Obtaining a mask corresponding to a region of a first target object in the image to be processed; Obtaining a mask corresponding to a region of a second target object in the reference image; The mask corresponding to the overlapping area is obtained according to the intersection of the mask corresponding to the area of ​​the first target object and the mask corresponding to the area of ​​the second target object.

7. The method according to claim 3 or 5, characterized in that: The image to be processed includes a background area and an area of ​​the first target object, the reference image includes a background area and an area of ​​the second target object, and the method further includes: The background areas of the image to be processed and the reference image are respectively replaced with the same background image.

8. An image processing device, characterized in that: include: A first obtaining module, used to obtain an image to be processed and a reference image for the image to be processed, wherein a similarity between a first target object in the image to be processed and a second target object in the reference image is greater than a first preset value; A first adjustment module, configured to adjust the image of the corresponding area in the image to be processed according to the image of the preset area in the reference image, until the similarity between the image to be processed and the reference image is greater than a second preset value; A first determination module, used for taking the adjusted image to be processed as a target image; A second obtaining module, configured to obtain a mask corresponding to an overlapping area between the area of ​​the first target object and the area of ​​the second target object when the overlap between the area of ​​the first target object and the area of ​​the second target object is not greater than a third preset value; The second adjustment module is used to adjust the image to be processed and the reference image respectively by using the mask corresponding to the overlapped area.

9. A terminal device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the image processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images