Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic equipment and storage medium. The image processing method comprises: acquiring a first image, wherein the first image comprises an object of a first gender; performing gender transformation on the object of the first gender by using a gender transformation model to obtain a second image comprising an object of a second gender; and performing image inpainting on a to-be-repaired region in the second image to obtain a third image, wherein the to-be-repaired region comprises an image region corresponding to a hair region of the object of the first gender in the second image which is processed due to gender transformation.

Description

Technical Field

[0001] The present disclosure relates to the field of signal processing, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] In image processing, changing the gender of people in an image can help generate images that don't exist in reality. Gender change is a very straightforward facial attribute editing task: after inputting a portrait image and performing gender change processing, the output is an image of the person with the reversed gender. However, existing models for gender change primarily focus on the differences in image features between the two genders in order to perform targeted changes. However, because male and female images don't differ much in areas like background and clothing, the model cannot learn this knowledge. Therefore, when generating gender-changed images, blurring and "fake at first glance" phenomena often occur, resulting in poor results for the gender-changed images. Summary of the Invention

[0003] The present disclosure provides an image processing method, an electronic device, and a storage medium to at least solve the problem in the related art that images obtained by gender conversion have poor effects.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: acquiring a first image, wherein the first image includes an object of a first gender; performing gender transformation on the object of the first gender using a gender transformation model to obtain a second image including an object of a second gender; and performing image restoration on an area to be restored in the second image to obtain a third image, wherein the area to be restored includes an image area in the second image corresponding to a hair area of ​​the object of the first gender that has been processed due to gender transformation.

[0005] Optionally, obtaining the first image includes: obtaining an input image, and obtaining an image of a predetermined size containing a facial area of ​​an object of the first gender based on the input image as the first image; the image processing method also includes: performing image fusion on the third image and the input image to obtain a fourth image.

[0006] Optionally, the image processing method further includes: performing image restoration on the hair area in the remaining image areas of the fourth image except the image area corresponding to the third image, to obtain an output image.

[0007] Optionally, performing image restoration on the area to be restored in the second image to obtain a third image includes: acquiring the area to be restored; and performing image restoration on the area to be restored based on features of a first background area in which the area to be restored is located to obtain the third image.

[0008] Optionally, the image inpainting on the to-be-inpainted region based on the feature of the first background region comprises: processing the to-be-inpainted region to have the same or similar feature as the first background region based on the feature of the first background region.

[0009] Optionally, the processing the to-be-inpainted region to have the same or similar feature as the first background region based on the feature of the first background region comprises: performing image inpainting on each sub-region in the to-be-inpainted region from outside to inside based on the feature of the first background region.

[0010] Optionally, if the first background region comprises at least a part of the region of the object, the feature of the first background region comprises a semantic feature of the at least a part of the region of the object.

[0011] Optionally, the image inpainting on the hair region in the remaining image region in the fourth image except for the image region corresponding to the third image to obtain an output image comprises: obtaining the hair region; and performing image inpainting on the hair region based on a feature of a second background region in which the hair region is located to obtain an output image.

[0012] Optionally, the image inpainting on the hair region based on the feature of the second background region in which the hair region is located comprises: processing the hair region to have the same or similar feature as the second background region based on the feature of the second background region.

[0013] Optionally, the processing the hair region to have the same or similar feature as the second background region based on the feature of the second background region comprises: performing image inpainting on each sub-region in the hair region from outside to inside based on the feature of the second background region.

[0014] Optionally, if the second background region comprises at least a part of the region of the object, the feature of the second background region comprises a semantic feature of the at least a part of the region of the object.

[0015] Optionally, the gender transformation model comprises a generator of a generative adversarial network, and the gender transformation on the object of the first gender by using the gender transformation model to obtain the second image containing the object of the second gender comprises: transforming a feature related to gender among features of the first image by using the generator in the generative adversarial network to generate the second image containing the object of the second gender.

[0016] According to a second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, which comprises: an image acquisition unit configured to acquire a first image, wherein the first image comprises an object of a first gender; a gender transformation unit configured to perform gender transformation on the object of the first gender by using a gender transformation model to obtain a second image comprising an object of a second gender; and an image inpainting unit configured to perform image inpainting on a to-be-inpainted region in the second image to obtain a third image, wherein the to-be-inpainted region comprises an image region in the second image corresponding to a hair region of the object of the first gender processed due to the gender transformation.

[0017] Optionally, the acquiring the first image comprises: acquiring an input image, and acquiring an image of a face region of the object of the first gender with a predetermined size based on the input image as the first image; and the image inpainting unit is further configured to: perform image fusion on the third image and the input image to obtain a fourth image.

[0018] Optionally, the image inpainting unit is further configured to: perform image inpainting on a hair region in a remaining image region in the fourth image except for an image region corresponding to the third image to obtain an output image.

[0019] Optionally, the performing image inpainting on the to-be-inpainted region in the second image to obtain the third image comprises: acquiring the to-be-inpainted region; and performing image inpainting on the to-be-inpainted region based on a feature of a first background region in which the to-be-inpainted region is located to obtain the third image.

[0020] Optionally, the performing image inpainting on the to-be-inpainted region based on the feature of the first background region comprises: processing the to-be-inpainted region to have the same or similar feature as the first background region based on the feature of the first background region.

[0021] Optionally, the processing the to-be-inpainted region to have the same or similar feature as the first background region based on the feature of the first background region comprises: performing image inpainting on each sub-region in the to-be-inpainted region from outside to inside based on the feature of the first background region.

[0022] Optionally, if the first background region comprises at least a part of the object, the feature of the first background region comprises a semantic feature of the at least a part of the object.

[0023] Optionally, the image inpainting of the hair region in the remaining image region of the fourth image except for the image region corresponding to the third image to obtain an output image comprises: obtaining the hair region; and performing image inpainting on the hair region based on a feature of a second background region in which the hair region is located to obtain the output image.

[0024] Optionally, the image inpainting of the hair region based on the feature of the second background region comprises: processing the hair region to have the same or similar feature as the second background region based on the feature of the second background region.

[0025] Optionally, the processing of the hair region to have the same or similar feature as the second background region based on the feature of the second background region comprises: performing image inpainting on each sub-region in the hair region from outside to inside based on the feature of the second background region.

[0026] Optionally, if the second background region comprises at least one partial region of an object, the feature of the second background region comprises a semantic feature of the at least one partial region of the object.

[0027] Optionally, the gender transformation model comprises a generator of a generative adversarial network, and the gender transformation of the object of the first gender to obtain the second image comprising the object of the second gender comprises: transforming a feature related to gender among features of the first image by using the generator in the generative adversarial network to generate the second image comprising the object of the second gender.

[0028] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, and the electronic device comprises: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the image processing method as described above.

[0029] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium storing instructions is provided, and the instructions, when executed by at least one processor, cause the at least one processor to perform the image processing method as described above.

[0030] According to a fifth aspect of embodiments of the present disclosure, a computer program product is provided, and the computer program product comprises computer instructions, and the computer instructions, when executed by a processor, implement the image processing method as described above.

[0031] The technical scheme provided by the embodiment of the present disclosure at least brings the following beneficial effects: according to the image processing method of the embodiment of the present disclosure, after the gender transformation model is used to perform gender transformation on the object of the first gender to obtain the second image containing the object of the second gender, the image region corresponding to the hair region of the object processed due to gender transformation in the second image is subjected to image inpainting to obtain the third image, and therefore, the image effect after gender transformation is effectively improved.

[0032] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are incorporated into and form part of the specification, illustrate exemplary embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0034] Figure 1 is an exemplary system architecture to which exemplary embodiments of the present disclosure can be applied;

[0035] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;

[0036] Figure 3 is a schematic diagram of an example of an image processing method according to an exemplary embodiment of the present disclosure;

[0037] Figure 4 is a block diagram showing an image processing device according to an exemplary embodiment of the present disclosure;

[0038] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order for those skilled in the art to better understand the technical scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0041] It is to be noted that "at least one of a plurality of items" appearing in the present disclosure means all the three cases of "any one of the plurality of items", "a combination of any two or more of the plurality of items", and "all of the plurality of items". For example, "including at least one of A and B" means all the three cases of (1) including A alone, (2) including B alone, and (3) including both of A and B. Also, for example, "performing at least one of step one and step two" means all the three cases of (1) performing step one alone, (2) performing step two alone, and (3) performing both of step one and step two.

[0042] Figure 1 An exemplary system architecture 100 in which exemplary embodiments of the present disclosure can be applied is shown.

[0043] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as image or video data upload requests, image or video data download requests), etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as audio / video communication software, audio / video recording software, instant messaging software, conference software, email client, social platform software, etc. In addition, various image or video shooting / editing applications can also be installed on the terminal devices 101, 102, and 103. The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with a display screen and capable of playing, recording, editing, etc. audio / video, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices, and can be implemented as multiple software or software modules (such as for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0044] The terminal devices 101, 102, and 103 can be installed with image acquisition devices (e.g., cameras) to acquire image or video data. In practice, the minimum visual unit constituting a video is a frame. Each frame is a static image. A sequence of frames that are continuous in time are synthesized together to form a dynamic video. In addition, the terminal devices 101, 102, and 103 can also be installed with components (e.g., speakers) for converting electrical signals into sound to play sound, and can also be installed with devices (e.g., microphones) for converting analog audio signals into digital audio signals to acquire sound. In addition, the terminal devices 101, 102, and 103 can communicate with each other in voice or video.

[0045] The server 105 can be a server providing various services, such as a background server providing support for multimedia applications installed on the terminal devices 101, 102, and 103. The background server can analyze, store, and the like, data such as received audio-video data upload requests, and can also receive audio-video data download requests sent by the terminal devices 101, 102, and 103, and feed back audio-video data indicated by the audio-video data download requests to the terminal devices 101, 102, and 103.

[0046] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., to provide distributed services), or as a single software or software module. No specific limitation is made herein.

[0047] It should be noted that the image processing method provided by the embodiments of the present disclosure is generally executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in cooperation. Accordingly, the image processing apparatus can be disposed in a terminal device, a server, or both a terminal device and a server.

[0048] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is merely illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers, and the present disclosure has no limitation thereon.

[0049] Figure 2 is a flowchart of the image processing method of the exemplary embodiments of the present disclosure.

[0050] Referring to Figure 2, in step S210, a first image is acquired. Here, the first image includes an object of the first gender. For example, the first gender may be female. As an example, the first image may be an original image captured and including an object of the first gender, or the first image may be an image including an object of the first gender obtained by cropping the captured original image. For example, the first image may be acquired in the following manner: first, an input image is acquired. Here, the input image may be an original image captured and including an object of the first gender. Secondly, an image of a predetermined size containing a facial area of ​​an object of the first gender is acquired based on the input image as the first image.

[0051] In the following, for the sake of clarity Figure 2 The image processing method, combined with Figure 3 Specifically, refer to Figure 3 For example, in order to save computing resources and improve the stability of the subsequent gender conversion model, an image containing the face area of ​​the object of the first gender (the female in the input image) can be cropped from the input image first, and then the cropped image can be scaled to a predetermined size (for example, to a resolution of 512*512, but not limited thereto), and then input into the gender conversion model for gender conversion. In addition, optionally, in order to make the gender conversion model perform gender conversion more stably, in the case where the head of the object of the first gender in the input image is not straight, such as Figure 3 As shown, the cropped image can also be rotated appropriately to ensure that the subject's face is straight.

[0052] After acquiring the first image, next, in step S220, the gender of the objects of the first gender is transformed using the gender transformation model to obtain a second image containing objects of the second gender. For example, the gender transformation model may include a generator of a generative adversarial network. In this case, step S220 may include using the generator of the generative adversarial network to transform gender-related features among the features of the input image to generate the second image containing objects of the second gender.

[0053] The basic principle of Generative Adversarial Network (GAN) is as follows: A Generative Adversarial Network can include two networks, namely the Generator (hereinafter referred to as G) and the Discriminator (hereinafter referred to as D). Their functions are:

[0054] G is a network that generates images. It receives a random noise z and generates an image from this noise, which is recorded as G(z).

[0055] D is a discriminative network that determines whether an image is "real." Its input parameter is x, where x represents an image. The output D(x) represents the probability that x is a real image. If the output is 1, it means that the image is 100% real, while the output is 0, it means that the image cannot be real.

[0056] During training, the goal of the generative network G is to generate realistic images to deceive the discriminative network D. Meanwhile, the goal of D is to distinguish the images generated by G from real images. Thus, G and D form a dynamic "game." Ideally, G can generate images G(z) that are sufficiently realistic to be mistaken for real. However, it is difficult for D to determine whether the images generated by G are realistic, so D(G(z)) = 0.5. Ultimately, after training, a generative model G is obtained, which can be used to generate images.

[0057] In this application, when training a gender conversion model, images containing objects of the first gender are input as training samples into G. This generates gender-converted images containing objects of the second gender. This image is then input into the discriminator, and the model parameters are continuously adjusted until the probability that the images generated by the generator containing objects of the second gender are judged to be real images reaches a predetermined probability, for example, 0.5. The trained generator of the generative adversarial network can then be used as a gender conversion model to perform gender conversion. That is, when an image containing objects of the first gender is input into the generator, the generator can output an image containing objects of the second gender.

[0058] Specifically, during the gender conversion process, the generator is used to transform gender-related features within the features of the first image. Gender-related features are generally features that can produce gender differences between men and women. For example, women generally have longer hair than men, and long hair can be processed during gender conversion.

[0059] It should be noted that although in the above example, a generative adversarial network is used to perform gender conversion, in fact, any gender conversion model in the prior art that is pre-trained to perform gender conversion can be used to perform gender conversion, and this disclosure does not limit this.

[0060] After obtaining a second image containing an object of the second gender through gender transformation, as mentioned in the background of this application, the model usually focuses on the differences in the image features of the two genders in order to perform targeted transformations. However, in areas such as background and clothing, since there is little difference between male and female images, the model cannot learn this knowledge. Therefore, in the process of generating the gender-transformed image, blurring and "fake at first glance" phenomena often occur, resulting in poor results for the gender-transformed image. Therefore,Figure 2 As shown, in step S230, image inpainting is performed on the to-be-inpainted region in the second image to obtain a third image. Here, the to-be-inpainted region includes an image region in the second image corresponding to a hair region of the object of the first gender processed due to the gender transformation. For example, as shown in FIG. 2B, the image region in the second image corresponding to the hair region of the object of the first gender processed due to the gender transformation is to-be-inpainted. Figure 3 As shown, after the gender transformation is performed, blurring occurs at the image region in the second image corresponding to the hair region of the object of the first gender processed due to the gender transformation, and the image looks unreal at a first glance. Therefore, performing image inpainting on such a region can improve the effect of the overall image after the gender transformation.

[0061] Specifically, for example, first, the to-be-inpainted region can be obtained. Specifically, for example, the hair region of the object processed due to the gender transformation can be determined first, and then an image region corresponding to the hair region of the object in the second image can be determined as the to-be-inpainted region. After the to-be-inpainted region is obtained, the to-be-inpainted region can be inpainted based on the characteristics of the first background region where the to-be-inpainted region is located to obtain a third image. For example, the to-be-inpainted region can be processed to have the same or similar characteristics as the first background region based on the characteristics of the first background region. For example, the to-be-inpainted region can be processed to have the same characteristics as a part of the first background region by directly filling the to-be-inpainted region with a part of the first background region. The to-be-inpainted region can be processed to have similar characteristics as a part of the first background region by further performing some image processing (for example, performing partial feathering processing so that the filled region is more realistic and natural) after filling the to-be-inpainted region with a part of the first background region. Alternatively, the to-be-inpainted region can be processed to have similar characteristics as the first background region by directly using a pre-trained image inpainting model according to the characteristics of the first background region. The present disclosure does not limit the specific way in which the to-be-inpainted region is processed to have the same or similar characteristics as the first background region. As an example, the to-be-inpainted region can be processed to have the same or similar color characteristics and texture characteristics as the first background region according to the color characteristics and texture characteristics of the first background region. In the foregoing, similar characteristics may, for example, mean that the difference between the characteristics (for example, color characteristics, texture characteristics, etc.) of the to-be-inpainted region after processing and the characteristics of the first background region is within a predetermined range. For example, if the background region is clothing, the to-be-inpainted region can be processed according to the color characteristics and texture characteristics of the clothing so that the difference between the color characteristics and texture characteristics of the region after inpainting and the color characteristics and texture characteristics of the background clothing is within a predetermined range.

[0062] Typically, the hair area of ​​an object of the first gender that is processed due to gender reversal is relatively large. If the entire area is processed directly based on the characteristics of the background area, blurry and unnatural phenomena may appear in the middle area. To this end, optionally, according to an exemplary embodiment of the present disclosure, image restoration can be performed on each sub-area in the area to be restored from the outside to the inside based on the characteristics of the first background area. For example, the peripheral sub-area in the area to be restored can be processed first based on the characteristics of the first background area, and then the sub-area that is closer to the inside than the peripheral sub-area can be further processed based on the characteristics of the peripheral sub-area and / or the characteristics of the first background area, and the process is gradually carried out inward until the entire area to be restored is processed. In this way, the image restoration effect can be further improved and a more natural image can be obtained.

[0063] In addition, since the background area where the area to be repaired is located may include at least a part of the area of ​​the object itself, for example Figure 3 As shown, the area to be repaired (i.e., Figure 3 The background area where the first background area (the white area in the image after gender conversion) is located includes the shoulder area of ​​the object. In this case, in order to better repair the area to be repaired, for example, to make the shoulder of the object look neat and natural after the image is repaired, in addition to repairing the area to be repaired based on the color features, texture features, etc. of the background area, the area to be repaired can also be repaired in combination with the semantic features of at least a part of the object (for example, semantic layout information, such as where the shoulder is). That is, if the first background area includes at least a part of the object, the features of the first background area include the semantic features of at least a part of the object. By considering the semantic features of the object during image repair, it is convenient to consider the appropriate image repair method for the body parts of the object in a targeted manner.

[0064] Through the above S230, since the image area corresponding to the hair area of ​​the subject of the first gender that is processed due to gender conversion in the second image is repaired, the overall effect of the gender-converted image can be effectively improved. Figure 3 As shown in Figure 1, after image restoration, the background and clothes near the character's face have been restored.

[0065] As described above, optionally, in order to save computing resources and improve the stability of the gender transformation model for gender transformation, the image input into the model can be an image of a face region of an object of a first gender with a predetermined size. For example, the first image can be obtained in the following manner: first, an input image is obtained. Here, the input image can be an original image of a photographed object of a first gender. Second, an image of a face region of an object of a first gender with a predetermined size is obtained based on the input image as the first image. In this case, after the image inpainting of the to-be-inpainted region in the second image in step S230 obtains a third image, it is necessary to paste back the original image (i.e., the input image), therefore, optionally, Figure 2 The image processing method shown can further include: image fusion of the third image and the input image to obtain a fourth image.

[0066] In addition, since the face region is mainly cropped when the first image is obtained in step S210, the hair of the object can be truncated, for example, when the image is changed from a female to a male, the hair will become shorter, but the hair outside the cropping frame in the original image cannot be processed, therefore, the end of the long hair of the female in the final output image will not be processed. For example, as shown in Figure 3 After the image obtained by step S230 is pasted back to the original image, it can be seen that the end of the female hair is not processed.

[0067] Therefore, in order to further improve the image effect after the final gender transformation, optionally, Figure 2 The image processing method shown can further include: image inpainting of a hair region in a remaining image region of the fourth image except for an image region corresponding to the third image to obtain an output image. Specifically, the hair region (for example, Figure 3 the white region in the image pasted back to the original image in FIG. 8) can be obtained first, and then the hair region can be inpainted based on the characteristics of a second background region where the hair region is located to obtain an output image. For example, the hair region can be processed to have the same or similar characteristics as the second background region based on the characteristics of the second background region. Here, similar to the processing of the to-be-inpainted region in step S230, each sub-region in the hair region can also be inpainted from outside to inside based on the characteristics of the second background region. Moreover, if the second background region includes at least a part of the object, the characteristics of the second background region can also include semantic characteristics of the at least a part of the object. That is, in addition to the inpainting of the to-be-inpainted region according to the color characteristics, texture characteristics, etc. of the background region, the to-be-inpainted region can also be inpainted in combination with the semantic characteristics of the at least a part of the object. After the above further image inpainting processing, Figure 3As shown, the hair region originally outside the cropping frame is further processed, and finally, a better gender transformation effect can be obtained.

[0068] The above has introduced the image processing method according to the example embodiment of the present disclosure with reference to the examples of Figure 2 and in combination with Figure 3 The image processing method described above can effectively improve the image effect after gender transformation. For example, the image processing method described above can improve the generation effect of the hair occlusion region after transformation, and optionally, since the hair occlusion region of the whole image is repaired, the transformation effect of the long-haired female in the female-to-male image processing task can be effectively improved.

[0069] Figure 4 is a block diagram illustrating an image processing apparatus according to an example embodiment of the present disclosure.

[0070] With reference to Figure 4 , the image processing apparatus 400 can include an image acquisition unit 410, a gender transformation unit 420, and an image inpainting unit 430. Specifically, the image acquisition unit 410 can be configured to acquire a first image, wherein the first image includes an object of a first gender. As an example, the image acquisition unit 410 can acquire an input image, and acquire an image of a face region of the object of the first gender with a predetermined size based on the input image as the first image. The gender transformation unit 420 can be configured to perform gender transformation on the object of the first gender by using a gender transformation model to obtain a second image including an object of a second gender. The image inpainting unit 430 can be configured to perform image inpainting on a to-be-inpainted region in the second image to obtain a third image, wherein the to-be-inpainted region includes an image region in the second image corresponding to a hair region of the object of the first gender processed due to gender transformation. Optionally, the image inpainting unit 430 can be further configured to perform image fusion on the third image and the input image to obtain a fourth image. In addition, optionally, the image inpainting unit 430 can be further configured to perform image inpainting on a hair region in a remaining image region of the fourth image except for an image region corresponding to the third image to obtain an output image.

[0071] The image processing method shown in Figure 2 can be performed by the image processing apparatus 400 shown in Figure 4 , and the image acquisition unit 410, the gender transformation unit 420, and the image inpainting unit 430 perform steps S210, S220, and S230 in Figure 2 respectively, therefore, any related details involved in the operations performed by the units in Figure 4 can be referred to the corresponding descriptions of Figure 2 , which will not be repeated here.

[0072] Further, it should be noted that although the above image processing apparatus 400 is divided into units for performing respective processes, it is clear to those skilled in the art that the processes performed by the above units can also be performed without any specific division of the image processing apparatus 400 or without clear demarcation between the units. In addition, the image processing apparatus 400 can further include other units, such as a storage unit, etc.

[0073] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0074] Referring to Figure 5 The electronic device 500 can include at least one memory 501 storing computer executable instructions and at least one processor 502, which, when executing the computer executable instructions, causes the at least one processor 502 to perform the image processing method according to an embodiment of the present disclosure.

[0075] As an example, the electronic device can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but can be a collection of any devices or circuits capable of executing the above instructions (or instruction set) individually or jointly. The electronic device can also be part of an integrated control system or a system manager, or can be configured as a portable electronic device that interfaces with a local or remote (e.g., via wireless transmission) interface.

[0076] In the electronic device, the processor can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller, or a microprocessor. As an example and not a limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0077] The processor can run instructions or codes stored in the memory, where the memory can also store data. The instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0078] The memory can be integrated with the processor, for example, arranging RAM or flash memory within an integrated circuit microprocessor, etc. In addition, the memory can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory and the processor can be operatively coupled or can communicate with each other, for example, through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0079] In addition, the electronic device can further include a video display such as a liquid crystal display and a user interaction interface such as a keyboard, a mouse, a touch input device, etc. All components of the electronic device can be connected to each other via a bus and / or a network.

[0080] According to embodiments of the disclosure, a computer readable storage medium storing instructions can also be provided, wherein, when the instructions are executed by at least one processor, the at least one processor is caused to perform the image processing method according to the exemplary embodiments of the disclosure. Examples of the computer readable storage medium here include read only memory (ROM), random access programmable read only memory (PROM), electrically erasable programmable read only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc memory, hard disk drive (HDD), solid state drive (SSD), card memory such as a multimedia card, secure digital (SD) card or extreme digital (XD) card, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer program in the above computer readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0081] According to embodiments of the disclosure, a computer program product can also be provided, the computer program including computer instructions which, when executed by a processor, implement the image processing method according to the exemplary embodiments of the disclosure.

[0082] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, wherein the first image includes an object of a first gender; performing gender transformation on the object of the first gender using a gender transformation model to obtain a second image including the object of the second gender; Performing image restoration on the area to be restored in the second image to obtain a third image, wherein the area to be restored includes an image area in the second image corresponding to the hair area of ​​the subject of the first gender that has been processed due to gender transformation, The acquiring of the first image comprises: acquiring an input image, and acquiring an image of a predetermined size containing a face region of the subject of the first gender based on the input image as the first image; The image processing method further includes: Performing image fusion on the third image and the input image to obtain a fourth image; Image restoration is performed on the hair area in the remaining image areas of the fourth image except the image area corresponding to the third image to obtain an output image.

2. The image processing method according to claim 1, wherein: The performing image restoration on the area to be restored in the second image to obtain a third image includes: Acquire the area to be repaired; Image restoration is performed on the area to be restored based on features of the first background area where the area to be restored is located, to obtain the third image.

3. The image processing method according to claim 2, wherein: The performing image restoration on the area to be restored based on the features of the first background area where the area to be restored is located includes: Based on the features of the first background area, the area to be repaired is processed to have the same or similar features as the first background area.

4. The image processing method according to claim 3, wherein: The processing of the area to be repaired to have the same or similar features as the first background area based on the features of the first background area includes: performing image repair on each sub-area in the area to be repaired from the outside to the inside based on the features of the first background area.

5. The image processing method according to claim 4, wherein: If the first background area includes at least a portion of an object, the feature of the first background area includes a semantic feature of the at least a portion of the object.

6. The image processing method according to claim 1, wherein: The performing image restoration on the hair region in the remaining image regions of the fourth image except the image region corresponding to the third image to obtain an output image includes: obtaining the hair region; Image restoration is performed on the hair region based on features of the second background region where the hair region is located to obtain an output image.

7. The image processing method according to claim 6, wherein: The performing image restoration on the hair region based on the features of the second background region where the hair region is located includes: Based on the features of the second background region, the hair region is processed to have features identical or similar to those of the second background region.

8. The image processing method according to claim 7, wherein: The processing of the hair region to have the same or similar features as the second background region based on the features of the second background region includes: performing image restoration on each subregion in the hair region from outside to inside based on the features of the second background region.

9. The image processing method according to claim 8, wherein: If the second background area includes at least a portion of an object, the feature of the second background area includes a semantic feature of the at least a portion of the object.

10. The image processing method according to claim 1, wherein: The gender transformation model includes a generator of a generative adversarial network, wherein using the gender transformation model to transform the gender of an object of the first gender to obtain a second image containing an object of the second gender includes: A generator in a generative adversarial network is used to transform gender-related features among the features of the first image to generate the second image containing an object of the second gender.

11. An image processing device, characterized in that: include: an image acquisition unit configured to acquire a first image, wherein the first image includes an object of a first gender; a gender transformation unit configured to transform the gender of the object of the first gender using a gender transformation model to obtain a second image containing the object of the second gender; an image restoration unit configured to perform image restoration on a region to be restored in the second image to obtain a third image, wherein the region to be restored includes an image region in the second image corresponding to a hair region of the subject of the first gender that has been processed due to gender conversion, The image acquisition unit is configured to: acquire an input image, and acquire an image of a predetermined size containing a face area of ​​an object of the first gender based on the input image as a first image, The image restoration unit is further configured to: Performing image fusion on the third image and the input image to obtain a fourth image; Image restoration is performed on the hair area in the remaining image areas of the fourth image except the image area corresponding to the third image to obtain an output image.

12. The image processing device according to claim 11, wherein The performing image restoration on the area to be restored in the second image to obtain a third image includes: Acquire the area to be repaired; Image restoration is performed on the area to be restored based on features of the first background area where the area to be restored is located, to obtain the third image.

13. The image processing device according to claim 12, wherein: The performing image restoration on the area to be restored based on the features of the first background area where the area to be restored is located includes: Based on the features of the first background area, the area to be repaired is processed to have the same or similar features as the first background area.

14. The image processing device according to claim 13, wherein The processing of the area to be repaired to have the same or similar features as the first background area based on the features of the first background area includes: performing image repair on each sub-area in the area to be repaired from the outside to the inside based on the features of the first background area.

15. The image processing device according to claim 14, wherein If the first background area includes at least a portion of an object, the feature of the first background area includes a semantic feature of the at least a portion of the object.

16. The image processing device according to claim 11, wherein The performing image restoration on the hair region in the remaining image regions of the fourth image except the image region corresponding to the third image to obtain an output image includes: obtaining the hair region; Image restoration is performed on the hair region based on features of the second background region where the hair region is located to obtain an output image.

17. The image processing device according to claim 16, wherein The performing image restoration on the hair region based on the features of the second background region where the hair region is located includes: Based on the features of the second background region, the hair region is processed to have features identical or similar to those of the second background region.

18. The image processing device according to claim 17, wherein The processing of the hair region to have the same or similar features as the second background region based on the features of the second background region includes: performing image restoration on each subregion in the hair region from outside to inside based on the features of the second background region.

19. The image processing device according to claim 18, wherein If the second background area includes at least a portion of an object, the feature of the second background area includes a semantic feature of the at least a portion of the object.

20. The image processing device according to claim 11, wherein The gender transformation model includes a generator of a generative adversarial network, wherein using the gender transformation model to transform the gender of an object of the first gender to obtain a second image containing an object of the second gender includes: A generator in a generative adversarial network is used to transform gender-related features among the features of the first image to generate the second image containing an object of the second gender.

21. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the image processing method according to any one of claims 1 to 10.

22. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the image processing method according to any one of claims 1 to 10.

23. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images