Image processing method and device

Abstract

An image processing method and device are disclosed. The image processing method includes: extracting a person portrait from a real-life image and inputting it to an encoder to obtain a first vector; training an editor to modify the first vector to obtain a second vector, wherein the modification refers to a description text in a natural language format constructed based on user needs, and the degree of closeness between the person portrait generated based on the second vector and the description text meets the set requirements; providing the second vector to generate a specified image processing model to obtain a corrected portrait; fusing the corrected portrait with the person portrait in the real-life image to obtain a new real-life image. Compared with the prior art, the embodiment of the present disclosure edits the hidden variables of the portrait through a natural language supervision editor to generate a person portrait that meets the requirements, and obtains the naturalness of the new real-life image through the portrait fusion.

Description

Technical Field

[0001] The present disclosure relates to an application field of a combination of neural networks and image processing, and in particular, to an image processing method and device. Background Art

[0002] For companies, reshooting the materials that have already been shot (including posters, videos, flyers, etc.) will require a lot of manpower and resources. However, in some cases, new materials have to be shot. For example, when a product enters a new country, it is often necessary to use local models to shoot new promotional posters.

[0003] With the rapid application of neural networks, image processing models can generate high-definition facial images. However, the generation and replacement of real-life model portraits still face the following difficulties: 1) Models in the original materials often have specific postures. In order to ensure a good replacement effect, it is difficult to controllably generate faces with the same postures. 2) Different demanders have different requirements for character images, such as face shape, hair color, eyes, etc. How to generate corresponding character portraits is another difficulty. Summary of the invention

[0004] In view of this, the present disclosure provides an image processing method and device to controllably generate a character portrait and replace the character portrait of a real-life model.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Extracting a character portrait from a real scene image and inputting it into an encoder to obtain a first vector;

[0007] The training editor modifies the first vector to obtain a second vector, wherein the modification refers to a description text in a natural language format constructed based on user needs, and the degree of closeness between the character avatar generated based on the second vector and the description text meets a set requirement;

[0008] Providing the second vector to a specified image processing model to obtain a corrected head image;

[0009] A new real-scene image is obtained by fusing the corrected portrait with the portrait of the person in the real-scene image.

[0010] In some embodiments, extracting the portrait of a person from the real-scene image includes: using facial key points to detect multiple key points of the face and capturing the portrait of the person accordingly.

[0011] In some embodiments, extracting a person's portrait from a real-scene image further includes: rotating and aligning the face according to key points of the detected face.

[0012] In some embodiments, the image processing method further includes:

[0013] Acquire multiple sample head images that partially match the description text from a picture library and input them into an encoder to obtain multiple third vectors;

[0014] Then, in the step of modifying the first vector by the training editor, a loss function is established based on the Euclidean distance between the plurality of third vectors and the second vector.

[0015] In some embodiments, the editor, the specified image processing model, the text encoder, the image encoder and the image-text similarity model constitute a first network structure, the editor is trained by training the first network structure, and the loss function of the first network structure is constructed based on the following items: the Euclidean distance between the second vector and the first vector, the facial similarity between the reconstructed avatar output by the specified image processing model and the character avatar of the real image, and the image-text correlation between the descriptive text and the reconstructed avatar output by the specified image processing model.

[0016] In some embodiments, the designated image processing model is a generative model in a generative adversarial network, and the first vector and the second vector meet the requirements of latent variables in a latent space of the generative adversarial network.

[0017] In some embodiments, the fusion based on the modified portrait and the portrait of the person in the real scene image includes: performing portrait fusion by means of secondary reconstruction and smooth transition.

[0018] In some embodiments, the description text comes from the user's request for the model.

[0019] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0020] An extraction encoding unit, used for extracting the character portrait from the real scene image and inputting it to the encoder to obtain a first vector, wherein the first vector meets the requirements of generating latent variables of the latent space of the adversarial network;

[0021] An editor training unit, used for training an editor to modify the first vector to obtain a second vector, wherein the modification refers to a description text in a natural language format constructed based on user needs, and the degree of closeness between the character avatar generated based on the second vector and the description text meets a set requirement;

[0022] An image generating unit, configured to provide the second vector to a generative adversarial network model to obtain a modified avatar;

[0023] The image fusion unit is used to fuse the corrected head portrait with the head portrait of the person in the real scene image to obtain a new real scene image.

[0024] In some embodiments, the loss function adopted by the editor training unit is established based on the Euclidean distance between multiple third vectors and the second vector, and the multiple third vectors are obtained by the following steps: obtaining multiple sample avatars that partially match the description text from a picture library and inputting them into the encoder to obtain the multiple third vectors.

[0025] In some embodiments, the editor training unit is trained on a first network structure, the first network structure including: the editor, a generative model of a generative adversarial network, a text encoder, an image encoder and a picture-text similarity model, and the loss function adopted by the editor training unit is constructed based on the following items: the Euclidean distance between the second vector and the first vector, the facial similarity between the reconstructed avatar output by the generative model of the generative adversarial network and the character avatar of the real-life image, and the picture-text correlation between the description and the reconstructed avatar output by the generative model of the generative adversarial network.

[0026] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising a memory, a scheduler and a plurality of model acceleration units, wherein the memory further stores computer instructions executable by the processor and the plurality of model acceleration units, and when the computer instructions are executed, any of the above-described image processing methods is implemented.

[0027] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium stores computer instructions executable by an electronic device, and when the computer instructions are executed, any of the above-mentioned image processing methods is implemented.

[0028] The disclosed embodiment aims to complete the replacement of the character portrait in the real-life image according to the specified requirements. To this end, improvements are made in the following three aspects: first, since the representation of the face from the image level cannot decouple various attribute features and is not conducive to the generation of new character portraits, the character portrait is mapped to the latent space of the generative adversarial network to represent it, which is conducive to the subsequent modification of the facial attributes in the character portrait; second, the latent variables corresponding to the character portrait are edited through a natural language supervised editor to generate a character portrait that meets the requirements, thereby realizing the control of the character portrait generation by natural language; third, the characteristics of the continuous interpolation latent space of the generative adversarial network are utilized to perform secondary reconstruction of the unnaturally spliced ​​fused portrait to achieve a smooth fusion effect.

[0029] The above solution can achieve the goal of relatively naturally replacing the person portrait in the real-scene image with the person portrait constructed according to the specified requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0031] Figure 1 It is a deployment diagram of the application service;

[0032] Figure 2 is a schematic diagram of an exemplary server;

[0033] Figure 3 Some online shopping pages of the e-commerce system are shown;

[0034] Figure 4 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0035] Figure 5 A schematic diagram showing the principle of generating adversarial networks;

[0036] Figure 6 It is a schematic diagram of the residual network model;

[0037] Figure 7 This is a schematic diagram of the network structure built for training editors;

[0038] Figure 8 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0040] Regarding neural networks, it generally refers to artificial neural networks (ANN), which are algorithmic networks that imitate the behavioral characteristics of animal neural networks and perform distributed parallel information processing. A classic neural network, which is also the simplest neural network structure, contains three levels: input layer, output layer, and middle layer (also called hidden layer). The input layer, output layer, and middle layer each include multiple nodes. A node is the smallest processing unit in a neural network. A very complex neural network structure can be formed through the extensive interconnection of a large number of simple function nodes. In a neural network, the nodes are mathematized to generate a mathematical model of the nodes. The mathematical models of a large number of nodes in a neural network constitute a neural network model.

[0041] Regarding pre-training, as we all know, model training usually requires a lot of resources, including but not limited to a large number of training samples that need to be labeled and computer resources for performing training, so model training is not easy to perform. However, through pre-training, a "semi-finished" model with better weight parameters can be provided, which we usually call a pre-trained model. Before applying the pre-trained model to the real scene, the pre-trained model can also be incrementally trained using sample data collected in the real scene. Through incremental training, the weight parameters in the pre-trained model can be fine-tuned to obtain a "finished" model suitable for the real scene, which we usually call a trained model.

[0042] Regarding model acceleration units, they are hardware execution units dedicated to specific neural network models, including NPU (embedded neural network processor), TPU (tensor processing unit), etc., which are produced by different companies.

[0043] Figure 1 As shown in the figure, the deployment diagram 100 includes a terminal 103 and a server cluster 102 connected via a network 101 .

[0044] The network 101 is one or a combination of various communication technologies based on exchanged signals, including but not limited to wired technologies using electrical and / or optical transmission cables, and wireless technologies using infrared, radio frequency and / or other forms. In different application scenarios, the network 101 can be the Internet, a wide area network or a local area network, and can be a wired network or a wireless network. For example, the network 101 is a local area network within a company.

[0045] The server cluster 102 is composed of multiple physical servers. The terminal 103 can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. Various application systems are deployed on the server cluster 102, and the terminal 103 can obtain services provided by these application systems via the network 101.

[0046] With the development of cloud computing, the server cluster 102 can deploy a cloud service system. The cloud service system can aggregate the software and hardware resources in the cloud server cluster and provide software and hardware services according to the request from the terminal 103. For example, the cloud service system can provide a server (which may be a virtual machine) with a specified configuration to the user, where the specified configuration includes such as processor model, memory size, hard disk size, operating system type, various software types deployed on the operating system (such as text processing software, video playback software), etc. The user remotely accesses and uses this server to complete various tasks through the terminal 103. For another example, a specific application system is deployed on a server (which may be a virtual machine) and a service portal is provided to the user so that the user can obtain the corresponding functions provided by the application system through the service portal.

[0047] Figure 2 2 is a schematic diagram of an exemplary server. As shown in the figure, the server 200 may include but is not limited to: a scheduler 201 coupled via a bus 205, a storage unit 203, an I / O interface 204, a field programmable gate array (FPGA) 208, a graphics processing unit (GPU) 202, a neural network acceleration unit (NPU) 202, and a data transfer unit (DTU) 207.

[0048] The storage unit 203 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit. The storage unit 203 may also include a readable medium in the form of a non-volatile storage unit, such as a read-only memory unit (ROM), a flash memory, and various disk memories.

[0049] The storage unit 203 can store various program modules and data, including operating systems, applications that provide functions such as text processing, video playback, software editing and compilation, etc. The executable codes of these applications are read out and executed by the scheduler 201 from the storage unit 203 to achieve the functions predetermined by these program modules. The scheduler 201 is generally a CPU.

[0050] Bus 205 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0051] The server 200 may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the server 200, and / or any device that enables the server 200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via input / output (I / O) interface 204. In addition, the server 200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter (not shown). For example, via a network adapter, Figure 1 The terminal 103 in the figure can access the server 200. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used based on the server 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0052] The neural network acceleration unit (NPU) 202 adopts a data-driven parallel computing architecture and is a processing unit for processing a large number of operations (such as convolution, pooling, etc.) of each neural network node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each neural network node are closely related in the entire calculation process and will be used frequently, using the existing CPU architecture, since the memory capacity in the CPU core is very small, a large number of external core memories must be frequently accessed, resulting in inefficient processing. With the use of NPU, each core has an on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to the memory outside the core, which can greatly improve processing efficiency and computing performance.

[0053] The data transmission unit (DTU) 207 is a wireless terminal device specially used to convert serial port data into IP data or convert IP data into serial port data for transmission through a wireless communication network. The main function of DTU is to transmit the data of the remote device back to the background center wirelessly. At the front end, DTU and the customer's device are connected through an interface. After the DTU is powered on, it first registers to the mobile GPRS network, and then establishes a socket connection with the background center set in the DTU. The background center serves as the server of the socket connection, and the DTU is the client of the socket connection. Therefore, DTU and the background software are used together. After the connection is established, the front-end device and the background center can perform wireless data transmission through DTU.

[0054] The graphics processing unit (GPU) 206 is a microprocessor that is specialized in image and graphics related computing. The GPU exploits the shortcoming of the CPU having too little space for computing units and uses a large number of computing units specialized for graphics computing, which reduces the graphics card's dependence on the CPU and takes on some of the computationally intensive graphics and image processing tasks that the CPU originally took on.

[0055] Field Programmable Gate Array (FPGA) 208 is a product further developed on the basis of programmable devices such as PAL and GAL. It appears as a semi-custom circuit in the field of Application Specific Integrated Circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0056] It should be noted that the above and Figure 2 It is only used to exemplify the server in the system, and is not used to limit the specific implementation of the server. The server may also include other components, and the various components of the server described above may also be appropriately omitted in actual applications.

[0057] Figure 3 It shows some online shopping pages of the e-commerce system. Figure 3 In the figure, part of the online shopping page 301 contains graphic and text introduction information of a specific product, and the poster design drawing 3011 is a real-life image of a domestic model. The user wants to use a foreign model to replace the Chinese model in the poster, but cannot re-shoot the poster. In this case, the disclosed embodiment provides a method for replacing the character portrait in the real-life image, which constructs a virtual character portrait of a foreign model. The fictional foreign model has some specific facial features, such as deep eye sockets and blue eyes, and then replaces the domestic model's portrait in the poster design drawing 3011 with the virtual character portrait. In this way, it looks like a foreign model has taken the poster. The method can be implemented as an application system that provides the above-mentioned image processing services through a computer program, and the computer program can be deployed on Figure 1 The server cluster 102 shown provides the above-mentioned image processing service to users accessing via the terminal 103 .

[0058] Of course, there are other scenarios where similar operations need to be performed. For example, in the field of film and television production, for a film or TV series that has been shot, it is necessary to replace the characters in it. In this case, a virtual character avatar can be constructed using the method provided in the embodiment of the present disclosure, and the character avatar can be replaced with it.

[0059] Figure 4 The present disclosure provides a flowchart of an image processing method. The method is used to replace a person image from a real scene image. The method specifically includes the following steps.

[0060] In step S01, a head portrait of a person is extracted from a real scene image and input to an encoder to obtain a first vector.

[0061] In some embodiments, facial key point detection can be used to first extract multiple key points of the face and capture the face, and then the face can be rotated and aligned based on the detected key points of the face. The alignment method is to solve the affine transformation matrix based on the coordinates of the corresponding key points in the two postures, and then the affine transformation matrix is ​​used to rotate and align the face, and finally the rotated and aligned face is encoded into a first vector through an encoder.

[0062] In some embodiments, the first vector meets the requirements of latent variables of the latent space of the generative adversarial network. The generative adversarial network can be any image generative adversarial network such as StyleGAN, BigGAN, etc., which is trained by a large number of high-definition human faces. Figure 5 The schematic diagram of the principle of the generative adversarial network is shown in the figure. As shown in the figure, during the training process, the generative model 501 receives the picture vector and generates a reconstructed picture. Both the reconstructed picture and the real picture are provided to the judgment model 502, and the judgment model 502 determines whether it is a reconstructed picture or a real picture. Step 503 adjusts the weight parameters of the generative model 501 and the judgment model 502 based on whether the output result of the judgment model 502 is correct: if the judgment model 502 judges correctly, the weight parameters of the generative model 501 are adjusted to make the generated reconstructed picture more realistic; if the judgment model 502 judges incorrectly, the weight parameters of the judgment model 502 are adjusted to avoid the next judgment error. The training continues until it enters a balanced and harmonious state. Finally, a trained generative model and judgment model are obtained.

[0063] In some embodiments, the encoder can be obtained through training of any convolutional neural network. During the training process, the image is converted into an input vector of the generative adversarial network through a convolutional neural network, and then the input vector is provided to a trained generative model to obtain a reconstructed face, and then the weight parameters of the encoder are adjusted based on the loss function. The loss function includes a face recognition similarity loss function and a perceptual loss function (Perceptual loss). A loss function can also be constructed based on the Euclidean distance between the reconstructed face and the original face. Face recognition similarity is to extract facial features using a face recognition model and calculate cosine similarity as a loss function. The perceptual loss function is to extract image features through a pre-trained convolutional neural network and use the Euclidean distance between image features as a loss function.

[0064] In step S02, the training editor modifies the first vector to obtain a second vector.

[0065] In some embodiments, the editor can be a pre-trained neural network model, whose input is a first vector, and whose output is a second vector modified based on the first vector. The first vector and the second vector are vectors that meet the input requirements of a specified image processing model (for example, the generative model in the generative adversarial network mentioned above), and the first vector and the second vector can be input into the specified image processing model respectively to generate a character portrait, and the character portrait generated by the first vector is substantially consistent with the character portrait extracted from the original real-life image.

[0066] The training of the editor in this step refers to the input description text in natural language format, and through training, the second vector is finally obtained, which is close to the description text to the set requirements. That is, the character portrait generated by inputting the final second vector into the specified image processing model is very close to the description text. The description text here comes from user needs, for example, Figure 3 In the description, the user puts forward specific requirements for the model in the poster design.

[0067] It should be understood that the usual model training is carried out in a laboratory, and then the trained model is used in an actual scenario. However, in the present disclosure, the editor must be trained for the description text provided in each case in an actual scenario to obtain an editor and a second vector that meet the description text.

[0068] In some embodiments, multiple sample avatars that partially match the description text can be selected from the image library based on the description text provided by the user, that is, a part of each sample avatar (such as skin color, eye shape, size and distance, cheekbones) matches some requirements of the description text; then the multiple sample images are subjected to some image preprocessing mentioned in step S01, including but not limited to: first, multiple key points of the face can be extracted and the face can be captured using face key point detection, and then the face is rotated and aligned according to the key points of the detected face; then the character avatar after image preprocessing is encoded into multiple third vectors through an encoder, and the multiple third vectors can be integrated into a fourth vector based on the information that each sample avatar matches the description text, and a loss function is established based on the Euclidean distance between the second vector and the fourth vector. During training, when the loss function converges to a set degree, it indicates that the character image generated based on the current second vector is very close to the character avatar required by the description text. At this time, the training can be stopped, and the current second vector is used as the second vector finally output in this step.

[0069] In some embodiments, the Figure 7The network structure shown. The network structure consists of an editor 701, a generative model 702 in a generative adversarial network, an image encoder 703, a text encoder 704, and a picture-text similarity model 705. The generative model 702, the image encoder 703, the text encoder 704, and the picture-text similarity model 705 can all be trained models. This network structure is constructed to train the editor 701. The text encoder 704 converts the input description text into a text vector. The image encoder 703 converts the input reconstructed image into an image vector. The picture-text similarity model 705 outputs a similarity index based on the input text vector and image vector. The loss function used here includes three parts: the Euclidean distance between the modified second vector and the original first vector, the face recognition similarity, and the picture-text correlation loss. The weight coefficient of the editor is continuously adjusted through such a loss function. Here, the text encoder 704, the image encoder 703, and the picture-text similarity model 705 are integrated together as a model for training.

[0070] In some embodiments, Figure 7 In the network structure shown, the image-text correlation model can use multimodal pre-training models such as CLIP and M6. The loss function remains unchanged, and the weight coefficients of the editor and the image-text correlation model are trained simultaneously through such a loss function. Among them, CLIP (Contrastive Language-Image Pre-Training) is an image-text correlation matching model. M6 (Multi-Modality to Multi-Modality Multitask Mega-transformer) is a self-developed ultra-large-scale Chinese pre-training model.

[0071] In some embodiments, the editor is composed of multiple residual neural networks, and offsets of different layers are obtained by regression, and then the offsets are added to the original first vector as the modified second vector. Figure 6 A schematic diagram of the structure of a residual network model is shown, Figure 6 As shown, the residual network model includes at least one residual module ( Figure 6 Only one is shown), the residual module includes a first layer convolution module 61 and a second layer convolution module 62, the first layer convolution module 61 and the second layer convolution module 62 are connected through an activation function (ReLU), when the input of the residual module is X, after being processed by the first layer convolution module 61 and the second layer convolution module 62, F(X) is obtained, and then X and F(X) are added and passed to the next residual module, and the residual module is also connected to the next residual module through RELU.

[0072] In step S03, the second vector is provided to a specified image processing model to obtain a modified avatar.

[0073] Through step S02, the second vector output by the editor is input to the specified image processing model, and the degree of proximity between the character image generated by the model and the description text meets the set requirements. The specified image processing model is, for example, a generative model in a generative adversarial network. In this step, the second vector output by step S02 is input to the generative model in the generative adversarial network, and the generative model in the generative adversarial network outputs a corrected image.

[0074] In step S04, the corrected head portrait is merged with the head portrait of the person in the real scene image to obtain a new real scene image.

[0075] In some embodiments, the corrected image and the head portrait of the person in the real scene image may be directly spliced ​​together to form a new real scene image, and the splicing may be performed using multiple key points detected from the face of the person.

[0076] In some embodiments, in order to solve the problem of inconsistent skin color and obvious splicing marks when directly splicing the generated faces, it is proposed to fuse the portraits by smooth transition and secondary reconstruction. The smooth transition includes: Gaussian blurring the original face portrait and the new face in the revised portrait, and for each position in the head area, multiplying the face pixels in the revised portrait by the blurred original face pixels and dividing them by the blurred face pixels to generate face pixels, so as to achieve the purpose of skin color correction, and then Gaussian filtering the regional binary mask of the face portrait after skin color correction, and then using the filtered binary mask to fuse the revised portrait and the original real scene image.

[0077] The secondary reconstruction includes: extracting the face portrait again from the fused image, and mapping it into a vector through an encoder, then generating a new face portrait through a generative model of a generative adversarial network, and finally using a regional mask after Gaussian filtering to merge the face to obtain the final effect.

[0078] In summary, the disclosed embodiment aims to complete the replacement of the character portrait in the real-life image according to the specified requirements. To this end, improvements are made in the following three aspects: first, since the representation of the face from the image level cannot decouple various attribute features and is not conducive to the generation of new character portraits, the character portrait is represented by mapping it to the latent space of the generative adversarial network, which is conducive to the subsequent modification of the facial attributes in the character portrait; second, the latent variables corresponding to the character portrait are edited through a natural language supervised editor to generate a character portrait that meets the requirements, thereby realizing the control of the character portrait generation by natural language; third, the characteristics of the continuous interpolation latent space of the generative adversarial network are used to reconstruct the unnaturally spliced ​​fusion portrait to achieve a smooth fusion effect. The solution provided by the disclosed embodiment can achieve a relatively natural replacement of the character portrait in the real-life image with a character portrait constructed according to the specified requirements.

[0079] Figure 8 80 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure. The image processing device comprises: an extraction coding unit 801 , an editor training unit 802 , an image generation unit 803 and an image fusion unit 804 .

[0080] The extraction coding unit 801 is used to extract the character portrait from the real scene image 901 and input it to the encoder to obtain a first vector 903. The first vector 903 meets the requirements of generating latent variables in the latent space of the adversarial network.

[0081] In some embodiments, the extraction coding unit 801 first detects and extracts multiple (for example, 68) key points of the face from the real-scene image 901 and captures the face, and then rotates and aligns the face according to the multiple key points. The alignment method is to solve the affine transformation matrix according to the coordinates of the corresponding key points in two postures, and then use the affine transformation matrix to rotate and align the face, and finally encode the rotated and aligned face into a first vector through the encoder.

[0082] The editor training unit 802 is used to train the editor to modify the first vector 901 to obtain the second vector 904. That is, the purpose of the training is to output the second vector 902 through the editor that meets the following conditions: the degree of closeness between the character portrait generated based on the second vector and the description text 902 input by the user meets the set requirements.

[0083] In some embodiments, the editor can be a pre-trained neural network model, whose input is a first vector, and whose output is a second vector modified based on the first vector. The first vector and the second vector are vectors of the same type, and the first vector and the second vector are respectively input into the image processing model to generate a character portrait. The character portrait generated by the first vector is generally consistent with the character portrait extracted from the original real-life image, but the character portrait generated by the second vector will change. The purpose of training the editor is to obtain a second vector that meets the following conditions through training: the second vector is input into an image processing model (for example, a generative model in a generative adversarial network), and the character image generated by the model is close enough to the description text to meet the set requirements. The description text referred to here is the one in Figure 3 In the description, the user puts forward specific requirements for the model in the design drawing.

[0084] The image generation unit 803 is used to provide the second vector to the generative adversarial network, and obtain the corrected avatar through the generation model in the generative adversarial network. Based on the functional description of the editor training unit 802, it can be seen that the degree of closeness between the corrected avatar obtained by this unit and the description text 902 input by the user meets the set requirements.

[0085] In some embodiments, the image generation unit 803 inputs the second vector output by the editor training unit 802 to a generation model in a generative adversarial network, and the generation model in the generation adversarial network outputs a corrected image.

[0086] The image fusion unit 804 is used to fuse the corrected head portrait with the head portrait of the person in the real scene image to obtain a new real scene image.

[0087] In some embodiments, since directly splicing the generated faces will result in inconsistent skin color and obvious splicing marks, it is proposed to perform head portrait fusion through secondary reconstruction and smooth transition. First, the UNet segmentation model and key point detection are used to obtain the hair and facial areas and align the key points through affine transformation. The original face and the new face are Gaussian blurred. For each position in the head area, the generated face pixels are multiplied by the blurred original face pixels and divided by the blurred generated face pixels to achieve the purpose of skin color correction. Then, the head area mask is Gaussian filtered, and then the filtered mask is used to fuse the generated face and the original model.

[0088] It should be understood that the image processing device provided in the embodiment of the present disclosure corresponds to the method, so the various implementations mentioned in the method are also applicable to the device, and the device will not be described in a relatively simple manner here.

[0089] The implementation methods of the various embodiments of the present disclosure may be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure may be embodied in the form of a software product, which may be stored on a computer-readable medium and include several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present disclosure.

[0090] A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by a processing unit, device or device, or used in combination therewith.

[0091] Computer readable signal media may include data signals propagated in baseband or as part of a chopped wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction system, apparatus, or device.

[0092] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0093] The computer program code for executing the disclosed embodiments may be written in one or more programming languages ​​or combinations. The programming languages ​​include object-oriented programming languages, such as JAVA, C++, and may also include conventional procedural programming languages, such as C. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0094] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Extracting a character portrait from a real scene image and inputting it into an encoder to obtain a first vector; The training editor modifies the first vector to obtain a second vector, wherein the modification refers to a description text in a natural language format constructed based on user needs, and the degree of closeness between the character avatar generated based on the second vector and the description text meets a set requirement; Providing the second vector to a specified image processing model to obtain a corrected head image; Based on the fusion of the modified head portrait and the head portrait of the person in the real scene image, a new real scene image is obtained; Among them, the training editor modifies the first vector to obtain the second vector, including: selecting multiple sample avatars that partially match the description text from the picture library according to the description text provided by the user; determining multiple third vectors based on the multiple sample avatars; establishing a loss function based on the Euclidean distance between the multiple third vectors and the second vector. During training, when the loss function converges to a set degree, the training is stopped, and the current second vector is used as the final output of the second vector.

2. The image processing method according to claim 1, wherein: The extracting of the person's head portrait from the real scene image includes: using the key points of the face to detect multiple key points of the face and intercepting the person's head portrait accordingly.

3. The image processing method according to claim 2, wherein: The step of extracting a person's head portrait from a real-scene image further includes: rotating and aligning the face according to key points of the detected face.

4. The image processing method according to claim 1, wherein: The image processing method further comprises: The plurality of sample avatars are input into an encoder to obtain the plurality of third vectors.

5. The image processing method according to claim 1, wherein: The editor, the specified image processing model, the text encoder, the image encoder and the image-text similarity model constitute a first network structure. The editor is trained by training the first network structure, and the loss function of the first network structure is constructed based on the following items: the Euclidean distance between the second vector and the first vector, the facial similarity between the reconstructed avatar output by the specified image processing model and the character avatar of the real-life image, and the image-text correlation between the description text and the reconstructed avatar output by the specified image processing model.

6. The image processing method according to any one of claims 1 to 5, wherein: The designated image processing model is a generative model in a generative adversarial network, and the first vector and the second vector meet the requirements of latent variables in a latent space of the generative adversarial network.

7. The image processing method according to claim 1, wherein the fusion of the corrected head portrait and the head portrait of the person in the real scene image comprises: Secondary reconstruction and smooth transition are used for portrait fusion.

8. An image processing device, comprising: An extraction encoding unit, used for extracting a head portrait of a person from a real scene image and inputting it into an encoder to obtain a first vector, wherein the first vector meets the requirements of generating latent variables of a latent space of a confrontation network; An editor training unit, used for training an editor to modify the first vector to obtain a second vector, wherein the modification refers to a description text in a natural language format constructed based on user needs, and the degree of closeness between the character avatar generated based on the second vector and the description text meets a set requirement; An image generating unit, configured to provide the second vector to a generative adversarial network model to obtain a modified avatar; An image fusion unit, configured to fuse the corrected head portrait with the head portrait of the person in the real scene image to obtain a new real scene image; Among them, the editor training unit is also used to select multiple sample avatars that partially match the description text from the picture library according to the description text provided by the user; determine multiple third vectors based on the multiple sample avatars; establish a loss function based on the Euclidean distance between the multiple third vectors and the second vector. During training, when the loss function converges to a set degree, the training is stopped, and the current second vector is used as the final output of the second vector.

9. The image processing apparatus according to claim 8, wherein: The plurality of third vectors are obtained by the following steps: inputting the plurality of sample avatars to an encoder to obtain the plurality of third vectors.

10. The image processing device according to claim 8, wherein: The editor training unit is trained on a first network structure, which includes: the editor, a generative model of a generative adversarial network, a text encoder, an image encoder and a picture-text similarity model, and the loss function adopted by the editor training unit is constructed based on the following items: the Euclidean distance between the second vector and the first vector, the facial similarity between the reconstructed avatar output by the generative model of the generative adversarial network and the character avatar of the real-life image, and the picture-text correlation between the description text and the reconstructed avatar output by the generative model of the generative adversarial network.

11. A server comprising a memory, a scheduler and a plurality of model acceleration units, wherein the memory further stores computer instructions executable by a processor and the plurality of model acceleration units, wherein when the computer instructions are executed, the image processing method as described in any one of claims 1 to 7 is implemented.

12. A computer-readable medium storing computer instructions executable by an electronic device, wherein when the computer instructions are executed, the image processing method according to any one of claims 1 to 7 is implemented.

Images