Image processing method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which relate to the field of computers. The method comprises: obtaining a target partial image including a face from an image to be processed; inputting the target partial image into a trained comic face generation model to obtain a comic face image; replacing the face area content in the target partial image with the face area content in the comic face image to obtain a target comic face image; replacing the area corresponding to the target partial image in the image to be processed with the target comic face image to obtain a result image of the face comic. Thus, based on the image to be processed and the comic face generation model, a result image of the face comic can be automatically obtained, and the face area in the result image has a natural transition with other areas.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to an image processing method, device, electronic device, and readable storage medium. Background Art

[0002] Comics are an art form that we can widely access in our daily lives. In addition to art itself, its application ranges from publishing media to children's education. However, hand-drawing faces in a cartoon comic style is a time-consuming and laborious task, and requires a lot of professional painting skills. In order to obtain a high-quality picture, the original artist must carefully draw each line and color each color area of ​​the face. However, the workload of hand-drawing faces is relatively large and inconvenient to use. For example, if hand-drawing is used, the cartoonization of faces in live broadcast scenes cannot be achieved. Therefore, how to automatically convert photos of real people into satisfactory comic-style pictures is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which can automatically obtain a result image of human face cartoonization based on the image to be processed and a cartoon face generation model, and the human face area in the result image has a natural transition with other areas.

[0004] The embodiment of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Obtaining a target partial image including a face from the image to be processed;

[0007] Inputting the target partial image into a trained comic face generation model to obtain a comic face image;

[0008] Replacing the face region content in the target partial image with the face region content in the cartoon face image to obtain a target cartoon face image;

[0009] The area in the image to be processed corresponding to the target partial image is replaced with the target cartoon face image to obtain a result image of the cartoonized face.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0011] A preprocessing module, used for obtaining a target partial image including a human face from an image to be processed;

[0012] A comic face acquisition module, used for inputting the target partial image into a trained comic face generation model to obtain a comic face image;

[0013] A processing module, used for replacing the face area content in the target partial image with the face area content in the cartoon face image to obtain a target cartoon face image;

[0014] The processing module is further used to replace the area corresponding to the target partial image in the image to be processed with the target cartoon face image to obtain a result image of the cartoonized face.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the image processing method described in any one of the aforementioned embodiments.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method as described in any one of the aforementioned embodiments.

[0017] The image processing method, device, electronic device and readable storage medium provided by the embodiment of the present application input the target partial image including the face obtained from the image to be processed into the trained comic face generation model to obtain the comic face image; then, replace the face area content in the target partial image with the face area content in the comic face image to obtain the target comic face image; finally, replace the area corresponding to the target partial image in the image to be processed with the target comic face image to obtain the result image of the face comic. Thus, the result image of the face comic can be automatically obtained based on the image to be processed and the comic face generation model, and no style image needs to be input in the process of obtaining the result image, which is convenient to use; and in the process of obtaining the result image, the target comic face image obtained by replacing the face area content is pasted back to the original position, instead of directly pasting the comic face image output by the comic face generation model back to the original position, so that the comic part and other parts in the result image can be avoided to have obvious color difference / texture difference, and the transition is more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 2 One of the flowcharts of the image processing method provided in the embodiment of the present application;

[0021] Figure 3 A schematic diagram of obtaining a result image provided in an embodiment of the present application;

[0022] Figure 4 The second flowchart of the image processing method provided in the embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a generator network provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a discriminator network provided in an embodiment of the present application;

[0025] Figure 7 One of the block diagrams of the image processing device provided in the embodiment of the present application;

[0026] Figure 8 The second block diagram of the image processing device provided in the embodiment of the present application.

[0027] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - image processing device; 210 - training module; 220 - pre-processing module; 230 - cartoon face acquisition module; 240 - processing module. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0030] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0031] From the perspective of computer vision algorithms, the goal of face caricature is to map real-life photos into a two-dimensional style while ensuring that the content remains unchanged. However, most current content style transfer algorithms require a content map (i.e., the image to be processed) and a style map as input, which results in a bloated network structure and is difficult to apply to real-time stylization. In addition, two-dimensional faces have characteristics such as smooth colors and clear boundaries, and the existing common style transfer algorithms cannot obtain style transfer results with two-dimensional characteristics.

[0032] Based on the above situation, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which can automatically and quickly obtain a result image of human face cartoonization based on the image to be processed and the cartoon face generation model, which can be applied to real-time stylization; and the face area in the result image has a natural transition with other areas.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 A block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be, but is not limited to, a smart phone, a computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0035] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0036] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, the memory 110 stores an image processing device 200, and the image processing device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the image processing device 200 in the embodiment of the present application, that is, implementing the image processing method in the embodiment of the present application.

[0037] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0038] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0039] Please refer to Figure 2 , Figure 2 This is one of the flow diagrams of the image processing method provided in the embodiment of the present application. The method can be applied to the above-mentioned electronic device 100. The specific flow of the image processing method is described in detail below. The method can include steps S120 to S150.

[0040] Step S120, obtaining a target partial image including a human face from the image to be processed.

[0041] In this embodiment, the image to be processed may be preprocessed to obtain a target partial image including a human face. The specific operation of the preprocessing may be determined by a trained comic face generation model. For example, if the comic face generation model has size requirements for the input image, the image to be processed may be processed according to the size requirements of the comic face generation model to obtain a target partial image that meets the size requirements. The comic face generation model may be trained in an unsupervised training manner or in a supervised training manner.

[0042] The content included in the target partial image may be less than the content included in the image to be processed, that is, the target partial image is a local area image of the image to be processed that includes a human face. This can not only meet the needs of face caricature, but also speed up the image processing speed and avoid irrelevant content affecting the speed. Of course, it is understandable that if the main content of the image to be processed is a human face, then the content included in the target partial image obtained from the image to be processed can be the same as the content included in the image to be processed.

[0043] Step S130, inputting the target partial image into a trained comic face generation model to obtain a comic face image.

[0044] Step S140, replacing the face area content in the target partial image with the face area content in the cartoon face image to obtain a target cartoon face image.

[0045] Step S150, replacing the area in the image to be processed corresponding to the target partial image with the target cartoon face image, to obtain a result image of the cartoonized face.

[0046] When the target partial image is obtained, the target partial image is input into the comic face generation model, and the output of the comic face generation model is the comic face image. The comic face image is generated by the comic face generation model, and may have a slight change in color compared to the image to be processed without any network processing. If the comic face is directly pasted back to the corresponding position in the image to be processed, there may be an obvious color / texture difference between the comic face image and the image to be processed.

[0047] Please refer to Figure 3 , Figure 3 A schematic diagram of obtaining a result image provided by an embodiment of the present application. When the comic face is directly pasted back to the corresponding position in the image to be processed, the color of area 1 and area 2 in the result image is different, there is an obvious color / texture difference between area 1 and area 2, and the background of the result image has a sense of fragmentation.

[0048] In order to eliminate this sense of discontinuity, in this embodiment, the face area content in the target partial image is replaced with the face area content in the comic face image to obtain a target comic face image, and then the target comic face image is pasted back to the corresponding position of the image to be processed, thereby obtaining a result image of face comics. Except for the face area content in the target comic face image, the rest of the content is the original content of the image to be processed, so that the color difference can be avoided, so that the face area can maintain the comic style content, and the transition between the comic and the original image is natural, that is, the background of the result image has no sense of disconnection. In this way, the image translation of the image to be processed can be completed, and the result image of face comics can be obtained.

[0049] Please refer to Figure 4 , Figure 4 The second flowchart of the image processing method provided in the embodiment of the present application is as follows. In the present embodiment, before step S120, the method may further include step S110, and the comic face generation model is obtained by step S110.

[0050] Step S110, training the initial network according to the sample face image and the sample comic face image corresponding to the sample face image to obtain the comic face generation model.

[0051] In this embodiment, a sample set may be first obtained, and the initial network may be obtained, and then the sample set may be used to train the initial network, so as to obtain the comic face generation model when the training stop condition is reached. The initial network may be an existing network (e.g., a pix2pix network and a pix2pixhd network, etc.), or a self-built network.

[0052] The generation result of the model obtained by using a supervised training method is relatively stable. Therefore, in one implementation of this embodiment, the sample set includes sample face images and sample cartoon face images corresponding to the sample face images, and the cartoon face generation model is obtained by training using a supervised training method based on the sample set.

[0053] In order to cover all face cartoons, the selected data can be as close as possible to real-life faces of various skin colors, ages, and angles. For real-life image libraries, the open source dataset FFHQ face dataset can be used, with 13k images. When the face dataset is prepared, the face dataset can be aligned to obtain multiple sample face images, and then based on each sample face image, a sample cartoon face image corresponding to each sample face image can be obtained by hand-drawing or other means.

[0054] Among them, the significance of face alignment is to normalize the input of the network model, greatly reduce the difficulty of network convergence, and improve the speed of network convergence. The process of face alignment can be as follows: detect facial key points on a face image; rotate the face image so that the direction of the line connecting the eye key points is horizontal; crop and scale the image so that the distance between the eye key points and the distance from the midpoint of the eyes to the nose are fixed to a uniform value, and the final image is a sample face image. It can be seen that the size of the sample face image can be a preset size, and the direction of the line connecting the eye key points is horizontal.

[0055] Optionally, in this embodiment, during the training process, when calculating the loss value of the initial network, the loss value includes a local loss value corresponding to the eye region. The local loss value may be, but is not limited to, an L1 loss. Thus, the eye region in the comic face image obtained based on the comic face generation model can be made clearer.

[0056] As an optional implementation, the initial network can be a lightweight generative adversarial network that can perform real-time reasoning and has good stylization effects. The initial network can be a network built based on a pix2pix network or a pix2pixhd network.

[0057] In one implementation of this embodiment, the initial network is a network built based on the pix2pixhd network. pix2pix is ​​based on the GAN framework. Unlike ordinary GAN, the input received by the D (discriminator) of pix2pix is ​​a<x,y> pair, rather than the generated image. Among them, x represents the real-life image, and y is the cartoon image. What D needs to judge is whether this pair is real, that is, whether the cartoon image is the comic image corresponding to (or matching) the real-life image, so that Mapping is guaranteed. Because the real-life image and the unreal cartoon image obviously cannot be identified as a pair, D already contains the information to generate the real image when judging whether they can be a pair. Among them, the real image represents the comic image corresponding to the real-life image (that is, the real cartoon image). During the training process, the mutual game between G and D makes the generated image continuously fit the groundtruth.

[0058] The initial network may include a generator network and a discriminator network. The cartoon face generation model is a trained generator network. Figure 5 As shown, the generator network (G) is used to generate a comic style image based on a sample face image, that is, to map the input image to a comic style. Figure 6 As shown, the discriminator network (D) is used to receive the real pair image and the generated pair image, and determine whether the real pair image and the generated pair image are matched.

[0059] The real pair of images includes the sample face image used this time and the sample comic face image corresponding to the sample face image, and the generated pair of images includes the sample face image used this time and the comic style image generated by the generator network based on the sample face image. The discriminator network determines whether the real pair of images matches, which means judging whether the comic style image matches the sample face image; the discriminator network determines whether the generated pair of images matches, which means judging whether the sample comic face image matches the sample face image; the two judgment results obtained through the above judgment can be used to indicate whether the comic style image is similar to the sample comic face image.

[0060] Optionally, when both discrimination results are matches, it can be determined that the sample comic face image and the generated comic style image are similar, that is, at this time, the discriminator network cannot distinguish the comic style image generated by the generator network from the sample comic face image. When the comic style image generated by the generator network is similar to the sample comic face image, the generator network can be used as a trained comic face generation model.

[0061] The following is a brief description of the method for obtaining the comic face generation model by training the initial network.

[0062] The initial network includes a generator network and a discriminator network. A sample face image can be input into the generator network to obtain a comic style image, which is an image generated by the generator network based on the input sample face image.

[0063] Next, the sample face image, the sample comic face image corresponding to the sample face image, and the comic style image obtained by the generator network are input into the discriminator network. Since the discriminator network judges a pair of images, the sample face image and the comic style image can be input into the discriminator network as a generated pair of images, and the sample face image and the sample comic face image can be input into the discriminator network as a real pair of images.

[0064] The discriminator network makes a judgment based on the input image and obtains two judgment results. Among the two judgment results, one judgment result indicates that the comic style image matches or does not match the sample face image; the other judgment result indicates that the sample comic face image matches or does not match the sample face image.

[0065] The generated pair of images is judged as mismatched, which means that the comic-style image generated by the generator network at this time is not infinitely close to the sample comic face image, and the generator network needs to be corrected at this time.

[0066] When the above two discrimination results are obtained, it can be determined whether to continue training based on the above two discrimination results and the set stop training conditions. For example, combining the two discrimination results obtained this time and the two discrimination results of the historical times, it is determined whether there are a preset number of times that the real pair of images are judged as a match and the generated pair of images are judged as a match. If so, it can be determined that the stop training conditions are met. Of course, it is understandable that other stop training conditions can also be set based on actual needs, as long as the following purpose can be achieved: the generator network finally trained can generate comic-style images that are infinitely close to the sample comic face images.

[0067] If it is determined to continue training, the loss value of the initial network can be calculated based on the set loss function, the comic style image, the sample face image, and the sample comic face image corresponding to the sample face image. Then, the generator network and / or the discriminator network are adjusted based on the loss value, for example, the parameters in the generator network and the discriminator network are adjusted to obtain the adjusted initial network.

[0068] Then, the sample face image can be input again to the adjusted initial network, and the above process can be repeated until it is determined to stop training. When it is determined to stop training, the generator network at the time of stopping training is used as the comic face generation model.

[0069] Alternatively, as a possible implementation, Figure 5 As shown in FIG. 1 , the generator network may include: 4 down-convolution blocks, 4 residual blocks, and 4 deconvolution blocks in sequence. The generator network starts with 4 consecutive down-convolution blocks to spatially compress and encode the input image, extracting useful local signals for style transfer at this stage. Then, 4 residual blocks ( Figure 5 The example in Figure 1 shows only two residual blocks) to construct content and style features. That is, four residual blocks are used for nonlinear mapping to convert the features of the input image into two-dimensional features. Finally, the output comic style image is reconstructed by four deconvolution blocks. In this way, a comic style image can be obtained based on the generator network, and the structure is lightweight.

[0070] Optionally, the discriminator network can be a conventional discriminator network structure or a lightweight structure. Determining whether an image is a comic is a less demanding task, so in one possible implementation, a simple discriminator with fewer parameters in D can be used instead of a conventional full-image discriminator. Unlike image classification tasks, comic style identification tasks rely on local features of the image, so the discriminator network is designed to be shallow. Figure 6 As shown, the discriminator network may include: a flat layer (i.e. Figure 6 The two vertical blocks closest to the true pair and the generated pair in the image), 2 strided convolution blocks, 2 feature construction blocks and 2 down-convolution blocks.

[0071] In the discriminator network, flat convolution is first performed to extract feature information of low-dimensional features. After flattening, two strided convolution blocks are used to reduce the resolution and encode to obtain basic local features for classification. Subsequently, two feature building blocks and two down-convolution blocks are used to obtain classification returns. Among them, the two feature building blocks are used to perform nonlinear transformations to increase the expressive power of the network. The two down-convolution blocks are used to perform downsampling and extract high-dimensional feature information. Among them, the two down-convolution blocks can be two 3×3 down-convolution blocks. Therefore, based on Figure 5 and Figure 6 The initial network shown ensures the generation effect and the model performance at the same time, so that the initial network and / or the comic face generation model can run in real time on small smart devices such as computers or smart phones.

[0072] In this embodiment, the loss function of the initial network is as follows:

[0073]

[0074] Where: L GAN (G,D k )=E[(logD k (y))]+E[log(1-D k (G(x)))]

[0075]

[0076] L VGG (G,F)=||F(y)-F(G(x))|| 1

[0077] L eye (G,y)=||G(y) eye_roi -y eye_roi || 1

[0078] Where G represents the generator network, D represents the discriminator network, and F represents the VGG network; x represents a real face (i.e., a sample face image), y represents a real anime face (i.e., a sample comic face image), G(x) represents a generated anime face (i.e., a comic style image generated by the generator network), k represents the kth D network, i represents the i-th layer in the D network, and T represents all layers in the D network; L GAN (G,D k ) represents the adversarial loss, L FM (G,D k) represents the feature matching loss of the discriminator network, L VGG (G, F) represents VGG loss, L eye (G,y) represents the loss of eye clarity. fm , vgg , eye All represent preset values. Optionally, λ fm =100,λ vgg =100,λ eye =100.

[0079] The comic face generation model can be trained based on the initial network of the lightweight structure, the sample set and the loss function. Optionally, the training can be determined to be completed when the calculated loss value of the initial network is less than a preset value or a preset number of training times or other set conditions are met. After the training is completed, comic stylization can be performed in real time through the comic face generation model.

[0080] Optionally, the image to be processed may be subjected to face alignment processing, thereby obtaining a target partial image of a preset size. When performing face alignment, the image to be processed may be subjected to face key point detection, thereby determining the positions of the face key points in the image to be processed. Then, based on the positions of the face key points, the image to be processed is rotated, cropped, and scaled to obtain the target partial image. For example, in accordance with the manner of obtaining a sample face image, the eye key points in the image to be processed are connected, and then the image to be processed is rotated so that the connection of the eye key points is horizontal; thereafter, the image to be processed after cropping, scaling, and rotation is performed so that the distance between the eye key points and the distance from the midpoint of the eyes to the nose are fixed to a uniform value, thereby obtaining a target partial image of the face being turned upright.

[0081] In order to avoid missing facial features during cropping, the content of the target partial image is more than the content of the facial area in the rotated image to be processed, but can be less than the content included in the image to be processed. For example, if the image to be processed includes the head and the body, the obtained target partial image may include the head but not the body.

[0082] It is understandable that the above method of obtaining the target partial image is only an example, and other methods can be used to obtain the image for input into the comic face generation model according to actual needs, which is not specifically limited here.

[0083] Then, the target partial image can be input into the comic face generation model to obtain the comic face image. Afterwards, the face area content in the target partial image can be replaced with the face area content in the comic face image in any manner to obtain the target comic face image.

[0084] As an optional implementation, a mask image may be pre-set, and the mask image includes a foreground area and a background area. Figure 3 As shown, the white part is the foreground area, and the black part is the background area. The size of the mask image can be the size of the input image required by the comic face generation face model. The mask image can be, but is not limited to, a Gaussian mask template.

[0085] The target partial image can be processed by the mask image to obtain a first background image, which includes content other than the face, that is, background content. The comic face image can be processed by the mask image to obtain a first foreground image, which includes content in the face area but does not include content outside the face area, that is, foreground content. Afterwards, the target comic face image can be obtained based on the first background image and the first foreground image to complete face fusion. The above process can be simply described as: multiplying the white part of the mask image by the comic face image, multiplying the black part of the mask image by the target partial image, and then obtaining a blended image, that is, the target comic face image. Therefore, the target comic face image can be quickly obtained using the mask image.

[0086] Of course, it can be understood that the above method is only an example, and the method for obtaining the target comic face image is not specifically limited here, and the target comic face image can also be obtained by other methods. For example, the facial key points in the target partial image and the facial key points in the comic face image are detected, and then based on the facial key points, the facial area content in the target partial image is replaced with the facial area content in the comic face image to obtain the target comic face image.

[0087] When the target comic face image is obtained, the target comic face image can be processed accordingly in the manner of obtaining the target partial image from the image to be processed, and the position of the processed target comic image in the image to be processed is determined, and then the processed target comic image is pasted back to the image to be processed based on the position, thereby obtaining the result image.

[0088] For example, in the case where the target partial image is obtained by face alignment, the target comic face image may be reversely processed according to the face alignment process (i.e. Figure 3 The target comic face image after reverse processing is obtained by performing the face alignment process, and the position of the target comic face image after reverse processing in the image to be processed is determined according to the face alignment process, and the target comic face image after reverse processing is fused into the image to be processed according to the position to obtain the result image.

[0089] Optionally, the affine transformation matrix used in face alignment can be saved, and when performing reverse processing, the target comic face image can be reflected and transformed based on the affine transformation matrix to restore the target comic face image to its original face angle and size. Afterwards, the target comic face image with restored angle and size is pasted back to its original position to obtain the result image. In this way, the reverse processing can be completed quickly to obtain the result image.

[0090] As an optional implementation, the image to be processed is a live broadcast image. After obtaining the result image, the result image can also be sent to the live broadcast receiving end for display. In this way, the face cartoonization of the live broadcast scene can be realized. This method can help users create content, reduce the cost of artistic creation, and enhance the interest of the content. It is worth noting that the live broadcast sending end can process the image to be processed to obtain the result image, and then send the result image to the live broadcast receiving end via the live broadcast server; or the live broadcast server can process the image to be processed to obtain the result image, and then send the result image to the live broadcast receiving end. It can be set specifically according to actual needs.

[0091] In order to execute the corresponding steps in the above embodiments and various possible methods, an implementation method of an image processing device 200 is given below. Optionally, the image processing device 200 can adopt the above Figure 1 The device structure of the electronic device 100 is shown. Figure 7 , Figure 7 One of the block diagrams of the image processing device 200 provided in the embodiment of the present application. It should be noted that the basic principle and technical effect of the image processing device 200 provided in this embodiment are the same as those in the above-mentioned embodiment. For the sake of brief description, for parts not mentioned in this embodiment, please refer to the corresponding content in the above-mentioned embodiment. The image processing device 200 can be used as an AI (Artificial Intelligence) video content creation tool to help users create content, reduce the cost of artistic creation, and enhance the interest of content. The image processing device 200 may include: a preprocessing module 220, a comic face acquisition module 230 and a processing module 240.

[0092] The pre-processing module 220 is used to obtain a target partial image including a human face from the image to be processed.

[0093] The comic face acquisition module 230 is used to input the target partial image into a trained comic face generation model to obtain a comic face image.

[0094] The processing module 240 is used to replace the face area content in the target partial image with the face area content in the cartoon face image to obtain a target cartoon face image.

[0095] The processing module 240 is further configured to replace the region in the image to be processed corresponding to the target partial image with the target cartoon face image to obtain a result image of the cartoonized face.

[0096] Please refer to Figure 8 , Figure 8 The second block diagram of the image processing device 200 provided in the embodiment of the present application. The image processing device 200 may further include a training module 210 .

[0097] The training module 210 is used to train the initial network according to the sample face image and the sample comic face image corresponding to the sample face image to obtain the comic face generation model. In the training process, the loss value of the initial network includes the local loss value corresponding to the eye area.

[0098] Optionally, in this embodiment, the image to be processed is a live broadcast image, and the processing module 240 is further configured to send the result image to a live broadcast receiving end for display.

[0099] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0100] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image processing method is implemented.

[0101] In summary, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, wherein a target partial image including a face obtained from an image to be processed is input into a trained comic face generation model to obtain a comic face image; then, the face area content in the target partial image is replaced with the face area content in the comic face image to obtain a target comic face image; finally, the area corresponding to the target partial image in the image to be processed is replaced with the target comic face image to obtain a result image of the face comic. Thus, based on the image to be processed and the comic face generation model, a result image of the face comic can be automatically obtained, and no style image needs to be input in the process of obtaining the result image, which is convenient to use; and in the process of obtaining the result image, the target comic face image obtained by replacing the face area content is pasted back to the original position, rather than directly pasting the comic face image output by the comic face generation model back to the original position, thereby avoiding obvious color difference / texture difference between the comic part and other parts in the result image, and the transition is more natural.

[0102] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0104] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0105] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a target partial image including a human face from the image to be processed; Inputting the target partial image into a trained comic face generation model to obtain a comic face image, wherein the loss value used in the process of training the comic face generation model includes a local loss value corresponding to an eye region, and the size of the comic face image is the same as the size of the target partial image; Replacing the face region content in the target partial image with the face region content in the cartoon face image to obtain a target cartoon face image; The area in the image to be processed corresponding to the target partial image is replaced with the target cartoon face image to obtain a result image of the human face cartoonization.

2. The method according to claim 1, characterized in that Before replacing the face area content in the target partial image with the face area content in the cartoon face image to obtain the target cartoon face image, the method further includes: The initial network is trained according to the sample face image and the sample cartoon face image corresponding to the sample face image to obtain the cartoon face generation model, wherein during the training process, the loss value of the initial network includes a local loss value corresponding to the eye area.

3. The method according to claim 2, characterized in that The initial network includes a generator network and a discriminator network. The initial network is trained according to the sample face image and the sample comic face image corresponding to the sample face image to obtain the comic face generation model, including: Inputting the sample face image into the generator network to obtain a comic style image; Inputting the comic style image, the sample face image and the sample comic face image corresponding to the sample face image into the discriminator network to obtain two discrimination results, wherein the two discrimination results are used to indicate whether the comic style image is similar to the sample comic face image; In the case where it is determined to continue training according to the two discrimination results, a loss value of the initial network is calculated according to the comic style image, the sample face image and the sample comic face image corresponding to the sample face image; The generator network and / or the discriminator network is adjusted according to the loss value, and the adjusted initial network is continuously trained until it is determined to stop the training, wherein the comic face generation model is the generator network obtained when the training is stopped.

4. The method according to claim 2, characterized in that: The initial network includes a generator network and a discriminator network, the cartoon face generation model is a trained generator network, and the generator network includes: 4 down-convolution blocks, 4 residual blocks and 4 deconvolution blocks in sequence.

5. The method according to claim 4, characterized in that The discriminator network includes: a flat layer, 2 strided convolution blocks, 2 feature construction blocks and 2 down-convolution blocks in sequence.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a target partial image including a face from the image to be processed comprises: Performing face alignment processing on the image to be processed to obtain the target partial image, wherein the size of the target partial image is a preset size; The step of replacing the region in the image to be processed corresponding to the target partial image with the target comic face image to obtain a result image of the human face comics includes: According to the face alignment process, the target comic face image is reversely processed to obtain a target comic face image after reverse processing; According to the face alignment process, the position of the target cartoon face image after reverse processing in the image to be processed is determined, and the target cartoon face image after reverse processing is fused into the image to be processed according to the position to obtain the result image.

7. The method according to claim 6, characterized in that The performing face alignment processing on the image to be processed to obtain the target partial image includes: Performing facial key point detection on the image to be processed to determine the facial key points; Based on the facial key points, the image to be processed is rotated, cropped, and scaled to obtain the target partial image.

8. The method according to claim 6, characterized in that The step of replacing the face region content in the target partial image with the face region content in the cartoon face image to obtain the target cartoon face image comprises: Processing the target partial image through a mask image to obtain a first background image, wherein the mask image includes a foreground area and a background area; Processing the comic face image by using the mask image to obtain a first foreground image; The target comic face image is obtained according to the first background image and the first foreground image.

9. The method according to claim 6, characterized in that The image to be processed is a live image, and the method further includes: The result image is sent to the live broadcast receiving end for display.

10. An image processing device, characterized in that: The device comprises: A preprocessing module, used for obtaining a target partial image including a human face from an image to be processed; A comic face acquisition module, used for inputting the target partial image into a trained comic face generation model to obtain a comic face image, wherein the loss value used in the process of training to obtain the comic face generation model includes a local loss value corresponding to the eye area, and the size of the comic face image is the same as the size of the target partial image; A processing module, used for replacing the face area content in the target partial image with the face area content in the cartoon face image to obtain a target cartoon face image; The processing module is further used to replace the area corresponding to the target partial image in the image to be processed with the target cartoon face image to obtain a result image of the cartoonized face.

11. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the image processing method according to any one of claims 1 to 9.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images