Image global cartooning method and device based on regional supervision, equipment and medium

Abstract

The application relates to an image global cartoonization method and device based on regional supervision, equipment and a medium, wherein the method comprises the following steps: acquiring a to-be-processed image comprising a face region, and cropping the face region in the to-be-processed image to obtain a face image; performing face feature extraction on the face image through a first encoder to obtain face features, and performing background feature extraction on the to-be-processed image through a second encoder to obtain background features; acquiring a true value cartoon image, and performing loss calculation on the face features, the background features and the true value cartoon image to obtain a target loss value; pasting the face features back into the background features to obtain global features; and inputting the global features into a decoder, and performing feature decoding on the global features through the decoder based on the target loss value to obtain a target cartoon image. The application takes into account the face features and the background features, improves the cartoonization effect, and improves the cartoon visual effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and medium for global image cartoonization based on region supervision. Background Technology

[0002] In recent years, the concept of the metaverse has gained popularity, with virtual humans based on 2D cartoon and 3D styles becoming increasingly sought after. Cartoonized images, in particular, can significantly enhance the fun and aesthetics of videos. Currently, mainstream cartoonization algorithms are widely used. For example, cartoon filters in beauty apps can highlight facial features, generating various styles of cartoon portraits; another approach is through cartoonization algorithms in image processing software, which process the entire image, including the background. However, these solutions still have the following shortcomings: 1. Existing face cartoonization techniques primarily learn from facial features, making it difficult to consider the background, and sometimes even creating fake faces in the background, greatly affecting the viewing experience; 2. Existing global background cartoonization networks cannot distinguish between faces, causing facial features to be discarded, resulting in a nearly flattened facial outline in the final global effect. Therefore, there is an urgent need for an image cartoonization method that can consider both facial and background features to improve the cartoonization effect and enhance the overall cartoon aesthetic. Summary of the Invention

[0003] The purpose of this application is to propose a method, apparatus, device, and medium for global image cartoonization based on region supervision, so as to take into account both facial features and background features, improve the cartoonization effect, and enhance the cartoon viewing experience.

[0004] To address the aforementioned technical problems, embodiments of this application provide a method for global image cartoonization based on region supervision, comprising:

[0005] Obtain an image to be processed that includes a face region, and crop the face region in the image to be processed to obtain a face image;

[0006] The face image is subjected to facial feature extraction by the first encoder to obtain facial features, and the image to be processed is subjected to background feature extraction by the second encoder to obtain background features.

[0007] A ground truth cartoon image is obtained, and a loss is calculated based on the face features, the background features, and the ground truth cartoon image to obtain a target loss value, wherein the target loss value includes a face cartoon loss value, a background cartoon loss value, and a global cartoon loss value;

[0008] The facial features are pasted back into the background features to obtain the global features;

[0009] The global features are input into the decoder, and based on the target loss value, the global features are decoded by the decoder to obtain the target cartoon image.

[0010] Further, the step of acquiring an image to be processed that includes a face region, and cropping the face region in the image to be processed to obtain a face image, includes:

[0011] The image to be processed, including the face region, is acquired, and the image to be processed is processed using a face detection algorithm to obtain facial key points;

[0012] Select preset key points from the facial key points as cropping key points;

[0013] Based on the cropping key points, the face region is cropped from the image to be processed to obtain an initial face image;

[0014] Based on the image to be processed, the initial face image is rotated and aligned to obtain the face image.

[0015] Furthermore, the first encoder and the second encoder are each composed of multiple connected coded convolutional blocks. The step of extracting facial features from the face image using the first encoder to obtain facial features, and extracting background features from the image to be processed using the second encoder to obtain background features, includes:

[0016] The face image is input into the first encoder, and the image to be processed is input into the second encoder;

[0017] In any coding convolutional block of the first encoder, facial features are extracted from the face image to obtain the facial feature extraction result. The facial feature extraction result is then used as the input of the next coding convolutional block until all the coding convolutional blocks in the first encoder have completed feature extraction, thus obtaining the facial features.

[0018] In any coded convolutional block of the second encoder, background features are extracted from the image to be processed to obtain a background feature extraction result. The background feature extraction result is then used as the input of the next coded convolutional block until all coded convolutional blocks in the second encoder have completed feature extraction, thus obtaining the background features.

[0019] Further, the step of extracting facial features from the face image in any coding convolution block of the first encoder to obtain the facial feature extraction result includes:

[0020] In any coding convolution block of the first encoder, the facial features are convolved to obtain the convolution result;

[0021] The convolution result is activated using a preset activation method to obtain the activation result;

[0022] The activation results are processed using max pooling to obtain the facial feature extraction results.

[0023] Further, the step of acquiring the ground truth cartoon image and performing loss calculation based on the facial features, the background features, and the ground truth cartoon image to obtain the target loss value includes:

[0024] The true value cartoon image is obtained, and the face region of the true value cartoon image is cropped to obtain the cartoon face region and the cartoon background region;

[0025] The similarity between the facial features and the cartoon face region is calculated to obtain the cartoon face loss value;

[0026] The similarity between the background features and the cartoon background region is calculated to obtain the background cartoon loss value;

[0027] The facial features are combined with the background features to obtain combined features, and the similarity between the combined features and the ground truth cartoon image is calculated to obtain the global cartoon loss value.

[0028] Furthermore, the step of pasting the facial features back into the background features to obtain global features includes:

[0029] Obtain the cropping position of the image to be processed;

[0030] Based on the cropping position, identify the original position corresponding to the facial features;

[0031] Based on the original position, the facial features are pasted back into the background features to obtain the global features.

[0032] Furthermore, the decoder includes multiple interconnected decoding convolutional blocks. The step of inputting the global features into the decoder and performing feature decoding on the global features based on the target loss value to obtain the target cartoon image includes:

[0033] The global features are input into the decoder, and for any one of the decoding convolutional blocks, the parameters of the decoding convolutional block are adjusted by the target loss value so that the decoding convolutional block performs feature decoding on the global features to obtain the decoding result;

[0034] The decoding result obtained from the previous decoding convolutional block is used as the input to the next decoding convolutional block for feature decoding. When the feature decoding of all the decoding convolutional blocks in the decoder is completed, the target cartoon image is obtained.

[0035] To address the aforementioned technical problems, embodiments of this application provide an image global cartoonization device based on region supervision, comprising:

[0036] The image to be processed acquisition module is used to acquire an image to be processed including a face region, and to crop the face region in the image to be processed to obtain a face image;

[0037] The feature extraction module is used to extract facial features from the face image using a first encoder to obtain facial features, and to extract background features from the image to be processed using a second encoder to obtain background features.

[0038] The target loss value calculation module is used to acquire the ground truth cartoon image and perform loss calculation based on the face features, the background features and the ground truth cartoon image to obtain the target loss value, wherein the target loss value includes the face cartoon loss value, the background cartoon loss value and the global cartoon loss value;

[0039] The face feature pasting module is used to paste the face features back into the background features to obtain global features;

[0040] The target cartoon image generation module is used to input the global features into the decoder, and based on the target loss value, perform feature decoding on the global features through the decoder to obtain the target cartoon image.

[0041] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the image global cartoonization method based on region supervision as described above.

[0042] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the image global cartoonization method based on region supervision as described above.

[0043] This invention provides a method, apparatus, device, and medium for global cartoonization of images based on region supervision. The method includes: acquiring an image to be processed, including a face region, and cropping the face region in the image to obtain a face image; extracting face features from the face image using a first encoder, and extracting background features from the image to be processed using a second encoder, obtaining background features; acquiring a ground truth cartoon image, and calculating a loss based on the face features, background features, and the ground truth cartoon image to obtain a target loss value, wherein the target loss value includes a face cartoon loss value, a background cartoon loss value, and a global cartoon loss value; pasting the face features back into the background features to obtain global features; inputting the global features into a decoder, and decoding the global features based on the target loss value to obtain the target cartoon image. This invention performs feature processing on the face region and background region separately to avoid mutual interference between face features and background features, thus avoiding excessive loss of facial details. It achieves a balance between face features and background features, improving the cartoonization effect and enhancing the cartoon visual appeal. Attached Figure Description

[0044] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of an implementation of the image global cartoonization method based on region supervision provided in this application embodiment;

[0046] Figure 2 This is a flowchart illustrating an implementation of a sub-process in the image global cartoonization method based on region supervision provided in this application embodiment;

[0047] Figure 3 This is another implementation flowchart of a sub-process in the image global cartoonization method based on region supervision provided in the embodiments of this application;

[0048] Figure 4 This is another implementation flowchart of a sub-process in the image global cartoonization method based on region supervision provided in the embodiments of this application;

[0049] Figure 5 This is a schematic diagram of the image to be processed provided in the embodiments of this application;

[0050] Figure 6 This is a schematic diagram of existing unsupervised generated cartoon images;

[0051] Figure 7This is a schematic diagram of the target cartoon image provided in the embodiments of this application;

[0052] Figure 8 This is a schematic diagram of an image global cartoonization device based on region supervision provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0055] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0057] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the image global cartoonization method based on region supervision provided in this application embodiment is generally executed by a server, and correspondingly, the image global cartoonization device based on region supervision is generally configured in the server.

[0059] Please see Figure 1 , Figure 1 A specific implementation of a region-supervised global cartoonization method for images is shown.

[0060] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 Limited to the order of the processes shown, this method includes the following steps:

[0061] S1: Obtain the image to be processed, including the face region, and crop the face region in the image to obtain the face image.

[0062] Specifically, since this embodiment of the application is for cartoonizing facial images, the image to be processed needs to include a facial region. By performing face detection on the image to be processed, facial key points are obtained. Then, the facial region in the image to be processed is cropped based on the facial key points to obtain a facial image. This facilitates the subsequent processing of facial features and background features separately, avoiding interference from background features on facial features, and making the generated cartoon image more beautiful and natural.

[0063] Please see Figure 2 , Figure 2 A specific implementation of step S1 is shown below:

[0064] S11: Obtain the image to be processed, which includes the face region, and perform face detection on the image to be processed using a face detection algorithm to obtain the face key points.

[0065] S12: Select the preset key points from the facial key points as the cropping key points.

[0066] S13: Based on the cropping key points, crop the face region from the image to be processed to obtain the initial face image.

[0067] S14: Based on the image to be processed, the initial face image is rotated and aligned to obtain the face image.

[0068] Specifically, the face detection algorithm used in this embodiment is the open-source Dlib algorithm. The Dlib algorithm is used to detect faces in the image to be processed, obtaining facial key points. Then, preset key points from these key points are selected as cropping key points. These preset key points include the chin position, the sides of the face, the left and right eye positions, the sides of the mouth, the nose position, and the forehead vertex position. Based on these cropping key points, the face region is cropped from the image to be processed to obtain an initial face image. Since the initial face image after cropping may have shifted or rotated, the image to be processed is used as a reference, and the initial face image is rotated and aligned to obtain the final face image.

[0069] S2: The face image is extracted by the first encoder to obtain the face features, and the background features are extracted by the second encoder to obtain the background features.

[0070] Specifically, this embodiment uses two encoders to extract features from facial features and background features respectively. The purpose of using two encoders is that faces possess significant structural and facial feature information; using a single encoder would cause facial features and background features to interfere with each other, resulting in excessive loss of facial details. The two encoders are a first encoder and a second encoder, and both encoders produce the same result, consisting of five consecutive coded convolutional blocks (ConvBlocks).

[0071] Please see Figure 3 , Figure 3 A specific implementation of step S2 is shown below:

[0072] S21: Input the face image into the first encoder and input the image to be processed into the second encoder.

[0073] S22: In any coding convolutional block of the first encoder, facial features are extracted from the face image to obtain the facial feature extraction result. The facial feature extraction result is then used as the input of the next coding convolutional block until all coding convolutional blocks in the first encoder have completed feature extraction, thus obtaining the facial features.

[0074] Specifically, in the first encoder, when a face image is input, facial features are extracted from the face image through the first coded convolutional block to obtain the facial feature extraction result. This result is then used as the input to the next coded convolutional block, which also extracts facial features, until all coded convolutional blocks have completed their extraction, thus obtaining the facial features. The resolution of the face image differs in different coded convolutional blocks; for example, the resolutions from the first to the fifth coded convolutional block are 512*512*3, 256*256*64, 128*128*128, 64*64*256, and 32*32*512, respectively. The resolutions in the subsequent decoding convolutional blocks in the decoder are exactly the opposite of those in the coded convolutional blocks, i.e., 32*32*512, 64*64*256, 128*128*128, 256*256*64, and 512*512*3, respectively.

[0075] Furthermore, a detailed implementation process for the step "extracting facial features from the face image in any coding convolution block of the first encoder to obtain the facial feature extraction result" is provided below:

[0076] In any coded convolutional block of the first encoder, the facial features are convolved to obtain the convolution result.

[0077] The convolution result is activated using a preset activation method to obtain the activation result.

[0078] Max pooling is used to process the activation results to obtain the facial feature extraction results.

[0079] Specifically, in each encoded convolutional block, facial features are convolved to obtain a convolution result. Then, a preset activation method is applied to activate the convolution result, resulting in an activation result. Finally, max pooling is applied to the activation result to obtain the facial feature extraction result. The preset activation method is the ReLU activation function. This embodiment of the application uses max pooling on the combined result to reduce feature redundancy, reduce model learning difficulty, and thus improve the efficiency of cartoon image generation.

[0080] S23: In any coded convolutional block of the second encoder, background features are extracted from the image to be processed to obtain the background feature extraction result. The background feature extraction result is then used as the input of the next coded convolutional block until all coded convolutional blocks in the second encoder have completed feature extraction, thus obtaining the background features.

[0081] Specifically, in the second encoder, when the image to be processed is input, background features are extracted from the image through the first coded convolutional block to obtain the background feature extraction result. This result is then used as the input to the next coded convolutional block, which also performs background feature extraction, until all coded convolutional blocks have completed their extraction, thus obtaining the background features. The background image resolution varies across different coded convolutional blocks; for example, the resolutions from the first to the fifth coded convolutional block are 512*512*3, 256*256*64, 128*128*128, 64*64*256, and 32*32*512, respectively.

[0082] Furthermore, a specific implementation process for the step "in any coding convolution block of the second encoder, background features are extracted from the image to be processed to obtain the background feature extraction result" is provided, detailed below:

[0083] In any coded convolutional block of the second encoder, the background features are convolved to obtain the convolution result.

[0084] The convolution result is activated using a preset activation method to obtain the activation result.

[0085] Max pooling is used to process the activation results to obtain the background feature extraction results.

[0086] Specifically, in each coded convolutional block, the background features are convolved to obtain the convolution result. Then, a preset activation method is used to activate the convolution result, resulting in an activation result. Finally, max pooling is applied to the activation result to obtain the background feature extraction result. The preset activation method is the ReLU activation function. This embodiment uses max pooling to reduce feature redundancy and model learning difficulty, thereby improving the efficiency of cartoon image generation.

[0087] S3: Obtain the ground truth cartoon image, and calculate the loss based on the face features, background features, and ground truth cartoon image to obtain the target loss value.

[0088] The target loss values ​​include face cartoon loss value, background cartoon loss value, and global cartoon loss value.

[0089] Specifically, in this embodiment, ground truth cartoon images are obtained from open-source cartoon face datasets; if no corresponding ground truth cartoon images are found in the open-source cartoon face dataset, the corresponding cartoon movie is downloaded for cartoon face detection, and the cartoon face images are cropped out and used as ground truth cartoon datasets.

[0090] Please see Figure 4 , Figure 4 A specific implementation of step S3 is shown below:

[0091] S31: Obtain the true value cartoon image and crop the face region of the true value cartoon image to obtain the cartoon face region and the cartoon background region.

[0092] S32: Calculate the similarity between the facial features and the cartoon face region to obtain the cartoon face loss value.

[0093] S33: Calculate the similarity between the background features and the cartoon background area to obtain the background cartoon loss value.

[0094] S34: Combine the facial features with the background features to obtain the combined features, and calculate the similarity between the combined features and the ground truth cartoon image to obtain the global cartoon loss value.

[0095] Specifically, this embodiment requires calculation of face cartoon loss, background cartoon loss, and global cartoon loss. After obtaining the ground truth cartoon image, the face region is cropped to obtain a cartoon face region and a cartoon background region. The similarity between the face features and the cartoon face region is calculated to obtain the face cartoon loss value. Similarly, the similarity between the background features and the cartoon background region is calculated to obtain the background cartoon loss value. Finally, the face features and background features are combined to obtain a combined feature, and the similarity between the combined feature and the ground truth cartoon image is calculated to obtain the global cartoon loss value. In this embodiment, the loss value is calculated using a similarity function, determining the similarity between the output image and the ground truth cartoon image, and using this result as the corresponding loss value. The face cartoon loss value and background cartoon loss value respectively supervise the cartoonization effect of the face and background, while the global cartoon loss value ensures the naturalness and aesthetics of the final fused image, avoiding obvious boundaries at the pasting point.

[0096] S4: Paste the facial features back into the background features to obtain the global features.

[0097] Furthermore, a specific implementation process for step S4 is provided, detailed below:

[0098] Obtain the cropping position of the image to be processed.

[0099] Based on the cropping position, identify the original location corresponding to the facial features.

[0100] Based on the original location, the facial features are pasted back into the background features to obtain the global features.

[0101] Specifically, in the image decoding part, this embodiment of the application pastes the face back into the background features according to the cropping position to obtain global features, so that the decoder can completely recover the entire image. Since the image to be processed has already been cropped in the above steps, it is necessary to obtain the cropping position of the image to be processed here, then match the face features with the cropping position one-to-one to obtain the original position, and finally paste the face features back into the background features based on the original position to obtain global features.

[0102] S5: Input the global features into the decoder, and based on the target loss value, perform feature decoding on the global features to obtain the target cartoon image.

[0103] Specifically, the decoder is a structure corresponding to the encoder, and it consists of five interconnected decoding convolutional blocks (DconvBlock). Within the decoder, the resolution of the features is continuously increased, ultimately outputting a cartoonish effect with the same resolution as the input image.

[0104] Furthermore, a specific implementation process for step S5 is provided, detailed below:

[0105] The global features are input into the decoder, and for any decoding convolutional block, the parameters of the decoding convolutional block are adjusted by the target loss value so that the decoding convolutional block can perform feature decoding on the global features and obtain the decoding result.

[0106] The decoding result obtained from the previous decoding convolutional block is used as the input for the next decoding convolutional block for feature decoding. When all the feature decoding of the decoding convolutional blocks in the decoder is completed, the target cartoon image is obtained.

[0107] Specifically, in the decoder, after the first decoding convolutional block obtains the global features, the parameters of the decoding convolutional block are adjusted according to the target loss value. Then, the global features are decoded to obtain the decoding result. The decoding result is then used as the input of the next decoding convolutional block. The next decoding convolutional block continues to perform feature decoding. When all the decoding convolutional blocks in the decoder have completed feature decoding, the target cartoon image is obtained.

[0108] Please see Figures 5 to 7 , Figure 5 This is a schematic diagram of the image to be processed provided in an embodiment of this application. Figure 6 This is a schematic diagram of existing unsupervised generated cartoon images. Figure 7 This is a schematic diagram of the target cartoon image provided in an embodiment of this application. Existing image cartoonization uses an unsupervised cartoon image generation method, which will... Figure 5 The image to be processed is converted into a cartoon image, that is, converted into... Figure 6 The method cannot distinguish human faces in cartoon images, causing the facial features to be directly discarded. In the final global effect, the facial contours are almost flattened, resulting in a lack of contour in the generated cartoon images. Figure 7 The cartoon image generated using the embodiments of this application takes into account both facial features and background features, thereby improving the cartoon effect and enhancing the cartoon visual experience.

[0109] In this embodiment, an image to be processed, including a face region, is acquired, and the face region in the image to be processed is cropped to obtain a face image. Face features are extracted from the face image using a first encoder to obtain face features, and background features are extracted from the image to be processed using a second encoder to obtain background features. A ground truth cartoon image is acquired, and a loss calculation is performed based on the face features, background features, and the ground truth cartoon image to obtain a target loss value. The target loss value includes a face cartoon loss value, a background cartoon loss value, and a global cartoon loss value. The face features are pasted back into the background features to obtain global features. The global features are input into a decoder, and based on the target loss value, the global features are decoded by the decoder to obtain the target cartoon image. This embodiment of the invention processes the face region and background region separately to avoid mutual interference between face features and background features, thus avoiding excessive loss of facial details. It achieves a balance between face features and background features, improving the cartoon effect and enhancing the cartoon visual appeal.

[0110] Please refer to Figure 8 As a response to the above Figure 1 To implement the method shown, this application provides an embodiment of a region-supervised global image cartoonization device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0111] like Figure 8 As shown, the image global cartoonization device based on region supervision in this embodiment includes: a to-be-processed image acquisition module 61, a feature extraction module 62, a target loss value calculation module 63, a face feature pasting module 64, and a target cartoon image generation module 65, wherein:

[0112] The image acquisition module 61 is used to acquire an image to be processed that includes a face region, and to crop the face region in the image to be processed to obtain a face image;

[0113] The feature extraction module 62 is used to extract facial features from the face image through the first encoder to obtain facial features, and to extract background features from the image to be processed through the second encoder to obtain background features.

[0114] The target loss value calculation module 63 is used to acquire the ground truth cartoon image and perform loss calculation based on the face features, background features and the ground truth cartoon image to obtain the target loss value, wherein the target loss value includes the face cartoon loss value, the background cartoon loss value and the global cartoon loss value.

[0115] The face feature pasting module 64 is used to paste face features back into the background features to obtain global features;

[0116] The target cartoon image generation module 65 is used to input global features into the decoder and, based on the target loss value, perform feature decoding on the global features to obtain the target cartoon image.

[0117] Furthermore, the image acquisition module 61 includes:

[0118] The facial landmark recognition unit is used to acquire an image to be processed that includes a facial region, and to perform face detection on the image to be processed using a face detection algorithm to obtain facial landmarks.

[0119] The cropping key point selection unit is used to select preset key points from the key points of the face as cropping key points;

[0120] The initial face image generation unit is used to crop the face region from the image to be processed based on the cropping key points to obtain the initial face image;

[0121] The rotation and alignment processing unit is used to perform rotation and alignment processing on the initial face image based on the image to be processed, so as to obtain the face image.

[0122] Furthermore, the feature extraction module 62 includes:

[0123] An image input unit is used to input a face image into a first encoder and to input an image to be processed into a second encoder;

[0124] The face feature extraction unit is used to extract face features from a face image in any coding convolution block of the first encoder, obtain the face feature extraction result, and use the face feature extraction result as the input of the next coding convolution block until the feature extraction of all coding convolution blocks in the first encoder is completed, and the face features are obtained.

[0125] The background feature extraction unit is used to extract background features from the image to be processed in any coded convolutional block of the second encoder, obtain the background feature extraction result, and use the background feature extraction result as the input of the next coded convolutional block until the feature extraction of all coded convolutional blocks in the second encoder is completed, and the background features are obtained.

[0126] Furthermore, the facial feature extraction unit includes:

[0127] The convolution result generation subunit is used to perform convolution processing on facial features in any coding convolution block of the first encoder to obtain the convolution result;

[0128] The activation processing subunit is used to activate the convolution result using a preset activation method to obtain the activation result;

[0129] The pooling processing subunit is used to perform pooling processing on the activation results using max pooling to obtain the face feature extraction results.

[0130] Furthermore, the target loss value calculation module 63 includes:

[0131] The true value cartoon image acquisition unit is used to acquire true value cartoon images and crop the face region of the true value cartoon images to obtain the cartoon face region and the cartoon background region.

[0132] The face cartoon loss value calculation unit is used to calculate the similarity between facial features and cartoon face regions to obtain the face cartoon loss value;

[0133] The background cartoon loss value calculation unit is used to calculate the similarity between background features and cartoon background regions to obtain the background cartoon loss value.

[0134] The global cartoon loss calculation unit is used to combine facial features with background features to obtain combined features, and then calculate the similarity between the combined features and the ground truth cartoon image to obtain the global cartoon loss value.

[0135] Furthermore, the facial feature pasting module 64 includes:

[0136] The cropping position acquisition unit is used to acquire the cropping position of the image to be processed;

[0137] The original location recognition unit is used to identify the original location corresponding to the facial features based on the cropping location; the global feature generation unit is used to paste the facial features back into the background features based on the original location to obtain the global features.

[0138] Furthermore, the target cartoon image generation module 65 includes:

[0139] The decoding result generation unit is used to input global features into the decoder and, for any decoding convolutional block, adjust the parameters of the decoding convolutional block by means of the target loss value so that the decoding convolutional block can perform feature decoding on the global features and obtain the decoding result.

[0140] The decoding completion unit is used to use the decoding result obtained from the previous decoding convolutional block as the input of the next decoding convolutional block for feature decoding. When all the feature decoding of the decoding convolutional blocks in the decoder is completed, the target cartoon image is obtained.

[0141] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0142] Computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that only a computer device 7 with these three components (memory 71, processor 72, and network interface 73) is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0143] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0144] The memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 7. Of course, the memory 71 may also include both internal storage units and external storage devices of the computer device 7. In this embodiment, the memory 71 is typically used to store the operating system and various application software installed on the computer device 7, such as the program code of a region-supervised image global cartoonization method. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0145] In some embodiments, processor 72 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 72 is typically used to control the overall operation of computer device 7. In this embodiment, processor 72 is used to run program code stored in memory 71 or process data, for example, to run the program code of the region-supervised global image cartoonization method described above, to implement various embodiments of the region-supervised global image cartoonization method.

[0146] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 7 and other electronic devices.

[0147] This application also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the region-supervised global cartoonization method described above.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0149] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for global cartoonization of images based on region supervision, characterized in that, include: A face image is obtained by acquiring an image to be processed that includes a face region, and the face region in the image to be processed is cropped to obtain a face image; The face image is subjected to facial feature extraction by the first encoder to obtain facial features, and the image to be processed is subjected to background feature extraction by the second encoder to obtain background features. A ground truth cartoon image is obtained, and a loss is calculated based on the face features, the background features, and the ground truth cartoon image to obtain a target loss value, wherein the target loss value includes a face cartoon loss value, a background cartoon loss value, and a global cartoon loss value; The facial features are pasted back into the background features to obtain the global features; The global features are input into the decoder, and based on the target loss value, the global features are decoded by the decoder to obtain the target cartoon image; The first encoder and the second encoder are each composed of multiple connected coded convolutional blocks. The process of extracting facial features from the face image using the first encoder and extracting background features from the image to be processed using the second encoder includes: The face image is input into the first encoder, and the image to be processed is input into the second encoder; In any coding convolutional block of the first encoder, facial features are extracted from the face image to obtain a facial feature extraction result. The facial feature extraction result is then used as the input of the next coding convolutional block until all coding convolutional blocks in the first encoder have completed feature extraction, thus obtaining the facial features. In any coding convolutional block of the second encoder, background features are extracted from the image to be processed to obtain a background feature extraction result. The background feature extraction result is then used as the input of the next coding convolutional block until all coding convolutional blocks in the second encoder have completed feature extraction, thus obtaining the background features. The step of pasting the facial features back into the background features to obtain global features includes: Obtain the cropping position of the image to be processed; Based on the cropping position, identify the original position corresponding to the facial features; Based on the original position, the facial features are pasted back into the background features to obtain the global features; The process of acquiring a ground truth cartoon image and performing loss calculations based on the facial features, background features, and the ground truth cartoon image to obtain a target loss value includes: The true value cartoon image is obtained, and the face region of the true value cartoon image is cropped to obtain the cartoon face region and the cartoon background region; The similarity between the facial features and the cartoon face region is calculated to obtain the cartoon face loss value; The similarity between the background features and the cartoon background region is calculated to obtain the background cartoon loss value; The facial features are combined with the background features to obtain combined features, and the similarity between the combined features and the ground truth cartoon image is calculated to obtain the global cartoon loss value.

2. The image global cartoonization method based on region supervision according to claim 1, characterized in that, The step of acquiring an image to be processed, including a face region, and cropping the face region from the image to obtain a face image includes: The image to be processed, including the face region, is acquired, and a face detection algorithm is used to perform face detection on the image to be processed to obtain facial key points; Select preset key points from the facial key points as cropping key points; Based on the cropping key points, the face region is cropped from the image to be processed to obtain an initial face image; Based on the image to be processed, the initial face image is rotated and aligned to obtain the face image.

3. The image global cartoonization method based on region supervision according to claim 1, characterized in that, The step of extracting facial features from the face image in any coding convolution block of the first encoder to obtain the facial feature extraction result includes: In any coding convolution block of the first encoder, the facial features are convolved to obtain the convolution result; The convolution result is activated using a preset activation method to obtain the activation result; The activation results are processed using max pooling to obtain the facial feature extraction results.

4. The image global cartoonization method based on region supervision according to any one of claims 1 to 3, characterized in that, The decoder includes multiple interconnected decoding convolutional blocks. The global features are input into the decoder, and based on the target loss value, the global features are decoded by the decoder to obtain the target cartoon image, including: The global features are input into the decoder, and for any one of the decoding convolutional blocks, the parameters of the decoding convolutional block are adjusted by the target loss value so that the decoding convolutional block performs feature decoding on the global features to obtain the decoding result; The decoding result obtained from the previous decoding convolutional block is used as the input to the next decoding convolutional block for feature decoding. When the feature decoding of all the decoding convolutional blocks in the decoder is completed, the target cartoon image is obtained.

5. A global cartoonization device for images based on region supervision, characterized in that, include: The image to be processed acquisition module is used to acquire an image to be processed including a face region, and to crop the face region in the image to be processed to obtain a face image; The feature extraction module is used to extract facial features from the face image using a first encoder to obtain facial features, and to extract background features from the image to be processed using a second encoder to obtain background features. The target loss value calculation module is used to acquire the ground truth cartoon image and perform loss calculation based on the face features, the background features and the ground truth cartoon image to obtain the target loss value, wherein the target loss value includes the face cartoon loss value, the background cartoon loss value and the global cartoon loss value; The face feature pasting module is used to paste the face features back into the background features to obtain global features; The target cartoon image generation module is used to input the global features into the decoder, and based on the target loss value, perform feature decoding on the global features through the decoder to obtain the target cartoon image; The first encoder and the second encoder are each composed of multiple connected coding convolutional blocks, and the feature extraction module includes: An image input unit is used to input the face image into the first encoder and the image to be processed into the second encoder; The face feature extraction unit is used to extract face features from the face image in any coding convolution block of the first encoder, obtain the face feature extraction result, and use the face feature extraction result as the input of the next coding convolution block until the feature extraction of all coding convolution blocks in the first encoder is completed, so as to obtain the face features. The background feature extraction unit is used to extract background features from the image to be processed in any coding convolution block of the second encoder, obtain the background feature extraction result, and use the background feature extraction result as the input of the next coding convolution block until the feature extraction of all coding convolution blocks in the second encoder is completed, thereby obtaining the background features. The facial feature pasting module includes: The cropping position acquisition unit is used to acquire the cropping position of the image to be processed; An original location recognition unit is used to identify the original location corresponding to the facial features based on the cropping location; A global feature generation unit is used to paste the face features back into the background features based on the original position to obtain the global features; The target loss value calculation module includes: The true value cartoon image acquisition unit is used to acquire the true value cartoon image and crop the face region of the true value cartoon image to obtain the cartoon face region and the cartoon background region. The face cartoon loss value calculation unit is used to calculate the similarity between the face features and the cartoon face region to obtain the face cartoon loss value; The background cartoon loss value calculation unit is used to calculate the similarity between the background features and the cartoon background region to obtain the background cartoon loss value. The global cartoon loss calculation unit is used to combine the face features with the background features to obtain combined features, and to calculate the similarity between the combined features and the ground truth cartoon image to obtain the global cartoon loss value.

6. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the region-supervised global cartoonization method for images as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the region-supervised global cartoonization method for images as described in any one of claims 1 to 4.

Images