A face jewelry generation method and device based on CNN and equipment

Abstract

The application discloses a face ornament generation method and device based on CNN, equipment and a storage medium, which comprises: inputting a to-be-processed image, pre-processing the to-be-processed image to obtain a first face image, and the to-be-processed image comprising a face region; determining a target region for modification in the first face image, and inputting a hand-drawn ornament superimposed on the target region to a pre-trained jewelry generation model to obtain a jewelry generation image; inputting the jewelry generation image and the first face image to a pre-trained jewelry fusion model for fusion to obtain a first result image; and performing portrait posture restoration and fusion processing on the first result image to obtain a second result image. The jewelry generation can make full use of the context information such as illumination conditions and portrait postures in the image, so that the generated jewelry is more natural and reasonable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a face ornament generation method and device based on CNN and equipment. BACKGROUND

[0002] Face ornaments include glasses, false eyelashes, beauty lenses, earrings and other facial decoration items. The appearance value of a portrait can be effectively improved by wearing ornaments. In the past, in order to find suitable ornaments, people often need to spend a lot of time to try offline or online, which is a time-consuming and laborious process. In order to save time, people have tried computer simulation to "stick" jewelry to the target face to achieve virtual jewelry wearing, so that the wearing effect of the jewelry can be previewed in advance. The traditional simulation method mainly pastes existing jewelry materials to the target area of the portrait, and cooperates with a certain fusion algorithm to make the final effect look more natural. However, the traditional method of adding ornaments to the portrait by pasting is prone to unnatural effects, and has the defects of high requirements for portrait shooting and inability to meet individual needs. SUMMARY

[0003] Therefore, the present application aims to provide a face ornament generation method and device based on CNN and equipment, which aims to solve the problems of unnatural effects and unnatural pasting effects of the existing method of adding ornaments to the portrait by pasting.

[0004] To achieve the above-mentioned purpose, the present application provides a face ornament generation method based on CNN, which comprises the following steps:

[0005] inputting a to-be-processed image, pre-processing the to-be-processed image to obtain a first face image, and the to-be-processed image comprising a face region;

[0006] determining a target region for modification in the first face image, superimposing a hand-drawn ornament diagram on the target region, and inputting the hand-drawn ornament diagram to a pre-trained jewelry generation model to obtain a jewelry generation image;

[0007] inputting the jewelry generation image and the first face image into a pre-trained jewelry fusion model for fusion to obtain a first result image;

[0008] performing portrait posture restoration and fusion processing on the first result image to obtain a second result image.

[0009] Preferably, the jewelry generation model comprises a hand-drawing mapping model, a context encoding model and a decoding model, wherein,

[0010] the hand-drawing mapping model maps the drawing process of the hand-drawn ornament diagram into jewelry features;

[0011] extracting a portrait feature around the jewelry hand-drawing through the context encoding model;

[0012] fusing the jewelry feature and the portrait feature through the decoding model to generate the jewelry generation image.

[0013] Preferably, the hand-drawing mapping model comprises a multi-layer perceptron and an adaptive mapping layer; the mapping of the drawing process of the jewelry hand-drawing to the jewelry feature through the hand-drawing mapping model comprises:

[0014] cutting the jewelry hand-drawing into a plurality of line segments, and dividing the plurality of line segments into Q groups, each group containing K connected line segments, inputting each group of line segments into the multi-layer perceptron to obtain a D-dimensional feature vector;

[0015] mapping the D-dimensional feature vector to a fixed-dimensional jewelry feature map through the adaptive mapping layer.

[0016] Preferably, the jewelry generation image and the first portrait image are input into a pre-trained jewelry fusion model for fusion to obtain a first result image, which comprises:

[0017] calculating the final output of the jewelry fusion model according to the formula O = I × (1 - O a ) + O rgb × O a ; wherein O represents the first result image, I represents the input picture, O rgb represents the jewelry generation image, and O a represents the fusion ratio.

[0018] Preferably, the training process of the jewelry generation model comprises:

[0019] supervising the result image output by the jewelry generation model by using a first loss function and a second loss function, wherein,

[0020] the first loss function is:

[0021] the second loss function is: O represents the result image output by the jewelry generation model, T represents a target image, H represents the height of the image, W represents the width of the image, W win represents the side length of a square window, O sketch represents the sketch drawing of the model output result, I′ sketch represents the sketch drawing image after extracting the key positions, λ i，j represents the window-related weight parameter.

[0022] Preferably, according to the formula calculating the lambda i，j wherein zeta is a sign function,

[0023] Preferably, the training process of the jewelry fusion model comprises:

[0024] The result image output by the jewelry fusion model is supervised by using a third loss function and a fourth loss function; wherein,

[0025] The third loss function is:

[0026] The fourth loss function is: F o denotes the result image output by the jewelry fusion model, F t denotes a target effect image, phi j denotes a feature map output by the last convolutional layer of the jth module of the VGG16 network, H denotes image height, and W denotes image width.

[0027] To achieve the above object, the application further provides a face jewelry generation device based on a CNN, which comprises:

[0028] A preprocessing unit is configured to input a to-be-processed image, pre-process the to-be-processed image, and obtain a first face image, wherein the to-be-processed image comprises a face region.

[0029] A jewelry generation unit is configured to determine a target region to be modified in the first face image, superimpose a jewelry hand-drawing on the target region, and input the target region to a pre-trained jewelry generation model to obtain a jewelry generation image.

[0030] A first fusion unit is configured to input the jewelry generation image and the first face image to a pre-trained jewelry fusion model to perform fusion and obtain a first result image.

[0031] A second fusion unit is configured to perform portrait posture restoration and fusion processing on the first result image to obtain a second result image.

[0032] To achieve the above object, the application further provides a device comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the face jewelry generation method based on the CNN.

[0033] In order to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of the CNN-based face ornament generation method.

[0034] Beneficial effects:

[0035] The above scheme selects a target region in the face image, generates face ornaments by using a jewelry generation model and a jewelry fusion model according to the hand-drawn ornament, generates ornaments directly in the target region of the face image, fully utilizes context information such as illumination conditions and portrait poses in the image to generate ornaments, and makes the generated ornaments more natural and reasonable.

[0036] The above scheme generates ornaments based on the hand-drawn ornament of the user, breaks away from the constraint of the material library, better utilizes background information of pixels around the ornament, makes the ornament wearing effect more reasonable and natural, realizes personal customization of the user by intervening in the hand-drawn ornament, obtains a personalized ornament effect, and better meets the personalized demand. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0038] Figure 1 A flowchart of a CNN-based face ornament generation method provided by an embodiment of the present application is shown.

[0039] Figure 2 A flowchart of a face ornament generation method provided by an embodiment of the present application is shown.

[0040] Figure 3 A schematic diagram of the overall structure of a jewelry generation model provided by an embodiment of the present application is shown.

[0041] Figure 4 A schematic diagram of the structure of an SMN provided by an embodiment of the present application is shown.

[0042] Figure 5 A schematic diagram of the structure of a CEN provided by an embodiment of the present application is shown.

[0043] Figure 6 A schematic diagram of the structure of a residual Fourier convolution block provided by an embodiment of the present application is shown.

[0044] Figure 7 A structure diagram of a DN provided by an embodiment of the present application.

[0045] Figure 8 A structure diagram of a UCL module provided by an embodiment of the present application.

[0046] Figure 9 A structure diagram of a jewelry fusion model provided by an embodiment of the present application.

[0047] Figure 10 A structure diagram of a face jewelry generation device based on a CNN provided by an embodiment of the present application.

[0048] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0050] In the description of the present application, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0051] The content of the present application will be described in detail below in combination with embodiments.

[0052] Since the existing face ornament generation method needs to prepare materials in advance, the user's selection is greatly limited by the material library, that is, it mainly depends on the materials and the content is limited; in addition, the user needs to select the one he likes from a large number of materials, which is a time-consuming process, resulting in time-consuming search, that is, the traditional method cannot meet the user's demand when the materials are few, or the user's personalized demand when the materials are many. In addition, the materials are irrelevant to the user's portrait, and the materials are independent of the specific portrait generation, so the lighting, face posture information of the portrait cannot be fully utilized, and therefore the shooting conditions of the portrait need to be limited.

[0053] Based on this, the face ornament generation method provided by the embodiment of the application selects a target region in a face image, generates a face ornament according to a jewelry drawing by using a jewelry generation model and a jewelry fusion model, can fully utilize the context information such as the lighting condition and the portrait posture in the image for jewelry generation, and make the generated jewelry more natural and reasonable; and can more meet the user's personalized demand by generating the jewelry based on the user's jewelry drawing.

[0054] Referring to Figure 1 Fig. 1 shows a flowchart of a face ornament generation method based on CNN provided by an embodiment of the application.

[0055] In this embodiment, the method comprises:

[0056] S11, input a to-be-processed image, pre-process the to-be-processed image to obtain a first face image, and the to-be-processed image comprises a face region.

[0057] S12, determine a target region for modification in the first face image, and input a jewelry drawing superimposed on the target region to a pre-trained jewelry generation model to obtain a jewelry generation image.

[0058] S13, input the jewelry generation image and the first face image to a pre-trained jewelry fusion model for fusion to obtain a first result image.

[0059] S14, perform portrait posture restoration and fusion processing on the first result image to obtain a second result image.

[0060] Referring to Figure 2 Fig. 2 shows a flowchart of a face ornament generation method. In this embodiment, the pre-processing of the to-be-processed image comprises: obtaining a face point set FP in the to-be-processed image I by a mature CNN-based face detection and face alignment method, calculating an external rectangle thereof, and expanding outward to obtain a face cutting rectangle; obtaining a rotation angle of the face by the face cutting rectangle, and cutting the face image F after the face is properly positioned from the to-be-processed image I. At the same time, the face point set FP is converted to the coordinates of the face image F, denoted as FP*.

[0061] The user selects the target area for the jewelry in the face image F and inputs a hand-drawn sketch of the jewelry. The jewelry generation model generates the jewelry based on the sketch and relevant background information. Notably, this is an interactive process; the user can intervene in the generated jewelry by modifying the sketch until the desired jewelry is obtained. After the user confirms that the generated jewelry meets expectations, the image of the jewelry area is re-fused back onto the face image F, and the final effect is optimized by a jewelry fusion model to make the generated effect more natural. Finally, the face is rotated back to the original pose using FP* and then fused. The facial jewelry is generated based on the user-drawn sketch. Throughout the process, the user can continuously add new texture information based on the model's generated effect, gradually refining the jewelry effect.

[0062] Furthermore, the jewelry generation model includes a hand-drawn mapping model, a context encoding model, and a decoding model, wherein,

[0063] The process of drawing the hand-drawn illustration of the jewelry is mapped to jewelry features through the hand-drawn mapping model.

[0064] The context encoding model is used to extract the human portrait features surrounding the hand-drawn illustration of the jewelry.

[0065] The jewelry features and the portrait features are fused using the decoding model to generate the jewelry image.

[0066] like Figure 3 The diagram shows the overall structure of the jewelry generation model. In this embodiment, the jewelry generation model (denoted as OrnamentNetwork, or OrN) consists of three sub-networks: a sketch mapping model (denoted as SketchMappingNet, or SMN), which maps the user's sketching process to jewelry features; a context encoding model (ContextEncodingNet, or CEN), which encodes image information near the sketch; and a decoding model (DecodeNet, or DN), which fuses the features extracted by the two sub-models to generate the jewelry image.

[0067] Furthermore, the hand-drawn mapping model includes a multilayer perceptron and an adaptive mapping layer; the process of mapping the drawing process of the hand-drawn jewelry sketch to jewelry features through the hand-drawn mapping model includes:

[0068] The hand-drawn picture of the ornament is cut into several line segments, and the several line segments are divided into Q groups, each group containing K connected line segments. Each group of line segments is input into the multilayer perceptron to obtain a D-dimensional feature vector.

[0069] The adaptive mapping layer maps the D-dimensional feature vector into a fixed-dimensional jewelry feature map.

[0070] In this embodiment, the hand-drawn jewelry illustration is input by the user, and in most cases, only a few areas have pixels, making it sparse enough to be directly used as input for a CNN. To make subsequent encoding more targeted and to extract the jewelry illustration more effectively, the SMN attempts to use the user-drawn edges as input to avoid the sparsity problem of the image input. The role of the SMN is to receive several edges as input and map them into a feature map of a fixed scale. In this process, the user's hand-drawn illustration is first divided into several line segments, each represented by a quadruple (x1, y1, x2, y2), and K connected edges are taken as input to the SMN each time. Since a hand-drawn illustration... Figure 1 Multiple edges can usually be extracted, and the total number of edges is generally not an integer multiple of K. In this case, the number of edges can be supplemented to an integer multiple of K by padding with 0. Let's assume that the number of edges after padding is Q times K.

[0071] See Figure 4 The diagram shows the structure of an SMN. An SMN consists of a Multilayer Perceptron (MLP) and an Adaptive Mapping Layer (ADPL). These two parts operate separately. The MLP receives K edges as input at a time to obtain a D-dimensional feature vector. When the total number of edges is Q*K, a Q*D-dimensional output is obtained through a fully connected layer. The ADPL maps the Q*D output to a fixed-dimensional output, facilitating fusion with the CEN output. The ADPL comprises an AdpPool and a fully connected layer. The AdpPool divides the Q*D input into P parts, averages each part, and obtains a P*1 output. The fully connected layer (Linear) maps the P*1 input to an H*W*1 feature map, which has the same scale as the CEN output and the same scale as the DN input.

[0072] See Figure 5 The diagram shows the structure of CEN. The context encoding model receives 4 channels of input, where the first 3 channels are RGB target region images, and the fourth channel is a hand-drawn image of the ornament. This sub-model consists of Residual Fourier Convolution (Res-FFT-ConvBlock) and stacked convolutional layers, used to extract portrait features around the hand-drawn image. The structure of the Residual Fourier Convolution block can be found in [link to relevant documentation]. Figure 6 As shown. The purpose of this structure is to extract contextual information around the hand-drawn image. Considering the combination of convolutional layers with different receptive fields, the structure consists of a set of Fourier convolution and residual convolution branches. The Fourier convolution branches can effectively reduce the computational load of the module while ensuring a larger receptive field. At the same time, a small convolutional kernel is used as a branch to extract image features with a small receptive field.

[0073] Referring to Figure 7 The main structure of the decoder DN is stacked by convolutional layers, which respectively receive the outputs from the context encoder (CEN), the hand-drawn mapping model (SMN) as input, while introducing noise signals layer by layer to increase the texture details of the model output. The output of the DN is an RGB image. The UCL module in the decoder DN is an upsampling module used to upsample the output of the SMN and filter out the noise signals therein. The structure of the UCL module can be seen from Figure 8 The UCL module mainly consists of a PixelShuffle module and a self-attention module. The PixelShuffle module can map the input of HxWx4C into 2Hx2WxC by rearranging the feature map, where H represents the height of the feature map, W represents the height of the feature map, and C represents the number of channels of the feature map. In the figure, Conv-ReLU is convolution-ReLu activation, Conv-BN-Sigmoid is convolution-batch normalization-Sigmoid activation, and Conv-Leaky is convolution-LeakyReLU activation.

[0074] Further, the jewelry generation image and the first face image are input into a pre-trained jewelry fusion model for fusion to obtain a first result image, including:

[0075] According to the formula O = I x (1 - O a )+ O rgb x O a to calculate the final output of the jewelry fusion model; wherein O represents the first result image, I represents an input picture, the input picture includes the first face image and the jewelry generation image, O rgb represents the jewelry generation image, and O a represents the fusion ratio.

[0076] Referring to Figure 9 The structure of the jewelry fusion model is shown in the figure. Since the role of OrN is to generate jewelry in the local wearing area, the generated picture also needs to be fused onto the portrait with a certain ratio. The traditional effect fusion is to fuse the RGB generated by the model onto the original picture with a certain ratio. Although this method is simple and convenient, it ignores the influence of the overall lighting condition of the portrait on the wearing effect of the jewelry. Therefore, the jewelry fusion model designed in this embodiment solves the above problems, and the structure of the jewelry fusion model consists of stacked convolution modules (ConvBlock) and residual modules (ResBlock). The input of the jewelry fusion model is a portrait and a jewelry region image (padded to the same size as the portrait), and the output is a fusion ratio O a . The output of the jewelry generation model is denoted as O rgbThe input picture (including the first face image and the jewelry generation image) is denoted as I, and the final output is denoted as O. The result can be expressed as:

[0077] O = I x (1 - O a ) + O rgb x O a

[0078] Further, the training process of the jewelry generation model includes:

[0079] The result image output by the jewelry generation model is supervised by using a first loss function and a second loss function, wherein

[0080] The first loss function is:

[0081] The second loss function is: O represents the result image output by the jewelry generation model, T represents the target image, H represents the image height, W represents the image width, W win represents the side length of the square window, O sketch represents the sketch of the result output by the model, I sketch represents the sketch image after extracting the key position, and λ i，j represents the window-related weight parameter.

[0082] In specific implementation, data acquisition and preparation are needed to train the model, wherein the data set is composed of a large number of data pairs <I, I d >, wherein I represents a portrait without jewelry, and I d represents a portrait with jewelry. In the embodiment, data acquisition is performed through two channels of model shooting and traditional pasting. I d will be obtained by a hand-drawing model (the purpose of the model is to convert a color picture into a hand-drawing (line drawing)), image edge detection, and the like. I sketch is a hand-drawing of the jewelry region. Then, the data is divided into a training set and a test set. During training, the lines on the hand-drawing are randomly erased, twisted, and straightened to simulate the case of manual drawing.

[0083] The output result of the jewelry generation model is supervised by constructing a loss function, and the output of the jewelry generation model is compared with the target image at the pixel level. The loss function includes:

[0084]

[0085] wherein O represents the model output, T represents the target image, H represents the image height, and W represents the image width.

[0086] The output result O is input into the hand-drawing model based on deep learning to obtain a hand-drawing O sketch , that is,

[0087] O sketch = Sketch(θ, O)

[0088] wherein Sketch represents the hand-drawing model, and θ is a parameter of the hand-drawing model. Due to the limitation of the input device and the user's drawing ability, the hand-drawing input by the user is generally rough and has a certain difference from the contour of the actual object. If the hand-drawing generated by the model is required to be completely consistent with the user input, it will be difficult to obtain a suitable jewelry image in actual use. Therefore, as long as the two are similar, in order to calculate the "similarity", the user input jewelry hand-drawing needs to be filtered first, and the key positions are extracted to calculate the loss. The process of extracting the key positions is as follows: Figure 1

[0089] A square window with a side length of W win is used to traverse the input hand-drawing. For a point (x, y) at the center of the window, the value v x，y is:

[0090] wherein,

[0091] The image after extracting the key positions is denoted as I′ skctch .

[0092] The result generated by the jewelry generation model generally has more details than the user's hand-drawing. Therefore, the hand-drawing O sketch obtained by taking the model output as the input of the hand-drawing model needs to cover the above-mentioned key points, and at the same time, it should allow appropriate increase of the details not in I sketch . Similarly, a square window with a side length of W win is used to traverse the image area. For convenience of description, the center position of the window is set as the coordinate origin. Then the hand-drawing part loss can be expressed as:

[0093]

[0094] wherein λ i，j is a window-related parameter, and its calculation formula is:

[0095]

[0096] wherein ζ is a symbol function:

[0097]

[0098] ​The window correlation function aims to give enough "attention" to the key points and appropriately relax the requirements for the surrounding area, so that the generated result is similar to the user's hand-drawn result.

[0099] Further, the training process of the jewelry fusion model comprises:

[0100] The result image output by the jewelry fusion model is supervised by using a third loss function and a fourth loss function.

[0101] The third loss function is:

[0102] The fourth loss function is: F o represents the result image output by the jewelry fusion model, F t represents the target effect image, and φ j represents the feature map output by the last convolutional layer of the jth module of the VGG16 network, H represents the image height, and W represents the image width.

[0103] In specific implementation, the final output of the jewelry fusion model also needs to be supervised. For ease of expression, F o represents the final output of the fusion model, F t represents the target effect, and is supervised by using the following loss function:

[0104] Fusion loss:

[0105]

[0106] This loss uses L1-Loss to supervise the final output result of the jewelry fusion model at the pixel level.

[0107] Perception loss:

[0108]

[0109] wherein φ j represents the feature map output by the last convolutional layer of the jth module of the VGG16 network.

[0110] Discrimination loss:

[0111] L dis =-log(D f (F o , F t ))

[0112] wherein D fFor the discriminator, the output is 1 when the picture is a real picture, and the purpose is to make the generated result closer to the real picture.

[0113] The embodiment generates jewelry by drawing jewelry on a face image according to a user inputted hand-drawn picture, breaks away from the constraint of a material library, can better utilize background information of pixels around the jewelry, makes the effect of wearing the jewelry more reasonable and natural, and realizes personal customization of the user by intervention of the hand-drawn picture, and obtains a personalized jewelry effect.

[0114] Referring to Figure 10 Fig. 1 shows a structure schematic diagram of a face jewelry generation device based on CNN provided by an embodiment of the present application.

[0115] In the embodiment, the device 10 comprises:

[0116] The preprocessing unit 101 is configured to input a to-be-processed image, pre-process the to-be-processed image, and obtain a first face image, wherein the to-be-processed image comprises a face region.

[0117] The jewelry generation unit 102 is configured to determine a target region to be modified in the first face image, superimpose a hand-drawn picture of jewelry on the target region, and input the hand-drawn picture of jewelry to a pre-trained jewelry generation model to obtain a jewelry generation image.

[0118] The first fusion unit 103 is configured to input the jewelry generation image and the first face image to a pre-trained jewelry fusion model to perform fusion, and obtain a first result image.

[0119] The second fusion unit 104 is configured to perform portrait posture restoration and fusion processing on the first result image, and obtain a second result image.

[0120] The units and modules of the device 10 can respectively execute the corresponding steps in the above method embodiments, and thus will not be described here in detail, and please refer to the above description of the corresponding steps.

[0121] The embodiment of the present application also provides a device comprising the face jewelry generation device based on CNN as described above, wherein the face jewelry generation device based on CNN can adopt Figure 10 The structure of the embodiment, which can correspondingly execute Figure 1 The technical solutions of the method embodiments shown in the figure are similar in implementation principle and technical effects, and details can be referred to the above description of the embodiments, and thus will not be described here in detail.

[0122] The device comprises a mobile phone, a digital camera, a tablet computer or other devices with a photographing function, or a device with an image processing function, or a device with an image display function. The device can comprise a memory, a processor, an input unit, a display unit, a power supply and other components.

[0123] The memory can be used to store software programs and modules, and the processor executes various functions and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and the input unit.

[0124] The input unit can be used to receive inputted digital or character or image information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls. Specifically, the input unit of the embodiment can include a camera in addition to a touch-sensitive surface (such as a touch display screen) and other input devices.

[0125] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, video and any combination thereof. The display unit can include a display panel, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface can cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it transmits to the processor to determine the type of touch event, and then the processor provides corresponding visual output on the display panel according to the type of touch event.

[0126] The embodiment of the present application also provides a computer readable storage medium, which can be a computer readable storage medium contained in the memory in the above-mentioned embodiments; or can exist separately and not be assembled into the device. The computer readable storage medium stores at least one instruction, which is loaded and executed by the processor to realize Figure 1 the CNN-based face ornament generation method shown. The computer readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0127] It should be noted that each of the embodiments described in this specification has a corresponding counterpart embodiment not mentioned explicitly. Therefore, each of the embodiments described in this specification should not be taken as excluding the other. For example, the method embodiments can be implemented by a device or apparatus, and the device or apparatus embodiments can be implemented by a method. Therefore, the device or apparatus embodiments are not mutually exclusive with the method embodiments.

[0128] Also, as used in the description of the embodiments of the application and the appended claims, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" or "the component" can include a plurality of such components unless the context clearly dictates otherwise.

[0129] The above description discloses and describes the preferred embodiments of the present application. It is understood that the present application is not limited to the disclosed embodiments, and is not to be construed as excluding other embodiments, but is applicable to various other combinations, modifications and environments, and can be modified and changed within the scope of the inventive concept disclosed herein by the above teachings or related art or knowledge, and the modifications and changes made by those skilled in the art without departing from the spirit and scope of the present application shall fall within the scope of the appended claims.

Claims

1. A method for generating facial ornaments based on CNN, characterized in that, The method includes: Input the image to be processed, preprocess the image to be processed to obtain a first face image, the image to be processed includes a face region; The target region for modification in the first face image is determined, and the hand-drawn jewelry image is superimposed on the target region and then input into the pre-trained jewelry generation model to obtain the jewelry generation image; The jewelry generation model includes a hand-drawn mapping model, a context encoding model, and a decoding model. The process of drawing the hand-drawn illustration of the jewelry is mapped to jewelry features through the hand-drawn mapping model. The context encoding model is used to extract the human portrait features surrounding the hand-drawn illustration of the jewelry. The jewelry features and the portrait features are fused using the decoding model to generate the jewelry image. The generated jewelry image and the first face image are input into a pre-trained jewelry fusion model for fusion to obtain the first result image; The step of inputting the generated jewelry image and the first face image into a pre-trained jewelry fusion model for fusion to obtain a first result image includes: According to the formula O=I×(1-O) α )+O rgb +O α The final output of the jewelry fusion model is calculated; where O represents the first result image, I represents the input image, and O rgb This indicates that the image of the jewelry has been generated, O α Indicates the fusion ratio; The first result image is subjected to human pose restoration and fusion processing to obtain the second result image.

2. The method for generating facial accessories based on CNN according to claim 1, characterized in that, The hand-drawn mapping model includes a multilayer perceptron and an adaptive mapping layer; the process of mapping the hand-drawn sketch of the jewelry into jewelry features through the hand-drawn mapping model includes: The hand-drawn picture of the ornament is cut into several line segments, and the several line segments are divided into Q groups, each group containing K connected line segments. Each group of line segments is input into the multilayer perceptron to obtain a D-dimensional feature vector. The adaptive mapping layer maps the D-dimensional feature vector into a fixed-dimensional jewelry feature map.

3. The method for generating facial accessories based on CNN according to claim 1, characterized in that, The training process of the jewelry generation model includes: The output image of the jewelry generation model is supervised by using a first loss function and a second loss function, wherein... The first loss function is: The second loss function is: O represents the result image output by the jewelry generation model, T represents the target image, H represents the image height, and W represents the image width. win O represents the side length of a square window. sketch A sketch representing the output of the model, I sketch λ represents the sketch image after extracting key locations. i，j This represents the window-related weight parameters.

4. The method for generating facial accessories based on CNN according to claim 3, characterized in that, According to the formula Calculate the λ i，j Where ζ is a sign function.

5. The method for generating facial accessories based on CNN according to claim 1, characterized in that, The training process of the jewelry fusion model includes: The output graph of the jewelry fusion model is supervised using a third and a fourth loss function; wherein... The third loss function is: The fourth loss function is: F o F represents the output image of the jewelry fusion model. t This represents the target effect diagram, φ j H represents the feature map output by the last convolutional layer of the j-th module of the VGG16 network, where H represents the image height and W represents the image width.

6. A CNN-based facial adornment generation device, characterized in that, The device includes: A preprocessing unit is used to input an image to be processed, preprocess the image to be processed, and obtain a first face image, wherein the image to be processed includes a face region; The jewelry generation unit is used to determine the target area to be modified in the first face image, and after superimposing the hand-drawn jewelry image on the target area, it is input into the pre-trained jewelry generation model to obtain the jewelry generation image; The jewelry generation model includes a hand-drawn mapping model, a context encoding model, and a decoding model. The process of drawing the hand-drawn illustration of the jewelry is mapped to jewelry features through the hand-drawn mapping model. The context encoding model is used to extract the human portrait features surrounding the hand-drawn illustration of the jewelry. The jewelry features and the portrait features are fused using the decoding model to generate the jewelry image. The first fusion unit is used to input the generated jewelry image and the first face image into a pre-trained jewelry fusion model for fusion to obtain a first result image; The first fusion unit is further configured to: According to the formula O=I×(1-O) α )+O rgb ×O α The final output of the jewelry fusion model is calculated; where O represents the first result image, I represents the input image, and O rgb This indicates that the image of the jewelry has been generated, O α The first unit represents the fusion ratio; the second fusion unit is used to perform portrait pose restoration and fusion processing on the first result image to obtain the second result image.

7. A device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a CNN-based face decoration generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of a CNN-based face decoration generation method as described in any one of claims 1 to 5.

Images