Image generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image generation method, device, electronic device and storage medium, the method comprising: extracting features from a target face image to obtain target face key points; inputting the target face key points into a key point prediction model to predict the key points of a sticker face to obtain target sticker face key points; generating a target sticker face image according to the target sticker face key points; performing semantic segmentation processing on the target sticker face image to obtain a target avatar area; intercepting the target sticker face image according to the target avatar area to obtain a target sticker avatar corresponding to the target avatar area; pasting the target sticker avatar into the target face image to generate a sticker image. The present disclosure generates a sticker avatar consistent with the facial posture in the target face image in real time according to the target face image, and pastes it into the face area in the target face image, so as to achieve natural and coordinated facial occlusion in the image and better protect the user's privacy.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] Face masking with sticker avatars refers to the technology of using sticker avatars to mask real faces, which not only presents an aesthetic effect but also protects the privacy of users. In related technologies, the face area in the image is masked with preset cartoon image materials. The materials are fixed and cannot match the user's posture, so natural and coordinated face masking cannot be achieved. Some parts of the face are often exposed, which cannot well protect the privacy of users. Summary of the invention

[0003] The present disclosure provides an image generation method, device, electronic device and storage medium to at least solve the problem in the related art that a sticker avatar consistent with the posture of a face image cannot be generated in real time. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

[0005] Extract features from the target face image to obtain the key points of the target face;

[0006] Inputting the target face key points into a key point prediction model to predict the mapping face key points, and obtaining the target mapping face key points; the target face corresponding to the target face key points has the same posture as the target mapping face corresponding to the target mapping face key points;

[0007] Generating a target map face image according to the target map face key points;

[0008] Performing semantic segmentation processing on the target face image to obtain a target head image region; the target head image region includes a face region and a hair region;

[0009] The target face image is intercepted according to the target avatar area to obtain a target face image corresponding to the target avatar area;

[0010] Paste the target sticker avatar into the target face image to generate a sticker image.

[0011] In an exemplary embodiment, extracting features from the target face image to obtain key points of the target face includes:

[0012] Inputting the target face image into a face key point detection network for feature extraction to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set;

[0013] Using the key points in the initial set of facial key points as calibration points, performing a face extraction operation on the target face image according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or filling;

[0014] The key points in the initial facial key point set are converted into the coordinate system constructed by the facial image to obtain a target facial key point set.

[0015] In an exemplary embodiment, the step of using key points in the initial facial key point set as calibration points and performing a face extraction operation on the target facial image according to a face alignment algorithm to obtain a facial image includes:

[0016] Determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point;

[0017] Determine the lip center point according to the lip contour point set;

[0018] The eye center point and the lip center point are used as calibration points, and a face extraction operation is performed on the target face image according to the face alignment algorithm to obtain the face image.

[0019] In an exemplary embodiment, the training method of the key point prediction model includes:

[0020] According to the sample face image set and the sample texture face image set, at least one set of sample image pairs is constructed; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same;

[0021] Detecting sample face key points of the sample face images in each group of sample image pairs according to a face key point detector;

[0022] Detecting sample texture key points of the sample texture face images in each set of sample image pairs according to a texture face key point detector;

[0023] Labeling the sample map key points in each group of sample image pairs onto the sample face key points in each group of sample image pairs to obtain first map key point labels for the sample face key points in each group of sample image pairs;

[0024] Input the sample face key points in each set of sample image pairs into a preset model and output the second map key point labels;

[0025] The preset model is trained according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

[0026] In an exemplary embodiment, constructing at least one set of sample image pairs based on the sample face image set and the sample texture face image set includes:

[0027] Determine the human face generation model and the texture face generation model;

[0028] Inputting a random facial feature set into the face generation model to perform face image prediction to obtain the sample face image set;

[0029] Inputting the random facial feature set into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set;

[0030] The sample face images and the sample textured face images corresponding to the same random facial features are taken as a set of sample image pairs.

[0031] In an exemplary embodiment, the method further comprises:

[0032] Using the preset facial features as the current facial features; the current facial features are marked with the current image face label;

[0033] Inputting the current facial features into the sticker face generation model to perform sticker face generation processing, and outputting the current sticker face;

[0034] If the output current mapped face does not match the annotated current mapped face label, determine current key point information of the current mapped face, and annotate the output current mapped face label with the current key point information;

[0035] The current key point information is used as the current facial feature again, and the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing is repeated to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, the current key point information of the current sticker face is determined, and the current sticker face label outputted by annotating the current key point information is outputted; until the current sticker face label outputted by the sticker face generation model matches the annotated current sticker face label;

[0036] The current key point information when the sticker face label output by the sticker face generation model matches the annotated current sticker face label is determined as the target facial feature.

[0037] In an exemplary embodiment, generating a target textured face image according to the target textured face key points includes:

[0038] Filtering out target key points of the same type as the target facial features from the target map face key points;

[0039] The target key points are input into the textured face generation model to generate the target textured face image.

[0040] In an exemplary embodiment, after the face extraction operation is performed on the target face image using the key points in the initial face key point set as calibration points according to the face alignment algorithm to obtain the face image, the method further includes:

[0041] Obtaining image processing parameters corresponding to the face extraction operation;

[0042] Performing inverse transformation processing on the image processing parameters to obtain inverse transformation parameters;

[0043] Pasting the target sticker avatar into the target face image to generate a sticker image includes:

[0044] Performing a reverse transformation process on the target sticker head portrait according to the reverse transformation parameters to obtain an image to be pasted;

[0045] The image to be pasted is pasted in the face avatar area of ​​the target face image to obtain the pasted image.

[0046] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0047] The target face key point determination module is configured to perform feature extraction on the target face image to obtain the target face key points;

[0048] The target mapping face key point determination module is configured to input the target face key points into the key point prediction model to predict the mapping face key points, and obtain the target mapping face key points; the target face corresponding to the target face key points has the same posture as the target mapping face corresponding to the target mapping face key points;

[0049] A target map face image generation module is configured to generate a target map face image according to the target map face key points;

[0050] A target avatar region determination module is configured to perform semantic segmentation processing on the target mapped face image to obtain a target avatar region; the target avatar region includes a face region and a hair region;

[0051] A sticker avatar generation module is configured to intercept the target sticker face image according to the target avatar area to obtain a sticker avatar corresponding to the target avatar area;

[0052] The image generation module is configured to execute pasting the target sticker avatar into the target face image to generate a sticker image.

[0053] In an exemplary embodiment, the target face key point determination module includes:

[0054] A feature extraction unit is configured to perform feature extraction by inputting the target face image into a face key point detection network to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set;

[0055] An image processing unit is configured to perform a face extraction operation on the target face image using the key points in the initial face key point set as calibration points according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or padding;

[0056] The target face key point set determination unit is configured to convert the key points in the initial face key point set into the coordinate system constructed by the facial image to obtain the target face key point set.

[0057] In an exemplary embodiment, the image processing unit includes:

[0058] The eye center point determination subunit is configured to determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point;

[0059] A lip center point determination subunit is configured to determine the lip center point according to the lip contour point set;

[0060] The image processing subunit is configured to perform a face extraction operation on the target face image according to the face alignment algorithm using the eye center point and the lip center point as calibration points to obtain the face image.

[0061] In an exemplary embodiment, the device further comprises:

[0062] The sample image pair construction module is configured to construct at least one set of sample image pairs according to the sample face image set and the sample texture face image set; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same;

[0063] A sample face key point detection module is configured to detect sample face key points of sample face images in each group of sample image pairs according to a face key point detector;

[0064] A sample map key point detection module is configured to detect sample map key points of a sample map face image in each set of sample image pairs according to a map face key point detector;

[0065] A map key point labeling module is configured to perform labeling of sample map key points in each group of sample image pairs onto sample face key points in each group of sample image pairs to obtain first map key point labels for sample face key points in each group of sample image pairs;

[0066] An output map key point label determination module is configured to input the sample face key points in each group of sample image pairs into a preset model and output a second map key point label;

[0067] The training module is configured to train the preset model according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

[0068] In an exemplary embodiment, the sample image pair construction module includes:

[0069] A model determination unit, configured to determine a human face generation model and a texture face generation model;

[0070] A sample face image set generating unit is configured to input a random face feature set into the face generation model to perform face image prediction to obtain the sample face image set;

[0071] A sample sticker face image set determination unit is configured to input the random facial feature set into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set;

[0072] The sample image pair construction unit is configured to execute the sample human face images and the sample textured face images corresponding to the same random facial features as a group of sample image pairs.

[0073] In an exemplary embodiment, the device further comprises:

[0074] The current facial feature determination module is configured to execute the preset facial feature as the current facial feature; the current facial feature is marked with the current mapped face label;

[0075] A current sticker face determination module is configured to execute sticker face generation processing by inputting the current facial features into the sticker face generation model, and output the current sticker face;

[0076] The current sticker face labeling module is configured to determine the current key point information of the current sticker face if the output current sticker face does not match the labeled current sticker face label, and label the output current sticker face label for the current key point information;

[0077] The step repeating module is configured to execute the steps of reusing the current key point information as the current facial feature, and repeating the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, determining the current key point information of the current sticker face, and annotating the current key point information with the output current sticker face label; until the current sticker face label output by the sticker face generation model matches the annotated current sticker face label;

[0078] The target facial feature determination module is configured to determine the current key point information when the sticker face label output by the sticker face generation model matches the marked current sticker face label as the target facial feature.

[0079] In an exemplary embodiment, the target texture face image generation module includes:

[0080] A target key point screening unit is configured to screen target key points of the same type as the target facial features from the target mapped face key points;

[0081] The target textured face image generating unit is configured to input the target key points into the textured face generating model to generate the target textured face image.

[0082] In an exemplary embodiment, the device further comprises:

[0083] An image processing parameter acquisition module is configured to execute and acquire image processing parameters corresponding to the face extraction operation;

[0084] an inverse transformation parameter determination module, configured to perform inverse transformation processing on the image processing parameters to obtain inverse transformation parameters;

[0085] The image generation module comprises:

[0086] A unit for determining an image to be pasted is configured to perform an inverse transformation process on the target image according to the inverse transformation parameters to obtain an image to be pasted;

[0087] The pasting image generating unit is configured to paste the image to be pasted in the face avatar area of ​​the target face image to obtain the pasting image.

[0088] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0089] processor;

[0090] a memory for storing instructions executable by the processor;

[0091] The processor is configured to execute the instructions to implement the image generation method as described above.

[0092] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image generating method as described above.

[0093] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which implements the image generating method as described above when executed by a processor.

[0094] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0095] The present invention performs feature extraction on a target face image to obtain target face key points; the target face key points are input into a key point prediction model to predict the target face key points to obtain target face key points; the target face corresponding to the target face key points has the same posture as the target face corresponding to the target face key points; a target face image is generated based on the target face key points; semantic segmentation is performed on the target face image to obtain a target avatar area; the target avatar area includes a face area and a hair area; the target face image is intercepted based on the target avatar area to obtain a target avatar corresponding to the target avatar area; the target avatar is pasted into the target face image to generate a map image. The present invention generates a map avatar consistent with the face posture in the image in real time based on the target face image, and pastes it into the face area in the target face image, thereby achieving natural and coordinated facial occlusion in the image and better protecting user privacy.

[0096] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0098] Figure 1 The figure is a diagram showing an application environment of an image generating method according to an exemplary embodiment.

[0099] Figure 2 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0100] Figure 3The present invention is a flowchart of a method for determining a target face key point set according to an exemplary embodiment.

[0101] Figure 4 The present invention is a flowchart of a method for training a key point prediction model according to an exemplary embodiment.

[0102] Figure 5 The figure is a flow chart of a method for constructing a sample image pair according to an exemplary embodiment.

[0103] Figure 6 The figure is a flow chart showing a method for determining target facial features according to an exemplary embodiment.

[0104] Figure 7 It is an algorithm flow chart of an image generating method according to an exemplary embodiment.

[0105] Figure 8 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0106] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0107] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0108] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0110] Sticker avatar face masking is a technology that uses cartoon effects to mask the face, which not only presents a beautiful effect but also protects the user's privacy. Smart sticker avatar generation refers to a technology that uses computers to automatically generate sticker avatars. In the process of sharing daily photos and using photo editing apps, many users are reluctant to show their faces due to some subjective or objective factors, which gave rise to the birth of this technology.

[0111] At present, various photo editing apps on the market are highly homogenized, with similar core functions and excessive focus on making users more beautiful and good-looking, ignoring common user pain points and core demands such as social phobia and privacy protection. Current apps often use fixed materials and lack diversity.

[0112] In order to provide a more natural, more coordinated and more diverse way of occluding a sticker avatar, the present disclosure provides an image generation method, device, electronic device and storage medium.

[0113] See also Figure 1 , which is a diagram of an application environment of an image generating method according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0114] Specifically, in the embodiments of this specification, the server 01 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, a memory, and the like. Specifically, the server 01 can be used to extract features from a target face image to obtain target face key points; input the target face key points into a key point prediction model to predict a map face key point to obtain target map face key points; the target face corresponding to the target face key points has the same posture as the target map face corresponding to the target map face key points; and generate a target map face image based on the target map face key points; perform semantic segmentation processing on the target map face image to obtain a target avatar area; the target avatar area includes a face area and a hair area; and intercept the target map face image based on the target avatar area to obtain a target map avatar corresponding to the target avatar area; and paste the target map avatar into the target face image to generate a map image; and send the map image to the client 02.

[0115] Specifically, in the embodiments of this specification, the client 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, smart wearable devices, and vehicle-mounted terminals, and may also include software running in physical devices, such as web pages provided by some service providers to users, or applications provided by these service providers to users. Specifically, the client 02 may be used to display sticker images.

[0116] Figure 2 is a flowchart of an image generating method according to an exemplary embodiment. Figure 2 As shown, this method can be applied to Figure 1 The client 02 shown includes the following steps.

[0117] In step S21, feature extraction is performed on the target face image to obtain target face key points.

[0118] In the embodiment of the present disclosure, the target facial image may be a facial image obtained by a camera or a photographing device, and the target facial key points may include but are not limited to key points such as eyes and lips in the target facial image.

[0119] In some embodiments, Figure 3 As shown, the feature extraction of the target face image to obtain the target face key points includes:

[0120] In step S211, the target face image is input into a face key point detection network for feature extraction to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set;

[0121] In the disclosed embodiments, face key point detection is also called face key point detection, positioning or face alignment, which means locating the key areas of the face, including eyebrows, eyes, nose, mouth, facial contour, etc., given a face image. There are three main methods for face key point detection: traditional methods based on ASM (Active Shape Model) and AAM (Active Appearance Model); methods based on cascade shape regression and methods based on deep learning. Face key point detection network (Net detect_human_lmk) can include ASM, AAM, etc. ASM is an algorithm based on the Point Distribution Model (PDM); in PDM, the geometric shapes of objects with similar appearances, such as faces, hands, hearts, lungs, etc., can be represented by connecting the coordinates of several key feature points (landmarks) in series to form a shape vector. AAM is based on ASM, and further statistically models the texture (the shape-independent image obtained by deforming the face image to the average shape), and further merges the two statistical models of shape and texture into an appearance model.

[0122] In the embodiment of the present disclosure, the eye contour points may include left eye contour points and right eye contour points; the input target face image may be recorded as I human , through a face key point detection network (denoted as Net detect_human_lmk ) detects the coordinates of the key points corresponding to the face area in the image (relative coordinates relative to the original image size), and records the obtained face key points as Lmk ori_human .

[0123] In step S212, using the key points in the initial set of facial key points as calibration points, a face extraction operation is performed on the target face image according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or filling;

[0124] In the disclosed embodiment, the face alignment algorithm is a method for extracting facial features based on facial key points. During the face extraction operation, feature extraction can be performed according to a preset ratio, and the preset ratio can be set according to the cropping standard of the Flickr-Faces-High-Quality (FFHQ) data set. The face extraction operation can be selected according to the actual situation, usually by cropping the target face image; if the face in the target face image has a tilt angle, it needs to be rotated, and the image is filled as needed so that the tilt angle of the face after rotation is 0; thereby obtaining a straightened face image with a low non-face area ratio.

[0125] In some embodiments, the step of using key points in the initial set of facial key points as calibration points and performing a face extraction operation on the target facial image according to a face alignment algorithm to obtain a facial image includes:

[0126] Determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point;

[0127] In the embodiment of the present disclosure, the eye contour point set may include a left eye contour point set and a right eye contour point set, and the eye center point may include a left eye center point and a right eye center point; the left eye contour point set in the key point may be used to calculate the left eye center point P of the face. left_eye_center The right eye contour point set in this key point can be used to calculate the right eye center point P of the face. right_eye_center .

[0128] Determine the lip center point according to the lip contour point set;

[0129] In the embodiment of the present disclosure, the lip contour point set in the key point can be used to calculate the lip center point P of the face. mout_h_center .

[0130] The eye center point and the lip center point are used as calibration points, and a face extraction operation is performed on the target face image according to the face alignment algorithm to obtain the face image.

[0131] In the embodiment of the present disclosure, the left eye contour point set, the right eye contour point set and the lip contour point set in the key points can be used to calculate the left eye center point P of the face. left_eye_center , right eye center point P right_eye_center and the lip center point P mouth_center .

[0132] In the embodiment of the present disclosure, the face extraction operation may include rotation, cropping and filling operations. The left eye center point, the right eye center point and the lip center point may be used as calibration points to rotate, crop and fill the face area according to a certain ratio (according to the cropping standard of the FFHQ dataset) (the corresponding operation parameters are Params aligned_human_i ). Finally, the aligned face cropping area image I is obtained aligned_human_i , where i represents the image I human For example, if the face in the target face image has a tilt angle, it needs to be rotated and filled as needed so that the tilt angle of the face after rotation is 0; thus, a straightened face image with a low proportion of non-face areas is obtained.

[0133] In the disclosed embodiment, features of the target facial image can be extracted to obtain two eye center points and one lip center point, and then the target facial image is processed using these three points as calibration points to obtain a processed facial image; thereby, non-face areas in the target facial image are eliminated to the greatest extent, and a processed facial image that can accurately represent the face area is obtained, thereby improving the accuracy of subsequent determination of the target facial key points.

[0134] In some embodiments, after the face extraction operation is performed on the target face image using the key points in the initial face key point set as calibration points according to the face alignment algorithm to obtain the face image, the method further includes:

[0135] Obtaining image processing parameters corresponding to the face extraction operation;

[0136] The image processing parameters are inversely transformed to obtain inverse transformation parameters.

[0137] In the disclosed embodiment, the image processing parameter is an image processing parameter corresponding to the face extraction operation, and the image processing parameter can be inversely transformed; for example, if the image processing parameter is a counterclockwise rotation of 30°, the inverse transformation is a clockwise rotation of 30°.

[0138] In the disclosed embodiment, the image processing parameters can be processed to obtain inverse transformation parameters, and then the generated target map avatar can be processed according to the inverse transformation parameters, so as to generate a target map avatar with a posture completely consistent with the facial image in real time, thereby improving the matching degree between the target map avatar and the target facial image, thereby achieving natural and coordinated facial occlusion.

[0139] In step S213, the key points in the initial facial key point set are converted into the coordinate system constructed by the facial image to obtain a target facial key point set.

[0140] In the disclosed embodiment, the detected facial key point coordinates are transformed into a relative coordinate system, from the original upper left corner of the image as the origin, the height as the x-axis, and the width as the y-axis to the face cropping area I aligned_human_i The upper left corner of the crop area is the origin, the height of the crop area is the x-axis, and the width of the crop area is the y-axis coordinate system. The key point after conversion is Lmk crop_human .

[0141] In the disclosed embodiment, feature extraction can be performed on a target facial image to obtain facial key points; then, a face extraction operation is performed on the target facial image using the facial key points as calibration points to obtain a facial image; the facial image may be an image that includes only the facial area; a coordinate system is constructed based on the facial image, and the facial key points are updated to obtain a target facial key point set, and based on the updated target facial key point set, the prediction accuracy of the target map facial key points can be improved.

[0142] In step S22, the target face key points are input into a key point prediction model to predict the mapping face key points, and the target mapping face key points are obtained; the target face corresponding to the target face key points has the same posture as the target mapping face corresponding to the target mapping face key points.

[0143] In the disclosed embodiment, the key point prediction model can predict the corresponding target texture face key points based on the input target face key points. The texture face can include but is not limited to cartoon face, sketch face, robot face, comic face, oil painting face, etc.

[0144] In some embodiments, Figure 4 As shown, the training method of the key point prediction model includes:

[0145] In step S41, at least one set of sample image pairs is constructed based on the sample face image set and the sample texture face image set; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same;

[0146] In the disclosed embodiment, during the training process of the key point prediction model, a sample face image set and a sample texture face image set may be first obtained, and multiple groups of sample image pairs may be constructed; each group of sample image pairs includes a sample face image and a corresponding sample texture face image.

[0147] In some embodiments, Figure 5 As shown, the method of constructing at least one set of sample image pairs based on the sample face image set and the sample texture face image set includes:

[0148] In step S411, a human face generation model and a texture face generation model are determined;

[0149] In the embodiment of the present disclosure, determining the human face generation model and the mapping face generation model includes:

[0150] Inputting facial features of the labeled facial image into a pre-trained model to perform training for generating facial images, so as to adjust model parameters of the pre-trained model until the facial image output by the pre-trained model matches the labeled facial image;

[0151] The pre-trained model corresponding to the model parameters when the output face image matches the annotated face image is used as the face generation model;

[0152] Inputting the facial features of the labeled face image into the face generation model to train the generated face image, so as to adjust the model parameters of the face generation model until the face image output by the face generation model matches the labeled face image;

[0153] The model parameters when the output sticker face image is matched with the labeled sticker face image and the corresponding face generation model are used as the sticker face generation model.

[0154] In the embodiment of the present disclosure, firstly, the required face data is collected and sorted through the network to construct a face data set Datasets anime Then, we use this data to train a StyleGANv2 model (denoted as Gen human ) to obtain a StyleGANv2 model (denoted as Gen finetune_anime ), that is, the sticker face generation model. The StyleGANv2 model can be replaced by any model that supports generation by sampling from the noise space (including but not limited to StyleGAN, StyleGANv3 and other models). Sticker face generation models may include but are not limited to CycleGAN, UGATIT and other models. Generative Adversarial Networks (GAN) is a very effective deep learning model, from which CycleGAN is derived. CycleGAN is essentially two mirror-symmetric GANs, forming a ring network. The two GANs share two generators and each has a discriminator, that is, there are two discriminators and two generators in total. UGATIT is an unsupervised generative attention network with adaptive layer instance normalization for image-to-image conversion, which is a new unsupervised image-to-image conversion method.

[0155] In step S412, the random facial feature set is input into the face generation model to perform face image prediction to obtain the sample face image set;

[0156] In the disclosed embodiment, the random facial feature set may include multiple random facial features (random numbers). Random numbers are used in machine learning. In order to grasp the generation characteristics of random numbers, certainty and order are obtained from the randomness and disorder of random numbers. A random number seed is used to achieve this goal. The random number seed can enable the entire program that introduces random numbers to obtain certain and consistent results in multiple runs.

[0157] In step S413, the random facial feature set is input into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set;

[0158] In the disclosed embodiment, the random facial feature set may be input into the human face generation model and the texture face generation model respectively to obtain a sample human face image set and a sample texture face image set respectively.

[0159] In step S414, the sample human face image and the sample texture face image corresponding to the same random facial feature are taken as a set of sample image pairs.

[0160] In the embodiment of the present disclosure, 50,000 sets of data are sampled from the two models before and after fine-tuning, the face generation model and the texture face generation model, respectively, by using a fixed random number seed to form a pair, and each set of sampled paired data is recorded as P_I i =(I sample_human_i , I sample_anime_i )(where i represents each set of paired data).

[0161] In the disclosed embodiment, fine-tuning training can be performed on the basis of the face generation model to obtain a textured face generation model; and based on the face generation model and the textured face generation model, multiple groups of sample image pairs are constructed, and random number seeds are used to ensure that the facial postures of the sample face images and the sample textured face images obtained by inputting the same random facial features into the two models are the same, thereby improving the prediction accuracy of the key point prediction model.

[0162] In step S42, the sample face key points of the sample face images in each group of sample image pairs are detected according to the face key point detector;

[0163] In the disclosed embodiment, the face key point detector Net detect_human_lmk The sample face images in each group of sample image pairs are detected to obtain the key points of the corresponding sample face images, namely, the sample face key points.

[0164] In step S43, the sample texture key points of the sample texture face images in each group of sample image pairs are detected according to the texture face key point detector;

[0165] In the disclosed embodiment, the sample map key points can be determined by a map face key detector; if the map face is a cartoon face, a cartoon key point detector (denoted as Net detect_anime_lmk ) Detect the sample textured face image in each group of sample image pairs to obtain the key points of the sample textured face image.

[0166] In step S44, the sample map key points in each group of sample image pairs are labeled onto the sample face key points in each group of sample image pairs to obtain the first map key point labels of the sample face key points in each group of sample image pairs;

[0167] In the disclosed embodiment, the first map key points may be annotated to the corresponding sample face key points, thereby facilitating model training.

[0168] In step S45, the sample face key points in each group of sample image pairs are input into a preset model, and a second map key point label is output;

[0169] In step S46, the preset model is trained according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

[0170] In the disclosed embodiment, a preset model can be trained based on a plurality of sample face key points to obtain a key point prediction model.

[0171] In the disclosed embodiment, key points in a sample face image set and a sample texture face image set can be extracted respectively, and then a preset model is trained based on these two key points to obtain the key point prediction model, so that the corresponding target texture face key points can be predicted based on the target face key points.

[0172] In step S23, a target mapping face image is generated according to the target mapping face key points.

[0173] In some embodiments, Figure 6 As shown, the method also includes:

[0174] In step S61, the preset facial features are used as current facial features; the current facial features are marked with the current image face label;

[0175] In the disclosed embodiment, preset facial features can be determined based on sampling points. The sampling points can be points generated by a random number generator, and one sampling point is a vector. A preset facial feature can be obtained based on one sampling point, and the preset facial feature can be labeled as a mapping face.

[0176] In step S62, the current facial features are input into the face mapping generation model to perform face mapping generation processing, and the current face mapping face is output;

[0177] In step S63, if the output current mapped face does not match the annotated current mapped face label, current key point information of the current mapped face is determined, and the current key point information is annotated with the output current mapped face label;

[0178] In some embodiments, the determining the current key point information of the current mapped face includes:

[0179] Predicting the current key point attribute information of the current mapped face according to the mapped face attribute predictor;

[0180] In the disclosed embodiments, key point attribute information may include but is not limited to gender, organ size, organ color, facial expression, and additional facial information; key point attribute information may be set according to actual needs; the mapped face attribute predictor may predict the attributes of the mapped face image to obtain key point attribute attributes.

[0181] Detecting the current key point identification information of the current mapped face according to the mapped face key point detector;

[0182] The current key point attribute information and the current key point identification information of the current mapped face are determined as the current key point information of the current mapped face.

[0183] In some embodiments, the method further comprises:

[0184] If the output current mapped face matches the annotated current mapped face label, the current facial feature is determined as the target facial feature.

[0185] In step S64, the current key point information is used as the current facial feature again, and the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing is repeated to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, the current key point information of the current sticker face is determined, and the current sticker face label outputted by annotating the current key point information is outputted; until the current sticker face label outputted by the sticker face generation model matches the annotated current sticker face label;

[0186] In step S65, the current key point information when the sticker face label output by the sticker face generation model matches the annotated current sticker face label is determined as the target facial feature.

[0187] In the disclosed embodiment, the model training process does not change the model parameters, but only changes the input sample features, and gradually selects the target sampling points (i.e., the target facial features). In the model training process, the preset facial features are first input to obtain a primary mapping face; the corresponding primary key point information is determined according to the primary mapping face; the primary key point information is matched with the primary mapping face; the primary key point information is then input into the model to obtain a secondary mapping face; if the secondary mapping face matches the primary mapping face, the primary key point is determined as the target facial feature; if the secondary mapping face does not match the primary mapping face, the secondary key point information is repeatedly determined according to the secondary mapping face, and the secondary key point information is input into the model to obtain a tertiary mapping face; it is continued to be determined whether the tertiary mapping face matches the secondary mapping face, and this cycle is repeated until the mapping face output by the model matches the labeled mapping face.

[0188] In the disclosed embodiment, the sample texture face image I and the corresponding sampling point Z, the sample texture face attribute A and the sample texture key point Lmk can be constructed into a sample data pair. sample_anime_i =(I sample_anime_i ,Z sample_anime_i ,A sample_anime_i ,Lmk sample_anime_i ).

[0189] In the disclosed embodiment, based on 100,000 paired data P_data sample_anime_i Perform model training (remember this model is Gen optim_anime Specifically, Z sample_anime_i As the starting point, find a point that meets (A sample_anime_i ,Lmk sample_anime_i ) target point Z optim_anime_i The SGF (Surrogate Gradient Field) algorithm can be used for training, which can be replaced by any attribute-guided image optimization algorithm.

[0190] In the disclosed embodiment, the target facial features can be determined by the model training method, and the accuracy of the generated target mapped face image can be improved by using the target facial features.

[0191] In some embodiments, generating a target textured face image according to the target textured face key points includes:

[0192] Filtering out target key points of the same type as the target facial features from the target map face key points;

[0193] The target key points are input into the textured face generation model to generate the target textured face image.

[0194] In the disclosed embodiment, after the target mapping face key points are determined, they can be further screened to obtain the target key points; and then a target mapping face image is generated based on the target key points, thereby improving the matching degree between the target mapping face image and the target face image.

[0195] In step S24, semantic segmentation is performed on the target face image to obtain a target head image region; the target head image region includes a face region and a hair region.

[0196] In some embodiments, the step of performing semantic segmentation processing on the target face image to obtain a target head portrait area includes:

[0197] Obtain a sample texture face image set with sample head image regions marked;

[0198] In the disclosed embodiment, a sample textured face image set may be obtained, and each sample textured face image in the sample textured face image set may be labeled to obtain a sample avatar region corresponding to each sample textured face image.

[0199] A sticker face semantic segmentation model is obtained by training the labeled sample sticker face image set;

[0200] In the disclosed embodiment, the sample mapped face image can be input into a preset model, and the model parameters can be adjusted until the avatar area output by the model is consistent with the labeled sample avatar area, and the preset model corresponding to the model parameters at this time is determined as the mapped face semantic segmentation model.

[0201] According to the semantic segmentation model of the mapped face, semantic segmentation processing is performed on the target mapped face image to obtain a target head portrait area.

[0202] In the disclosed embodiment, the target textured face image may include an avatar area and a non-avatar area. In practical applications, usually only the avatar area in the target textured face image needs to be used. The target textured face image may be input into a textured face semantic segmentation model to obtain a target avatar area, which includes a face area and a hair area.

[0203] In the disclosed embodiment, a sticker face semantic segmentation model can be obtained through training, and then semantic segmentation processing is performed on the target sticker face image according to the sticker face semantic segmentation model to quickly determine the target head portrait area, thereby improving the generation efficiency of the sticker face image.

[0204] In step S25, the target sticker face image is intercepted according to the target avatar area to obtain a target sticker face image corresponding to the target avatar area.

[0205] In some embodiments, the method further comprises:

[0206] Determine a foreground mask layer based on the target head portrait area;

[0207] In the embodiment of the present disclosure, the foreground mask layer can be determined according to the target head image area, that is, the foreground mask layer is obtained by intercepting the face and hair area, which is recorded as M target_anime .

[0208] The target textured face image is intercepted according to the foreground mask layer to obtain a target textured head portrait corresponding to the target head portrait area.

[0209] In the disclosed embodiment, the mask layer is used to map the face image result I optim_anime Cut out and get the texture avatar I including only the face and hair area masked_anime .

[0210] In the disclosed embodiment, the foreground mask layer can be determined by the target avatar area, and then the sticker avatar can be obtained based on the screenshot of the mask layer, so as to quickly generate the sticker avatar, thereby improving the generation efficiency of the sticker face image.

[0211] In step S26, the target sticker avatar is pasted into the target face image to generate a sticker image.

[0212] In some embodiments, pasting the target sticker avatar into the target face image to generate the sticker image includes:

[0213] Performing a reverse transformation process on the target sticker head portrait according to the reverse transformation parameters to obtain an image to be pasted;

[0214] The image to be pasted is pasted in the face avatar area of ​​the target face image to obtain the pasted image.

[0215] In the disclosed embodiment, the generated target textured avatar can be processed according to the inverse transformation parameters and then pasted into the face avatar area of ​​the target face image to obtain the textured image, thereby achieving natural and coordinated facial occlusion in the image.

[0216] In the disclosed embodiments, the PyTorch deep learning framework can be used for neural network training, and the Python language can be used for algorithm integration. For example, for a face image with a resolution of 512x512, it takes about 0.7 seconds to generate the corresponding texture image. PyTorch is an open source Python machine learning library for applications such as natural language processing. Python was designed by Guido van Rossum of the Netherlands Institute for Mathematical and Computer Science Research in the early 1990s as a replacement for a language called ABC. Python provides efficient high-level data structures and simple and effective object-oriented programming. Python syntax and dynamic typing, as well as the nature of an interpreted language, make it a programming language for writing scripts and rapidly developing applications on most platforms.

[0217] In a specific embodiment, Figure 7 As shown, Figure 7 The present invention provides an algorithm flow chart of an image generation method provided in the present embodiment, wherein the mapped face is a cartoon face, the input face image is a photo, and then face detection and extraction are performed, and the key points of the cartoon face are predicted based on the extracted face key points; then a cartoon image is generated based on the cartoon face key points; the cartoon image is segmented to obtain a cartoon avatar; then the cartoon avatar is subjected to a series of operations such as posture twisting according to image processing parameters, and then pasted on the face area of ​​the original face image, thereby generating a corresponding cartoon face image.

[0218] This embodiment provides a technology that can automatically occlude a real person's face using a sticker face. The sticker face is adjusted and segmented based on a given target posture to generate a sticker face result with a corresponding posture to the original portrait, ensuring that a variety of sticker face results that are consistent with the real person's posture, consistent in scale, and beautiful can be generated; while providing a rich selection space, it also protects the user's privacy well. Compared with the existing solutions, this technical solution has brought about significant improvements in the diversity and coordination of results. It can adaptively generate any number of sticker face results with diversity, support single-image multi-person occlusion, and provide users with a rich selection space.

[0219] The present invention performs feature extraction on a target face image to obtain target face key points; the target face key points are input into a key point prediction model to predict the target face key points to obtain target face key points; the target face corresponding to the target face key points has the same posture as the target face corresponding to the target face key points; a target face image is generated based on the target face key points; semantic segmentation is performed on the target face image to obtain a target avatar area; the target avatar area includes a face area and a hair area; the target face image is intercepted based on the target avatar area to obtain a target avatar corresponding to the target avatar area; the target avatar is pasted into the target face image to generate a map image. The present invention generates a map avatar consistent with the face posture in the image in real time based on the target face image, and pastes it into the face area in the target face image, thereby achieving natural and coordinated facial occlusion in the image and better protecting user privacy.

[0220] Figure 8 is a block diagram of an image generating device according to an exemplary embodiment. Figure 8 , the device comprises:

[0221] The target face key point determination module 810 is configured to perform feature extraction on the target face image to obtain the target face key points;

[0222] The target mapping face key point determination module 820 is configured to input the target face key points into the key point prediction model to predict the mapping face key points, and obtain the target mapping face key points; the target face corresponding to the target face key points has the same posture as the target mapping face corresponding to the target mapping face key points;

[0223] The target mapping face image generation module 830 is configured to generate a target mapping face image according to the target mapping face key points;

[0224] The target avatar region determination module 840 is configured to perform semantic segmentation processing on the target texture face image to obtain a target avatar region; the target avatar region includes a face region and a hair region;

[0225] The sticker avatar generation module 850 is configured to intercept the target sticker face image according to the target avatar area to obtain the sticker avatar corresponding to the target avatar area;

[0226] The image generation module 860 is configured to execute pasting the target sticker avatar into the target face image to generate a sticker image.

[0227] In some embodiments, the target face key point determination module includes:

[0228] A feature extraction unit is configured to perform feature extraction by inputting the target face image into a face key point detection network to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set;

[0229] An image processing unit is configured to perform a face extraction operation on the target face image using the key points in the initial face key point set as calibration points according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or padding;

[0230] The target face key point set determination unit is configured to convert the key points in the initial face key point set into the coordinate system constructed by the facial image to obtain the target face key point set.

[0231] In some embodiments, the image processing unit includes:

[0232] The eye center point determination subunit is configured to determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point;

[0233] A lip center point determination subunit is configured to determine the lip center point according to the lip contour point set;

[0234] The image processing subunit is configured to perform a face extraction operation on the target face image according to the face alignment algorithm using the eye center point and the lip center point as calibration points to obtain the face image.

[0235] In some embodiments, the apparatus further comprises:

[0236] The sample image pair construction module is configured to construct at least one set of sample image pairs according to the sample face image set and the sample texture face image set; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same;

[0237] A sample face key point detection module is configured to detect sample face key points of sample face images in each group of sample image pairs according to a face key point detector;

[0238] A sample map key point detection module is configured to detect sample map key points of a sample map face image in each set of sample image pairs according to a map face key point detector;

[0239] A map key point labeling module is configured to perform labeling of sample map key points in each group of sample image pairs onto sample face key points in each group of sample image pairs to obtain first map key point labels for sample face key points in each group of sample image pairs;

[0240] An output map key point label determination module is configured to input the sample face key points in each group of sample image pairs into a preset model and output a second map key point label;

[0241] The training module is configured to train the preset model according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

[0242] In some embodiments, the sample image pair construction module includes:

[0243] A model determination unit, configured to determine a human face generation model and a texture face generation model;

[0244] A sample face image set generating unit is configured to input a random face feature set into the face generation model to perform face image prediction to obtain the sample face image set;

[0245] A sample sticker face image set determination unit is configured to input the random facial feature set into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set;

[0246] The sample image pair construction unit is configured to execute the sample human face images and the sample textured face images corresponding to the same random facial features as a group of sample image pairs.

[0247] In some embodiments, the apparatus further comprises:

[0248] The current facial feature determination module is configured to execute the preset facial feature as the current facial feature; the current facial feature is marked with the current mapped face label;

[0249] A current sticker face determination module is configured to execute sticker face generation processing by inputting the current facial features into the sticker face generation model, and output the current sticker face;

[0250] The current sticker face labeling module is configured to determine the current key point information of the current sticker face if the output current sticker face does not match the labeled current sticker face label, and label the output current sticker face label for the current key point information;

[0251] The step repeating module is configured to execute the steps of reusing the current key point information as the current facial feature, and repeating the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, determining the current key point information of the current sticker face, and annotating the current key point information with the output current sticker face label; until the current sticker face label output by the sticker face generation model matches the annotated current sticker face label;

[0252] The target facial feature determination module is configured to determine the current key point information when the sticker face label output by the sticker face generation model matches the marked current sticker face label as the target facial feature.

[0253] In some embodiments, the target texture face image generation module includes:

[0254] A target key point screening unit is configured to screen target key points of the same type as the target facial features from the target mapped face key points;

[0255] The target textured face image generating unit is configured to input the target key points into the textured face generating model to generate the target textured face image.

[0256] In some embodiments, the apparatus further comprises:

[0257] An image processing parameter acquisition module is configured to execute and acquire image processing parameters corresponding to the face extraction operation;

[0258] an inverse transformation parameter determination module, configured to perform inverse transformation processing on the image processing parameters to obtain inverse transformation parameters;

[0259] The image generation module comprises:

[0260] A unit for determining an image to be pasted is configured to perform an inverse transformation process on the target image according to the inverse transformation parameters to obtain an image to be pasted;

[0261] The pasting image generating unit is configured to paste the image to be pasted in the face avatar area of ​​the target face image to obtain the pasting image.

[0262] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0263] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig. 9 is a block diagram of an electronic device according to an exemplary embodiment. Fig. 9 As shown, the server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the server 900. The server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or one or more operating systems 921, such as Windows ServerTM, Mac OSXTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0264] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0265] It can be understood by those skilled in the art that Fig. 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown.

[0266] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 930 including instructions, and the instructions can be executed by the processor 910 of the device 900 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0267] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the training method of the multi-service data prediction model or the multimedia information pushing method provided in any of the above embodiments.

[0268] In an exemplary embodiment, there is also provided an electronic device, comprising:

[0269] A processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the above-mentioned image generation method.

[0270] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the above image generation method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0271] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned image generating method is implemented.

[0272] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0273] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0274] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: include: Extract features from the target face image to obtain the key points of the target face; Inputting the target face key points into the key point prediction model to predict the key points of the mapped face, and obtaining the target mapped face key points; The target face corresponding to the target face key points has the same posture as the target texture face corresponding to the target texture face key points; Generating a target map face image according to the target map face key points; Performing semantic segmentation processing on the target face image to obtain a target head image region; the target head image region includes a face region and a hair region; The target face image is intercepted according to the target avatar area to obtain a target face image corresponding to the target avatar area; Paste the target sticker avatar into the target face image to generate a sticker image.

2. The method according to claim 1, characterized in that The feature extraction of the target face image to obtain the target face key points includes: Inputting the target face image into a face key point detection network for feature extraction to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set; Using the key points in the initial set of facial key points as calibration points, performing a face extraction operation on the target face image according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or filling; The key points in the initial facial key point set are converted into the coordinate system constructed by the facial image to obtain a target facial key point set.

3. The method according to claim 2, characterized in that The step of using the key points in the initial face key point set as calibration points and performing a face extraction operation on the target face image according to a face alignment algorithm to obtain a face image includes: Determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point; Determine the lip center point according to the lip contour point set; The eye center point and the lip center point are used as calibration points, and a face extraction operation is performed on the target face image according to the face alignment algorithm to obtain the face image.

4. The method according to claim 1, characterized in that: The training method of the key point prediction model includes: According to the sample face image set and the sample texture face image set, at least one set of sample image pairs is constructed; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same; Detecting sample face key points of the sample face images in each group of sample image pairs according to a face key point detector; Detecting sample texture key points of the sample texture face images in each set of sample image pairs according to a texture face key point detector; Labeling the sample map key points in each group of sample image pairs onto the sample face key points in each group of sample image pairs to obtain first map key point labels for the sample face key points in each group of sample image pairs; Input the sample face key points in each set of sample image pairs into a preset model and output the second map key point labels; The preset model is trained according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

5. The method according to claim 4, characterized in that The step of constructing at least one set of sample image pairs according to the sample face image set and the sample texture face image set comprises: Determine the human face generation model and the texture face generation model; Inputting a random facial feature set into the face generation model to perform face image prediction to obtain the sample face image set; Inputting the random facial feature set into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set; The sample face images and the sample textured face images corresponding to the same random facial features are taken as a set of sample image pairs.

6. The method according to claim 5, characterized in that The method further comprises: Using the preset facial features as the current facial features; the current facial features are marked with the current image face label; Inputting the current facial features into the sticker face generation model to perform sticker face generation processing, and outputting the current sticker face; When the output current mapped face does not match the annotated current mapped face label, determining current key point information of the current mapped face, and annotating the current key point information with the output current mapped face label; The current key point information is used as the current facial feature again, and the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing is repeated to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, the current key point information of the current sticker face is determined, and the current sticker face label outputted by annotating the current key point information is outputted; until the current sticker face label outputted by the sticker face generation model matches the annotated current sticker face label; The current key point information when the sticker face label output by the sticker face generation model matches the annotated current sticker face label is determined as the target facial feature.

7. The method according to claim 6, characterized in that The step of generating a target textured face image according to the target textured face key points comprises: Filtering out target key points of the same type as the target facial features from the target map face key points; The target key points are input into the textured face generation model to generate the target textured face image.

8. The method according to claim 2, characterized in that: After the key points in the initial facial key point set are used as calibration points and a face extraction operation is performed on the target face image according to a face alignment algorithm to obtain a face image, the method further includes: Obtaining image processing parameters corresponding to the face extraction operation; Performing inverse transformation processing on the image processing parameters to obtain inverse transformation parameters; Pasting the target sticker avatar into the target face image to generate a sticker image includes: Performing a reverse transformation process on the target sticker head portrait according to the reverse transformation parameters to obtain an image to be pasted; The image to be pasted is pasted in the face avatar area of ​​the target face image to obtain the pasted image.

9. An image generating device, characterized in that: include: The target face key point determination module is configured to perform feature extraction on the target face image to obtain the target face key points; A target facial key point determination module is configured to input the target facial key points into a key point prediction model to perform facial key point prediction and obtain target facial key points; The target face corresponding to the target face key points has the same posture as the target texture face corresponding to the target texture face key points; A target map face image generation module is configured to generate a target map face image according to the target map face key points; A target avatar region determination module is configured to perform semantic segmentation processing on the target mapped face image to obtain a target avatar region; the target avatar region includes a face region and a hair region; A sticker avatar generation module is configured to intercept the target sticker face image according to the target avatar area to obtain a sticker avatar corresponding to the target avatar area; The image generation module is configured to execute pasting the target sticker avatar into the target face image to generate a sticker image.

10. The device according to claim 9, characterized in that The target face key point determination module includes: A feature extraction unit is configured to perform feature extraction by inputting the target face image into a face key point detection network to obtain an initial face key point set; the initial face key point set includes an eye contour point set and a lip contour point set; An image processing unit is configured to perform a face extraction operation on the target face image using the key points in the initial face key point set as calibration points according to a face alignment algorithm to obtain a face image; the face extraction operation includes at least one of rotation, cropping or padding; The target face key point set determination unit is configured to convert the key points in the initial face key point set into the coordinate system constructed by the facial image to obtain the target face key point set.

11. The device according to claim 10, characterized in that The image processing unit comprises: The eye center point determination subunit is configured to determine the eye center point according to the eye contour point set; the eye center point includes the left eye center point and the right eye center point; A lip center point determination subunit is configured to determine the lip center point according to the lip contour point set; The image processing subunit is configured to perform a face extraction operation on the target face image according to the face alignment algorithm using the eye center point and the lip center point as calibration points to obtain the face image.

12. The device according to claim 9, characterized in that The device also includes: The sample image pair construction module is configured to construct at least one set of sample image pairs according to the sample face image set and the sample texture face image set; each set of sample image pairs includes a sample face image and a sample texture face image; the postures of the sample face image and the sample texture face image in each set of sample image pairs are the same; A sample face key point detection module is configured to detect sample face key points of sample face images in each group of sample image pairs according to a face key point detector; A sample map key point detection module is configured to detect sample map key points of a sample map face image in each set of sample image pairs according to a map face key point detector; A map key point labeling module is configured to perform labeling of sample map key points in each group of sample image pairs onto sample face key points in each group of sample image pairs to obtain first map key point labels for sample face key points in each group of sample image pairs; An output map key point label determination module is configured to input the sample face key points in each group of sample image pairs into a preset model and output a second map key point label; The training module is configured to train the preset model according to the difference between the first map key point label and the second map key point label to obtain the key point prediction model.

13. The device according to claim 12, characterized in that The sample image pair construction module comprises: A model determination unit, configured to determine a human face generation model and a texture face generation model; A sample face image set generating unit is configured to input a random face feature set into the face generation model to perform face image prediction to obtain the sample face image set; A sample sticker face image set determination unit is configured to input the random facial feature set into the sticker face generation model to perform sticker face image prediction to obtain the sample sticker face image set; The sample image pair construction unit is configured to execute the sample human face images and the sample textured face images corresponding to the same random facial features as a group of sample image pairs.

14. The device according to claim 13, characterized in that The device also includes: The current facial feature determination module is configured to execute the preset facial feature as the current facial feature; the current facial feature is marked with the current mapped face label; A current sticker face determination module is configured to execute sticker face generation processing by inputting the current facial features into the sticker face generation model, and output the current sticker face; The current sticker face labeling module is configured to determine the current key point information of the current sticker face if the output current sticker face does not match the labeled current sticker face label, and label the output current sticker face label for the current key point information; The step repeating module is configured to execute the steps of reusing the current key point information as the current facial feature, and repeating the step of inputting the current facial feature into the sticker face generation model to perform sticker face generation processing to obtain the current sticker face; and if the current sticker face does not match the annotated preset sticker face label, determining the current key point information of the current sticker face, and annotating the current key point information with the output current sticker face label; until the current sticker face label output by the sticker face generation model matches the annotated current sticker face label; The target facial feature determination module is configured to determine the current key point information when the sticker face label output by the sticker face generation model matches the marked current sticker face label as the target facial feature.

15. The device according to claim 14, characterized in that The target map face image generation module comprises: A target key point screening unit is configured to screen target key points of the same type as the target facial features from the target mapped face key points; The target textured face image generating unit is configured to input the target key points into the textured face generating model to generate the target textured face image.

16. The device according to claim 10, characterized in that The device also includes: An image processing parameter acquisition module is configured to execute and acquire image processing parameters corresponding to the face extraction operation; an inverse transformation parameter determination module, configured to perform inverse transformation processing on the image processing parameters to obtain inverse transformation parameters; The image generation module comprises: A unit for determining an image to be pasted is configured to perform an inverse transformation process on the target image according to the inverse transformation parameters to obtain an image to be pasted; The pasting image generating unit is configured to paste the image to be pasted in the face avatar area of ​​the target face image to obtain the pasting image.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image generating method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image generating method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 8 is implemented.

Images