Image processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method and device, a storage medium and a program product. According to the embodiment of the invention, the method comprises the steps: carrying out the segmentation of the five sense organ regions of a human face image, obtaining a target mask image containing the five target sense organs, combining a plurality of preset mask images containing the same five target sense organs and the types of the corresponding five target sense organs, and carrying out the classification of the five target sense organs according to the similarity between the target mask image and the plurality of preset mask images. And the category of the target five sense organs in the face image can be determined through the category of the target five sense organs corresponding to the preset mask image. Through the mode, the categories of the five sense organs in the face image can be determined based on the mask images corresponding to the five sense organs, the method does not depend on a deep learning model specially used for determining the categories of the five sense organs, and does not need to prepare and mark a large amount of sample data for model training, so that the recognition accuracy of the categories of the five sense organs is ensured. The overall time consumption and the implementation cost are reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and program product. Background Art

[0002] The facial features are diverse. For example, eyebrows can be divided into willow-shaped eyebrows, sword-shaped eyebrows, straight eyebrows, and triangular eyebrows. For example, eyes can be divided into peach blossom eyes, almond eyes, phoenix eyes, and round eyes. In scenarios such as face-based image processing, video processing, or virtual images, it is of great significance to perform more fine-grained recognition of facial features. For example, in an online makeup trial scenario, by identifying the type of facial features, the effect of cosmetics on the face can be better simulated. For another example, in a virtual image dress-up interactive game, by identifying the type of facial features, a virtual image that is more suitable for the user can be generated.

[0003] In the prior art, traditional computer vision algorithms can be used to extract facial features from facial images, such as eyebrow curvature, eyebrow flatness, eyebrow area, etc., and identify facial features based on the extracted feature information. However, traditional computer vision algorithms have always had the problem of low image recognition accuracy, so a solution for facial feature type recognition based on deep learning has emerged, that is, a sample data set is made and the sample data is annotated, and then a facial feature type recognition model is trained based on the sample data set and its annotated results, and then the facial feature type in the facial image is recognized based on the trained facial feature type recognition model.

[0004] Relatively speaking, the solution based on the facial feature type recognition model can improve the accuracy of the recognition results to a certain extent, but the production of the initial sample data set is time-consuming and the labeling work is also cumbersome, resulting in a high implementation cost of the solution. Summary of the invention

[0005] Multiple aspects of the present application provide an image processing method, device, storage medium and program product to reduce the overall time consumption and implementation cost while ensuring the accuracy of facial feature category recognition.

[0006] An embodiment of the present application provides an image processing method, including: acquiring a first facial image, the first facial image including target facial features; performing facial feature area segmentation processing on the first facial image to obtain a target mask image; determining the category of the target facial features included in the first facial image based on the similarity between the target mask image and multiple preset mask images, wherein different preset mask images include target facial features of different categories.

[0007] An embodiment of the present application also provides a virtual image generation method, comprising: in response to a virtual image generation operation, obtaining a facial image of a target user; determining target facial features in the facial image that the target user requests to pay attention to; determining the category of the target facial features in the facial image, and based on the category of the target facial features, generating a virtual image corresponding to the target user; wherein the category of the target facial features is determined according to the image processing method.

[0008] An embodiment of the present application also provides an online makeup trial method, comprising: responding to an online makeup trial trigger operation, collecting a facial image of a target user, the facial image including target facial features to be tried, and the target facial features corresponding to the makeup product to be tried; determining the category of the target facial features, and based on the category of the target facial features, generating a makeup trial effect image corresponding to the makeup product to be tried on the facial image; wherein the category of the target facial features is determined according to the image processing method.

[0009] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the image processing method.

[0010] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the image processing method.

[0011] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the steps in the image processing method are executed.

[0012] In this embodiment, by performing segmentation processing on the facial features of the face image, a target mask image containing the target facial features is obtained, and multiple preset mask images containing the same target facial features and their corresponding target facial features are combined. According to the similarity between the target mask image and the multiple preset mask images, the category of the target facial features in the face image can be determined by the category of the target facial features corresponding to the preset mask image. In this way, the category of the facial features in the face image can be determined based on the mask image corresponding to the facial features, without relying on a deep learning model specifically used for determining the category of facial features, nor does it need to prepare and annotate a large amount of sample data for model training, thereby reducing the overall time consumption and implementation cost while ensuring the accuracy of facial features category recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 A flowchart of an image processing method provided by an exemplary embodiment of the present application;

[0015] Figure 2 A schematic diagram of a mask image provided for an exemplary embodiment of the present application;

[0016] Figure 3a A schematic diagram of an inverted residual structure provided by an exemplary embodiment of the present application;

[0017] Figure 3b A flowchart of an image processing method in a practical scenario provided by an exemplary embodiment of the present application;

[0018] Figure 4 A schematic diagram of a flow chart of a method for generating a virtual image provided by an exemplary embodiment of the present application;

[0019] Figure 5 A schematic diagram of a flow chart of an online makeup trial method provided by an exemplary embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application;

[0021] Figure 7 A schematic diagram of the structure of a virtual image generating device provided by an exemplary embodiment of the present application;

[0022] Figure 8 A schematic diagram of the structure of an online makeup trial device provided by an exemplary embodiment of the present application;

[0023] Fig. 9 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0025] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0026] In face-related scenarios, traditional computer vision algorithms can be used to extract facial features from facial images, such as eyebrow curvature, eyebrow flatness, eyebrow area, etc., and identify facial features based on the extracted feature information. However, traditional computer vision algorithms have always had the problem of low image recognition accuracy, so there is a solution for facial feature type recognition based on deep learning, that is, to make a sample data set and annotate the sample data, and then train a facial feature type recognition model based on the sample data set and its annotation results, and then identify the facial feature type in the face image based on the trained facial feature type recognition model.

[0027] Relatively speaking, the solution based on the facial feature type recognition model can improve the accuracy of the recognition results to a certain extent, but the production of the initial sample data set is time-consuming and the labeling work is also cumbersome, resulting in a high implementation cost of the solution.

[0028] In view of the above technical problems, in an embodiment of the present application, a target mask image containing target facial features can be obtained by segmenting the facial features of a facial image, and a plurality of preset mask images containing the same target facial features and their corresponding target facial features categories can be combined. According to the similarity between the target mask image and the plurality of preset mask images, the category of the target facial features in the facial image can be determined by the category of the target facial features corresponding to the preset mask image. In this way, the category of the facial features in the facial image can be determined based on the mask image corresponding to the classified facial features, without relying on a deep learning model specifically used for determining the category of facial features, and without the need to prepare and annotate a large amount of sample data for model training, thereby reducing the overall time consumption and implementation cost while ensuring the accuracy of facial feature category recognition.

[0029] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0030] Figure 1 A flowchart of an image processing method provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include:

[0031] Step 11: Acquire a first facial image, where the first facial image includes target facial features.

[0032] Step 12: Segment the facial features of the first face image to obtain a target mask image.

[0033] Step 13: Determine the category of the target facial features included in the first facial image based on the similarity between the target mask image and a plurality of preset mask images, wherein different preset mask images include target facial features of different categories.

[0034] In this embodiment, the first facial image may be an image containing a real face, or an image containing a virtual face of a virtual character, and this embodiment does not impose any restrictions. The first facial image may include target facial features, wherein the target facial features may be one or more of eyebrows, glasses, ears, nose and mouth, and this embodiment does not impose any restrictions. The first facial image may be an image obtained by photographing the user's face, or it may be an image containing a virtual character's face generated based on a neural network model, and this embodiment does not impose any restrictions on the source of the first facial image. Afterwards, the first facial image may be segmented into facial feature areas to obtain a target mask image. Among them, the obtained target mask image refers to the mask image corresponding to the target facial features.

[0035] The embodiments of the present application do not limit the specific implementation method of the segmentation processing. In some exemplary embodiments, the facial feature areas can be segmented from the first face image based on a pre-trained deep learning model with image segmentation capabilities; the facial feature areas can also be segmented based on an image segmentation algorithm. The image segmentation algorithm can be used to identify and mark different parts in the first face image, such as facial feature areas such as eyes or nose.

[0036] Among them, the facial features region obtained by the segmentation process can be masked to obtain a target mask image. Masking is a masking process that can be used to select and highlight specific facial features regions in the first face image. In other words, the mask image corresponding to the target facial features to be classified can be used to highlight the target facial features. For example, Figure 2 A mask image is shown exemplarily, in which the facial features of a person can be highlighted, wherein the target facial features can be Figure 2 The eyebrows shown use different colors to distinguish the left eyebrow and the right eyebrow.

[0037] After obtaining the target mask image, the category of the target facial features included in the first face image can be determined according to the similarity between the target mask image and a plurality of preset mask images. Specifically, a plurality of preset mask images can be determined, and different preset mask images include target facial features of different categories. That is to say, the target facial features have been classified in advance in the preset mask image, and different preset mask images classify the target facial features into different categories. Taking the target facial features as eyebrows as an example, it is assumed that there are three preset mask images, namely, preset mask image M1, preset mask image M2 and preset mask image M3, and these three preset mask images all include eyebrows. The category of eyebrows in the preset mask image M1 is willow-leaf eyebrows, the category of eyebrows in the preset mask image M2 is sword eyebrows, and the category of eyebrows in the preset mask image M3 is straight eyebrows. After determining a plurality of preset mask images, the category of the target facial features included in the first face image can be determined according to the similarity between the target mask image and a plurality of preset mask images. Continuing with the above example, assuming that the similarities between the target mask image and the three preset mask images, namely, preset mask image M1, preset mask image M2 and preset mask image M3, are 10%, 60% and 95% respectively, then the category of straight eyebrows in the preset mask image M3 can be determined as the category of eyebrows included in the first face image.

[0038] In this way, the categories of the facial features in the face image can be determined based on the mask images corresponding to the facial features. It does not rely on a deep learning model specifically used to determine the categories of the facial features, nor does it require the preparation and labeling of a large amount of sample data for model training. This reduces the overall time and implementation cost while ensuring the accuracy of facial feature category recognition.

[0039] Considering that there may be a posture inconsistency between the first face image and the preset mask image, it may cause the target mask image obtained by the segmentation process to be unable to perform similarity calculation with the preset mask image in the same posture, thereby affecting the accuracy of the similarity calculation result. Therefore, when the first face image is segmented to obtain the target mask image, the first face image can be aligned from the first posture to the second posture of multiple preset mask images to obtain the second face image. Among them, the second face image refers to the face image obtained after posture adjustment. Afterwards, the second face image can be segmented to obtain the target mask image. In this way, since the first face image has been aligned to obtain the second face image, the processing result of the segmentation of the face area of ​​the second face image can be more accurate. In the embodiment of the present application, the second posture is not limited. Optionally, the second posture can be a normal posture, that is, a posture of looking straight ahead in the face image.

[0040] The embodiments of the present application do not limit the specific implementation method of posture alignment. In some exemplary embodiments, it can be implemented based on the following steps R1 to R3:

[0041] Step R1, performing face detection on the first face image to determine the face region in the first face image. Face detection refers to identifying the position and size of a face from a face image to determine the face region in the face image. The face detection process may include steps such as image preprocessing, face candidate region generation, feature extraction, and classification.

[0042] Specifically, the image preprocessing may include at least one of the following processing methods: ① Grayscale: converting the face image into a grayscale image to reduce the computational complexity. ② Scaling: adjusting the size of the face image to adapt it to subsequent processing. ③ Histogram equalization: enhancing the image contrast to make the face image clearer. ④ Noise removal: using a filter to remove noise in the image. The filter may be a Gaussian filter or a median filter, etc., which is not limited in this embodiment.

[0043] When generating candidate face regions, a sliding window or other method may be used to determine regions that may contain faces, i.e., determine candidate regions. Feature extraction refers to extracting useful features from candidate regions for further classification. Among them, the salient features of faces can be extracted based on deep learning algorithms. Among them, the deep learning algorithm can be a convolutional neural network (CNNs), a recurrent neural network (RNNs), or any other network, which is not limited in this embodiment. Among them, recurrent neural networks are a special type of neural network model that can be used to process sequence data. Among them, convolutional neural networks are a type of deep learning model that is suitable for processing data with a grid structure, such as images and videos. Classification refers to determining whether a candidate region is a face based on the extracted features, thereby determining a candidate region belonging to a face from the candidate region as a face region. Among them, the Adaboost classifier can be used for classification, and the pre-trained deep learning model can also be used for classification, which is not limited in this embodiment. Among them, Adaboost is an iterative algorithm that can train different weak classifiers for the same training set, and then combine these weak classifiers to form a stronger final classifier, that is, the Adaboost classifier mentioned above. Through the above steps, face detection can be performed on the first face image to more accurately determine the face area in the first face image.

[0044] Step R2: perform key point detection on the face area in the first face image to obtain the face key points and their position coordinates in the first posture.

[0045] Among them, the key point detection for the face area refers to further locating the positions of the key points of the face (such as eyes, nose, mouth, etc.) on the basis of detecting the face. The embodiments of the present application do not limit the specific implementation method of key point detection. In some exemplary embodiments, face key point detection can be performed based on the built-in regression model of Dlib (an open source C++ library). Among them, the goal of the regression model is to learn a function that can map from the input face image to the face key points and their position coordinates in the first posture; in other exemplary embodiments, the Hourglass Network (a deep learning architecture) can also be used to detect face key points. Hourglass Network is a deep learning architecture, which is particularly suitable for face key point detection tasks. It can achieve more accurate key point positioning through multi-level feature extraction and fusion, thereby outputting the face key points and their position coordinates in the first posture.

[0046] Step R3: Based on the position coordinates of the facial key points in the first posture, the facial region in the first facial image is adjusted to a second posture to obtain a second facial image.

[0047] Among them, the posture information of the second posture can be obtained, and the posture information may include a pitch angle, a yaw angle, and a roll angle, and a transformation matrix is ​​calculated according to the posture information of the second posture and the position coordinates of the key points of the face in the first posture. The transformation matrix can be an affine transformation matrix or a perspective transformation matrix, which is not limited in this embodiment. Afterwards, the transformation matrix can be used to adjust the face area in the first face image to the second posture to obtain the second face image.

[0048] After the posture of the face area in the first face image is adjusted relatively accurately through the above steps R1 to R3, the facial features area of ​​the second face image can be segmented in the following manner to obtain a target mask image:

[0049] Firstly, the second face image is input into the pixel segmentation model, and the feature map of the second face image is extracted by using the inverse residual structure and the depthwise separable convolutional layer. Secondly, the facial features in the second face image can be identified according to the feature map, and at least the facial features corresponding to the target facial features are masked to obtain the target mask image.

[0050] Specifically, the pixel segmentation model may include a feature extraction network, a facial features recognition network, and a mask generation network. The feature extraction network may use an inverted residual structure, and a depth-separable convolutional layer is used in the inverted residual structure for feature extraction. Based on the architecture of the pixel segmentation model, the above method can be implemented based on steps K1 to K3:

[0051] Step K1: input the second face image into the feature extraction network, and use the depthwise separable convolutional layer in the inverse residual structure to extract features of the second face image to obtain a feature map.

[0052] It should be noted that the residual block in the traditional residual structure usually contains one or more convolutional layers, and the output of these convolutional layers is directly added to the input to form the so-called "residual connection"; when performing feature extraction based on this residual connection, the dimension of the feature space can be reduced first, the corresponding feature extraction can be performed, and then the dimension of the feature space can be increased to extract the feature map; and the activation function used in the residual connection is usually the ReLU activation function, and the ReLU activation function only needs to determine whether the input is greater than zero.

[0053] Different from the residual structure, refer to Figure 3a As shown in the schematic diagram, the inverted residual structure of the feature extraction network may or may not use a jump connection, which is not limited in this embodiment. The inverted residual structure may include: an expansion layer (ExpansionLayer), a depthwise separable convolution layer (Depthwise Separable Convolution) and a contraction layer (ReductionLayer). When the depthwise separable convolution layer in the inverted residual structure is used to extract features of the second face image, the expansion layer may be used to first increase the dimension of the feature space of the second face image, the depthwise separable convolution layer may be used to perform corresponding feature extraction, and then the contraction layer may be used to reduce the dimension of the feature space to extract a feature map.

[0054] Specifically, the expansion layer can increase the number of channels of the feature map corresponding to the second face image through 1x1 convolution, thereby expanding the feature representation capability.

[0055] The depthwise separable convolution layer may include: depthwise convolution and pointwise convolution; when performing corresponding feature extraction through the depthwise separable convolution layer, depthwise convolution may be performed first, that is, each input channel is convolved through an independent convolution kernel to obtain an intermediate feature map, which can reduce the number of parameters and the amount of calculation while retaining local features; then pointwise convolution is performed, that is, multiple intermediate feature maps generated by the depthwise convolution are mixed through 1x1 convolution to obtain a mixed feature map, which can change the number of channels of the feature map to make it more suitable for subsequent processing.

[0056] The shrinkage layer can reduce the number of channels of the mixed feature map through 1x1 convolution, restoring it to a number of channels close to the input, which can further reduce the number of parameters of the feature extraction network and reduce the complexity of the feature extraction network, thereby more efficiently completing the feature extraction of the second face image and obtaining the corresponding feature map.

[0057] In addition, the activation functions used in the inverted residual connection are usually linear activation functions and ReLU6 activation functions. Among them, the ReLU6 activation function does not only need to determine whether the input is greater than zero like the ReLU activation function, but also imposes more restrictions on the output, which helps to introduce more nonlinearity in the feature extraction network and reduce the output precision loss. Among them, the linear activation function refers to the activation function whose output is linearly related to the input. Compared with the ReLU activation function, it can reduce the precision loss of the output feature map.

[0058] Compared with the residual structure, the inverted residual structure can increase the number of channels through the expansion layer, thereby extracting more feature information at a shallower level, allowing the feature extraction network to extract rich features more efficiently; through the combination of "expansion layer + depth-separable convolution layer + contraction layer", the number of parameters is significantly reduced while maintaining the performance of the feature extraction network, thereby achieving more efficient feature extraction at a lower cost. In addition, the activation function used by the inverted residual connection can effectively reduce the loss of output accuracy.

[0059] Step K2, input the feature map into the facial features recognition network, and generate a candidate box on the feature map based on the sliding window, and the candidate box corresponds to the facial features area in the second face image. This embodiment does not limit the size of the sliding window. In some exemplary embodiments, the size of the sliding window can be adjusted according to the size of the target facial features. Specifically, the facial features recognition network can control the sliding window to move on the feature map, and generate a series of rectangular boxes of different sizes and proportions at each position to cover possible targets. The size and proportion of the rectangular box can be pre-set or dynamically adjusted, and this embodiment does not limit this.

[0060] After that, each rectangular box can be sent to the classifier of the facial features recognition network, and the classifier can output a corresponding score for each rectangular box, which can be used to determine whether each rectangular box contains a facial feature area. Since the sliding window may generate overlapping candidate boxes, the NMS (Non-Maximum Suppression) technology can be used to remove duplicate detection results, and the rectangular boxes whose scores meet the preset score conditions are used as candidate boxes. Each candidate box corresponds to the facial feature area in the second face image.

[0061] Step K3: input the feature map with the candidate box into the mask generation model, and perform mask processing on at least the facial feature area corresponding to the target facial features in the second face image according to the position information of the candidate box in the feature map to obtain the target mask image.

[0062] Among them, according to the position information of the candidate box in the feature map, the sub-region corresponding to the candidate box can be cropped from the feature map; then a mask of the sub-region is generated, and the generated mask is applied to the facial feature region corresponding to the target facial features in the second face image to obtain the target mask image.

[0063] Through the above steps K1 to K3, the inverse residual structure and the depthwise separable convolutional layer used in the pixel segmentation model can be used to extract the feature map more efficiently and at a lower cost; based on the facial feature recognition network, the candidate boxes corresponding to the facial feature areas can be more accurately determined on the feature map; and based on the feature map with the candidate boxes, the mask generation model can be used to more efficiently perform mask processing on the facial feature areas in the second face image to obtain a more accurate target mask image.

[0064] Based on the above embodiments, the above step 13 "determining the category of the target facial features included in the first facial image according to the similarity between the target mask image and the plurality of preset mask images" can be implemented based on the following steps 131 to 133:

[0065] Step 131, calculating the similarity between the target facial features in the target mask image and the target facial features in a plurality of preset mask images. This step can be implemented in the following manner:

[0066] First, a reference point in the target facial features may be selected. The reference point may be any point in the target facial features, such as the upper left corner, upper right corner, lower left corner, lower right corner or other points, and this implementation does not impose any restrictions. For example, if the target facial features are eyebrows, the reference point of the eyebrows may be the left eye corner.

[0067] Secondly, the target mask image can be aligned with the multiple preset mask images according to the positions of the reference part points in the target mask image and the positions in the multiple preset mask images. Specifically, the target mask image can be translated according to the positions of the reference part points in the target mask image and the positions in the multiple preset mask images, so that the target mask image is aligned with the multiple preset mask images.

[0068] Afterwards, for each preset mask image, the ratio between the intersection area and the union area between the target facial features in the target mask image and the target facial features in the preset mask image can be used as the similarity. In other words, the similarity can be determined using the Intersection over Union (IoU) method.

[0069] Specifically, the intersection between the target facial features in the target mask image and the target facial features in the preset mask image, that is, the intersection area, can be calculated; then the union between the target facial features in the target mask image and the target facial features in the preset mask image, that is, the merged area, can be calculated; finally, the ratio between the intersection area and the merged area is calculated to obtain the similarity.

[0070] Step 132: According to the similarity, a preset mask image whose similarity meets a set condition is selected from multiple preset mask images as a reference mask image. The set condition can be set to any condition according to actual design requirements, and this embodiment does not limit it; preferably, the set condition can be the highest similarity.

[0071] Step 133: The category corresponding to the target facial features in the reference mask image is used as the category of the target facial features in the first face image. For example, if the category corresponding to the eyebrows in the reference mask image is straight eyebrows, then the category corresponding to the eyebrows in the first face image is straight eyebrows.

[0072] Based on the above steps 131 to 133 , the categories of the target facial features included in the first facial image can be determined more accurately according to the similarities between the target mask image and a plurality of preset mask images.

[0073] The embodiments of the present application do not limit the specific method for obtaining the preset mask image. In some optional embodiments, the target facial features can be classified in advance to obtain target facial features of different categories; and different preset mask images can be generated for target facial features of different categories; in other optional embodiments, the preset mask image can also be generated in response to the user's manual drawing operation, which is not limited in this embodiment.

[0074] The following will refer to Figure 3b , the above image processing method is further explained in combination with actual scenes:

[0075] exist Figure 3b In the figure, the image processing method is illustrated by taking eyebrows as an example. Figure 3b As shown, the user can input a first face image and perform face detection and alignment. Figure 3b The first face image shown is an image of a real person's face. It should be noted that for the protection of personal information, Figure 3bIn the first face image, some facial features are blurred. Figure 3b The face detection in this corresponds to the step of “performing face detection on the first face image” in the aforementioned embodiment; the face alignment here corresponds to the step of “aligning the first face image from the first posture to the second posture” in the aforementioned embodiment.

[0076] Afterwards, facial key point detection and eyebrow segmentation can be performed based on the aligned first facial image to obtain facial key point detection and facial eyebrow segmentation results. For details, please refer to the relevant records of "performing key point detection on the facial area in the first facial image to obtain facial key points and their position coordinates in the first posture" and "performing segmentation processing on the facial features area of ​​the second facial image to obtain a target mask image" in the aforementioned embodiment, which will not be repeated here.

[0077] When the deviation between the eyebrow position in the target mask image and the eyebrow position in the preset mask image is greater than the preset range, the target mask image and the preset mask image can be aligned in the following manner: select a reference point in the eyebrow, such as the left eye corner of the eyebrow; take the left eye corner of the eyebrow as the reference, and translate the target mask image according to the position of the left eye corner of the eyebrow in the target mask image and in multiple preset mask images, so that the target mask image is aligned with the positions of multiple preset mask images respectively. In this way, the eyebrow position in the target mask image can be made to coincide with the eyebrow position in the preset mask image, thereby completing the alignment. Figure 3b The process of aligning real eyebrows (ie, eyebrows in the target mask image) with the template (ie, the preset mask image) is shown.

[0078] like Figure 3b As shown, the preset mask image in the embodiment of the present application can be obtained by marking the eyebrow key point template. For details, please refer to the relevant records in the aforementioned embodiment of "preliminarily classifying the target facial features to obtain target facial features of different categories; generating different preset mask images for target facial features of different categories", which will not be repeated here.

[0079] Afterwards, the mask IoU (mask intersection over union) between the target mask image and the plurality of preset mask images, i.e., the similarity, can be calculated using the intersection over union (IoU) method. The specific implementation method has been described in detail in the aforementioned embodiment and will not be repeated here. On this basis, the preset mask image with the largest mask IoU can be used as a reference mask image, and the category corresponding to the classified eyebrows in the reference mask image can be used as the category of the eyebrows in the first face image.

[0080] In this way, the categories of the facial features in the face image can be determined based on the mask images corresponding to the facial features. It does not rely on a deep learning model specifically used to determine the categories of the facial features, nor does it require the preparation and labeling of a large amount of sample data for model training. This reduces the overall time and implementation cost while ensuring the accuracy of facial feature category recognition.

[0081] Based on the above embodiments, the image processing method provided in the embodiments of the present application can also be applied to more scenarios, which will be described in detail below.

[0082] like Figure 4 As shown, the embodiment of the present application also provides a method for generating a virtual image, which may include the following steps:

[0083] Step 41: In response to the virtual image generation operation, a facial image of the target user is obtained.

[0084] Step 42: Determine the target facial features in the facial image that the target user requests to focus on.

[0085] Step 43, determine the category of the target facial features in the facial image, and generate a virtual image corresponding to the target user based on the category of the target facial features; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiments.

[0086] In this embodiment, a digital human generation service interface may be provided to the target user, and the digital human generation service interface may allow the user to generate his or her own virtual image. Correspondingly, in response to the virtual image generation operation of the target user, a facial image acquisition request may be sent to the target user. After receiving the consent instruction issued by the target user, the target user may be photographed to obtain his or her facial image, or the facial image of the target user may be obtained from a local or cloud server, which is not limited in this embodiment. Afterwards, the target facial features in the facial image that the target user requests to focus on may be determined, and the target facial features may be one or more of eyebrows, glasses, ears, nose, and mouth.

[0087] The category of the target facial features is determined according to the methods described in the aforementioned embodiments, and based on the category of the target facial features, a virtual image corresponding to the target user is generated. Specifically, the corresponding modeling elements can be called according to the category of the target facial features, and based on the called modeling elements, a virtual image corresponding to the target user is generated. The virtual image can be a digital human, or an anime character, etc., which is not limited in this embodiment. A digital human, also known as a virtual human, a digital character or a digital avatar, is a virtual human image.

[0088] The modeling elements can be models or maps, etc. Take the target facial features including eyebrows, glasses, ears, nose and mouth as an example: according to the target user's phoenix eyes, the eye model corresponding to the phoenix eyes category can be called; according to the target user's crescent lips, the mouth model corresponding to the crescent lips can be called; according to the target user's straight eyebrows, the eyebrow model corresponding to the straight eyebrows category can be called; according to the target user's protruding ears, the ear model corresponding to the protruding ears category can be called; according to the target user's hooked nose, the nose model corresponding to the hooked nose category can be called; finally, according to the relative positions of the target facial features in the face image, the called models can be fused to obtain the virtual image corresponding to the target user.

[0089] In this way, the category of the target facial features can be determined more accurately based on the methods recorded in the aforementioned embodiments. Since the category of the target facial features has been determined, the corresponding modeling elements can be selected for the target facial features more accurately, and the generated virtual image is more accurate.

[0090] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0091] like Figure 5 As shown, the embodiment of the present application also provides an online makeup trial method, which may include the following steps:

[0092] Step 51: In response to the online makeup trial trigger operation, a facial image of the target user is collected, the facial image includes the target facial features to be tried, and the target facial features correspond to the makeup product to be tried. The target facial features can be any of the following: eyebrows, glasses, ears, nose and mouth. For example, the makeup product to be tried corresponding to the eyebrows can be an eyebrow pencil, and the makeup product to be tried corresponding to the mouth can be a lipstick.

[0093] Step 52, determine the category of the target facial features, and based on the category of the target facial features, generate a trial makeup effect image corresponding to the makeup product to be tried on the face image; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiments.

[0094] Specifically, a face image acquisition page may be provided to the target user, through which the target user may upload a face image of his / her own, or choose to use a camera to capture a face image in real time. After the face image is acquired, the target user may jump from the face image acquisition page to an online makeup trial page, which may display a variety of products and face images on the same screen. The target user may select a product to be tried from a variety of products.

[0095] Among them, when generating a trial makeup effect image corresponding to the makeup product to be tried on the face image, the corresponding drawing element can be called according to the category of the target facial features, and based on the called drawing element, the trial makeup effect image corresponding to the makeup product to be tried on the face image is generated. In this way, the category of the target facial features can be more accurately determined based on the methods recorded in the above embodiments. Since the category of the target facial features has been determined, the corresponding drawing elements can be selected for the target facial features more accurately, so that the trial makeup effect image can be generated more accurately.

[0096] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 13 can be device A; for another example, the execution subject of steps 11 and 12 can be device A, and the execution subject of step 13 can be device B; and so on.

[0097] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 12, 13, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0098] Figure 6 FIG. 1 is a structural diagram of an image processing device provided by another exemplary embodiment of the present application. Figure 6 As shown, the device comprises:

[0099] An acquisition module 601 is used to acquire a first facial image, wherein the first facial image includes target facial features; a segmentation module 602 is used to segment the facial feature regions of the first facial image to obtain a target mask image; a similarity module 603 is used to determine the category of the target facial features included in the first facial image based on the similarity between the target mask image and a plurality of preset mask images, wherein different preset mask images include target facial features of different categories.

[0100] Optionally, the first facial image has a first posture, and the multiple preset mask images have a second posture; when the segmentation module 602 performs segmentation processing on the facial features area of ​​the first facial image to obtain a target mask image, it is specifically used to: align the first facial image from the first posture to the second posture to obtain a second facial image; and perform segmentation processing on the facial features area of ​​the second facial image to obtain the target mask image.

[0101] Optionally, when the segmentation module 602 aligns the first facial image from the first posture to the second posture to obtain the second facial image, it is specifically used to: perform face detection on the first facial image to determine the facial area in the first facial image; perform key point detection on the facial area in the first facial image to obtain facial key points and their position coordinates in the first posture; and adjust the facial area in the first facial image to the second posture based on the position coordinates of the facial key points in the first posture to obtain the second facial image.

[0102] Optionally, when the segmentation module 602 performs segmentation processing on the facial feature areas of the second facial image to obtain the target mask image, it is specifically used to: input the second facial image into a pixel segmentation model, and use an inverse residual structure and a depth-separable convolutional layer to extract a feature map of the second facial image; identify the facial feature areas in the second facial image according to the feature map, and at least mask the facial feature areas corresponding to the target facial features to obtain the target mask image.

[0103] Optionally, the pixel segmentation model includes a feature extraction network, a facial feature recognition network and a mask generation network, the feature extraction network adopts an inverted residual structure, and a depth-separable convolutional layer is used in the inverted residual structure to extract features; the segmentation module 602 inputs the second face image into the pixel segmentation model, and uses the inverted residual structure and the depth-separable convolutional layer to extract a feature map of the second face image; the facial feature areas in the second face image are identified according to the feature map, and at least the facial feature areas corresponding to the target facial features are masked to obtain the target mask image, which is specifically used to: The facial image is input into the feature extraction network, and the depthwise separable convolutional layer in the inverse residual structure is used to extract features of the second facial image to obtain the feature map; the feature map is input into the facial feature recognition network, and a candidate box is generated on the feature map based on a sliding window, and the candidate box corresponds to the facial feature area in the second facial image; the feature map with the candidate box is input into the mask generation model, and according to the position information of the candidate box in the feature map, at least the facial feature area corresponding to the target facial feature in the second facial image is masked to obtain the target mask image.

[0104] Optionally, when the similarity module 603 determines the category of the target facial features included in the first facial image based on the similarity between the target mask image and a plurality of preset mask images, it is specifically used to: calculate the similarity between the target facial features in the target mask image and the target facial features in the plurality of preset mask images; based on the similarity, select a preset mask image whose similarity meets a set condition from the plurality of preset mask images as a reference mask image; and use the category corresponding to the target facial features in the reference mask image as the category of the target facial features in the first facial image.

[0105] Optionally, when the similarity module 603 calculates the similarity between the target facial features in the target mask image and the target facial features in the multiple preset mask images, it is specifically used to: select reference part points in the target facial features; align the target mask image with the multiple preset mask images according to the positions of the reference part points in the target mask image and the positions in the multiple preset mask images; for each preset mask image, use the ratio of the intersecting area and the merged area between the target facial features in the target mask image and the target facial features in the preset mask image as the similarity.

[0106] Optionally, when the similarity module 603 aligns the target mask image with the multiple preset mask images respectively according to the positions of the reference part points in the target mask image and the multiple preset mask images, it is specifically used to: with the reference part points in the target mask image respectively overlapping with the reference part points in the multiple preset mask images, translate the target mask image according to the positions of the reference part points in the target mask image and the multiple preset mask images, so that the target mask image is aligned with the multiple preset mask images respectively.

[0107] Optionally, the acquisition module 601 is further used to: pre-classify the target facial features to obtain target facial features of different categories; and generate different preset mask images for target facial features of different categories.

[0108] In this embodiment, a target mask image containing target facial features can be obtained by segmenting the facial features of a face image, and multiple preset mask images containing the same target facial features and their corresponding target facial features can be combined. According to the similarity between the target mask image and the multiple preset mask images, the category of the target facial features in the face image can be determined by the category of the target facial features corresponding to the preset mask image. In this way, the category of the facial features in the face image can be determined based on the mask image corresponding to the facial features, without relying on a deep learning model specifically used for determining the category of facial features, and without the need to prepare and annotate a large amount of sample data for model training, thereby reducing the overall time consumption and implementation cost while ensuring the accuracy of facial feature category recognition.

[0109] Figure 7 FIG. 1 is a schematic diagram of a virtual image generation device provided by another exemplary embodiment of the present application. Figure 7 As shown, the device comprises:

[0110] The acquisition module 701 is used to: respond to the virtual image generation operation and obtain the facial image of the target user; the facial features determination module 702 is used to: determine the target facial features in the facial image that the target user requests to pay attention to; the category determination module 703 is used to: determine the category of the target facial features in the facial image, and based on the category of the target facial features, generate a virtual image corresponding to the target user; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiment.

[0111] Figure 8 FIG. 1 is a schematic diagram of the structure of an online makeup trial device provided by another exemplary embodiment of the present application. Figure 8 As shown, the device comprises:

[0112] The acquisition module 801 is used to: respond to an online makeup trial trigger operation, collect a facial image of a target user, the facial image including target facial features to be tried, and the target facial features correspond to the makeup product to be tried; the determination module 802 is used to: determine the category of the target facial features, and based on the category of the target facial features, generate a makeup trial effect image corresponding to the makeup product to be tried on the facial image; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiments.

[0113] The above describes the internal functions and structures of each device, such as Fig. 9 As shown, in practice, each of the above devices can be implemented as an electronic device, including: a memory 901 , a processor 902 , and a communication component 903 .

[0114] The memory 901 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0115] Memory 901 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0116] In some embodiments, the processor 902 is coupled to the memory 901 and is used to execute a computer program in the memory 901 to: obtain a first facial image, the first facial image including target facial features; perform facial feature area segmentation processing on the first facial image to obtain a target mask image; determine the category of the target facial features included in the first facial image based on the similarity between the target mask image and multiple preset mask images, wherein different preset mask images include target facial features of different categories.

[0117] Optionally, the first facial image has a first posture, and the multiple preset mask images have a second posture; when the processor 902 performs segmentation processing on the facial features area of ​​the first facial image to obtain a target mask image, it is specifically used to: align the first facial image from the first posture to the second posture to obtain a second facial image; perform segmentation processing on the facial features area of ​​the second facial image to obtain the target mask image.

[0118] Optionally, when the processor 902 aligns the first facial image from the first posture to the second posture to obtain the second facial image, it is specifically used to: perform face detection on the first facial image to determine the facial area in the first facial image; perform key point detection on the facial area in the first facial image to obtain facial key points and their position coordinates in the first posture; and adjust the facial area in the first facial image to the second posture based on the position coordinates of the facial key points in the first posture to obtain the second facial image.

[0119] Optionally, when the processor 902 performs segmentation processing on the facial feature areas of the second facial image to obtain the target mask image, it is specifically used to: input the second facial image into a pixel segmentation model, and use an inverse residual structure and a depth-separable convolutional layer to extract a feature map of the second facial image; identify the facial feature areas in the second facial image according to the feature map, and at least mask the facial feature areas corresponding to the target facial features to obtain the target mask image.

[0120] Optionally, the pixel segmentation model includes a feature extraction network, a facial feature recognition network and a mask generation network, the feature extraction network adopts an inverted residual structure, and a depth-separable convolutional layer is used in the inverted residual structure to perform feature extraction; the processor 902 inputs the second face image into the pixel segmentation model, and uses the inverted residual structure and the depth-separable convolutional layer to extract a feature map of the second face image; the facial feature area in the second face image is identified according to the feature map, and at least the facial feature area corresponding to the target facial feature is masked to obtain the target mask image, which is specifically used to: The facial image is input into the feature extraction network, and the depthwise separable convolutional layer in the inverse residual structure is used to extract features of the second facial image to obtain the feature map; the feature map is input into the facial feature recognition network, and a candidate box is generated on the feature map based on a sliding window, and the candidate box corresponds to the facial feature area in the second facial image; the feature map with the candidate box is input into the mask generation model, and according to the position information of the candidate box in the feature map, at least the facial feature area corresponding to the target facial feature in the second facial image is masked to obtain the target mask image.

[0121] Optionally, when the processor 902 determines the category of the target facial features included in the first facial image based on the similarity between the target mask image and a plurality of preset mask images, it is specifically used to: calculate the similarity between the target facial features in the target mask image and the target facial features in the plurality of preset mask images; based on the similarity, select a preset mask image whose similarity meets a set condition from the plurality of preset mask images as a reference mask image; and use the category corresponding to the target facial features in the reference mask image as the category of the target facial features in the first facial image.

[0122] Optionally, when the processor 902 calculates the similarity between the target facial features in the target mask image and the target facial features in the multiple preset mask images, it is specifically used to: select reference part points in the target facial features; align the target mask image with the multiple preset mask images according to the positions of the reference part points in the target mask image and the positions in the multiple preset mask images; for each preset mask image, use the ratio of the intersecting area and the merged area between the target facial features in the target mask image and the target facial features in the preset mask image as the similarity.

[0123] Optionally, when the processor 902 aligns the target mask image with the multiple preset mask images respectively according to the positions of the reference part points in the target mask image and the positions of the multiple preset mask images, it is specifically used to: with the reference part points in the target mask image and the reference part points in the multiple preset mask images respectively overlapping as the goal, translate the target mask image according to the positions of the reference part points in the target mask image and the positions of the multiple preset mask images, so that the target mask image is aligned with the multiple preset mask images respectively.

[0124] Optionally, the processor 902 is further used to: pre-classify the target facial features to obtain target facial features of different categories; and generate different preset mask images for target facial features of different categories.

[0125] In some other embodiments, the processor 902 is coupled to the memory 901 and is used to execute the computer program in the memory 901 to: in response to the avatar generation operation, obtain a facial image of a target user to determine target facial features in the facial image that the target user requests to focus on;

[0126] Determine the category of the target facial features in the facial image, and based on the category of the target facial features, generate a virtual image corresponding to the target user; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiments.

[0127] In other embodiments, the processor 902 is coupled to the memory 901, and is used to execute the computer program in the memory 901, so as to: respond to an online makeup trial trigger operation, collect a facial image of a target user, wherein the facial image includes target facial features of the makeup to be tried, and the target facial features correspond to the makeup product to be tried; determine the category of the target facial features, and based on the category of the target facial features, generate a makeup trial effect picture corresponding to the makeup product to be tried on the facial image; wherein the category of the target facial features is determined according to the method described in the aforementioned embodiments.

[0128] Further, if Fig. 9 As shown, the electronic device also includes: a display 904, a power component 905, an audio component 906 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 In addition, Fig. 9 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the working node. The working node of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server-side device such as a conventional server, a cloud server, or a server array. If the working node of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Fig. 9 If the working node of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Fig. 9 Components within the dashed box.

[0129] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0130] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, executes the steps in the online makeup trial method, the virtual image generation method or the image processing method.

[0131] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0132] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0133] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0134] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0135] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first facial image, wherein the first facial image includes target facial features; Performing facial feature area segmentation processing on the first face image to obtain a target mask image; According to the similarity between the target mask image and a plurality of preset mask images, the category of the target facial features included in the first facial image is determined, wherein different preset mask images include target facial features of different categories.

2. The method according to claim 1, characterized in that The first face image has a first posture, and the plurality of preset mask images have a second posture; Segmenting the facial features of the first face image to obtain a target mask image includes: aligning the first facial image from the first posture to the second posture to obtain a second facial image; The second facial image is segmented into facial features to obtain the target mask image.

3. The method according to claim 2, characterized in that Aligning the first facial image from the first posture to the second posture to obtain a second facial image includes: Performing face detection on the first face image to determine a face area in the first face image; Performing key point detection on the face area in the first face image to obtain the face key points and their position coordinates in the first posture; Based on the position coordinates of the facial key points in the first posture, the facial region in the first facial image is adjusted to the second posture to obtain the second facial image.

4. The method according to claim 2, characterized in that: Segmenting the facial features of the second face image to obtain the target mask image includes: The second facial image is input into a pixel segmentation model, and a feature map of the second facial image is extracted using an inverse residual structure and a depthwise separable convolutional layer; facial feature areas in the second facial image are identified based on the feature map, and at least facial feature areas corresponding to the target facial features are masked to obtain the target mask image.

5. The method according to claim 4, characterized in that The pixel segmentation model includes a feature extraction network, a facial features recognition network and a mask generation network. The feature extraction network adopts an inverted residual structure, and a depth-separable convolutional layer is used in the inverted residual structure to extract features. Inputting the second face image into a pixel segmentation model, extracting a feature map of the second face image using an inverse residual structure and a depthwise separable convolutional layer; identifying facial features in the second face image according to the feature map, and performing masking on at least the facial features corresponding to the target facial features to obtain the target mask image, including: Inputting the second face image into the feature extraction network, and performing feature extraction on the second face image using the depthwise separable convolutional layer in the inverse residual structure to obtain the feature map; Inputting the feature map into the facial feature recognition network, generating a candidate box on the feature map based on a sliding window, wherein the candidate box corresponds to a facial feature region in the second facial image; The feature map with the candidate box is input into the mask generation model, and according to the position information of the candidate box in the feature map, at least the facial feature area corresponding to the target facial features in the second facial image is masked to obtain the target mask image.

6. The method according to any one of claims 1 to 5, characterized in that: Determining the category of the target facial features included in the first facial image according to the similarity between the target mask image and a plurality of preset mask images, comprises: Calculating the similarity between the target facial features in the target mask image and the target facial features in the plurality of preset mask images; According to the similarity, selecting a preset mask image whose similarity meets a set condition from the plurality of preset mask images as a reference mask image; The category corresponding to the target facial features in the reference mask image is used as the category of the target facial features in the first facial image.

7. The method according to claim 6, characterized in that Calculating the similarity between the target facial features in the target mask image and the target facial features in the plurality of preset mask images, comprising: Selecting reference points in the target facial features; According to the positions of the reference part points in the target mask image and in the multiple preset mask images, aligning the target mask image with the multiple preset mask images respectively; For each preset mask image, the ratio between the intersection area and the merged area between the target facial features in the target mask image and the target facial features in the preset mask image is used as the similarity.

8. The method according to claim 7, characterized in that According to the position of the reference part point in the target mask image and the position of the reference part point in the multiple preset mask images, the target mask image is respectively aligned with the multiple preset mask images, including: With the goal of making the reference part points in the target mask image coincide with the reference part points in the multiple preset mask images respectively, the target mask image is translated according to the positions of the reference part points in the target mask image and in the multiple preset mask images so that the target mask image is aligned with the positions of the multiple preset mask images respectively.

9. The method according to any one of claims 1 to 5, characterized in that: Also includes: Classify the target facial features in advance to obtain different categories of target facial features; Different preset mask images are generated for different categories of target facial features.

10. A method for generating a virtual image, characterized in that: include: In response to the avatar generation operation, a face image of a target user is obtained; Determining target facial features in the face image that the target user requests to focus on; Determining the category of the target facial features in the facial image, and generating a virtual image corresponding to the target user based on the category of the target facial features; Wherein, the category of the target facial features is determined according to the method described in any one of claims 1-9.

11. An online makeup trial method, characterized in that: include: In response to an online makeup trial trigger operation, a facial image of a target user is collected, the facial image including target facial features to be tried on, the target facial features corresponding to the makeup product to be tried on; Determine the category of the target facial features, and based on the category of the target facial features, generate a trial makeup effect image corresponding to the makeup product to be tried on the face image; Wherein, the category of the target facial features is determined according to the method described in any one of claims 1-9.

12. An electronic device, characterized in that: include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method described in any one of claims 1-9, claim 10 or claim 11.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 9, claim 10 or claim 11.

14. A computer program product, characterized in that The method comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps in the method of any one of claims 1 to 9, claim 10 or claim 11 are performed.

Images