Method for intelligent expression management in later period of portrait photography

Abstract

The invention discloses a method for intelligent expression management in the later period of portrait photography. The method comprises the following steps: performing mask processing on a face region in a target image to obtain a face mask; performing recognition processing on a face region in the target image to obtain an expression label; and according to the target image, the face mask, the expression label and an expected expression label, performing first form adjustment on the face area in the target image, and generating a first face image paired with the expected expression label. The method aims at automatically detecting and correcting the smiling problem in the portrait image, and particularly overcomes the defects that in the current smiling optimization technology, the naturalness is poor, the individuation is insufficient, the processing effect is not coordinated, and the definition is insufficient. The processed image not only improves a smile visual effect, but also avoids common over-modification and unnatural phenomena in traditional manual editing, so that a more natural, real and affinity portrait photo is provided for a user.

Description

Technical Field

[0001] The present invention relates to the technical field of portrait photography post-production, and in particular to a method for intelligent expression management in portrait photography post-production. Background Art

[0002] In portrait image editing, the optimization and beautification of smiles is one of the key factors in improving photo quality. As an important element in expressing emotions and affinity, smiles directly affect the transmission and visual effects of character images. With the rise of social platforms, e-commerce, and online education, more and more users want to show their best smiles through portrait photos, especially in scenes such as portrait photography and personal portraits. The naturalness and harmonious beauty of smiles have become the focus of users' pursuit. However, in reality, due to the influence of shooting angles, lighting, facial muscle control and other factors, smiles often appear unnatural or asymmetrical when photographed, resulting in unsatisfactory overall portrait effects.

[0003] In traditional image post-processing, smile optimization usually relies on manual operations, and editors need to manually adjust the corners of the mouth, eyes, facial expressions and other areas to make detailed corrections. This process is not only cumbersome and time-consuming, but also due to the lack of intelligent support, the adjusted smile often has the risk of being unnatural or over-modified, and cannot fully show the individual's emotional characteristics and natural smile.

[0004] Therefore, an automated intelligent processing method is needed that can intelligently judge the naturalness and expression of a smile based on facial features, and automatically adjust and optimize the smile to make it more in line with individual characteristics and scene requirements. This can not only improve the overall aesthetics of the photo, but also avoid the stiffness caused by manual adjustments, and enhance the affinity and natural expression of the characters. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a method for intelligent expression management in the post-production of portrait photography. The method aims to automatically detect and correct the smile problem in the portrait image, especially to solve the defects of poor naturalness, insufficient personalization, inconsistent processing effect, insufficient clarity, etc. in the current smile optimization technology. Through intelligent algorithms, the type, naturalness and emotional expression of the smile in the portrait are automatically analyzed, and the smile is appropriately adjusted to make it more harmonious, natural and in line with the personalized needs of the user. The processed image not only improves the visual effect of the smile, but also avoids the over-modification and unnatural phenomena common in traditional manual editing, thereby providing users with more natural, real and friendly portrait photos.

[0006] According to one aspect of the present invention, a method for intelligent expression management in post-portrait photography is provided, the method comprising:

[0007] Mask the face area in the target image to obtain a face mask;

[0008] Recognize the facial area in the target image and obtain the expression label;

[0009] According to the target image, the face mask, the expression label, and the expected expression label, a first morphological adjustment is performed on the face area in the target image to generate a first face image paired with the expected expression label.

[0010] In the above technical scheme, this research scheme aims to achieve accurate replacement of facial expressions in the target image through advanced image processing technology. First, this scheme performs mask processing on the facial area in the target image to generate a high-resolution facial mask containing key parts of the face (such as eyes, mouth, etc.). This process can effectively identify and segment the facial area in the image, providing accurate facial feature positioning for subsequent expression adjustment. Furthermore, this scheme introduces facial expression recognition technology, which is based on machine learning algorithms and can analyze facial expressions in static images or dynamic videos and accurately identify the specific expression state of individuals, such as happiness, sadness, surprise, etc. Through this step, the expression label of the face in the target image can be obtained, providing necessary reference information for subsequent expression replacement. Based on the target image, the generated face mask and the identified expression label, this scheme uses conditional generative adversarial networks (cGANs) to perform morphological adjustment on the facial area in the target image. cGANs technology can generate a new face image that matches the expected expression under the condition of a given face mask and the expected expression label, thereby realizing the accurate replacement of the facial expression in the target image. Finally, through the above-mentioned morphological adjustment process, this scheme can generate a first face image that matches the expected expression label. This method has broad application prospects in the fields of face synthesis, portrait editing, face swapping, etc. Through the processing of this scheme, a new image that is similar in appearance to the face in the original image but has a different expression can be generated, which effectively solves the problems of poor naturalness, insufficient personalization, inconsistent processing effects, and insufficient clarity in the current smile optimization technology. This scheme automatically analyzes the type, naturalness and emotional expression of the smile in the portrait through an intelligent algorithm, and makes appropriate adjustments to achieve a more harmonious, natural and personalized smile effect that meets the personalized needs of users. The processed image not only improves the visual effect of the smile, but also avoids the over-modification and unnatural phenomena commonly seen in traditional manual editing, providing users with more natural, real and friendly portrait photos.

[0011] In some embodiments, performing a first morphological adjustment on a facial region in the target image according to the target image, the face mask, the expression label, and the expected expression label to generate a first facial image paired with the expected expression label includes:

[0012] Constructing a first expression data set, the first data set includes a plurality of expression pairing data pairs; the expression pairing data pairs include: a first expression label image and a second expression label image constitute a data pair; the second expression label is an expected expression type;

[0013] Training a first neural network based on the constructed first expression data set;

[0014] The face mask, the expression label, and the expected expression label are input into the pre-trained first neural network to generate a first face image paired with the target expression label.

[0015] In the above technical solution, a neural network is trained by constructing a data set containing expression pairs to achieve the conversion of facial expressions. The constructed data set consists of pairs of images, each pair of images containing an original expression image (first expression label image) and a target expression image (second expression label image). These paired images will be used as training samples to train the neural network to recognize the differences between expressions and learn how to map one expression to another. Based on the construction of the data set, a neural network model is further trained, which can recognize the expression in the input image and generate the corresponding output image according to the given target expression label. During the training process, the model will learn how to use the input face mask and expression label information, as well as the target expression label, to adjust the expression of the face image. After the training is completed, the input face mask, expression label and target expression label are input into the trained neural network model. The model will generate an output face image that matches the target expression label based on these input information. In the above scheme, a conditional generative adversarial network (CGANs) can be used as the structure of the first neural network model, which can generate data with specific attributes according to conditional variables, the corresponding conditional variables are expression labels, and the generated data are face images.

[0016] In some embodiments, training the first neural network based on the constructed plurality of expression pairing data pairs includes:

[0017] Input the constructed data pair into the first neural network, and input the pre-trained expression classifier based on the image generated by the first neural network;

[0018] Calculate the difference between the generated image and the second expression label image based on the classifier, and use the difference as the first loss value;

[0019] The first neural network is trained based on the first loss value.

[0020] In the above technical solution, first, the constructed expression pairing data pair is input into the first neural network. The task of this neural network is to generate images based on the input data pair, and these images should be able to reflect the transition from the first expression label to the second expression label (i.e., the expected expression type). The generated images are then input into a pre-trained expression classifier. The task of this classifier is to identify the expression label of the generated image to verify whether the image generated by the neural network matches the expected expression label. Based on the output of the pre-trained classifier, the difference value between the generated image and the second expression label image (expected expression type) is calculated. This difference value reflects the deviation between the generated image and the target expression and can be used as the first loss value for neural network training. This first loss value is used to adjust and optimize the parameters of the first neural network. By minimizing this loss value, the neural network can learn how to more accurately generate images that match the expected expression label. The above solution achieves the purpose of generating specific expression images by constructing expression pairing data pairs, calculating loss values ​​using pre-trained classifiers, and training and optimizing neural networks based on these loss values. This process can be iterated continuously until the recognition result of the generated image in the pre-trained classifier matches the expected expression label, thereby achieving accurate conversion of expressions. .

[0021] In some embodiments, according to the target image, the face mask, the expression label, and the expected expression label, a first morphological adjustment is performed on the face region in the target image to generate a first face image paired with the expected expression label, and then the method further includes:

[0022] According to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image.

[0023] In the above technical solution, the image generated in the first stage has achieved partial conversion to the target expression. In order to further refine this process, the target expression image, the face mask, the original expression label and the first face image are used as inputs, aiming to make more refined morphological adjustments to the facial area in the first face image. In this stage, the goal is to optimize the first face image so that it can match the expected expression more accurately while maintaining its naturalness. Specifically, the adjustment of the facial area of ​​the first face image aims to enhance the expression features in the image, making it closer to the target expression, while retaining other significant features of the original face image. After the adjustment in the second stage, the resulting images have significantly improved the accuracy of the expression and more realistically reflect the characteristics of the expected expression. In addition, these images also show high fidelity in maintaining other features of the original face image. At the same time, it can effectively improve the performance of the expression image generation task and achieve a smooth transition from the original expression to the target expression.

[0024] In some embodiments, based on the target image, the face mask, the expression label, and the first face image, performing a second morphological adjustment on the face region in the first face image to generate a second face image corresponding to the first face image includes:

[0025] Based on the first expression data set, data enhancement is performed to construct a second data set, the second data set includes a plurality of expression pairing data sequences; the plurality of data sequences include: a first expression label image, a plurality of images transitioning from the first expression to the second expression, and a second expression label image;

[0026] Training a second neural network based on the constructed second data set;

[0027] The target image, the face mask, the expression label, and the first face image are input into the pre-trained second neural network to generate a second face image corresponding to the first face image.

[0028] In the above technical solution, in the field of machine learning and computer vision, data enhancement technology has been widely recognized as an effective strategy. It can expand the training data set without increasing the actual data collection cost, thereby improving the generalization ability of the model. In this case, an expression pairing data sequence is constructed, which contains transition images from the first expression to the second expression. The design of this data sequence helps the model capture the continuity of expression changes, thereby generating a more natural expression transition effect. Based on the constructed second data set, the neural network is trained to enable it to more accurately capture the characteristics of expression changes, thereby improving the accuracy of expression adjustment. By taking the target image, face mask, expression label and the first face image as input, the model can comprehensively consider facial features and expression changes to generate a more accurate second face image. The final generated second face image not only matches the expected expression label, but also meets the user's demand for a specific expression. In addition, in the process of adjusting the expression, special attention is paid to maintaining other features of the face, such as identity features, to ensure that the generated image is visually consistent with the original face. This processing method not only improves the naturalness of expression transitions, but also ensures that the individual identification features of the generated images are retained, thereby maintaining the authenticity and credibility of the image while meeting the user's specific expression needs.

[0029] In some embodiments, according to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image, and then the method further includes:

[0030] According to the target image, the face mask and the second face image, the facial muscle optimization adjustment is performed on the face area in the second face image to generate a third face image corresponding to the second face image.

[0031] In the above technical solution, after the second stage of morphological adjustment, the resulting image is significantly close to the target expression. In order to further improve the naturalness and realism of the expression, the facial muscles in the face area of ​​the second face image are optimized and adjusted. This step involves the simulation of facial muscle movement to ensure that the expression changes are consistent with the actual movement rules of human facial muscles. Specifically, facial muscle simulation technology is used to carefully adjust the movement of facial muscles. Through this method, the desired expression changes can be achieved while maintaining the original facial features, and the naturalness and realism of the expression are significantly improved. The third face image finally generated is not only visually consistent with the original face, but also significantly improved in terms of the naturalness and realism of the expression. This solution effectively enhances the realism of the expression through the optimization and adjustment of facial muscles, making the generated image more visually convincing. This facial muscle adjustment not only improves the naturalness of the expression, but also ensures the coherence and authenticity of the generated image during the expression conversion process, thereby providing users with high-quality expression conversion results.

[0032] In some embodiments, according to the target image, the face mask and the second face image, optimizing and adjusting the facial muscles of the face area in the second face image to generate a third face image corresponding to the second face image includes:

[0033] Constructing a third expression data set, the third data set comprising a plurality of muscle pairing data pairs; the muscle pairing data pairs comprising: an expression image and a corresponding expression label and a facial muscle movement label;

[0034] training a third neural network based on the constructed third data set;

[0035] The target image, the face mask and the second face image are input into the pre-trained third neural network to generate a third face image corresponding to the second face image.

[0036] In the above technical solution, expression images and their corresponding expression labels and facial muscle movement labels are collected to construct a detailed data set. This data set provides the neural network with detailed information on the relationship between expression and muscle movement, thereby providing the model with a basis for in-depth understanding of the muscle change mechanism behind the expression. This data set containing muscle movement labels can help the model to more accurately capture and simulate the muscle dynamics behind the expression, thereby generating more accurate and natural expression adjustments. By using the third data set for training, the neural network can focus on the movement characteristics of facial muscles, so that the generated third face image is more visually natural and realistic. During the training process, we take the target image, face mask and second face image as input to ensure that all relevant information of the original image is fully considered during the adjustment process, which helps to generate an adjusted image that matches the original character features while maintaining the naturalness and authenticity of the expression adjustment. The third neural network optimizes the facial muscles of the second face image, which helps to correct possible unnatural expressions so that the final expression is more in line with the muscle movement laws of human natural expressions. Through this optimization and adjustment, the final generated third face image not only achieves the desired expression changes while maintaining the original face features, but also significantly improves the naturalness and authenticity of the expression.

[0037] In some embodiments, the third neural network adopts a biological neural network model.

[0038] In the above technical solution, biological models (such as FACS, 3D Morphable Models, etc.) are used to provide guidance for the generator on facial muscle movement. The generation process of the network can simulate the dynamic characteristics of facial muscle deformation, movement, etc.

[0039] According to another aspect of the present invention, a system for intelligent expression management in post-portrait photography is provided, based on the above method; comprising:

[0040] A mask module is used to perform mask processing on the face area in the target image to obtain a face mask;

[0041] The recognition module is used to recognize the facial area in the target image and obtain the expression label;

[0042] The morphology adjustment module is used to perform a first morphology adjustment on the face area in the target image according to the target image, the face mask and the expression label, and the expected expression label, so as to generate a first face image paired with the expected expression label.

[0043] In the above technical scheme, in order to better use the above method, the present application proposes a system for intelligent expression management in post-portrait photography, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0044] According to another aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0045] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flow chart of an embodiment of a method for intelligent expression management in the later stage of portrait photography according to the present invention;

[0048] Figure 2 This is a network architecture diagram of an embodiment of a method for intelligent expression management in the later stage of portrait photography according to the present invention;

[0049] Figure 3 It is a schematic diagram of processing results at different stages of an embodiment of a method for intelligent expression management in late portrait photography of the present invention;

[0050] Figure 4 It is a structural schematic diagram of an embodiment of a system for intelligent expression management in post-portrait photography of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0052] The present invention provides a method for intelligent expression management in the post-production of portrait photography, which aims to automatically detect and correct the smile problem in the portrait image, especially to solve the defects of poor naturalness, insufficient personalization, uncoordinated processing effect, insufficient clarity, etc. in the current smile optimization technology. Through an intelligent algorithm, the type, naturalness and emotional expression of the smile in the portrait are automatically analyzed, and the smile is appropriately adjusted to make it more harmonious, natural and in line with the personalized needs of the user. The processed image not only improves the visual effect of the smile, but also avoids the over-modification and unnatural phenomena commonly seen in traditional manual editing, thereby providing users with more natural, real and friendly portrait photos.

[0053] Embodiment 1

[0054] See also Figure 1 , a method for intelligent expression management in the later stage of portrait photography, the method comprising:

[0055] S1, masking the face area in the target image to obtain a face mask;

[0056] S2, recognize and process the facial area in the target image to obtain an expression label;

[0057] S3. Performing a first morphological adjustment on the facial region in the target image according to the target image, the face mask, the expression label, and the expected expression label, to generate a first facial image paired with the expected expression label.

[0058] In this embodiment, this research scheme aims to achieve accurate replacement of facial expressions in the target image through advanced image processing technology. First, this scheme performs mask processing on the facial area in the target image to generate a high-resolution facial mask containing key parts of the face (such as eyes, mouth, etc.). This process can effectively identify and segment the facial area in the image, providing accurate facial feature positioning for subsequent expression adjustment. Further, this scheme introduces facial expression recognition technology, which is based on machine learning algorithms and can analyze facial expressions in static images or dynamic videos, and accurately identify the specific expression state of individuals, such as happiness, sadness, surprise, etc. Through this step, the expression label of the face in the target image can be obtained, providing necessary reference information for subsequent expression replacement. Based on the target image, the generated face mask and the identified expression label, this scheme uses conditional generative adversarial networks (cGANs) to perform morphological adjustment on the facial area in the target image. cGANs technology can generate a new face image that matches the expected expression under the condition of a given face mask and the expected expression label, thereby realizing the accurate replacement of the facial expression in the target image. Finally, through the above-mentioned morphological adjustment process, this scheme can generate a first face image that matches the expected expression label. This method has broad application prospects in the fields of face synthesis, portrait editing, face swapping, etc. Through the processing of this scheme, a new image that is similar in appearance to the face in the original image but has a different expression can be generated, which effectively solves the problems of poor naturalness, insufficient personalization, inconsistent processing effects, and insufficient clarity in the current smile optimization technology. This scheme automatically analyzes the type, naturalness and emotional expression of the smile in the portrait through an intelligent algorithm, and makes appropriate adjustments to achieve a more harmonious, natural and personalized smile effect that meets the personalized needs of users. The processed image not only improves the visual effect of the smile, but also avoids the over-modification and unnatural phenomena commonly seen in traditional manual editing, providing users with more natural, real and friendly portrait photos.

[0059] In this embodiment, S3, performing a first morphological adjustment on the facial region in the target image according to the target image, the face mask, the expression label, and the expected expression label to generate a first facial image paired with the expected expression label, includes:

[0060] S31, constructing a first expression data set, the first data set comprising a plurality of expression pairing data pairs; the expression pairing data pairs comprising: a first expression label image and a second expression label image forming a data pair; the second expression label is an expected expression type;

[0061] S32, training a first neural network based on the constructed first expression data set;

[0062] S33, inputting the face mask, the expression label, and the expected expression label into the pre-trained first neural network to generate a first face image paired with the target expression label.

[0063] In this embodiment, a neural network is trained by constructing a data set containing expression pairs to achieve the conversion of facial expressions. The constructed data set consists of paired images, each pair of images containing an original expression image (first expression label image) and a target expression image (second expression label image). These paired images will be used as training samples to train the neural network to recognize the differences between expressions and learn how to map one expression to another. Based on the construction of the data set, a neural network model is further trained, which can recognize the expression in the input image and generate the corresponding output image according to the given target expression label. During the training process, the model will learn how to use the input face mask and expression label information, as well as the target expression label, to adjust the expression of the face image. After the training is completed, the input face mask, expression label and target expression label are input into the trained neural network model. The model will generate an output face image that matches the target expression label based on these input information. In the above scheme, a conditional generative adversarial network (CGANs) can be used as the structure of the first neural network model, which can generate data with specific attributes according to conditional variables, the corresponding conditional variables are expression labels, and the generated data are face images.

[0064] In this embodiment, S32, training a first neural network based on the constructed plurality of expression pairing data pairs, comprises:

[0065] S321, inputting the constructed data pair into the first neural network, and inputting the image generated based on the first neural network into the pre-trained expression classifier;

[0066] S322, calculating a difference between the generated image and the second expression label image based on the classifier, and using the difference as a first loss value;

[0067] S323: Train the first neural network based on the first loss value.

[0068] In this embodiment, first, the constructed expression pairing data pair is input into the first neural network. The task of this neural network is to generate images based on the input data pair, and these images should be able to reflect the transition from the first expression label to the second expression label (i.e., the expected expression type). The generated images are then input into a pre-trained expression classifier. The task of this classifier is to identify the expression label of the generated image to verify whether the image generated by the neural network matches the expected expression label. Based on the output of the pre-trained classifier, the difference value between the generated image and the second expression label image (expected expression type) is calculated. This difference value reflects the deviation between the generated image and the target expression and can be used as the first loss value for neural network training. This first loss value is used to adjust and optimize the parameters of the first neural network. By minimizing this loss value, the neural network can learn how to more accurately generate images that match the expected expression label. The above scheme achieves the purpose of generating specific expression images by constructing expression pairing data pairs, calculating loss values ​​using pre-trained classifiers, and training and optimizing neural networks based on these loss values. This process can be iterated continuously until the recognition result of the generated image in the pre-trained classifier matches the expected expression label, thereby achieving accurate conversion of expressions. .

[0069] In this embodiment, S3, according to the target image, the face mask and the expression label, and the expected expression label, a first morphological adjustment is performed on the face area in the target image to generate a first face image paired with the expected expression label, and then the following steps are further included:

[0070] S4. According to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image.

[0071] In this embodiment, the image generated in the first stage has achieved partial conversion to the target expression. In order to further refine this process, the target expression image, the face mask, the original expression label and the first face image are used as inputs, aiming to make more refined morphological adjustments to the facial area in the first face image. In this stage, the goal is to optimize the first face image so that it can match the expected expression more accurately while maintaining its naturalness. Specifically, the adjustment of the facial area of ​​the first face image is aimed at enhancing the expression features in the image, making it closer to the target expression, while retaining other significant features of the original face image. After the adjustment in the second stage, the resulting images have significantly improved the accuracy of the expression and more realistically reflect the features of the expected expression. In addition, these images also show high fidelity in maintaining other features of the original face image. At the same time, it can effectively improve the performance of the expression image generation task and achieve a smooth transition from the original expression to the target expression.

[0072] In this embodiment, S4, according to the target image, the face mask and the expression label, and the first face image, performing a second morphological adjustment on the face area in the first face image to generate a second face image corresponding to the first face image, includes:

[0073] S41, performing data enhancement based on the first expression data set to construct a second data set, the second data set comprising a plurality of expression pairing data sequences; the plurality of data sequences comprising: a first expression label image, a plurality of images transitioning from the first expression to the second expression, and a second expression label image;

[0074] S42, training a second neural network based on the constructed second data set;

[0075] S43, inputting the target image, the face mask, the expression label, and the first face image into the pre-trained second neural network to generate a second face image corresponding to the first face image.

[0076] In this embodiment, in the field of machine learning and computer vision, data enhancement technology has been widely recognized as an effective strategy. It can expand the training data set without increasing the actual data collection cost, thereby improving the generalization ability of the model. In this case, an expression pairing data sequence is constructed, which contains transition images from the first expression to the second expression. The design of this data sequence helps the model capture the continuity of expression changes, thereby generating a more natural expression conversion effect. Based on the constructed second data set, the neural network is trained to enable it to more accurately capture the characteristics of expression changes, thereby improving the accuracy of expression adjustment. By taking the target image, face mask, expression label and the first face image as input, the model can comprehensively consider facial features and expression changes and generate a more accurate second face image. The final generated second face image not only matches the expected expression label, but also meets the user's demand for a specific expression. In addition, in the process of adjusting the expression, special attention is paid to maintaining other features of the face, such as identity features, to ensure that the generated image is visually consistent with the original face. This processing method not only improves the naturalness of expression transitions, but also ensures that the individual identification features of the generated images are retained, thereby maintaining the authenticity and credibility of the image while meeting the user's specific expression needs.

[0077] In this embodiment, S4, according to the target image, the face mask and the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image, and then the following steps are further included:

[0078] S5. Perform facial muscle optimization adjustment on the face area in the second face image according to the target image, the face mask and the second face image, to generate a third face image corresponding to the second face image.

[0079] In this embodiment, after the second stage of morphological adjustment, the resulting image is significantly close to the target expression. In order to further improve the naturalness and realism of the expression, the facial muscles in the face area of ​​the second face image are optimized and adjusted. This step involves the simulation of facial muscle movement to ensure that the expression changes are consistent with the actual movement rules of human facial muscles. Specifically, facial muscle simulation technology is used to carefully adjust the movement of facial muscles. Through this method, the desired expression changes can be achieved while maintaining the original facial features, and the naturalness and realism of the expression are significantly improved. The third face image finally generated is not only visually consistent with the original face, but also significantly improved in terms of the naturalness and realism of the expression. This solution effectively enhances the realism of the expression through the optimization and adjustment of facial muscles, making the generated image more visually convincing. This facial muscle adjustment not only improves the naturalness of the expression, but also ensures the coherence and realism of the generated image during the expression conversion process, thereby providing users with high-quality expression conversion results.

[0080] In this embodiment, S5, performing facial muscle optimization adjustment on the face area in the second face image according to the target image, the face mask and the second face image to generate a third face image corresponding to the second face image, includes:

[0081] S51, constructing a third expression data set, the third data set comprising a plurality of muscle pairing data pairs; the muscle pairing data pairs comprising: an expression image and a corresponding expression label and a facial muscle movement label;

[0082] S52, training a third neural network based on the constructed third data set;

[0083] S53: Input the target image, the face mask and the second face image into the pre-trained third neural network to generate a third face image corresponding to the second face image.

[0084] In this embodiment, expression images and their corresponding expression labels and facial muscle movement labels are collected to construct a detailed data set. This data set provides the neural network with detailed information about the relationship between expression and muscle movement, thereby providing the model with a basis for in-depth understanding of the muscle change mechanism behind the expression. This data set containing muscle movement labels can help the model to more accurately capture and simulate the muscle dynamics behind the expression, thereby generating more accurate and natural expression adjustments. By using the third data set for training, the neural network can focus on the movement characteristics of facial muscles, so that the generated third face image is more natural and realistic visually. During the training process, we use the target image, the face mask and the second face image as input to ensure that all relevant information of the original image is fully considered during the adjustment process, which helps to generate an adjusted image that matches the original character features while maintaining the naturalness and authenticity of the expression adjustment. The third neural network optimizes the facial muscles of the second face image, which helps to correct possible unnatural expressions so that the final expression is more in line with the muscle movement laws of human natural expressions. Through this optimization and adjustment, the third face image finally generated not only achieves the desired expression changes while maintaining the original face features, but also significantly improves the naturalness and authenticity of the expression.

[0085] In this embodiment, the third neural network adopts a biological neural network model. A biological model (such as FACS, 3D Morphable Models, etc.) is used to provide guidance for facial muscle movement for the generator. The generation process of the network can simulate dynamic characteristics such as deformation and movement of facial muscles.

[0086] In order to better explain the present invention, a detailed description will be given based on one of the embodiments, as follows:

[0087] See also Figure 2 The network architecture of the method of this embodiment includes two parts: a recognition module and a processing module. The recognition module includes a face positioning model and a facial expression recognition model; the processing module includes a face morphology adjustment model, a facial expression synthesis model, and a facial muscle optimization model.

[0088] 1. Identification Module

[0089] The recognition module includes a face positioning model and a facial expression recognition model; the face positioning model needs to identify the facial area of ​​the person in the image; the facial expression recognition model needs to recognize the expression of the facial area.

[0090] The face location model needs to label a batch of data, which consists of the facial area mask of the person. The facial expression recognition model also needs to label a batch of data, which consists of the classification data of common expressions, including: happy, sad, angry, etc.

[0091] The face location model uses a segmentation network for training, and the training method is consistent with common segmentation networks, such as u-net. The face expression recognition model uses a classification network for training, and the training method is consistent with common classification networks, such as resnet.

[0092] 2. Processing Module

[0093] The processing modules include face morphology adjustment model, face expression synthesis model and facial muscle optimization model.

[0094] 1) Face shape adjustment model:

[0095] a. Model training data: It needs to contain diverse high-quality facial images, especially image datasets with multiple expression changes. The training data should include training samples of expression conversion. That is, paired data of original expressions and target expressions. For example, image pairs from "anger" to "smile", from "sadness" to "surprise", the richer the paired data, the better. In order to enhance data diversity, it is better to have some data with lighting changes, head angle changes, and other environmental factors.

[0096] b. Model training method: Expression consistency loss is added during training: the difference between the generated image and the expected expression category is calculated through the expression recognition network (pre-trained expression classifier) ​​to ensure that the generated image conforms to the target expression.

[0097] c. Model reasoning process: The input is the original image (see Figure 3 a), the mask area of ​​the face in the original image, the expression type of the face in the original image, and the expected expression category; the output is a generated face image (see Figure 3 b), the facial image needs to ensure that the overall expression category is consistent with the expected expression category.

[0098] 2) Facial expression synthesis model: Based on the generated face morphology, combined with the detailed texture of the original image, a smile image is synthesized; see Figure 3 Figure abcd; Figure a is the original image; Figure b is the smile image generated by the face morphology adjustment model, but the detail texture of the image is inconsistent with the original image; Figure c is the image synthesized by the face expression synthesis model, and the smile morphology is the same as that of Figure b. Figure 1 but the texture is similar to a Figure 1 To;

[0099] a. Model training data: The training data is similar to the data of the face morphology adjustment model; however, images of expression transitions are generated through data enhancement, such as from anger to smile, from expressionless to smile, etc. At the same time, the smile details are enhanced to simulate different lighting, viewing angles, local facial features, etc.

[0100] b. Model training method: On the one hand, the generative adversarial loss (GAN loss) is used to make the results tend to the real facial expression images. On the other hand, the mean square error loss (L2 loss) is used to supervise the model to generate the specified expression according to the guidance.

[0101] c. Model reasoning process: The input is the original image (see Figure 3 a), the mask area of ​​the face in the original image, the expression type of the face in the original image, and the result of the face shape adjustment model (see Figure 3 b); the output is a generated face image (see Figure 3 c) The facial image needs to further improve the smile texture and detail texture while ensuring the consistency of the overall expression with the results of the face shape adjustment model.

[0102] 3) Facial muscle optimization model:

[0103] a. Model training data: In order to train the facial muscle optimization model, high-quality facial dynamic expression data is required, especially data related to facial muscle movement. These datasets should contain well-labeled dynamic expressions and facial muscle movement markers, such as the appearance of zygomatic muscles when smiling.

[0104] b. Model training method: Biological models (such as FACS, 3D Morphable Models, etc.) are used to provide guidance for the generator on facial muscle movement. The network generation process can simulate dynamic features such as deformation and movement of facial muscles. It is mainly processed by adversarial generative networks, but biological models are added to provide guidance for the network.

[0105] c. Model reasoning process: The input is the original image (see Figure 3 a), the mask area of ​​the face in the original image, the result of the facial expression synthesis model; the output is a generated face image (see Figure 3 d), based on the results of the facial expression synthesis model, combined with the original image to ensure the naturalness and coordination of the expression.

[0106] The advantages of the present invention are specifically as follows:

[0107] 1. The degree of expression meets the actual needs of users: The present invention can automatically adjust the intensity and changes of expressions according to the personalized needs and situations of users. Specifically, the system can intelligently generate expressions of appropriate degrees according to the emotional preferences or scene requirements input by the user, thereby ensuring that the expression of expressions meets expectations, without distortion or exaggeration.

[0108] 2. Natural expression, in line with muscle activity trends: The present invention simulates the changes in facial expressions through the facial muscle movement model and accurately generates natural expressions. Through dynamic analysis and modeling of facial muscle activities, it can ensure that the expression transitions naturally during the movement process, conforms to physiological laws, avoids the generation of stiff or unreal smiles, etc., and improves the user experience.

[0109] 3. Preserve image clarity and facial details: While generating a specified expression, the present invention can preserve facial details, such as local details of the eyes, mouth, nose, etc., through advanced image optimization technology, while ensuring that the clarity of the image is not affected. By adopting a high-precision image processing algorithm, blur or loss of details caused by the expression generation or optimization process is avoided, thereby ensuring the realism and layering of facial expressions.

[0110] Embodiment 2

[0111] See also Figure 4 , a system for intelligent expression management in post-portrait photography, based on the method described in one of the embodiments; comprising:

[0112] A mask module is used to perform mask processing on the face area in the target image to obtain a face mask;

[0113] The recognition module is used to recognize the facial area in the target image and obtain the expression label;

[0114] The first morphology adjustment module is used to perform a first morphology adjustment on the face area in the target image according to the target image, the face mask and the expression label, and the expected expression label, so as to generate a first face image paired with the expected expression label.

[0115] In this embodiment, the system further includes:

[0116] a second morphology adjustment module, configured to perform a second morphology adjustment on the facial region in the first facial image according to the target image, the facial mask, the expression label, and the first facial image, so as to generate a second facial image corresponding to the first facial image;

[0117] The third morphology adjustment module is used to optimize the facial muscles of the face area in the second face image according to the target image, the face mask and the second face image, so as to generate a third face image corresponding to the second face image.

[0118] In this embodiment, in order to better use the method described in one of the embodiments, the present application proposes a system for intelligent expression management in post-portrait photography, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0119] Embodiment 3

[0120] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in one of the embodiments is implemented.

[0121] In the above technical solution, in order to better run and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and a processor is used to implement the method described in one of the embodiments. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0122] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for intelligent expression management in the later stage of portrait photography, characterized in that: The method comprises: Mask the face area in the target image to obtain a face mask; Recognize the facial area in the target image and obtain the expression label; According to the target image, the face mask, the expression label, and the expected expression label, a first morphological adjustment is performed on the face area in the target image to generate a first face image paired with the expected expression label.

2. The method for intelligent expression management in late portrait photography as claimed in claim 1, characterized in that: According to the target image, the face mask, the expression label, and the expected expression label, a first morphological adjustment is performed on the face region in the target image to generate a first face image paired with the expected expression label, including: Constructing a first expression data set, the first data set includes a plurality of expression pairing data pairs; the expression pairing data pairs include: a first expression label image and a second expression label image constitute a data pair; the second expression label is an expected expression type; Training a first neural network based on the constructed first expression data set; The face mask, the expression label, and the expected expression label are input into the pre-trained first neural network to generate a first face image paired with the target expression label.

3. A method for intelligent expression management in late portrait photography as claimed in claim 2, characterized in that: The first neural network is trained based on several constructed expression pair data pairs, including: Input the constructed data pair into the first neural network, and input the pre-trained expression classifier based on the image generated by the first neural network; Calculate the difference between the generated image and the second expression label image based on the classifier, and use the difference as the first loss value; The first neural network is trained based on the first loss value.

4. The method for intelligent expression management in the later stage of portrait photography as claimed in claim 1, characterized in that: According to the target image, the face mask, the expression label, and the expected expression label, a first morphological adjustment is performed on the face area in the target image to generate a first face image paired with the expected expression label, and then the method further includes: According to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image.

5. The method for intelligent expression management in the later stage of portrait photography as claimed in claim 4, characterized in that: According to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face region in the first face image to generate a second face image corresponding to the first face image, including: Based on the first expression data set, data enhancement is performed to construct a second data set, the second data set includes a plurality of expression pairing data sequences; the plurality of data sequences include: a first expression label image, a plurality of images transitioning from the first expression to the second expression, and a second expression label image; Training a second neural network based on the constructed second data set; The target image, the face mask, the expression label, and the first face image are input into the pre-trained second neural network to generate a second face image corresponding to the first face image.

6. The method for intelligent expression management in the later stage of portrait photography as claimed in claim 4, characterized in that: According to the target image, the face mask, the expression label, and the first face image, a second morphological adjustment is performed on the face area in the first face image to generate a second face image corresponding to the first face image, and then the method further includes: According to the target image, the face mask and the second face image, the facial muscle optimization adjustment is performed on the face area in the second face image to generate a third face image corresponding to the second face image.

7. The method for intelligent expression management in the later stage of portrait photography as claimed in claim 6, characterized in that: According to the target image, the face mask and the second face image, optimizing and adjusting the facial muscles of the face area in the second face image to generate a third face image corresponding to the second face image includes: Constructing a third expression data set, the third data set comprising a plurality of muscle pairing data pairs; the muscle pairing data pairs comprising: an expression image and a corresponding expression label and a facial muscle movement label; training a third neural network based on the constructed third data set; The target image, the face mask and the second face image are input into the pre-trained third neural network to generate a third face image corresponding to the second face image.

8. The method for intelligent expression management in the later stage of portrait photography as claimed in claim 7, characterized in that: The third neural network adopts a biological neural network model.

9. A system for intelligent expression management in post-portrait photography, characterized in that: The method according to any one of claims 1 to 8; comprising: A mask module is used to perform mask processing on the face area in the target image to obtain a face mask; The recognition module is used to recognize the facial area in the target image and obtain the expression label; The morphology adjustment module is used to perform a first morphology adjustment on the face area in the target image according to the target image, the face mask and the expression label, and the expected expression label, so as to generate a first face image paired with the expected expression label.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images