Model training methods and related equipment

Abstract

The disclosed embodiments provide a model training method and related equipment. The method includes: obtaining an initial two-dimensional object image carrying a real label; processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; processing the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; constructing a first loss function based on the real label and its predicted label, and constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image; and training the three-dimensional reconstruction network model using the first loss function and the second loss function. In this method, the three-dimensional reconstruction network model can be used as a part of the image generator in the generative adversarial network mode, and the training of the three-dimensional reconstruction network model is achieved through adversarial training between the image generator and the discriminator.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a model training method, an image recognition method, a model training device, an image recognition device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Currently, during video shooting or live broadcasting, image processing technology can be used to deform the two-dimensional object part in the collected initial two-dimensional object image, such as beautifying the face in a two-dimensional face image.

[0003] In the related art, when performing image processing on an initial two-dimensional object image, such as a two-dimensional face image, the planar two-dimensional object portion in the initial two-dimensional object image is usually directly deformed. However, since the human face is three-dimensional, the results obtained by the processing method of the related art often present unreasonable visual effects such as distorted curves and abrupt transitions. It can be seen that the related art cannot reasonably deform the initial two-dimensional object image.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a model training method, a model training device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem in the related art that an object in a two-dimensional object image cannot be reasonably deformed. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, the method comprising: obtaining an initial two-dimensional object image carrying a real label; processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; processing the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; constructing a first loss function based on the real label and its predicted label, and constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image; and training the three-dimensional reconstruction network model using the first loss function and the second loss function.

[0007] In some exemplary embodiments of the present disclosure, the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; the step of processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image includes: processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image.

[0008] In some exemplary embodiments of the present disclosure, the step of processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image includes: processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; processing the shape parameters and albedo parameters through a three-dimensional face statistical model to obtain three-dimensional mesh data and albedo data corresponding to meshes in the three-dimensional mesh data, and then using the three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters as the three-dimensional image data; and,

[0009] The step of rendering the three-dimensional image data of the initial two-dimensional object image to obtain the target two-dimensional object image includes: using a differentiable rendering layer to render the three-dimensional mesh data, albedo data, expression parameters, posture parameters and lighting parameters of the initial two-dimensional facial image to obtain the target two-dimensional facial image.

[0010] In some exemplary embodiments of the present disclosure, the step of constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image includes: constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; constructing a semantic loss function based on the difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; constructing a key point loss function based on the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; and determining the second loss function based on at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

[0011] In some exemplary embodiments of the present disclosure, the step of constructing a reconstruction loss function based on the pixel difference between an initial two-dimensional facial image and a target two-dimensional facial image includes: determining an initial error-allowed region and a non-initial error-allowed region of the initial two-dimensional facial image; determining a target error-allowed region and a non-target error-allowed region of the target two-dimensional facial image; calculating the pixel difference between the initial error-allowed region and the target error-allowed region to construct an error-allowed region loss function; calculating the pixel difference between the non-initial error-allowed region and the non-target error-allowed region to construct a non-error-allowed region loss function; determining a reconstruction loss function based on the error-allowed region loss function and the non-error-allowed region loss function, wherein the error-allowed region loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

[0012] In some exemplary embodiments of the present disclosure, the step of determining an initial error-allowed area and a non-initial error-allowed area of ​​an initial two-dimensional facial image includes: determining a facial area in the initial two-dimensional facial image; determining a lip center key point, a nose tip key point, and a distance between the lip center key point and the nose tip key point in the facial area; determining an initial error-allowed area of ​​the initial two-dimensional facial image based on the lip center key point and the distance, and determining an area outside the initial error-allowed area in the facial area as a non-initial error-allowed area.

[0013] In some exemplary embodiments of the present disclosure, the true label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes a first label and a second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

[0014] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: acquiring a two-dimensional image to be processed; processing the two-dimensional image to be processed by a trained three-dimensional reconstruction network model to obtain three-dimensional image data of an object to be processed in the two-dimensional image to be processed; rendering the three-dimensional image data of the two-dimensional image to be processed to obtain a target two-dimensional image; extracting a background image of the two-dimensional image to be processed, and fusing the background image and the target two-dimensional image to obtain a processed image of the two-dimensional image to be processed; wherein the trained three-dimensional reconstruction network model is trained based on an initial two-dimensional object image carrying a true label, as well as a first loss function and a second loss function; the first loss function is constructed based on the true label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image by the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image by a discriminator.

[0015] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, comprising: a first acquisition module, configured to execute acquisition of an initial two-dimensional object image carrying a real label; a first processing module, configured to execute processing of the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; a first rendering module, configured to execute rendering processing of the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; a prediction module, configured to execute processing of the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; a construction module, configured to execute construction of a first loss function based on the real label and its predicted label, and construction of a second loss function based on the initial two-dimensional object image and the target two-dimensional object image; a training module, configured to execute training of the three-dimensional reconstruction network model using the first loss function and the second loss function.

[0016] In some exemplary embodiments of the present disclosure, the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; the first processing module executes the step of processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image, including: processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image.

[0017] In some exemplary embodiments of the present disclosure, the first processing module executes the step of processing an initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image, including: processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; processing the shape parameters and albedo parameters through a three-dimensional face statistical model to obtain three-dimensional mesh data and albedo data corresponding to a mesh in the three-dimensional mesh data, and then using the three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters as three-dimensional image data; and,

[0018] The first rendering module performs rendering processing on the three-dimensional image data of the initial two-dimensional object image to obtain the target two-dimensional object image step, including: using a differentiable rendering layer to render the three-dimensional mesh data, albedo data, expression parameters, posture parameters and lighting parameters of the initial two-dimensional facial image to obtain the target two-dimensional facial image.

[0019] In some exemplary embodiments of the present disclosure, a construction module executes the step of constructing a second loss function based on an initial two-dimensional object image and a target two-dimensional object image, including: constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; constructing a semantic loss function based on the difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; constructing a key point loss function based on the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; and determining a second loss function based on at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

[0020] In some exemplary embodiments of the present disclosure, a construction module executes the step of constructing a reconstruction loss function according to the pixel difference between an initial two-dimensional facial image and a target two-dimensional facial image, including: determining an initial error-allowed area and a non-initial error-allowed area of ​​the initial two-dimensional facial image; determining a target error-allowed area and a non-target error-allowed area of ​​the target two-dimensional facial image; calculating the pixel difference between the initial error-allowed area and the target error-allowed area to construct an error-allowed area loss function; calculating the pixel difference between the non-initial error-allowed area and the non-target error-allowed area to construct a non-error-allowed area loss function; determining a reconstruction loss function according to the error-allowed area loss function and the non-error-allowed area loss function, wherein the error-allowed area loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

[0021] In some exemplary embodiments of the present disclosure, a construction module executes the steps of determining an initial error-allowed area and a non-initial error-allowed area of ​​an initial two-dimensional facial image, including: determining a facial area in the initial two-dimensional facial image; determining a lip center key point, a nose tip key point, and a distance between the lip center key point and the nose tip key point in the facial area; determining an initial error-allowed area of ​​the initial two-dimensional facial image based on the lip center key point and the distance, and determining an area outside the initial error-allowed area in the facial area as a non-initial error-allowed area.

[0022] In some exemplary embodiments of the present disclosure, the true label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes a first label and a second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

[0023] According to a fourth aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: a second acquisition module, configured to execute acquisition of a two-dimensional image to be processed; a second processing module, configured to execute processing of the two-dimensional image to be processed by a trained three-dimensional reconstruction network model, and obtain three-dimensional image data of the object to be processed in the two-dimensional image to be processed; a second rendering module, configured to execute rendering processing on the three-dimensional image data of the two-dimensional image to be processed, and obtain a target two-dimensional image; a fusion module, configured to execute extraction of a background image of the two-dimensional image to be processed, and perform fusion processing on the background image and the target two-dimensional image, and obtain a processed image of the two-dimensional image to be processed; wherein the trained three-dimensional reconstruction network model is trained based on an initial two-dimensional object image carrying a true label, as well as a first loss function and a second loss function, the first loss function is constructed based on the true label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image by the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image by a discriminator.

[0024] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the model training method or image processing method as described in the above embodiments.

[0025] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the model training method or image processing method as described in the above embodiments.

[0026] According to the seventh aspect of the embodiments of the present disclosure, a computer program product, a computer program or an instruction is provided, characterized in that when the computer program or the instruction is executed by a processor, the model training method or the image processing method described in the above embodiments is implemented.

[0027] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0028] The model training method provided by the embodiment of the present disclosure can, on the one hand, process a two-dimensional initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image, and the three-dimensional image data can be used to characterize the characteristics of the two-dimensional object in the initial two-dimensional object image in a three-dimensional scene. Since the extracted three-dimensional image data can be adjusted according to the parameters in the three-dimensional reconstruction network model, the three-dimensional image data may not be data that completely restores the two-dimensional object. Therefore, when the target two-dimensional object image is rendered according to the three-dimensional image data, there may be deformation between the target two-dimensional object image and the initial two-dimensional object image, so that the three-dimensional reconstruction network model can realize deformation processing of the image based on the two-dimensional image data; and the deformation processing realized by the three-dimensional reconstruction network model includes a two-dimensional-three-dimensional-two-dimensional conversion method, which can make the dimensional information used in the deformation processing process richer, so that the obtained target two-dimensional object image can present a more realistic and reasonable deformation effect. On the other hand, in the training method of using a discriminator to process a target two-dimensional object image to obtain a predicted label of the target two-dimensional object image, constructing a first loss function based on the real label of the initial two-dimensional object image and the predicted label of the target two-dimensional object image, then constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image, and then training the three-dimensional reconstruction network model based on the first loss function and the second loss function, the three-dimensional reconstruction network model can be used as a part of the generator in the generative adversarial network mode. Therefore, by using the adversarial training of the three-dimensional reconstruction network model and the discriminator, the target two-dimensional object image obtained by processing the initial two-dimensional object image using the trained three-dimensional reconstruction network model can further present a more realistic and reasonable deformation effect; at the same time, by using the three-dimensional reconstruction network model to process the two-dimensional initial two-dimensional object image carrying the real label, it is possible to avoid constructing a large amount of three-dimensional information pairing data during the training stage. Since the initial two-dimensional object image is easier to obtain than the three-dimensional information pairing data, the cost can be reduced.

[0029] Furthermore, the model training method provided by the embodiments of the present disclosure can be trained based on an initial two-dimensional facial image, for example, a two-dimensional face image. Since a reconstruction loss function, a semantic loss function, and a key point loss function are constructed during the training process, and the model training is performed based on at least one of these three loss functions, a natural and reasonable deformation processing effect can also be obtained for the case where the face in the two-dimensional face image is in a side face, half side face, or other postures, so that the deformed facial parts conform to the laws of real faces.

[0030] Furthermore, in the image processing method provided by the embodiment of the present disclosure, the trained three-dimensional reconstruction network model can be used to first process the two-dimensional image to be processed to obtain a target two-dimensional image with a real and reasonable deformation effect, and then the background image of the two-dimensional image to be processed is extracted, and the background image and the target two-dimensional image are fused to obtain a processed image of the two-dimensional image to be processed. Since the target two-dimensional image with a real and reasonable deformation effect can be obtained first, and then the background image extracted from the two-dimensional image to be processed is fused with the target two-dimensional image, the effect of the background image can be retained, so that the background part in the above-mentioned processed image maintains the background effect in the two-dimensional image to be processed, so that both the object part and the background part in the processed image have real and reasonable effects.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0033] Figure 1 It is a schematic diagram of an exemplary system architecture of a model training method according to an exemplary embodiment.

[0034] Figure 2 The figure is a flowchart of a model training method according to an exemplary embodiment.

[0035] Figure 3 It is a flowchart of obtaining a target two-dimensional facial image in a model training method according to an exemplary embodiment.

[0036] Figure 4 It is a schematic diagram showing determination of an initial error-allowed region and a non-initial error-allowed region of an initial two-dimensional facial image in a model training method according to an exemplary embodiment.

[0037] Figure 5 It is a network architecture diagram for implementing a model training method according to an exemplary embodiment.

[0038] Figure 6 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0039] Figure 7 It is a block diagram of a model training device according to an exemplary embodiment.

[0040] Figure 8It is a schematic diagram of the structure of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] Figure 1 A schematic diagram of an exemplary system architecture to which the model training method according to an embodiment of the present disclosure can be applied is shown.

[0044] like Figure 1 As shown, the system architecture may include a server 101, a network 102, and a client 103. The network 102 is used to provide a medium for a communication link between the client 103 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0045] In some optional embodiments, the server 101 may be a server that provides various services, and may be used to train a 3D reconstruction network model, and deploy the trained second feature extraction model on the server 101 or the client 103, and then the trained 3D reconstruction network model may be used to process the to-be-processed 2D image obtained from the client 103 to obtain a processed image of the to-be-processed 2D image. In some optional embodiments, the client 103 may also be used to train a 3D reconstruction network model, and deploy the trained second feature extraction model on the server 101 or the client 103.

[0046] Further, taking the server 101 training the three-dimensional reconstruction network model as an example, in some optional embodiments, the process of the server 101 for training the three-dimensional reconstruction network model can be: the server 101 obtains an initial two-dimensional object image carrying a real label; the server 101 processes the initial two-dimensional object image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; the server 101 renders the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; the server 101 processes the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; the server 101 constructs a first loss function based on the real label and its predicted label, and constructs a second loss function based on the initial two-dimensional object image and the target two-dimensional object image; the server 101 trains the three-dimensional reconstruction network model using the first loss function and the second loss function.

[0047] The server 101 can also provide a service for processing a two-dimensional image to be processed, wherein the two-dimensional image to be processed can be sent to the server 101 by the client 103. The process of the server 101 processing the two-dimensional image to be processed can be: the server 101 obtains the two-dimensional image to be processed; the server 101 processes the image to be processed through the three-dimensional reconstruction network model to obtain the three-dimensional image data of the object to be processed in the two-dimensional image to be processed; the server 101 renders the three-dimensional image data of the two-dimensional image to be processed to obtain the target two-dimensional image; the server 101 extracts the background image of the two-dimensional image to be processed, and fuses the background image and the target two-dimensional image to obtain the processed image of the two-dimensional image to be processed. After obtaining the processed image of the two-dimensional image to be processed, the server 101 can also return the processed image to the client 103 so that the client 103 can display the processed image to the user.

[0048] In some optional embodiments, server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0049] In some optional embodiments, the client 103 may include but is not limited to electronic devices such as smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, smart wearable devices, etc., or the client 103 may also be a personal computer, such as a laptop computer and a desktop computer, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0050] In addition, it should be noted that Figure 1 What is shown is merely one application environment of the model training method provided by the present disclosure. Figure 1 The numbers of clients, networks and servers in the embodiment are merely illustrative, and any number of clients, networks and servers may be provided according to actual needs.

[0051] In order to enable ordinary persons in the art to better understand the technical solution of the present disclosure, each step of the model training method in the exemplary embodiment of the present disclosure will be described in more detail below with reference to the accompanying drawings and embodiments.

[0052] Figure 2 is a flowchart of a model training method according to an exemplary embodiment. Figure 2 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The server 101 in the embodiment, but the present disclosure is not limited thereto.

[0053] like Figure 2 As shown, the model training method provided by the embodiment of the present disclosure may include the following steps:

[0054] Step S201, obtaining an initial two-dimensional object image carrying a real label.

[0055] In the embodiments of the present disclosure, the object may be a building, a plant, a human body, a human face, an animal body, an animal face, a landscape, etc., and the initial two-dimensional object image may be a photograph of an architecture, a landscape painting, a painting containing an animal body, an image containing a human face, etc., which is not limited in the embodiments of the present disclosure. In some practical applications, the initial two-dimensional object image may be obtained from a specified database, such as: the specified database may be a database on a work sharing platform (such as a video platform or a photography work display platform), which may store historical works uploaded by users; the database may also be a pre-set sample database for model training. The number of initial two-dimensional object images used for training may be adjusted according to actual conditions, which is not limited in the embodiments of the present disclosure.

[0056] In addition, the real label can be used to indicate whether the initial two-dimensional object image meets the requirements, such as whether it meets the user's visual experience requirements. In some embodiments, the initial two-dimensional object image is taken as an initial two-dimensional facial image as an example for explanation, wherein the face can be a human face, and the real label of the initial two-dimensional facial image can include a first label and a second label; the predicted label of the target two-dimensional facial image can include a first label and a second label; wherein the first label can indicate that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label can indicate that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line. In some practical applications, the user's visual experience requirement can be "feeling that the face is good-looking", and usually the visual effect of "the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line" can make the user have a better visual experience and make the user "feel that the face is good-looking", thereby meeting the user's visual experience requirements; therefore, in some practical applications, the requirement can be set as the above-mentioned first label.

[0057] Step S203 , processing the initial two-dimensional object image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image.

[0058] In the disclosed embodiment, the 3D reconstruction network model can be used to determine the 3D image data of the object in the 3D space based on the initial 2D object image. The 3D reconstruction network model can be a neural network model, such as a convolutional neural network CNN (Convolutional Neural Network) model, an R-CNN (Region-CNN) model, a Fast RCNN model, a target detection algorithm SSD (Single Shot MultiBox Detector) model, etc., which is not limited in the disclosed embodiment. In some practical applications, the 3D reconstruction network model can also be used to construct a 3D model of the object in the 3D space based on the initial 2D object image, wherein the 3D model and the above-mentioned 3D image data can correspond.

[0059] Step S205 , rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image.

[0060] In the embodiment of the present disclosure, after obtaining the three-dimensional image data of the object in the initial two-dimensional object image, rendering can be performed according to the three-dimensional image data to obtain a target two-dimensional object image of the object. In this process, the three-dimensional image data can be changed so that the target two-dimensional object image rendered according to the three-dimensional image data produces a natural and reasonable deformation relative to the initial two-dimensional object image, thereby making the target two-dimensional object image present a natural and reasonable visual effect.

[0061] Step S207: Process the target two-dimensional object image through the discriminator to obtain a predicted label of the target two-dimensional object image.

[0062] In the embodiment of the present disclosure, the discriminator can be used to determine the predicted label of the object in the target two-dimensional object image. The three-dimensional reconstruction network model can be a neural network model, such as a convolutional neural network CNN (Convolutional Neural Network) model, an R-CNN (Region-CNN) model, a Fast RCNN model, a target detection algorithm SSD (Single Shot MultiBox Detector) model, etc., which is not limited in the embodiment of the present disclosure. In the model training method proposed in the present disclosure, the three-dimensional reconstruction network model can be used as a part of the generator in the generative adversarial network mode, so the three-dimensional reconstruction network model and the discriminator can be trained adversarially in the generative adversarial network mode, thereby realizing the training of the three-dimensional reconstruction network model. Among them, the predicted label of the target two-dimensional facial image may include a first label and a second label; the discriminator may have the ability to identify whether the predicted label of the target two-dimensional object image is the first label or the second label.

[0063] Specifically, in the adversarial training process between the 3D reconstruction network model and the discriminator in the generative adversarial network mode, the discriminator can be trained first. When the training of the discriminator reaches the preset conditions of the current stage, the parameters in the discriminator can be fixed to train the 3D reconstruction network model. The preset conditions can be, for example: determining that the predicted label of the target 2D object image identified by the discriminator is consistent with the true label of the corresponding initial 2D object image by more than 80%; when the training of the 3D reconstruction network model reaches the preset conditions of the current stage, the parameters in the 3D reconstruction network model can be fixed to train the discriminator again; in this way, the discriminator and the 3D reconstruction network model can constitute a dynamic "game process", that is, the adversarial training process. When it is determined in the preset stage that the training of the discriminator reaches the preset conditions of the preset stage, the adversarial training can be stopped, and the 3D reconstruction network model at this time can be considered as a 3D reconstruction network model that has been trained.

[0064] Step S209: construct a first loss function according to the true label and the predicted label thereof, and construct a second loss function according to the initial two-dimensional object image and the target two-dimensional object image.

[0065] In the embodiment of the present disclosure, in the process of constructing the second loss function according to the initial two-dimensional object image and the target two-dimensional object image, feature information of the initial two-dimensional object image and the target two-dimensional object image can be extracted respectively, and then the second loss function can be constructed according to the feature information. The first loss function and the second loss function can be any one of a 0-1 loss function, an absolute value loss function, a logarithmic loss function, a square loss function, an exponential loss function, a hinge loss function, a perceptual loss function, a cross entropy loss function, a mean square error loss function, etc., which is not limited in the embodiment of the present disclosure.

[0066] In some practical applications, the hinge loss function or the perceptron loss function can be used to construct the first loss function. Both the hinge loss function and the perceptron loss function can be used to reflect whether the prediction result is satisfactory. The target label corresponding to "satisfied" can be set in advance. When the predicted label is judged to be equal to the target label, the loss is 0. Taking the hinge loss function as an example, the target label corresponding to "satisfactory" can be set in advance for the first loss function. For example, the target label is set to the first label, then when the predicted label of the target two-dimensional object image is the first label, the loss is 0, and when the predicted label of the target two-dimensional object image is the second label, the loss is 1; the actual meaning of the first loss function in training can be: the training effect of the three-dimensional reconstruction network model is expected to be able to generate a target two-dimensional object image that enables the discriminator to recognize that the predicted label is the target label; for example, when constructing the first loss function, it is determined that the target label indicates the feature that "the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line", then it can be considered that in the current training, it is hoped that the target two-dimensional object image processed by the three-dimensional reconstruction network model has the feature that "the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line".

[0067] Step S211, using the first loss function and the second loss function to train the 3D reconstruction network model.

[0068] In the embodiment of the present disclosure, the 3D reconstruction network model may be trained according to the first loss function and the second loss function calculated in step S209. In some practical applications, the 3D reconstruction network model may include model parameters, and the 3D reconstruction network model may be trained according to the first loss function and the second loss function, and the model parameters may be adjusted to obtain target model parameters of the 3D reconstruction network model, and the 3D reconstruction network model having the target model parameters may be determined as the trained 3D reconstruction network model.

[0069] Through the above-mentioned steps S201 to S211, the initial two-dimensional object image carrying the real label can be first processed by the three-dimensional reconstruction network model to obtain the three-dimensional image data of the initial two-dimensional object image, and then the three-dimensional image data of the initial two-dimensional object image can be rendered to obtain the target two-dimensional object image, and then the predicted label of the target two-dimensional object image is identified by the discriminator, so as to construct a first loss function according to the real label and its predicted label, and construct a second loss function according to the initial two-dimensional object image and the target two-dimensional object image, and finally the first loss function and the second loss function are used to train the three-dimensional reconstruction network model.

[0070] The model training method process provided by the present disclosure can be regarded as a part of the training mode of the Generative Adversarial Network (GAN), and the three-dimensional reconstruction network model in the embodiment of the present disclosure can be regarded as a part of the generator in the Generative Adversarial Network; specifically, in the method provided by the present disclosure, the predicted label of the target two-dimensional object image can be identified by the discriminator in the Generative Adversarial Network, and the first loss function can be calculated according to the predicted label obtained by identification and the real label carried by the initial two-dimensional object image, wherein the real label and the predicted label can be used to indicate whether the image meets the requirements, and the first loss function participates in the training so that the deformation generated by the three-dimensional reconstruction network model in the process of processing the initial two-dimensional object image meets the requirements; and in the method provided by the present disclosure, the second loss function can be calculated according to the target two-dimensional object image and the corresponding initial two-dimensional object image, and the second loss function participates in the training so that the generated target two-dimensional object image and the corresponding initial two-dimensional object image form a self-supervisory effect, so that the generated target two-dimensional object image and the initial two-dimensional object image are similar in visual effect. In the above manner, the adversarial training of the discriminator and the generator can be realized in the Generative Adversarial Network mode, and then the training of the three-dimensional reconstruction network model in the generator is realized. At the same time, in the model training method provided by the present invention, the initial two-dimensional object image undergoes a two-dimensional-three-dimensional-two-dimensional transformation. After obtaining the three-dimensional image data of the object in the initial two-dimensional object image, the three-dimensional image data can be changed so that the target two-dimensional object image rendered according to the three-dimensional image data produces a natural and reasonable deformation relative to the initial two-dimensional object image, thereby making the target two-dimensional object image present a natural and reasonable visual effect.

[0071] It can be seen that through the model training method provided by the present invention, the adversarial training of the discriminator and the generator can be realized in the generative adversarial network mode, and then the training of the 3D reconstruction network model in the generator can be realized, so that when the trained 3D reconstruction network model is used to process the initial two-dimensional object image, the following effects can be achieved: on the one hand, the object in the initial two-dimensional object image can be deformed to meet the requirements, and the target two-dimensional object image generated after the deformation can be similar to the initial two-dimensional object image in visual effect; on the other hand, through the 2D-3D-2D conversion method in the present invention, the generated target two-dimensional object image can present a natural and reasonable visual effect while generating a deformation that meets the requirements.

[0072] The following takes the initial two-dimensional object image as an initial two-dimensional facial image, that is, the object in the initial two-dimensional object image is a human face as an example to illustrate the method provided by the embodiment of the present disclosure:

[0073] In some embodiments, the target two-dimensional object image is a target two-dimensional facial image, and the target two-dimensional object image can be a target two-dimensional facial image; the initial two-dimensional facial image can be processed by a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image.

[0074] Furthermore, in some embodiments, the step of processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain three-dimensional mesh data and albedo data of the initial two-dimensional facial image may include: processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; processing the shape parameters and albedo parameters through a three-dimensional face statistical model to obtain three-dimensional mesh data, and albedo data corresponding to the mesh in the three-dimensional mesh data, and then the three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters can be used as the three-dimensional image data.

[0075] In the embodiment of the present disclosure, the three-dimensional image data obtained by processing the initial two-dimensional facial image through the three-dimensional reconstruction network model may include facial description information in various aspects such as three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters. For example, the three-dimensional mesh data may be a 99-dimensional vector data, the albedo data may be a 99-dimensional vector data, the expression parameters may be a 27-dimensional vector data, the posture parameters may be a 9-dimensional vector data, and the illumination parameters may be a 27-dimensional vector data, wherein the vector length in each data or parameter may be adjusted or set according to the actual situation, and the embodiment of the present disclosure does not limit it.

[0076] In some practical applications, three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters can be regarded as information that characterizes the facial features in the initial two-dimensional facial image from different aspects; specifically, the three-dimensional mesh data can characterize the shape model of the face, which shape model can be a face model in an expressionless state; the albedo data corresponding to the mesh in the three-dimensional mesh data can characterize the skin condition of the face; the expression parameters can characterize the facial expressions presented by the face, such as a certain degree of smile, sadness, etc.; the posture parameters can characterize the posture of the face in the two-dimensional image, such as a certain degree of side view, front view, etc.; the illumination parameters can characterize the light and dark state of the face in the two-dimensional image.

[0077] In some embodiments, the step of rendering the three-dimensional image data of the initial two-dimensional object image to obtain the target two-dimensional object image may include: using a differentiable rendering layer to render the three-dimensional mesh data, albedo data, expression parameters, posture parameters and lighting parameters of the initial two-dimensional facial image to obtain the target two-dimensional facial image.

[0078] The differentiable rendering layer may be a tool that can render according to scene parameters to obtain a new scene image. The differentiable rendering layer may be implemented using a pipeline rendering algorithm model or a physics-based rendering algorithm model, etc., which is not limited in the embodiments of the present disclosure. Specifically, in the embodiments of the present disclosure, the differentiable rendering layer may be used to render according to the above-mentioned various facial description information to obtain a target two-dimensional facial image.

[0079] Figure 3 FIG. 1 is a flow chart showing a method for obtaining a target two-dimensional facial image in a model training method according to an exemplary embodiment. Figure 3 As shown, Figure 2 Steps S203 to S205 in the embodiment may further include the following steps:

[0080] Step S301, obtaining an initial two-dimensional facial image.

[0081] Step S303, processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image.

[0082] Step S305, processing the shape parameters and albedo parameters through the three-dimensional face statistical model to obtain three-dimensional mesh data and albedo data corresponding to the meshes in the three-dimensional mesh data. The three-dimensional face statistical model can be a 3DMM (3D Morphable Face Model), which can be used to simulate a three-dimensional face model in the form of a three-dimensional mesh according to the shape parameters as the three-dimensional mesh data; and the albedo corresponding to each mesh in the three-dimensional mesh can also be obtained according to the corresponding albedo parameters as the albedo data.

[0083] Step S307 , using a differentiable rendering layer to render the three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image to obtain a target two-dimensional facial image.

[0084] Figure 3 In the embodiment shown, Figure 2 The same or similar steps in the embodiment shown can be referred to in Figure 2 The textual description of the illustrated embodiments will not be repeated in this disclosure.

[0085] It can be seen that the implementation Figure 3 The method shown can transform a two-dimensional image such as an initial two-dimensional facial image from two-dimensional to three-dimensional to two-dimensional. This process provides scene support for the natural and reasonable deformation of the face in the two-dimensional image, that is, by changing the three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters in the three-dimensional scene, a natural deformation can be achieved in the three-dimensional scene, so that the target two-dimensional facial image obtained after rendering is naturally and reasonably deformed relative to the initial two-dimensional facial image, thereby making the target two-dimensional facial image present a natural and reasonable visual effect. In addition, since the three-dimensional image data of the initial two-dimensional facial image is changed in the embodiment of the present disclosure, and the target two-dimensional facial image is rendered according to the three-dimensional image data after the change, a natural and reasonable processing effect can also be obtained for the face in the postures of the side face, half side face, etc., so that the deformed facial part conforms to the law of the real face.

[0086] In some embodiments, the step of constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image may include: constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; extracting semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image to construct a semantic loss function based on the difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; locating key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image to construct a key point loss function based on the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; and determining the second loss function based on at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

[0087] In the disclosed embodiment, the reconstruction loss function can reflect the difference in pixel level between images, the semantic loss function can reflect the difference in semantic level between images, and the key point loss function can reflect the difference in key point level between images; for example, for the semantic loss function, the initial semantic features of the initial two-dimensional facial image and the target semantic features of the target two-dimensional facial image can be extracted respectively by the image semantic feature extraction model, and then the semantic loss function is constructed based on the initial semantic features and the target semantic features. For the key point loss function, the initial key point features of the initial two-dimensional facial image and the target key point features of the target two-dimensional facial image can be extracted respectively by the face key point detection model, and then the key point loss function is constructed based on the initial key point features and the target key point features, wherein the key point features can represent a certain number (such as: 68, 128, etc.) of key points on the face.

[0088] In some practical applications, the reconstruction loss function, the semantic loss function, and the key point loss function may be any one of a 0-1 loss function, an absolute value loss function, a logarithmic loss function, a square loss function, an exponential loss function, a perceptual loss function, a cross entropy loss function, a mean square error loss function, etc., which is not limited in the embodiments of the present disclosure. The second loss function may be constructed based on one, two, or three of the reconstruction loss function, the semantic loss function, and the key point loss function. When two or three of the above three loss functions are used to construct the second loss function, the corresponding weight values ​​of each loss function may be obtained first, and then the second loss function may be determined based on each loss function and the corresponding weight values. The second loss function may be used to ensure that the initial two-dimensional facial image and the target two-dimensional facial image are similar, and then the initial two-dimensional facial image and the target two-dimensional facial image generated therefrom may be used to achieve the effect of self-supervised training.

[0089] In some embodiments, the step of constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image may include: determining an initial error-allowed area and a non-initial error-allowed area of ​​the initial two-dimensional facial image; determining a target error-allowed area and a non-target error-allowed area of ​​the target two-dimensional facial image; calculating the pixel difference between the initial error-allowed area and the target error-allowed area to construct an error-allowed area loss function; calculating the pixel difference between the non-initial error-allowed area and the non-target error-allowed area to construct a non-error-allowed area loss function; determining the reconstruction loss function based on the error-allowed area loss function and the non-error-allowed area loss function, wherein the error-allowed area loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

[0090] Since the face needs to be deformed during the training process, the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image is inevitable. The inevitable pixel difference can be offset by introducing an error tolerance threshold. For example, in the process of constructing a reconstruction loss function based on pixel differences, an error tolerance region (including an initial error tolerance region and a target error tolerance region) and a corresponding non-error tolerance region (including a non-initial error tolerance region and a non-target error tolerance region) can be introduced to further construct an error tolerance region loss function and a corresponding non-error tolerance region loss function, wherein the error tolerance region loss function can include a constraint condition that is less than or equal to the error tolerance threshold, thereby achieving the effect of offsetting the inevitable pixel difference through the error tolerance threshold set in the constraint condition.

[0091] Further, taking determining the initial error-allowed area and the non-initial error-allowed area of ​​the initial two-dimensional facial image as an example, in some embodiments, the step of determining the initial error-allowed area and the non-initial error-allowed area of ​​the initial two-dimensional facial image may include: determining the facial area in the initial two-dimensional facial image; determining the lip center key point, the nose tip key point, and the distance between the lip center key point and the nose tip key point in the facial area; determining the initial error-allowed area of ​​the initial two-dimensional facial image based on the lip center key point and the distance, and determining the area outside the initial error-allowed area in the facial area as the non-initial error-allowed area.

[0092] The step of determining the initial error tolerance region of the initial two-dimensional facial image based on the lip center key point and the distance may be: determining a closed region with the lip center key point as the center of the circle and the distance as the parameter; and performing boundary smoothing processing on the closed region using a filtering algorithm to obtain the initial error tolerance region of the initial two-dimensional facial image. The process of determining the closed region may be, for example, determining a circular region with the lip center key point as the center of the circle and the distance as the radius, and using the circular region as the closed region.

[0093] It should be noted that the application of various key points on the face mentioned in the present disclosure (including the nose tip key point, lip center key point, lip bead key point and chin center key point mentioned above) can be adjusted and set according to actual applications. For example, in the "nose tip key point, lip bead key point and chin center key point in the facial image are in a straight line" indicated by the aforementioned first label, the lip bead key point can also be the lip center key point; for another example, in the step of determining the initial error allowable area, the distance between the lip bead key point and the nose tip key point can also be determined first, and then the initial error allowable area is determined based on the distance calculated this time and the lip bead key point.

[0094] In some practical applications, since the lower part of the human head structure can have a large deformation, such as: for the mouth, there can be a variety of forms such as opening the mouth, smiling, laughing, etc., while the upper part of the human head structure usually does not undergo a large deformation, such as: the position of the brow bone, the shape of the cheekbone, etc., therefore, a circular area can be determined with the lip center key point as the center and the distance as the radius to serve as the initial error allowable area of ​​the initial two-dimensional facial image. Accordingly, the area outside the initial error allowable area in the facial area can be determined as the non-initial error allowable area. Through the above settings, the upper half of the deformed target two-dimensional facial image can be made as consistent as possible with the upper half of the initial two-dimensional facial image, while the lower half of the deformed target two-dimensional facial image can be slightly different from the lower half of the initial two-dimensional facial image, thereby achieving a visually similar effect between the target two-dimensional facial image and the initial two-dimensional facial image.

[0095] Figure 4 FIG. 1 is a schematic diagram showing a method for determining an initial error-allowed region and a non-initial error-allowed region of an initial two-dimensional facial image in a model training method according to an exemplary embodiment. Figure 4 As shown, the following steps may be included:

[0096] Step S401, determining a facial region in an initial two-dimensional facial image.

[0097] Step S403, determining the lip center key point, the nose tip key point, and the distance between the lip center key point and the nose tip key point in the facial area.

[0098] Step S405, determining a circular area with the lip center key point as the center and the distance as the radius.

[0099] Step S407, using a filtering algorithm to perform boundary smoothing processing on the circular area to obtain an initial error tolerance region of the initial two-dimensional facial image.

[0100] Step S409: determine the area outside the initial error allowable area in the facial area as a non-initial error allowable area.

[0101] Figure 5 is a network architecture diagram for implementing a model training method according to an exemplary embodiment. Figure 5 As shown, the following steps may be specifically included:

[0102] The server / terminal may first input the initial two-dimensional facial image 501 carrying the real label 512 into the three-dimensional reconstruction network model 502, and obtain three-dimensional mesh data 503, albedo data 504, expression parameters 505, posture parameters 506, and illumination parameters 507;

[0103] Then, the server / terminal may render the three-dimensional mesh data 503, the albedo data 504, the expression parameter 505, the posture parameter 506, and the illumination parameter 507 through a differentiable rendering layer 508, and may obtain a target two-dimensional facial image 509;

[0104] Next, the server / terminal may input the target two-dimensional facial image 509 into a discriminator 510, and may output a predicted label 511 of the target two-dimensional facial image 509;

[0105] Furthermore, the server / terminal may construct a first loss function according to the real label 512 and the predicted label 511 thereof, and construct a reconstruction loss function 512, a semantic loss function 513 and a key point loss function 514 according to the initial two-dimensional facial image 501 and the target two-dimensional facial image 509;

[0106] Finally, the server / terminal may train the 3D reconstruction network model 502 according to the first loss function, the reconstruction loss function 512 , the semantic loss function 513 , and the key point loss function 514 .

[0107] In the embodiment of the present disclosure, after the three-dimensional reconstruction network model training is completed, it can be deployed online for use. For example, a face image can be obtained as a two-dimensional image to be processed, and the face image can be processed by the trained three-dimensional reconstruction network model 502 to obtain three-dimensional image data of the face. The three-dimensional image data is then processed by a differentiable rendering layer to obtain a processed image of the face image.

[0108] Figure 6 is a flowchart of an image processing method according to an exemplary embodiment. Figure 6 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The server 101 in the embodiment, but the present disclosure is not limited thereto.

[0109] like Figure 6 As shown, the image processing method provided by the embodiment of the present disclosure may include the following steps:

[0110] Step S601, obtaining a two-dimensional image to be processed, wherein the two-dimensional image to be processed may be a face image obtained from a client, and the face in the face image may be in various postures such as front, side, upward or downward.

[0111] Step S603, processing the two-dimensional image to be processed by the trained three-dimensional reconstruction network model to obtain three-dimensional image data of the object to be processed in the two-dimensional image to be processed.

[0112] Among them, the three-dimensional image data can be three-dimensional mesh data, albedo data, expression parameters, posture parameters and lighting parameters of a face image; the trained three-dimensional reconstruction network model can be trained based on an initial two-dimensional object image carrying a true label, and a first loss function and a second loss function; the first loss function is constructed based on the true label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image with the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image with a discriminator.

[0113] Step S605, rendering is performed on the three-dimensional image data of the two-dimensional image to be processed to obtain a target two-dimensional image. The rendering process can be implemented by a differentiable rendering layer.

[0114] Step S607 , extracting the background image of the two-dimensional image to be processed, and performing fusion processing on the background image and the target two-dimensional image to obtain a processed image of the two-dimensional image to be processed.

[0115] Through the image processing method provided by the embodiment of the present disclosure, the trained three-dimensional reconstruction network model can be used to first process the two-dimensional image to be processed to obtain a target two-dimensional image with a real and reasonable deformation effect, and then the background image of the two-dimensional image to be processed can be extracted, and the background image and the target two-dimensional image can be fused to obtain a processed image of the two-dimensional image to be processed; since the target two-dimensional image with a real and reasonable deformation effect can be obtained first, and then the background image extracted from the two-dimensional image to be processed is fused with the target two-dimensional image, the unchanged background image in the two-dimensional image to be processed can be retained, so that the background part in the above-mentioned processed image maintains the background effect in the two-dimensional image to be processed, so that both the object part and the background part in the processed image present a real and reasonable effect.

[0116] It can be seen that implementation Figure 6 The method shown can perform correction processing on a two-dimensional image to be processed, such as a face image, and output the processed image for other systems to call or present to the user.

[0117] Figure 7is a block diagram of a model training device according to an exemplary embodiment. Figure 7 The model training device 700 provided by the embodiment of the present disclosure may include: a first acquisition module 710, configured to execute acquisition of an initial two-dimensional object image carrying a real label; a first processing module 720, configured to execute processing of the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; a first rendering module 730, configured to execute rendering processing of the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; a prediction module 740, configured to execute processing of the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; a construction module 750, configured to execute construction of a first loss function according to the real label and its predicted label, and construction of a second loss function according to the initial two-dimensional object image and the target two-dimensional object image; a training module 760, configured to execute training of the three-dimensional reconstruction network model using the first loss function and the second loss function.

[0118] In some embodiments, the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; the first processing module 720 executes the step of processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image, including: processing the initial two-dimensional facial image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image.

[0119] In some embodiments, the first processing module 720 executes the step of processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image, including: processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; processing the shape parameters and albedo parameters through the three-dimensional face statistical model to obtain three-dimensional mesh data, and albedo data corresponding to the mesh in the three-dimensional mesh data, and then using the three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters as three-dimensional image data; and the first rendering module 730 executes the step of rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image, including: using a differentiable rendering layer to render the three-dimensional mesh data, albedo data, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image to obtain a target two-dimensional facial image.

[0120] In some embodiments, the construction module 750 performs the step of constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image, including: constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; extracting semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image to construct a semantic loss function based on the difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; locating key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image to construct a key point loss function based on the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; determining the second loss function based on at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

[0121] In some embodiments, the construction module 750 performs the step of constructing a reconstruction loss function based on the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image, including: determining an initial error-allowed area and a non-initial error-allowed area of ​​the initial two-dimensional facial image; determining a target error-allowed area and a non-target error-allowed area of ​​the target two-dimensional facial image; calculating the pixel difference between the initial error-allowed area and the target error-allowed area to construct an error-allowed area loss function; calculating the pixel difference between the non-initial error-allowed area and the non-target error-allowed area to construct a non-error-allowed area loss function; determining a reconstruction loss function based on the error-allowed area loss function and the non-error-allowed area loss function, wherein the error-allowed area loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

[0122] In some embodiments, the construction module 750 performs the steps of determining an initial error-allowed area and a non-initial error-allowed area of ​​an initial two-dimensional facial image, including: determining a facial area in the initial two-dimensional facial image; determining a lip center key point, a nose tip key point, and a distance between the lip center key point and the nose tip key point in the facial area; determining an initial error-allowed area of ​​the initial two-dimensional facial image based on the lip center key point and the distance, and determining an area outside the initial error-allowed area in the facial area as a non-initial error-allowed area.

[0123] In some embodiments, the true label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes a first label and a second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

[0124] It can be seen that the implementation Figure 7The device shown, on the one hand, can process a two-dimensional initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image, and the three-dimensional image data can be used to characterize the characteristics of the two-dimensional object in the initial two-dimensional object image in a three-dimensional scene. Since the extracted three-dimensional image data can be adjusted according to the parameters in the three-dimensional reconstruction network model, the three-dimensional image data may not be data that completely restores the two-dimensional object. Therefore, when the target two-dimensional object image is rendered according to the three-dimensional image data, there may be deformation between the target two-dimensional object image and the initial two-dimensional object image, so that the three-dimensional reconstruction network model can realize deformation processing of the image based on the two-dimensional image data; and the deformation processing realized by the three-dimensional reconstruction network model includes a two-dimensional-three-dimensional-two-dimensional conversion method, which can make the dimensional information used in the deformation processing process richer, so that the obtained target two-dimensional object image can present a more realistic and reasonable deformation effect. On the other hand, in the training method of using a discriminator to process a target two-dimensional object image to obtain a predicted label of the target two-dimensional object image, constructing a first loss function based on the real label of the initial two-dimensional object image and the predicted label of the target two-dimensional object image, then constructing a second loss function based on the initial two-dimensional object image and the target two-dimensional object image, and then training the three-dimensional reconstruction network model based on the first loss function and the second loss function, the three-dimensional reconstruction network model can be used as a part of the generator in the generative adversarial network mode. Therefore, by using the adversarial training of the three-dimensional reconstruction network model and the discriminator, the target two-dimensional object image obtained by processing the initial two-dimensional object image using the trained three-dimensional reconstruction network model can further present a more realistic and reasonable deformation effect; at the same time, by using the three-dimensional reconstruction network model to process the two-dimensional initial two-dimensional object image carrying the real label, it is possible to avoid constructing a large amount of three-dimensional information pairing data during the training stage. Since the initial two-dimensional object image is easier to obtain than the three-dimensional information pairing data, the cost can be reduced.

[0125] In an exemplary embodiment, an image processing device is also provided, including: a second acquisition module, configured to execute acquisition of a two-dimensional image to be processed; a second processing module, configured to execute processing of the two-dimensional image to be processed by a trained three-dimensional reconstruction network model, and obtain three-dimensional image data of the object to be processed in the two-dimensional image to be processed; a second rendering module, configured to execute rendering processing on the three-dimensional image data of the two-dimensional image to be processed, and obtain a target two-dimensional image; a fusion module, configured to execute extraction of a background image of the two-dimensional image to be processed, and perform fusion processing on the background image and the target two-dimensional image, and obtain a processed image of the two-dimensional image to be processed; wherein the trained three-dimensional reconstruction network model is based on an initial two-dimensional object image carrying a true label, and a first loss function and a second loss function; the first loss function is constructed based on the true label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image by the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image by the discriminator.

[0126] Through the image processing device provided by the present invention, the trained three-dimensional reconstruction network model can be used to first process the two-dimensional image to be processed to obtain a target two-dimensional image with a real and reasonable deformation effect, and then the background image of the two-dimensional image to be processed can be extracted, and the background image and the target two-dimensional image can be fused to obtain a processed image of the two-dimensional image to be processed. Since the target two-dimensional image with a real and reasonable deformation effect can be obtained first, and then the background image extracted from the two-dimensional image to be processed is fused with the target two-dimensional image, the effect of the background image can be retained, so that the background part in the above-mentioned processed image maintains the background effect in the two-dimensional image to be processed, so that both the object part and the background part in the processed image have real and reasonable effects.

[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] Refer to the following Figure 8 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0130] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 The method steps shown in .

[0131] For example, electronic devices can achieve Figure 2 The steps shown.

[0132] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 821 and / or a cache memory unit 822 , and may further include a read-only memory unit (ROM) 823 .

[0133] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, such program modules 825 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0134] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0135] The electronic device 800 may also communicate with one or more external devices 870 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0136] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0137] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0138] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, which implements the model training method in the above embodiment when the computer program / instruction is executed by a processor.

[0139] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0140] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Obtain an initial two-dimensional object image with a true label; Processing the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; Rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; Processing the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; Constructing a first loss function according to the true label and the predicted label thereof, and constructing a second loss function according to the initial two-dimensional object image and the target two-dimensional object image; Training the 3D reconstruction network model using the first loss function and the second loss function; Wherein, the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; The step of processing the initial two-dimensional object image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image includes: Processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image; Among them, the real label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes the first label and the second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

2. The model training method according to claim 1, characterized in that: The step of processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image includes: Processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; Processing the shape parameters and the albedo parameters through a three-dimensional face statistical model to obtain three-dimensional mesh data and albedo data corresponding to a mesh in the three-dimensional mesh data, and then using the three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters as the three-dimensional image data; and, The step of rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image comprises: The three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters of the initial two-dimensional facial image are rendered using a differentiable rendering layer to obtain the target two-dimensional facial image.

3. The model training method according to claim 1, characterized in that: The step of constructing a second loss function according to the initial two-dimensional object image and the target two-dimensional object image comprises: constructing a reconstruction loss function according to a pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; constructing a semantic loss function according to a difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; constructing a key point loss function according to the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; The second loss function is determined according to at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

4. The model training method according to claim 3, characterized in that: The step of constructing a reconstruction loss function according to the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image comprises: Determine an initial error allowable region and a non-initial error allowable region of the initial two-dimensional facial image; Determine a target error-allowed region and a non-target error-allowed region of the target two-dimensional facial image; Calculating the pixel difference between the initial error-allowed region and the target error-allowed region to construct an error-allowed region loss function; Calculating the pixel difference between the non-initial error allowed region and the non-target error allowed region to construct a non-error allowed region loss function; The reconstruction loss function is determined according to the error-allowed region loss function and the non-error-allowed region loss function, wherein the error-allowed region loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

5. The model training method according to claim 4, characterized in that: The step of determining an initial error-allowed region and a non-initial error-allowed region of the initial two-dimensional facial image comprises: determining a facial region in the initial two-dimensional facial image; Determine a lip center key point, a nose tip key point, and a distance between the lip center key point and the nose tip key point in the facial region; An initial error-allowed region of the initial two-dimensional facial image is determined according to the lip center key point and the distance, and a region outside the initial error-allowed region in the facial region is determined as the non-initial error-allowed region.

6. An image processing method, characterized in that: include: Obtaining a two-dimensional image to be processed; Processing the two-dimensional image to be processed by the trained three-dimensional reconstruction network model to obtain three-dimensional image data of the object to be processed in the two-dimensional image to be processed; Rendering the three-dimensional image data of the two-dimensional image to be processed to obtain a target two-dimensional image; Extracting a background image of the two-dimensional image to be processed, and fusing the background image with the target two-dimensional image to obtain a processed image of the two-dimensional image to be processed; Among them, the trained three-dimensional reconstruction network model is trained based on an initial two-dimensional object image carrying a real label, as well as a first loss function and a second loss function, the first loss function is constructed based on the real label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image with the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image with a discriminator; the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; the real label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes the first label and the second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

7. A model training device, characterized in that: include: A first acquisition module is configured to acquire an initial two-dimensional object image carrying a real label; A first processing module is configured to process the initial two-dimensional object image through a three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image; A first rendering module is configured to perform rendering processing on the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image; A prediction module is configured to process the target two-dimensional object image through a discriminator to obtain a predicted label of the target two-dimensional object image; A construction module is configured to construct a first loss function according to the true label and the predicted label thereof, and to construct a second loss function according to the initial two-dimensional object image and the target two-dimensional object image; A training module, configured to train the 3D reconstruction network model using the first loss function and the second loss function; Wherein, the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; The first processing module executes the step of processing the initial two-dimensional object image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional object image, including: Processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image; Among them, the real label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes the first label and the second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

8. The model training device according to claim 7, characterized in that: The first processing module executes the step of processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain three-dimensional image data of the initial two-dimensional facial image, including: Processing the initial two-dimensional facial image through the three-dimensional reconstruction network model to obtain shape parameters, albedo parameters, expression parameters, posture parameters and illumination parameters of the initial two-dimensional facial image; Processing the shape parameters and the albedo parameters through a three-dimensional face statistical model to obtain three-dimensional mesh data and albedo data corresponding to a mesh in the three-dimensional mesh data, and then using the three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters as the three-dimensional image data; and, The first rendering module performs the step of rendering the three-dimensional image data of the initial two-dimensional object image to obtain a target two-dimensional object image, including: The three-dimensional mesh data, the albedo data, the expression parameters, the posture parameters and the illumination parameters of the initial two-dimensional facial image are rendered using a differentiable rendering layer to obtain the target two-dimensional facial image.

9. The model training device according to claim 7, characterized in that: The construction module executes the step of constructing a second loss function according to the initial two-dimensional object image and the target two-dimensional object image, including: constructing a reconstruction loss function according to a pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image; constructing a semantic loss function according to a difference between the semantic information of the initial two-dimensional facial image and the semantic information of the target two-dimensional facial image; constructing a key point loss function according to the difference between the key points of the initial two-dimensional facial image and the key points of the target two-dimensional facial image; The second loss function is determined according to at least one of the reconstruction loss function, the semantic loss function, and the key point loss function.

10. The model training device according to claim 9, characterized in that: The construction module performs the step of constructing a reconstruction loss function according to the pixel difference between the initial two-dimensional facial image and the target two-dimensional facial image, including: Determine an initial error allowable region and a non-initial error allowable region of the initial two-dimensional facial image; Determine a target error-allowed region and a non-target error-allowed region of the target two-dimensional facial image; Calculating the pixel difference between the initial error-allowed region and the target error-allowed region to construct an error-allowed region loss function; Calculating the pixel difference between the non-initial error allowed region and the non-target error allowed region to construct a non-error allowed region loss function; The reconstruction loss function is determined according to the error-allowed region loss function and the non-error-allowed region loss function, wherein the error-allowed region loss function includes a constraint condition that is less than or equal to an error-allowed threshold.

11. The model training device according to claim 10, characterized in that: The construction module performs the step of determining the initial error allowable region and the non-initial error allowable region of the initial two-dimensional facial image, including: determining a facial region in the initial two-dimensional facial image; Determine a lip center key point, a nose tip key point, and a distance between the lip center key point and the nose tip key point in the facial region; An initial error-allowed region of the initial two-dimensional facial image is determined according to the lip center key point and the distance, and a region outside the initial error-allowed region in the facial region is determined as the non-initial error-allowed region.

12. An image processing device, characterized in that: include: A second acquisition module is configured to acquire a two-dimensional image to be processed; A second processing module is configured to execute the trained three-dimensional reconstruction network model to process the two-dimensional image to be processed, so as to obtain three-dimensional image data of the object to be processed in the two-dimensional image to be processed; A second rendering module is configured to perform rendering processing on the three-dimensional image data of the two-dimensional image to be processed to obtain a target two-dimensional image; a fusion module, configured to extract a background image of the two-dimensional image to be processed, and perform fusion processing on the background image and the target two-dimensional image to obtain a processed image of the two-dimensional image to be processed; Among them, the trained three-dimensional reconstruction network model is trained based on an initial two-dimensional object image carrying a real label, as well as a first loss function and a second loss function, the first loss function is constructed based on the real label and the predicted label, and the second loss function is constructed based on the initial two-dimensional object image and the target two-dimensional object image; the target two-dimensional object image is generated by processing the initial two-dimensional object image with the three-dimensional reconstruction network model, and the predicted label is obtained by processing the target two-dimensional object image with a discriminator; the initial two-dimensional object image is an initial two-dimensional facial image, and the target two-dimensional object image is a target two-dimensional facial image; the real label of the initial two-dimensional facial image includes a first label and a second label; the predicted label of the target two-dimensional facial image includes the first label and the second label; the first label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are in a straight line; the second label indicates that the nose tip key point, the lip bead key point and the chin center key point in the facial image are not in a straight line.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 5 or the method according to claim 6.

14. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 5 or the method according to claim 6.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 5 or the method according to claim 6 is implemented.

Images