Image highlight processing method, device, electronic device and readable storage medium

Abstract

The present application discloses a method, device, electronic device, and readable storage medium for image highlight processing. The method comprises: obtaining a facial image to be processed and determining first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters; determining first facial mesh data corresponding to the facial image; determining a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameters, and the first facial mesh data; determining a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map; and performing pixel-by-pixel subtraction between the facial image and the facial highlight image to obtain a facial image after highlight removal. The image highlight processing method of the present application is generally applicable to facial images and improves the robustness of highlight removal.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image highlight processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] When photographing faces under certain lighting conditions, the resulting facial images often contain highlights. Because highlights in these areas obscure local facial features such as shape, color, and texture, they significantly interfere with facial image detection and recognition. Therefore, highlight removal is an important method for improving facial recognition rates. This process removes highlights from highlight areas and restores the facial color to its natural, non-highlighted state.

[0003] In the prior art, image features are usually extracted based on a facial image and a highlight area mask map and a highlight brightness map of the facial image are predicted; based on the above-extracted image features, the predicted highlight area mask map and the highlight brightness map of the facial image, a facial image after highlight removal is predicted.

[0004] However, in the prior art, when training neural network models for predicting the highlight region mask map, highlight brightness map, and highlight-removed face image, respectively, supervised learning is required using ground-truth data of face images without highlights as learning targets. However, due to the data distribution constraints of the ground-truth data, the highlight removal algorithm is only applicable to face images with the same data distribution as the ground-truth data and cannot be generalized to any face image, resulting in insufficient robustness in highlight removal for face images. For example, the face images collected for training with ground-truth data were all collected under laboratory lighting conditions. When the face image highlight removal model trained with this ground-truth data is applied to face images collected under outdoor lighting conditions, the output face image has poor highlight removal performance. Summary of the Invention

[0005] The present application provides an image highlight processing method, device, electronic device and computer-readable storage medium to improve the robustness of highlight removal in facial images.

[0006] In a first aspect, an embodiment of the present application provides a method for image highlight processing, the method comprising:

[0007] Acquire a facial image to be processed, and determine first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters;

[0008] Determining first face mesh data corresponding to the face image, where the first face mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the face image;

[0009] Determining a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter, and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize a highlight region of the facial image in the facial image, and the facial highlight ambient lighting map is used to characterize ambient lighting of the highlight region in the facial image;

[0010] determining a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map;

[0011] The face image and the face highlight image are subtracted pixel by pixel to obtain the face image after highlight removal.

[0012] In a second aspect, an embodiment of the present application provides an image highlight processing device, the device comprising:

[0013] An acquisition module, configured to acquire a facial image to be processed and determine first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters;

[0014] a first determining module, configured to determine first facial mesh data corresponding to the facial image, wherein the first facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the facial image;

[0015] a second determination module, configured to determine a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter, and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize a highlight region of the facial image, and the facial highlight ambient lighting map is used to characterize ambient lighting of the highlight region of the facial image;

[0016] a third determining module, configured to determine a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient illumination map;

[0017] The processing module is used to perform pixel-by-pixel subtraction on the face image and the face highlight image to obtain the face image after removing the highlights.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0019] A memory and a processor, wherein the memory and the processor are coupled;

[0020] The memory is used to store one or more computer instructions;

[0021] The processor is used to execute the one or more computer instructions to implement the image highlight processing method described in any one of the first aspects above.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having one or more computer instructions stored thereon, characterized in that the instruction is executed by a processor to implement the image highlight processing method described in any one of the above-mentioned first aspects.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image highlight processing method described in any one of the first aspects above.

[0024] Compared with the prior art, this application has the following advantages:

[0025] The image highlight processing method provided in the present application obtains a facial image to be processed and determines the first three-dimensional model parameters corresponding to the facial image. The first three-dimensional model parameters include first posture parameters; and the first facial mesh data corresponding to the facial image is determined. Based on the facial image, the first posture parameters, and the first facial mesh data, the facial highlight reflectance image and the facial highlight ambient lighting map corresponding to the facial image are determined. Based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map, the facial highlight image corresponding to the facial image is determined. The facial image and the facial highlight image are subtracted pixel by pixel to obtain the facial image after the highlights are removed.

[0026] Compared with the prior art, the present application determines the first three-dimensional model parameters corresponding to the facial image by means of three-dimensional reconstruction. Based on the first three-dimensional model parameters corresponding to the facial image, the facial highlight reflectance image and the facial highlight ambient lighting map corresponding to the facial image can be further determined by a deep learning method. Since the facial highlight reflectance image can represent the highlight reflectance of the facial image in the facial image, and the facial highlight ambient lighting map can represent the highlight intensity of the facial image, the highlight area of ​​the facial image and the ambient lighting intensity of the highlight area in the facial image are obtained. Based on the facial highlight reflectance image, the facial highlight ambient lighting map, and the first facial mesh data, a microscopic mirror rendering process is performed to obtain a facial highlight image corresponding to the facial image. The facial image and the facial highlight image are subtracted pixel by pixel to obtain a facial image after highlight removal. The image highlight processing method of the present application is universally applicable to any facial image, which improves the robustness of highlight removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0028] Figure 1 This is a flowchart of the image highlight processing method provided in the first embodiment of the present application;

[0029] Figure 2 One of the flowcharts of model training for the first prediction model and the second prediction model provided in the first embodiment of the present application;

[0030] Figure 3 The second flowchart of the model training process for the first prediction model and the second prediction model provided in the first embodiment of the present application is as follows;

[0031] Figure 4 This is a second flow chart of the image highlight processing method provided in the first embodiment of the present application;

[0032] Figure 5 A schematic diagram of the input and output of the first prediction model and the second prediction model provided in the first embodiment of the present application;

[0033] Figure 6 Schematic diagram of a low-frequency diffuse reflectance image of a face and a highlight-removed image provided in the first embodiment of the present application;

[0034] Figure 7 A schematic structural diagram of an image highlight processing device provided in a second embodiment of the present application;

[0035] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in the third embodiment of the present application.

[0036] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0037] To make the purposes, advantages, and features of this application more clear, the following clearly and completely describes this application in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of this application. However, the described embodiments are only some of the embodiments of this application, not all of them. All other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0038] It should be noted that, in the description of this application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance, or a specific order or precedence. For those skilled in the art, the specific meanings of the above terms in this application can be understood in specific circumstances. In addition, in the description of this application, unless otherwise specified, the term "plurality" refers to two or more. The term "and / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0039] In order to facilitate understanding of the technical solution of this application, the relevant concepts involved in this application are first introduced.

[0040] Monocular 2D image: A 2D image captured by a single camera.

[0041] Unconstrained face images: Face images have large variability in facial expression, scale, posture, appearance, etc.

[0042] Monocular Face Highlight Removal: Given an unconstrained monocular 2D image containing a face, remove the highlight portion of the face image, estimate, and restore the image region containing the original highlight portion to a non-highlight state. Highlight refers to the specular reflection phenomenon where a bright point light source illuminates a luminous surface, where the illuminated area on the surface reflects the highlight. Non-highlight refers to the diffuse reflection phenomenon where a bright point light source illuminates a non-luminous surface, where the bright light is dispersed across the illuminated area on the surface.

[0043] 3D Morphable Model (3DMM): This model consists of an average 3D model, a set of shape bases, and a set of texture bases. It linearly blends the shape and texture bases by inputting shape and texture parameters to generate a 3D face model with a new face shape and texture. The 3DMM is a classic statistical model of 3D face shape and texture, widely used in face analysis, model fitting, image synthesis, and other fields.

[0044] 3DMM reconstruction: Given an unconstrained monocular 2D image containing a face, the 3DMM reconstruction algorithm predicts 3DMM shape parameters and 3DMM texture parameters, reconstructing the face shape and low-frequency color image. Low-frequency color refers to skin color.

[0045] Differentiable rendering: A rendering method that uses differential equations to render 3D models into 2D images while preserving the gradient information of the rendered image. This method can be used to optimize neural networks based on the errors in the rendered image. It works by dividing the rendering process into multiple small steps, each of which can be expressed as a differential equation. The results of each small step are then obtained by solving the differential equation. Based on the rendering process and results, differentiable rendering is further divided into differentiable specular rendering and differentiable diffuse rendering. Differentiable specular rendering focuses on surface reflections and highlights. It uses physically based ray tracing technology to simulate the reflection and refraction of light on a surface, thereby generating images with realistic specular reflections. Differentiable diffuse rendering focuses on surface scattering and diffuse reflection, using scattering coefficient-based techniques to simulate the scattering and absorption of material surfaces, thereby generating low-frequency images with soft morphology.

[0046] Deep learning algorithm: A machine learning algorithm composed of large-scale neurons. Based on its advantage of being able to effectively solve complex nonlinear problems, it is currently widely used in many fields such as computer vision, speech recognition, and natural language processing.

[0047] Mean squared error: A loss function commonly used in deep learning to measure the bi-normal distance between two tensors.

[0048] Binary cross entropy error: A loss function commonly used in deep learning to measure the error of 0-1 classification results.

[0049] Mapping coordinates: A coordinate system used to align 2D images with 3D meshes. Each triangle in a 3D mesh model corresponds to a 2D coordinate on the 2D image. The mapping coordinate system is UVW, where U and V are two perpendicular vectors on the image plane, and W is a vector perpendicular to the image plane. In other words, in an image, U represents the horizontal direction, V represents the vertical direction, and W represents the direction perpendicular to the image plane.

[0050] Cubemap: A method of mapping three-dimensional direction vectors to points on the surface of a cube. The mapping is determined by emitting a ray from the center of the cube along each vector direction and the first point on the cube surface that hits it. Cubemaps can be used to model ambient lighting.

[0051] To address the problems of the prior art, the present application provides an image highlight processing method, an image highlight processing device corresponding to the method, an electronic device capable of implementing the image highlight processing method, and a computer-readable storage medium. The following examples provide detailed explanations of the method, device, electronic device, and computer-readable storage medium.

[0052] In order to make the purpose and technical solution of the present application clearer and more intuitive, the method provided by the embodiment of the present application will be described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It is understood that the following embodiments may exist separately, and the following embodiments and features in the embodiments may be combined with each other when there is no conflict between the embodiments provided in the present application. For the same or similar content, the description will not be repeated in different embodiments. In addition, the step sequence in the following method embodiments is only an example and is not strictly limited. In some cases, the steps shown or described may be performed in a different order from this.

[0053] The present application provides an image highlight processing method, device, electronic device and computer-readable storage medium. Specifically, the image highlight processing method of one embodiment of the present application can be executed by a computer device, wherein the computer device can be a terminal or server device. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer, a personal digital assistant, etc. The terminal can also include a client, which can be an image highlight processing client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0054] Next, combine Figure 1 , the image highlight processing method provided by the first embodiment of the present application is described, Figure 1 This is one of the flowcharts of the image highlight processing method provided in the first embodiment of the present application.

[0055] like Figure 1 As shown, steps S101-S105 are included:

[0056] S101: Acquire a face image to be processed, and determine first three-dimensional model parameters corresponding to the face image.

[0057] In the embodiment of the present application, the face image to be processed is a two-dimensional face image that needs to be subjected to image highlight removal processing. The two-dimensional face image can be any face image with highlight distribution.

[0058] In an optional implementation, a three-dimensional face reconstruction algorithm may be used on the facial image to be processed to predict the first three-dimensional model parameters corresponding to the facial image. The three-dimensional face reconstruction algorithm may predict the three-dimensional model parameters corresponding to the facial image based on the facial image. For example, the three-dimensional model parameters include three-dimensional posture parameters. The three-dimensional posture parameters are used to control the posture transformation of the three-dimensional face model in three-dimensional space, and include translation parameters and rotation parameters. The translation parameters are used to control the position of the three-dimensional model in the world coordinate system, and the position of the three-dimensional model is adjusted by adjusting the translation parameters; the rotation parameters are parameters used to control the rotation of the three-dimensional model in the world coordinate system, and are usually expressed in Euler angles. The rotation parameters are adjusted to achieve the rotation change of the model in three-dimensional space, thereby adjusting the posture of the three-dimensional model. For example, the three-dimensional face reconstruction algorithm may be a Deep 3D algorithm.

[0059] In an embodiment of the present application, the first posture parameter corresponding to the face image to be processed can be predicted by a three-dimensional face reconstruction algorithm.

[0060] It should be noted that the 3D Deformable Model (3DMM) is a universal 3D face model that represents a face using a fixed number of vertices and a collection of polygonal faces connected between them. Its core concept is that 3D faces can be matched one-to-one in 3D space and can be derived by weighted linear addition of shape bases and texture bases of many other face meshes. A shape base is a set of deviation models relative to a mean shape. In the 3DMM, different shape bases represent 3D faces of varying shapes. By weighted linear superposition of these shape bases, 3D face models of various sizes and shapes can be generated. The texture base represents variations in color and texture attributes and is used to describe the texture characteristics of an object's surface. The texture base captures variations in the mean texture. In the 3DMM, weighted linear superposition of these texture bases generates different texture maps to represent variations in texture attributes and color distribution.

[0061] The basic attributes of a 3D face model are shape and texture, so each 3D face can be represented as a linear superposition of a shape vector and a texture vector. Texture information can reflect facial color information.

[0062] Among them, the shape vector S=(x1,y1,z1,x2,y2,z2,...,x n ,y n ,z n), texture vector T=(R1,G1,B1,R2,G2,B2,...,R n ,G n ,B n ).

[0063] Any 3D face model can be formed by weighted combination of N 3D face models in the 3DMM algorithm model library. As follows:

[0064] in,

[0065] Where: S i and T i is the shape vector and texture vector of the i-th 3D face model in the 3DMM algorithm model library, a i and b i are the parameters of the shape vector and texture vector of the i-th 3D face model respectively.

[0066] Since the S i and T i They are not orthogonally related, so S and T cannot be used directly to build the model. We need to use principal component analysis (PCA) to reduce the dimension of S and T and get the following:

[0067]

[0068] in, and are the average values ​​of shape vectors and texture vectors of all 3D face models in the 3DMM algorithm model library, s i and t i S i and T i The eigenvectors of the covariance matrix after subtracting their respective means, the three-dimensional shape parameters of the 3DMM = (α1, α2, ..., α m-1 ), 3D texture parameters = (β1,β2,...,β m-1 ).

[0069] S102: Determine first face mesh data corresponding to the face image, where the first face mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the face image.

[0070] In an embodiment of the present application, a first shape parameter corresponding to the facial image to be processed can be predicted using a 3D face reconstruction algorithm. The first shape parameter is a 3D shape parameter used to reconstruct the shape of a 3D face model for the facial image to be processed. The 3D shape parameters include identity parameters and expression parameters. Identity parameters are parameters used to represent the identity of a person in a person image, such as an elderly person, child, man, or woman. Expression parameters are parameters used to represent the facial expression of a person in a person image, such as crying, laughing, or anger.

[0071] In an embodiment of the present application, a face mesh is a method for representing a three-dimensional face shape, which represents the shape of a three-dimensional face by using a surface formed by connecting vertices.

[0072] It should be noted that the face mesh is the 3D mesh that constitutes the three-dimensional face model. The mesh is composed of polygons, and a complex polygon is composed of multiple triangular faces. Therefore, the surface of the 3D model is composed of multiple interconnected triangular faces. The set of points that constitute the triangular faces and the edges of the triangles in three-dimensional space is the mesh. Mesh data contains vertices, faces, and boundaries. Among them, a vertex is a point in space that describes the shape of the three-dimensional face model. Each vertex has three coordinate values, representing its position in the x, y, and z directions. A face is a planar polygon composed of three or more vertices, which describes the surface of the three-dimensional model. Each face is composed of one or more triangles, which includes the sequence of vertices that make up each triangle. The boundary is the line connecting the vertices and describes the edge of the three-dimensional model.

[0073] In an embodiment of the present application, the first shape parameter of the face image can be used by a 3D face reconstruction algorithm to obtain first face mesh data corresponding to the face image. The 3D face reconstruction algorithm includes but is not limited to the Deep 3D algorithm.

[0074] S103: Determine a facial highlight reflectance image and a facial highlight ambient illumination map corresponding to the facial image according to the facial image, the first posture parameter, and the first facial mesh data.

[0075] Under lighting conditions, certain parts of the face (such as the nose, forehead, cheeks, etc.) will reflect light, and these reflected lights are called highlights.

[0076] In an embodiment of the present application, a facial highlight reflectance image is used to characterize the highlight area of ​​a facial image in a facial image. That is, the facial highlight reflectance image can reflect the highlight reflection of the facial image in the facial image. A facial highlight ambient lighting map is used to characterize the ambient lighting of the highlight area in the facial image. It is understandable that when taking a facial photo, different light source directions cause the highlight area of ​​the face to reflect different brightness and colors. Therefore, by obtaining ambient lighting information of the highlight area in the facial image, a more accurate understanding of the highlight area of ​​the facial image can be achieved.

[0077] Optionally, the implementation of step S103 may include steps S1031 - S1033 .

[0078] S1031. Perform posture adjustment on the first facial mesh data according to the first posture parameter to obtain second facial mesh data corresponding to the facial image, where the second facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with the shape of the face in the facial image and a posture consistent with the posture of the face in the facial image.

[0079] The first posture parameter includes a translation parameter and a rotation parameter.

[0080] In an embodiment of the present application, first facial mesh data can be calculated using a 3D facial reconstruction algorithm based on the first shape parameter corresponding to the facial image. Vertex coordinates of the mesh in the first facial mesh data are vertex positions in a standard pose. Based on the predicted first pose parameter corresponding to the facial image, the facial mesh represented by the first facial mesh data is translated and rotated to generate second facial mesh data. In other words, the pose of the facial mesh represented by the second facial mesh data is consistent with the pose of the face in the facial image.

[0081] S1032. Determine a UV rearrangement map and a cube rearrangement map corresponding to the face image based on the second face mesh data and the face image, wherein the UV rearrangement map is a UV position map corresponding to the facial image in the face image, and the cube rearrangement map is a panoramic map corresponding to the cube image of the facial image in the face image.

[0082] Optionally, the implementation of step S1032 includes steps S1-S2.

[0083] S1. Align the second face mesh data with the face image to determine the mapping coordinates corresponding to each pixel in the face image. The second face mesh data includes multiple vertices, and each vertex corresponds to a mapping coordinate.

[0084] S2. Determine a UV rearrangement map and a cube rearrangement map corresponding to the face image according to the mapping coordinates corresponding to each pixel of the face image and the face image.

[0085] Optionally, the mapping coordinates of the second facial mesh data are UV expanded to obtain a UV position map; for each first mapping coordinate in the UV position map corresponding to a pixel in the facial image, the pixel color of the first mapping coordinate in the UV position map is determined as the pixel color of the pixel corresponding to the first mapping coordinate. For each second mapping coordinate in the UV position map other than the first mapping coordinate, the pixel color of the second mapping coordinate in the UV position map is determined as the noise pixel color. The UV position map, after determining the pixel colors of each first mapping coordinate and each second mapping coordinate, is determined as a UV rearrangement map corresponding to the facial image.

[0086] A noise pixel refers to random noise in an image. Its pixel value is randomly determined. Compared to non-noise pixels, the pixel value of a noise pixel is typically higher, potentially many times higher than the values ​​of neighboring pixels, resulting in a significant difference. The color of a noise pixel is the color corresponding to the pixel value of the noise pixel.

[0087] In the embodiment of the present application, each second mapping coordinate in the UV position map can be filled by selecting different types of noise pixels. Different types of noise pixels include but are not limited to Gaussian noise, salt and pepper noise, etc., and the embodiment of the present application does not impose any limitation on this.

[0088] In some embodiments, based on the camera parameters used in the three-dimensional face reconstruction algorithm (which is a fixed set of parameters), the second face mesh corresponding to the second face mesh data after posture alignment is projected onto the face image, and each pixel in the facial image on the face image corresponds to a point on the surface of the second face mesh. For each vertex on the second face mesh, the mapping coordinates are fixed in advance. Therefore, for each pixel, the coordinates of the vertex corresponding to the pixel are determined as the mapping coordinates corresponding to the pixel. Therefore, the mapping coordinates of the face image aligned to the second face mesh are obtained. Subsequently, the facial pixel colors in the two-dimensional image are arranged according to the mapping coordinates, and noise pixels are filled in the empty areas to obtain the UV rearrangement map corresponding to the face image.

[0089] In some embodiments, a line of sight reflection direction for each target mapping coordinate in the first facial mesh data is determined, where the target mapping coordinate is the mapping coordinate corresponding to a pixel in the facial image. For each target mapping coordinate in the first facial mesh data, a pixel value of the image at the target mapping coordinate in the first facial mesh data is determined as the pixel value of the pixel corresponding to the target mapping coordinate. A cube image corresponding to the facial region in the first facial mesh data is determined. The cube image is converted into a panoramic image to obtain a permuted cube image corresponding to the facial image.

[0090] In this embodiment of the present application, for any point on the first facial mesh, its sight line direction is the vector from the point's position to the camera's position. The camera position can be determined based on position parameters from a fixed set of camera parameters. The symmetric direction of the sight line direction, with the surface normal of the point as the axis of symmetry, is determined as the sight line reflection direction of the point. After determining the sight line reflection direction for each point in the first facial mesh, the facial pixel colors in the facial image are mapped and expanded into a panoramic image to obtain a cube permutation image.

[0091] Specifically, given a cube centered at the origin and with an axis length of 1, a ray is emitted from each point on the first face grid along the corresponding line of sight reflection direction. This ray intersects the cube at a point on one of the cube's six faces. The color of the face corresponding to that point on the first face grid is assigned to that pixel on the cube. Finally, the six faces of the cube are unfolded and converted into a panoramic image, resulting in the permuted cube image.

[0092] S1033 : Determine a facial highlight reflectance image and a facial highlight ambient illumination image corresponding to the facial image according to the UV rearrangement map and the cube rearrangement map.

[0093] In some embodiments, the first prediction model and the second prediction model can be pre-trained and set. The first prediction model and the second prediction model can be obtained based on generative adversarial network training. The first prediction model and the second prediction model are both composed of a combination of an encoder and a decoder. The encoder is used to extract feature information of the input image, which features each pixel of the input image and compresses it into a low-dimensional feature vector. The decoder is used to gradually restore the feature vector processed by the encoder into an output image. Each module in the decoder consists of an upsampling layer (upsampling), a convolutional layer, and a conditional initialization module.

[0094] Optionally, the UV rearrangement map is input into a first prediction model, so that the first prediction model outputs a facial highlight reflectance image corresponding to the facial image. The first prediction model is a model for predicting the facial highlight reflectance image. The facial highlight reflectance image is used to represent highlight areas of the facial image in the facial image.

[0095] Optionally, the cube permutation image is input into a second prediction model, so that the second prediction model outputs a facial highlight reflectance image corresponding to the facial image. The second prediction model is a model for predicting a facial highlight ambient lighting map. The facial highlight ambient lighting map is used to represent the ambient lighting of the highlight region of the facial image in the facial image.

[0096] S104: Determine a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient illumination map.

[0097] An optional implementation method performs a differentiable specular reflection rendering process based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient illumination map to obtain a facial highlight image corresponding to the facial image. The differentiable specular reflection rendering process operates by inputting the geometric information of the scene into an image generation model, and then outputting a rendered image after model inference. Specifically, the image generation model inputs the scene's geometric information, lighting conditions, material properties, etc., and outputs a realistic rendered image. During the training process of the image generation model, a large amount of real data is used to train the model to improve its rendering effect and quality.

[0098] In the embodiments of the present application, the differentiable specular reflection rendering algorithm used can be implemented based on the NVdiffrast library and the Cook-Torrance BRDF illumination reflection model. The NVdiffrast library can utilize GPU-accelerated ray tracing operations to achieve efficient image rendering. The Cook-Torrance BRDF model is a lighting model used to calculate material surface reflections.

[0099] S105 , performing pixel-by-pixel subtraction on the face image and the face highlight image to obtain a face image with highlights removed.

[0100] The image highlight processing method provided in the embodiment of the present application obtains a facial image to be processed and determines a first three-dimensional model parameter corresponding to the facial image; wherein the first three-dimensional model parameter includes a first posture parameter; determines first facial mesh data corresponding to the facial image; determines a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize the highlight reflectance of the facial image in the facial image, and the facial highlight ambient lighting map is used to characterize the highlight intensity of the facial image; determines a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image and the facial highlight ambient lighting map; performs pixel-by-pixel subtraction on the facial image and the facial highlight image to obtain a facial image with the highlights removed.

[0101] Compared with the existing technology, the present application determines the first three-dimensional model parameters corresponding to the facial image by means of three-dimensional reconstruction. Based on the first three-dimensional model parameters corresponding to the facial image, the facial highlight reflectance image and the facial highlight ambient lighting map corresponding to the facial image can be further determined by a deep learning method. Since the facial highlight reflectance image can represent the highlight area of ​​the facial image in the facial image, and the facial highlight ambient lighting map can represent the ambient lighting of the highlight area of ​​the facial image, the highlight area of ​​the facial image in the facial image and the ambient lighting intensity of the highlight area are obtained. Based on the facial highlight reflectance image, the facial highlight ambient lighting map, and the first facial mesh data, a microscopic mirror reflection rendering process is performed to obtain the facial highlight image corresponding to the facial image. The facial image and the facial highlight image are subtracted pixel by pixel to obtain the facial image after the highlight is removed. The image highlight processing method of the present application is universally applicable to any facial image, which improves the robustness of highlight removal.

[0102] Next, combine Figure 2 , the training process of the first prediction model and the second prediction model provided in the first embodiment of the present application is described, Figure 2 This is one of the flowcharts for performing model training on the first prediction model and the second prediction model provided in the first embodiment of the present application.

[0103] like Figure 2 As shown, steps S201-S204 are included:

[0104] S201: Construct a first prediction model and a second prediction model.

[0105] In the embodiment of the present application, the first prediction model and the second prediction model may be composed of an encoder and a decoder. Specifically, the network structure of the first prediction model and the second prediction model may be the U-NET network structure in the Pix2pixHD image generation model.

[0106] S202: Obtain a sample face image.

[0107] S203: Determine second three-dimensional model parameters corresponding to the face sample image.

[0108] In an embodiment of the present application, the second three-dimensional model parameters corresponding to the face sample image may include three-dimensional posture parameters, three-dimensional shape parameters, three-dimensional texture parameters, and diffuse reflection lighting parameters, namely, second posture parameters, second shape parameters, second texture parameters and second diffuse reflection lighting parameters.

[0109] S204: Train the first prediction model and the second prediction model according to the face sample image and the second three-dimensional model parameters to obtain trained first prediction models and second prediction models.

[0110] Optionally, step S204 includes the following steps S2041-S2047.

[0111] S2041. Determine, based on the face sample image and the second three-dimensional model parameters, a face low-frequency diffuse reflection image, a UV rearrangement map, and a cube rearrangement map corresponding to the face sample image.

[0112] Optionally, third facial mesh data corresponding to the facial sample image is calculated based on the second shape parameter; a low-frequency facial color image corresponding to the facial sample image is calculated based on the second texture parameter. Based on the second diffuse illumination parameter, the third facial mesh data, and the low-frequency facial color image, differentiable diffuse reflection rendering is performed to obtain a low-frequency facial diffuse reflection image corresponding to the facial sample image.

[0113] In the embodiment of the present application, the implementation method of calculating the third face mesh data corresponding to the face sample image based on the second shape parameter can refer to the description and explanation of step S102, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0114] In the embodiment of the present application, the low-frequency color image of the face corresponding to the face sample image is used to represent the overall required color information of the facial image in the face sample image.

[0115] In some embodiments, the base color of each triangle can be solved by interpolation using the triangular mesh structure of the three-dimensional face model and the second texture parameters of the face sample image. Afterwards, these base colors are smoothed using a certain method and combined with the edge information of the triangle to generate a low-frequency color image of the face corresponding to the face sample image. Based on the second diffuse reflection illumination parameter, the third face mesh data and the low-frequency color image of the face, a differentiable diffuse reflection rendering algorithm is used to obtain a low-frequency diffuse reflection image of the face. The differentiable diffuse reflection rendering algorithm can be a rendering and shading method using a Lambert surface. Furthermore, in order to achieve further modification and optimization of the differentiable properties, the NVdiffrast library can be used for differentiable diffuse reflection rendering.

[0116] Optionally, regarding the description and explanation of "determining the UV rearrangement map and the cube rearrangement map" in step S2041, you can refer to the above description and explanation of step S1032, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0117] S2042: Input the UV rearrangement map corresponding to the face sample image into the first prediction model, so that the first prediction model outputs a face highlight reflectance image corresponding to the face sample image.

[0118] S2043: Input the cube rearrangement map corresponding to the face sample image into the second prediction model, so that the second prediction model outputs the face highlight environment illumination map corresponding to the face sample image.

[0119] S2044: Determine third face mesh data corresponding to the face sample image, where the third face mesh data is mesh data of a three-dimensional mesh having a shape consistent with the face shape in the face sample image;

[0120] S2045: Determine a facial highlight image corresponding to the facial sample image based on the third facial grid data, the facial highlight reflectance image corresponding to the facial sample image, and the facial highlight ambient illumination map.

[0121] In an embodiment of the present application, a 3D face reconstruction algorithm can be used to obtain first face mesh data corresponding to the face image using the second shape parameter of the face sample image. Based on the third face mesh data, the face highlight reflectance image corresponding to the face sample image, and the face highlight ambient illumination map, a microscopic mirror reflection rendering process is performed to obtain a face highlight image corresponding to the face sample image.

[0122] S2046. Based on a preset loss function, calculate a loss value according to the low-frequency diffuse reflection image and the highlight image of the face corresponding to the face sample image.

[0123] In an embodiment of the present application, the preset loss function includes but is not limited to a mean square error function and a binary cross entropy error function.

[0124] For example, using the mean squared error (MSE) loss function as the default loss function, the low-frequency color image and the highlight image of the face are added pixel by pixel to obtain a first sample face image. The first sample face image and the sample face image are subtracted pixel by pixel to obtain a target difference value corresponding to each pixel. Using the MSE loss function, the target difference value corresponding to each pixel is solved for the MSE value to obtain the loss value.

[0125] S2047. Update the model parameters of the first prediction model and the second prediction model based on the loss value.

[0126] In this embodiment of the present application, based on the loss value, the parameters of the model parameters of the first prediction model and the second prediction model are updated simultaneously. The training is completed until the number of training times reaches a preset iteration threshold. Otherwise, steps S201-S204 are repeated.

[0127] Next, combine Figure 3 and one A specific example is used to further illustrate the training process of the first prediction model and the second prediction model provided in the embodiment of the present application. Figure 3The second flowchart of the model training of the first prediction model and the second prediction model provided in the first embodiment of the present application.

[0128] S301, obtaining a two-dimensional face image I t .

[0129] S302, two-dimensional face image I t A 3D face reconstruction algorithm (ie, a 3DMM reconstruction algorithm) is used to predict 3DMM parameters used to establish a 3D face model, including 3DMM pose parameters, 3DMM shape parameters, 3DMM texture parameters, and diffuse illumination parameters.

[0130] S303, based on the 3DMM shape parameters, establish a three-dimensional face model S; based on the 3DMM texture parameters, calculate the low-frequency color image T of the face D ; Use the differentiable diffuse rendering method to transform the 3D face model S and the low-frequency color image T of the face D , rendered as a low-frequency diffuse reflection image of the face

[0131] S304: Use the 3DMM posture parameters in step S303 to align the three-dimensional face model S so that the posture of the three-dimensional face model is consistent with the facial posture in the face image.

[0132] S305: Align the 2D face image with the 3D face model after adjusting the posture in step S304 to obtain the mapping coordinates of the 2D image aligned with the 3D face model. Arrange the face pixel colors in the 2D image according to the mapping coordinates and fill the empty areas with noise pixels to obtain the UV rearrangement map M. uv .

[0133] S306: Map the color of the face pixels in the two-dimensional image according to the line of sight reflection direction of each point of the three-dimensional model S, and expand it into a panoramic image to obtain a cube rearrangement image M. cube .

[0134] S307, rearrange the UV generated in step S305 into the graph M uv As input, the trained neural network NET_1 (i.e., the first prediction model) is used to predict the high light reflectance image T of the face s .

[0135] S308, rearrange the cube generated in step S306 into graph M cube As input, the trained neural network NET_2 (i.e., the second prediction model) is used to predict the face highlight environment illumination map T of the face l .

[0136] S309, (Calculate loss function) Based on the three-dimensional face shape S obtained in step S303 above, predict the high light reflectance image T of the face s And the predicted face highlight ambient lighting map T l , the highlight image of the face is obtained by differentiable specular reflection rendering

[0137] S310, calculating the face highlight image and low-frequency diffuse reflection image of the face The sum of the addition and the input two-dimensional image I t The pixel-by-pixel error between as the rendering loss function.

[0138] S311. Use the above steps S301-S310 for each input image to calculate the rendering loss function L1, and use the gradient descent algorithm to simultaneously train the neural networks NET_1 and NET_2 to optimize the parameters of the neural networks NET_1 and NET_2.

[0139] S312: Determine whether the maximum number of iterations has been reached. If so, execute step S312 to complete the training and save the parameters of the neural networks NET_1 and NET_2. Otherwise, execute step S301.

[0140] Next, combine Figure 4-Figure 6 The image highlight processing method provided by the embodiment of the present application is further explained with a specific example. Figure 4 This is the second flow chart of the image highlight processing method provided in the first embodiment of the present application. Figure 5 Schematic diagram of the input and output of the first prediction model and the second prediction model provided in the first embodiment of the present application, Figure 6 Schematic diagram of the low-frequency diffuse reflection image and highlight-removed image of a face provided in the first embodiment of the present application.

[0141] S401, obtaining a two-dimensional face image I t .

[0142] S402, two-dimensional face image I t Perform a three-dimensional face reconstruction algorithm (3DMM reconstruction algorithm) to predict the 3DMM parameters used to establish a three-dimensional face model, including 3DMM posture parameters and 3DMM shape parameters.

[0143] S403: Align the three-dimensional face model S using the 3DMM posture parameters in step S401 so that the posture of the three-dimensional face model is consistent with the facial posture in the face image.

[0144] S404: Align the 2D face image with the 3D face model after adjusting the posture in step S403 to obtain the mapping coordinates of the 2D image aligned with the 3D face model. Arrange the face pixel colors in the 2D image according to the mapping coordinates and fill the empty areas with noise pixels to obtain the UV rearrangement map M. uv .

[0145] S405: Map the color of the face pixels in the two-dimensional image according to the line of sight reflection direction of each point of the three-dimensional model S, and expand it into a panoramic image to obtain a cube rearrangement image M. cube ,

[0146] S406, rearrange the UV generated in step S404 into the graph M uv As input, the trained neural network NET_1 (i.e., the first prediction model) is used to predict the high light reflectance image T of the face s .

[0147] S407, rearrange the cube generated in step S405 into the graph M cube As input, the trained neural network NET_2 (i.e., the second prediction model) is used to predict the face highlight environment illumination map T l .

[0148] like Figure 5 As shown, the UV permutation map corresponding to the two-dimensional face image is input to the neural network NET_1, and the neural network NET_1 outputs the predicted face highlight reflectance image corresponding to the two-dimensional face image. Also, the cube permutation map corresponding to the two-dimensional face image is input to the neural network NET_2, and the neural network NET_2 outputs the predicted face highlight ambient illumination map corresponding to the two-dimensional face image.

[0149] S408, using microscopic reflection rendering, the three-dimensional face shape S obtained in step S403 is used to predict the high light reflectance image T of the face. s And the predicted face highlight ambient lighting map T l , rendered as a facial highlight image

[0150] S409: input the face image I t The highlight rendering image of the face obtained in step S408 above Perform pixel-by-pixel subtraction to obtain the image after removing the face highlights

[0151] like Figure 6 As shown in the figure, highlight removal processing is performed on two two-dimensional face images. The first column is the original two-dimensional face image, the middle column is the low-frequency diffuse reflection image of the face corresponding to the original two-dimensional face image, and the last column is the two-dimensional face image after highlight removal processing is performed on the original two-dimensional face image.

[0152] The image highlight processing device provided in the present application is described below. The image highlight processing device described below and the image highlight processing method described above can be referenced to each other.

[0153] Figure 7 This is a schematic diagram of the structure of the image highlight processing device provided in the second embodiment of the present application. Figure 7 As shown, the image highlight processing device includes: an acquisition module 701 , a first determination module 702 , a second determination module 703 , a third determination module 704 and a processing module 705 .

[0154] An acquisition module, configured to acquire a facial image to be processed and determine first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters;

[0155] a first determining module, configured to determine first facial mesh data corresponding to the facial image, wherein the first facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the facial image;

[0156] a second determination module, configured to determine a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter, and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize a highlight region of the facial image, and the facial highlight ambient lighting map is used to characterize ambient lighting of the highlight region of the facial image;

[0157] a third determining module, configured to determine a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map;

[0158] The processing module is used to perform pixel-by-pixel subtraction on the face image and the face highlight image to obtain the face image after removing the highlights.

[0159] Optionally, the third determining module is specifically configured to:

[0160] Based on the first facial mesh data, the facial highlight reflectance image and the facial highlight ambient lighting map, microscopic mirror reflection rendering processing is performed to obtain a facial highlight image corresponding to the facial image.

[0161] Optionally, the second determining module is specifically configured to:

[0162] performing posture adjustment on the first facial mesh data according to the first posture parameter to obtain second facial mesh data corresponding to the facial image, wherein the second facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with the shape of the face in the facial image and a posture consistent with the posture of the face in the facial image;

[0163] Determining, based on the second facial mesh data and the facial image, a UV rearrangement map and a cube rearrangement map corresponding to the facial image, wherein the UV rearrangement map is a UV position map corresponding to the facial image in the facial image, and the cube rearrangement map is a panoramic map corresponding to the cube image of the facial image in the facial image;

[0164] According to the UV rearrangement map and the cube rearrangement map, a facial highlight reflectance image and a facial highlight ambient illumination map corresponding to the facial image are determined.

[0165] Optionally, the second determining module is specifically configured to:

[0166] Inputting the UV rearrangement map into a first prediction model so that the first prediction model outputs a facial highlight reflectance image corresponding to the facial image; wherein the first prediction model is a model for predicting a facial highlight reflectance image;

[0167] The cube rearrangement map is input into a second prediction model so that the second prediction model outputs a facial highlight ambient lighting map corresponding to the facial image; wherein the second prediction model is a model for predicting a facial highlight ambient lighting map.

[0168] Optionally, the second determining module is specifically configured to:

[0169] Aligning the second facial mesh data with the facial image to determine a mapping coordinate corresponding to each pixel in the facial image; wherein the second facial mesh data includes a plurality of vertices, each vertex corresponding to a mapping coordinate;

[0170] According to the mapping coordinates corresponding to each pixel of the facial image and the facial image, a UV rearrangement map and a cube rearrangement map corresponding to the facial image are determined.

[0171] Optionally, the second determining module is specifically configured to:

[0172] Perform UV unfolding on the second facial mesh data to obtain a UV position map;

[0173] For each first mapping coordinate in the UV position map corresponding to a pixel in the facial image, determining a pixel color of the first mapping coordinate in the UV position map as a pixel color of the pixel corresponding to the first mapping coordinate;

[0174] For each second mapping coordinate other than the first mapping coordinate in the UV position map, a pixel color of the second mapping coordinate in the UV position map is determined as a noise pixel color to obtain a UV rearrangement map corresponding to the face image.

[0175] Optionally, the second determining module is specifically configured to:

[0176] Determining a line of sight reflection direction of each target mapping coordinate in the first facial mesh data, where the target mapping coordinate is a mapping coordinate corresponding to a pixel in the facial image;

[0177] For each target mapping coordinate in the first face mesh data, determining a pixel value of an image at the target mapping coordinate in the first face mesh data as a pixel value of the pixel corresponding to the target mapping coordinate;

[0178] determining a cube image corresponding to a facial region in the first face grid data;

[0179] The cube image is converted into a panoramic image to obtain a cube rearrangement image corresponding to the face image.

[0180] Optionally, the device further includes a training module, wherein the training module is specifically configured to:

[0181] Constructing the first prediction model and the second prediction model;

[0182] Obtain a sample face image;

[0183] Determining second three-dimensional model parameters corresponding to the face sample image;

[0184] The first prediction model and the second prediction model are trained according to the face sample image and the second three-dimensional model parameters to obtain the trained first prediction model and the second prediction model.

[0185] Optionally, the training module is specifically used to:

[0186] Determining, based on the sample face image and the second three-dimensional model parameters, a low-frequency diffuse reflection image, a UV rearrangement map, and a cube rearrangement map corresponding to the sample face image; wherein the low-frequency diffuse reflection image is used to characterize the reflection of low-frequency components in the facial image under diffuse reflection lighting conditions;

[0187] Inputting the UV rearrangement map corresponding to the face sample image into the first prediction model, so that the first prediction model outputs a face highlight reflectance image corresponding to the face sample image;

[0188] Inputting the cube rearrangement map corresponding to the face sample image into the second prediction model, so that the second prediction model outputs a face highlight environment illumination map corresponding to the face sample image;

[0189] Determining third face mesh data corresponding to the face sample image, wherein the third face mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the face sample image;

[0190] Determining a facial highlight image corresponding to the facial sample image based on the third facial grid data, the facial highlight reflectance image corresponding to the facial sample image, and the facial highlight ambient illumination map;

[0191] Calculating a loss value based on a preset loss function and the low-frequency diffuse reflection image and the highlight image of the face corresponding to the face sample image;

[0192] Based on the loss value, model parameters of the first prediction model and the second prediction model are updated.

[0193] Optionally, the second three-dimensional model parameters include a second shape parameter, a second texture parameter, and a second diffuse reflection illumination parameter;

[0194] The training module is specifically used for:

[0195] Calculating third face mesh data corresponding to the face sample image according to the second shape parameter;

[0196] Calculating a low-frequency color image of the face corresponding to the face sample image according to the second texture parameter;

[0197] Based on the second diffuse reflection illumination parameter, a differentiable diffuse reflection rendering process is performed on the third face mesh data and the low-frequency color image of the face to obtain a low-frequency diffuse reflection image of the face corresponding to the face sample image.

[0198] The image highlight processing device provided in this embodiment can be used to execute the technical solution of the above-mentioned image highlight processing method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0199] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in the third embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 800 of this embodiment includes: a processor 801 and a memory 802;

[0200] Memory 802, for storing computer-executable instructions;

[0201] The processor 801 is configured to execute computer-executable instructions stored in the memory to implement the various steps of the image highlight processing method in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0202] Optionally, the memory 802 may be independent or integrated with the processor 801 .

[0203] When the memory 802 is independently provided, the electronic device further includes a bus 803 for connecting the memory 802 and the processor 801 .

[0204] The fourth embodiment of the present application further provides a computer-readable storage medium, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the technical solution corresponding to the image highlight processing method in any of the above embodiments executed by the above electronic device is implemented.

[0205] The fifth embodiment of the present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the technical solution corresponding to the image highlight processing method in any of the above embodiments.

[0206] It should be noted that for the detailed description of the devices, electronic devices, computer-readable storage media, and computer program products provided in the second, third, fourth, and fifth embodiments of the present application, reference can be made to the relevant description of the first embodiment of the present application, and no further details will be given here.

[0207] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0209] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in various embodiments of the present application.

[0210] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0211] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0212] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0213] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0214] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing image highlights, characterized in that: The method comprises: Acquire a facial image to be processed, and determine first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters; Determining first face mesh data corresponding to the face image, where the first face mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the face image; Determining a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter, and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize a highlight region of the facial image in the facial image, and the facial highlight ambient lighting map is used to characterize ambient lighting of the highlight region in the facial image; determining a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map; Perform pixel-by-pixel subtraction on the facial image and the facial highlight image to obtain the facial image after highlight removal.

2. The method according to claim 1, characterized in that The determining, based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient illumination map, of a facial highlight image corresponding to the facial image includes: Based on the first facial mesh data, the facial highlight reflectance image and the facial highlight ambient lighting map, microscopic mirror reflection rendering processing is performed to obtain a facial highlight image corresponding to the facial image.

3. The method according to claim 2, characterized in that The determining, based on the facial image, the first posture parameter, and the first facial mesh data, a facial highlight reflectance image and a facial highlight ambient illumination map corresponding to the facial image includes: performing posture adjustment on the first facial mesh data according to the first posture parameter to obtain second facial mesh data corresponding to the facial image, wherein the second facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with the shape of the face in the facial image and a posture consistent with the posture of the face in the facial image; Determining a UV rearrangement map and a cube rearrangement map based on the second face mesh data and the face image, wherein the UV rearrangement map is a UV position map corresponding to the facial image in the face image, and the cube rearrangement map is a panoramic map corresponding to the cube image of the facial image in the face image; According to the UV rearrangement map and the cube rearrangement map, a facial highlight reflectance image and a facial highlight ambient illumination map corresponding to the facial image are determined.

4. The method according to claim 3, characterized in that The determining, based on the UV rearrangement map and the cube rearrangement map corresponding to the face image, a face highlight reflectance image and a face highlight ambient illumination map corresponding to the face image, includes: Inputting the UV rearrangement map into a first prediction model so that the first prediction model outputs a facial highlight reflectance image corresponding to the facial image; wherein the first prediction model is a model for predicting a facial highlight reflectance image; The cube rearrangement map is input into a second prediction model so that the second prediction model outputs a facial highlight ambient lighting map corresponding to the facial image; wherein the second prediction model is a model for predicting a facial highlight ambient lighting map.

5. The method according to claim 3, characterized in that The determining, based on the second facial mesh data and the facial image, a UV rearrangement map and a cube rearrangement map corresponding to the facial image includes: Aligning the second facial mesh data with the facial image to determine a mapping coordinate corresponding to each pixel in the facial image; wherein the second facial mesh data includes a plurality of vertices, each vertex corresponding to a mapping coordinate; According to the mapping coordinates corresponding to each pixel of the facial image and the facial image, a UV rearrangement map and a cube rearrangement map corresponding to the facial image are determined.

6. The method according to claim 5, characterized in that The determining, based on the mapping coordinates corresponding to each pixel in the facial image and the facial image, a UV rearrangement map corresponding to the facial image, includes: Perform UV unfolding on the second facial mesh data to obtain a UV position map; For each first mapping coordinate in the UV position map corresponding to a pixel in the facial image, determining a pixel color of the first mapping coordinate in the UV position map as a pixel color of the pixel corresponding to the first mapping coordinate; For each second mapping coordinate other than the first mapping coordinate in the UV position map, a pixel color of the second mapping coordinate in the UV position map is determined as a noise pixel color to obtain a UV rearrangement map corresponding to the face image.

7. The method according to claim 5, characterized in that The step of determining a cube permutation diagram corresponding to the face image based on the mapping coordinates corresponding to each pixel in the face image and the face image comprises: Determining a line of sight reflection direction of each target mapping coordinate in the first facial mesh data, where the target mapping coordinate is a mapping coordinate corresponding to a pixel in the facial image; For each target mapping coordinate in the first face mesh data, determining a pixel value of an image at the target mapping coordinate in the first face mesh data as a pixel value of the pixel corresponding to the target mapping coordinate; determining a cube image corresponding to a facial region in the first face grid data; The cube image is converted into a panoramic image to obtain a cube rearrangement image corresponding to the face image.

8. The method according to claim 4, characterized in that The method further comprises: Constructing the first prediction model and the second prediction model; Obtain a sample face image; Determining second three-dimensional model parameters corresponding to the face sample image; The first prediction model and the second prediction model are trained according to the face sample image and the second three-dimensional model parameters to obtain the trained first prediction model and the second prediction model.

9. The method according to claim 8, characterized in that The step of training the first prediction model and the second prediction model according to the face sample image and the second three-dimensional model parameters to obtain the trained first prediction model and the second prediction model includes: Determining, based on the sample face image and the second three-dimensional model parameters, a low-frequency diffuse reflection image, a UV rearrangement map, and a cube rearrangement map corresponding to the sample face image; wherein the low-frequency diffuse reflection image is used to characterize the reflection of low-frequency components in the facial image under diffuse reflection lighting conditions; Inputting the UV rearrangement map corresponding to the face sample image into the first prediction model, so that the first prediction model outputs a face highlight reflectance image corresponding to the face sample image; Inputting the cube rearrangement map corresponding to the face sample image into the second prediction model, so that the second prediction model outputs a face highlight environment illumination map corresponding to the face sample image; Determining third face mesh data corresponding to the face sample image, wherein the third face mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the face sample image; Determining a facial highlight image corresponding to the facial sample image based on the third facial grid data, the facial highlight reflectance image corresponding to the facial sample image, and the facial highlight ambient illumination map; Calculating a loss value based on a preset loss function and the low-frequency diffuse reflection image and the highlight image of the face corresponding to the face sample image; Based on the loss value, model parameters of the first prediction model and the second prediction model are updated.

10. The method according to claim 9, characterized in that The second three-dimensional model parameters include a second shape parameter, a second texture parameter and a second diffuse reflection illumination parameter; The determining, based on the face sample image and the second three-dimensional model parameters, a low-frequency diffuse reflection image of the face corresponding to the face sample image includes: Calculating third face mesh data corresponding to the face sample image according to the second shape parameter; Calculating a low-frequency color image of the face corresponding to the face sample image according to the second texture parameter; Based on the second diffuse reflection illumination parameter, the third face mesh data and the low-frequency color image of the face, a differentiable diffuse reflection rendering process is performed to obtain a low-frequency diffuse reflection image of the face corresponding to the face sample image.

11. An image highlight processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a facial image to be processed and determine first three-dimensional model parameters corresponding to the facial image; wherein the first three-dimensional model parameters include first posture parameters; a first determining module, configured to determine first facial mesh data corresponding to the facial image, wherein the first facial mesh data is mesh data of a three-dimensional mesh having a shape consistent with a face shape in the facial image; a second determination module, configured to determine a facial highlight reflectance image and a facial highlight ambient lighting map corresponding to the facial image based on the facial image, the first posture parameter, and the first facial mesh data; wherein the facial highlight reflectance image is used to characterize a highlight region of the facial image, and the facial highlight ambient lighting map is used to characterize ambient lighting of the highlight region of the facial image; a third determining module, configured to determine a facial highlight image corresponding to the facial image based on the first facial mesh data, the facial highlight reflectance image, and the facial highlight ambient lighting map; The processing module is used to perform pixel-by-pixel subtraction on the face image and the face highlight image to obtain the face image after removing the highlights.

12. An electronic device, characterized in that: The electronic device comprises: processor; and The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the image highlight processing method according to any one of claims 1 to 10 is executed.

13. A computer-readable storage medium, characterized in that A data processing program is stored, and the program is run by a processor to execute the image highlight processing method according to any one of claims 1 to 10.

Images