Image processing method and device and related equipment

Abstract

The invention discloses an image processing method and device and related equipment, and relates to the technical field of image processing. Obtaining an image including the face of the collected object, and reconstructing according to the image to obtain a three-dimensional face structure of the collected object; based on the three-dimensional face structure of the collected object, a first neural network is used for reasoning to obtain non-linearly-changed face features, the face features are used for indicating one or more features of the shape, expression and posture of the collected object, and therefore the non-linearly-changed face features are used for identifying the three-dimensional face structure of the collected object. And adjusting the three-dimensional face structure of the collected object. Therefore, the facial features of the finally obtained three-dimensional facial structure have nonlinear change characteristics and accord with the nonlinear change rule of the facial features of the collected object in a real scene, so that the reconstructed three-dimensional facial structure better fits the actual facial structure of the collected object; therefore, the reconstruction precision of the face in the three-dimensional modeling process can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus and related equipment. Background Art

[0002] 3D modeling refers to the use of computer vision technology and 3D modeling technology to create 3D models, which are widely used in scenarios such as face analysis, film and television synthesis, and game design. For example, in game design scenarios, you can create corresponding 3D face models for game characters in the virtual game world based on the facial features of real people to enrich the character image in the game.

[0003] At present, in the process of three-dimensional modeling of the face, the collected face is scanned in three dimensions to obtain at least one set of point cloud data, which includes information such as the three-dimensional coordinate data of each position point of the face in a certain coordinate system during the three-dimensional scanning process. Then, from the point cloud data, the principal component analysis (PCA) method is used to construct an average three-dimensional face shape and a set of linear attribute bases. Among them, the shape of any face can be represented by the average three-dimensional face shape and the linear attribute base. In this way, by iteratively solving the coefficients of the attribute base, a three-dimensional face model can be constructed. In practical applications, the constructed face model can be called a three-dimensional deformable statistical model (3D morphable models, 3DMM).

[0004] However, in the process of building a 3DMM model, linear feature changes are used to simulate facial features such as shape, expression, and posture, which makes the reconstruction accuracy of the face in the constructed 3DMM model low, that is, there are large differences between the facial features in the 3DMM model and the real facial features.

[0005] Therefore, there is an urgent need for a method to improve the reconstruction accuracy of the face in the 3D modeling process. Summary of the invention

[0006] The present application provides an image processing method to improve the reconstruction accuracy of a face in a three-dimensional modeling process. In addition, the present application also provides an image processing device, a computing device, a computer-readable storage medium, and a computer program product.

[0007] In a first aspect, the present application provides an image processing method, which, when performing three-dimensional modeling of a face, obtains an image including the face of a captured object, and reconstructs the three-dimensional facial structure of the captured object based on the image. The obtained three-dimensional facial structure is a result of preliminary reconstruction, and its accuracy may be low. Therefore, based on the three-dimensional facial structure of the captured object, a first neural network is used for reasoning to obtain non-linear facial features, which are used to indicate one or more features of the shape, expression and posture of the captured object, so as to adjust the three-dimensional facial structure of the captured object using the non-linear facial features to obtain the final three-dimensional facial structure.

[0008] In this way, by using the nonlinearly changing facial features to adjust the three-dimensional facial structure obtained by the preliminary reconstruction, the facial features of the final three-dimensional facial structure can be made to conform to the nonlinear changing laws of the facial features of the collected object in the real scene. For example, the eyes, eyebrows, facial contours and other features of the collected object show nonlinear changing laws in the real scene, which makes the reconstructed three-dimensional facial structure more consistent with the actual facial structure of the collected object, thereby improving the reconstruction accuracy of the face in the three-dimensional modeling process.

[0009] In a possible implementation, the facial features include a first facial shape feature, a first facial expression feature, and a facial posture feature, and the first neural network used for inference includes an encoding network, a first decoding network, and a second decoding network. Then, when inference is performed based on the three-dimensional facial structure obtained by preliminary reconstruction, the three-dimensional facial structure of the collected object can be firstly obtained by encoding the second facial shape feature, the second facial expression feature, and the facial posture feature using the encoding network, and then the second facial shape feature is decoded using the first decoding network to obtain the first facial shape feature, and the second facial expression feature is decoded using the second decoding network to obtain the first facial expression feature. The decoded first facial shape feature and the first facial expression feature can be, for example, a three-dimensional grid map. In this way, the nonlinear facial features of the collected object can be inferred using the first neural network including the encoding network and the decoding network, so that the accuracy of the three-dimensional facial structure subsequently reconstructed based on the nonlinear facial features can be higher.

[0010] In a possible implementation, when performing coding network coding to obtain the second facial shape feature, the second facial expression feature, and the facial posture feature, specifically, multiple key points may be first extracted from the three-dimensional facial structure of the collected object, and the multiple key points correspond to multiple positions of the face of the collected object, such as multiple positions of the facial features and facial contours used to locate and mark the facial features of the collected object; then, the coding network is used to infer the multiple key points to obtain the second facial shape feature, the second facial expression feature, and the facial posture feature. In this way, multiple key points in the three-dimensional facial structure may be used to infer nonlinear facial features.

[0011] In a possible implementation, when the coding network is used to perform reasoning based on multiple key points, the positions of the multiple key points may be optimized using the second neural network model to obtain multiple optimized key points, and then the coding network may be used to perform reasoning based on the optimized multiple key points. In this way, the accuracy of the extracted key points may be improved by optimizing the key points, thereby improving the accuracy of the nonlinearly changing facial features inferred based on the optimized multiple key points, and further improving the accuracy of reconstructing the three-dimensional facial structure of the captured object.

[0012] In a possible implementation, when the positions of the multiple key points are optimized using the second neural network, the multiple key points, the shape parameters in the three-dimensional facial structure obtained by preliminary reconstruction, and the expression parameters in the three-dimensional facial structure are first input into the second neural network to obtain the optimized multiple key points output by the second neural network, wherein the shape parameters are used to indicate the facial shape features of the collected object, and the expression parameters are used to indicate the facial expression features of the collected object. In this way, the accuracy of the positions of the multiple key points determined can be improved by the shape parameters and the expression parameters in the three-dimensional facial structure, thereby improving the accuracy of the three-dimensional facial structure of the collected object finally reconstructed.

[0013] In a possible implementation, when the positions of multiple key points are optimized using the second neural network, the multiple key points may be first inferred using the second neural network to obtain residual values ​​corresponding to the multiple key points; then, the positions of the multiple key points are optimized using the residual values. In this way, the accuracy of the positions of the multiple key points determined may be improved by performing residual calculations on the multiple key points, thereby improving the accuracy of the three-dimensional facial structure of the captured object that is finally reconstructed.

[0014] In a possible implementation, the encoding network may specifically be a multi-layer perceptron MLP.

[0015] In a possible implementation, after adjusting the three-dimensional facial structure of the collected object obtained by preliminary reconstruction, the adjusted three-dimensional facial structure of the collected object can also be presented, such as being presented to a user through a client; or, a target processing result can be presented, which is obtained by performing a preset operation on the adjusted three-dimensional facial structure of the collected object. The preset operation can, for example, be an operation of adding texture information or lighting information to create a game character, or can be an operation of creating a facial structure of a cartoon character based on the three-dimensional facial structure, etc.

[0016] In a second aspect, the present application provides an image processing device, which includes various modules for executing the image processing method in the first aspect or any possible implementation of the first aspect.

[0017] In a third aspect, the present application provides a computing device, the computing device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the computing device performs the image processing method in the first aspect or any one of the implementations of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the operating steps of the image processing method described in the first aspect or any one of the implementations of the first aspect.

[0019] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operating steps of the image processing method described in the first aspect or any one of the implementations of the first aspect.

[0020] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an exemplary application scenario provided for this application;

[0022] Figure 2 A schematic diagram of another exemplary application scenario provided for this application;

[0023] Figure 3 A flowchart of an image processing method provided in this application;

[0024] Figure 4 A schematic diagram of the process of twice reconstructing the three-dimensional facial structure provided in this application;

[0025] Figure 5 A schematic diagram of the structure of an image processing device provided by the present application;

[0026] Figure 6 A schematic diagram of the hardware structure of a computing device provided for this application. DETAILED DESCRIPTION

[0027] In order to improve the reconstruction accuracy of the face in the three-dimensional modeling process, the present application provides an image processing method, which uses the nonlinear changing facial features obtained by neural network reasoning to adjust the three-dimensional facial structure obtained by the first round of reconstruction, so that the facial features of the three-dimensional facial structure finally reconstructed have the characteristics of nonlinear change, which conforms to the changing law of the facial features of the collected object in the real scene, thereby improving the reconstruction accuracy of the face in the three-dimensional modeling process.

[0028] The technical solution in the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0029] See also Figure 1 , is a schematic diagram of an exemplary application scenario provided in the embodiment of the present application. Figure 1 As shown, an image processing device 101 may be deployed on the cloud platform 100, and the image processing device 101 may provide a cloud service for reconstructing a three-dimensional facial structure. The cloud platform 100 may communicate with one or more clients through a network, such as exchanging data through a hypertext transfer protocol (HTTP) or other network protocols. For ease of understanding and explanation, Figure 1 The example in which the cloud platform 100 provides cloud services to the clients 201 to 204 is used for explanation.

[0030] The client may be a web browser provided by the cloud platform 100 to interact with users, such as users 301 to 304 may access the cloud platform 100 through corresponding clients. Alternatively, the client may be an application running on a user terminal.

[0031] Taking the example of user 301 using client 201 to request cloud platform 100 to provide a cloud service for reconstructing a three-dimensional facial structure, user 301 can import an image (such as a human face image) including the face of the collected object to client 201, and instruct to reconstruct the three-dimensional facial structure of the collected object on client 201. Client 201 can generate a request including the image, and send the request to cloud platform 100 to request cloud platform 100 to feedback the corresponding three-dimensional facial structure reconstruction result.

[0032] The image processing device 101 on the cloud platform 100 parses the image from the request and reconstructs the three-dimensional facial structure of the collected object based on the image. At this time, the facial features in the reconstructed three-dimensional facial structure may not conform to the nonlinear change law in the real scene, such as the lines of facial muscles in the three-dimensional facial structure change linearly. Therefore, the image processing device 101 uses a neural network (such as a multi-layer perceptron + convolutional neural network, etc.) to infer based on the three-dimensional facial structure to obtain nonlinear facial features. The facial features are used to indicate one or more features of the shape, expression and posture of the face of the collected object, so that the image processing device 101 uses the facial features to adjust the reconstructed three-dimensional facial structure to obtain a new three-dimensional facial structure. At this time, the features of the face shape, expression or posture in the new three-dimensional facial structure are adjusted to nonlinear changes. Finally, the image processing device 101 returns the new three-dimensional facial structure to the client 201 so that the client 201 presents it to the user 301.

[0033] Since the three-dimensional facial structure obtained by the preliminary reconstruction is adjusted by using the facial features with nonlinear changes, the facial features of the final three-dimensional facial structure can be made to conform to the nonlinear change law of the facial features of the collected object in the real scene, that is, the reconstructed three-dimensional facial structure is more consistent with the actual facial structure of the collected object, thereby improving the reconstruction accuracy of the face in the three-dimensional modeling process.

[0034] Exemplarily, the above-mentioned image processing device 101 can be implemented by software or hardware.

[0035] In the first example, when implemented by software, the image processing device 101 can be implemented by at least one of a virtual machine, a container, a bare metal server (BMS), a computing engine, etc.

[0036] In the second example, when implemented by hardware, the image processing device 101 can be implemented by at least one physical device including a processor, such as a server, etc. The processor can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), or any combination thereof. In addition, the number of processors included in the image processing device 101 can be one or more, and the type of processors included can be one or more. The number and type of processors can be set according to the business requirements of the actual application, and this embodiment does not limit this.

[0037] It is worth noting that the above Figure 1 The application scenarios shown are only for illustrative purposes and are not intended to be limiting. For example, in actual applications, the number of clients connected to the cloud platform 100 can be any number, and the cloud platform 100 can also provide other types of cloud services, such as face recognition, expression recognition, etc. Figure 2 In the application scenario shown, the image processing device 101 can also be deployed as software (such as a software development kit, etc.) on the terminal device 102 on the user side, so that the user 103 can import an image including the captured object into the terminal device 102, and use the terminal device 102 to reconstruct the three-dimensional facial structure of the captured object.

[0038] For ease of understanding, an embodiment of the image processing method provided by the present application is described below in conjunction with the accompanying drawings.

[0039] See also Figure 3 , Figure 3 A flowchart of an image processing method provided in an embodiment of the present application, which can be applied to Figure 1The application scenarios described above may also be applied to other applicable application scenarios. Figure 1 The image processing device 101 in the application scenario is taken as an example for exemplary description.

[0040] in, Figure 3 The image processing method shown may specifically include:

[0041] S301: The image processing device 101 acquires an image, which includes the face of the captured object.

[0042] The face of the object being collected may specifically be a human face, such as Figure 1 Alternatively, the face of the collected object may be the appearance of other objects, such as the face of an animal, the face of a character model, etc.

[0043] In this embodiment, the following examples of implementing image acquisition are provided.

[0044] In the first implementation example, user 301 can conduct a video conference with other users through client 201. At this time, client 201 can be configured with a camera module, such as a camera, and client 201 can use the camera module to shoot the face of user 301 to obtain a video stream, and send the video stream to the client used by other users in a wired or wireless manner to achieve a video conference between users. In this process, image processing device 101 can capture a frame of image in the video stream, and the frame of image includes the face of user 301, that is, part of the image content in the frame of image is the face of user 301.

[0045] In the second implementation example, the client 201 may present an interactive interface to the user 301 so that the user 301 may import an image on the interactive interface. The imported image may be an image including the face of the user 301 or an image including the face of another object.

[0046] The above implementation is only for illustrative purposes. In other embodiments, the image processing device 101 may also acquire an image including the face of the captured object in other ways.

[0047] S302: The image processing device 101 reconstructs the three-dimensional facial structure of the captured object according to the acquired image.

[0048] In this embodiment, the image processing device 101 can first preliminarily reconstruct the three-dimensional facial structure of the captured object. At this time, the facial features in the obtained three-dimensional facial structure may not be accurate, such as the facial features showing linear changes. Exemplarily, this embodiment provides the following two implementation examples of preliminarily reconstructing the three-dimensional facial structure.

[0049] In a first implementation example, the image processing device 101 may reconstruct a three-dimensional facial structure based on a 3DMM model.

[0050] Specifically, after acquiring an image, the image processing device 101 may first determine the pixel area of ​​the face of the captured object in the image, such as by using a pre-trained artificial intelligence (AI) model, such as a face detection model, to identify the face area in the image, and then extract a sub-image including the face pixel area from the image. Then, the image processing device 101 may input the sub-image into an encoder for encoding to obtain a latent space vector, such as Figure 4 As shown, the latent space vector contains features such as the shape, expression, and posture of the face of the collected object.

[0051] Among them, the encoder, for example, can be implemented by a trained convolutional neural network (CNN), or can be implemented by other types of networks, and there is no limitation on this. In addition, in actual application scenarios, the size of the image input to the encoder is usually a fixed size, and the size of the sub-image cut out from the image may not be consistent with the fixed size. For example, the size of the image input to the encoder is required to be 512×512, and the size of the cut sub-image may be 256×256, etc. Therefore, after the image processing device 101 cuts out the sub-image, it can first scale the sub-image so that the size of the scaled sub-image meets the size requirement of the encoder for the input image; then, the image processing device inputs the scaled sub-image into the encoder for encoding.

[0052] Next, the image processing device 101 can extract shape parameters, expression parameters, and posture parameters from the latent space vector output by the encoder. Among them, shape parameters are used to characterize the facial shape features of the collected object; expression parameters are used to characterize the facial expression features of the collected object; posture parameters are used to characterize the facial posture features of the collected object. For example, the image processing device 101 can use the fully connected layer 1 to extract shape parameters; use the linear fully connected layer 2 to extract expression parameters; use the linear fully connected layer 3 to extract posture parameters, etc. In other possible implementations, the linear fully connected layer 1, the fully connected layer 2, and the fully connected layer 3 can also be integrated in the encoder, so that after the image processing device 101 inputs the sub-image into the encoder, the encoder can directly output the corresponding shape parameters, expression parameters, and posture parameters.

[0053] In this way, the image processing device 101 can preliminarily reconstruct the three-dimensional facial structure based on the extracted shape parameters, expression parameters, and posture parameters. For example, the image processing device 101 can reconstruct a three-dimensional mesh (3Dmesh) for representing the three-dimensional facial structure of the captured object based on the following formula (1), that is, the formula for reconstructing the three-dimensional facial structure in the 3DMM model, such as Figure 4 As shown. A three-dimensional mesh is a data structure used in computer graphics to build models for various irregular objects, that is, a large number of small polygons (such as triangles, quadrilaterals, etc.) can be used to describe the outer surface of a three-dimensional object. Taking the triangular mesh in a three-dimensional mesh as an example, the triangular mesh is a list of triangles, and any surface in the real world can be composed of a large number of small triangular patches, so that these small triangular patches can be used to simulate the outer surfaces of various objects in the real world.

[0054]

[0055] V is a three-dimensional grid; f is a scaling factor, which is a preset fixed value; R is a rotation matrix; t is a translation matrix; is the average three-dimensional face shape; α shp is the shape parameter; α exp is the expression parameter; p is the posture parameter; A shp is the main component of the face shape obtained by principal component analysis (PCA); A exp is the main component of the facial expression obtained by PCA. Among them, the rotation matrix R and the translation matrix t are calculated according to the posture parameter p.

[0056] In the second implementation example, the image processing device 101 can input the acquired image including the face of the captured object into a pre-trained neural network model, and the neural network model directly outputs the three-dimensional facial structure of the face of the captured object. The neural network model can be, for example, a deep neural network (DNN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, etc., or other types of neural network models, which are not limited here.

[0057] In this embodiment, the accuracy of the three-dimensional facial structure initially reconstructed based on the above implementation is usually low. For example, when the three-dimensional facial structure is reconstructed based on the 3DMM model, the features such as the shape, expression and posture of the face in the three-dimensional facial structure usually change linearly, and the reconstruction accuracy is low. When the three-dimensional facial structure is reconstructed based on the neural network model, due to the high convergence difficulty of the neural network model in actual application scenarios, the features such as the shape, expression or posture of the face in the three-dimensional facial structure output by the neural network model usually deviate greatly from the real features of the face of the collected object, resulting in low reconstruction accuracy.

[0058] To this end, in this embodiment, the image processing device 101 may further improve the accuracy of the three-dimensional facial structure finally reconstructed by continuing to perform the following steps.

[0059] S303: The image processing device 101 uses the first neural network to perform reasoning based on the three-dimensional facial structure of the captured object to obtain nonlinear facial features, which are used to indicate one or more features of the shape, expression, and posture of the face of the captured object.

[0060] S304: The image processing device 101 adjusts the three-dimensional facial structure of the captured object using non-linear facial features.

[0061] Among them, the first neural network is used to predict the nonlinear change characteristics of the face of the collected object in the real scene, for example, it can be implemented by a multilayer perceptron (MLP), a convolutional neural network, an artificial neural network (ANN), or can be implemented by other types of neural networks.

[0062] In a possible implementation, the image processing device 101 can extract multiple key points from a three-dimensional facial structure (specifically, it can be a 3D mesh), and the multiple key points correspond to multiple positions on the face of the collected object, and can be used to locate and annotate the facial features and facial contours of the collected object. For example, the image processing device 101 can extract 68 key points that meet the screening conditions based on the positions of each point in the 3D mesh (the number of key points can also be more or less), and the 68 key points obtained by screening can be used to locate the positions of the eyebrows, eyes, nose, mouth, facial contours, etc. of the collected object on the face.

[0063] After extracting multiple key points, the image processing device 101 can use the first neural network that has been trained in advance to perform reasoning based on the multiple key points, such as inputting the position information of the multiple key points into the first neural network, so that the first neural network outputs nonlinear facial features, and uses the nonlinear facial features to adjust the three-dimensional facial structure obtained by preliminary reconstruction. In this embodiment, the nonlinear facial features can include one or more of facial shape features, facial expression features, and facial posture features. At this time, the facial features output by the first neural network conform to the nonlinear change rules of the shape, expression or posture of the face in the real scene, and are more in line with the real features of the face of the collected object.

[0064] In a specific implementation, taking the example of nonlinear facial features including the shape, expression and posture of the face of the collected object, the first neural network includes an encoding network, a first decoding network and a second decoding network. Figure 4 As shown, the image processing device 101 can first input multiple key points extracted from the three-dimensional facial structure into the encoding network, and use the encoding network to encode and obtain the second facial shape feature, the second facial expression feature, and the facial posture feature. Then, the image processing device 101 can use the first decoding network to decode the second facial shape feature to obtain the first facial shape feature, and use the second decoding network to decode the second facial expression feature to obtain the first facial expression feature. Thus, the image processing device 101 can adjust the three-dimensional facial structure of the captured object according to the facial posture feature and the decoded first facial shape feature and first facial expression feature.

[0065] Exemplarily, the first facial shape feature may be expressed as a UV map, which includes a plurality of points in a UV coordinate system, wherein the coordinates of the points in the UV coordinate system include coordinates in a horizontal direction (U) and coordinates in a vertical direction (V), and the coordinates of each point may correspond to a pixel on a two-dimensional image plane.

[0066] Generally, there is a mapping relationship between the coordinates of points in the image plane and the vertices in the three-dimensional grid, and the mapping relationship can be set in advance by a technician.

[0067] In this way, the image processing device 101 can map the UV map corresponding to the first facial shape feature to obtain the three-dimensional mesh map M corresponding to the first facial shape feature. shp ,like Figure 4 shown.

[0068] Similarly, the first facial expression feature is also expressed as a UV map, so the image processing device 101 can refer to the above-mentioned similar method, and map the UV map corresponding to the first facial expression shape feature to obtain the three-dimensional grid map M corresponding to the first facial expression shape. exp ,like Figure 4 shown.

[0069] Finally, the image processing device 101 can adjust the three-dimensional facial structure according to the following formula (2) to obtain the final three-dimensional facial structure, such as Figure 4 shown.

[0070]

[0071] Among them, V final is the final 3D facial structure reconstructed; f is the scaling factor, which is a preset fixed value; R′ is the updated rotation matrix; t′ is the updated translation matrix; is the average three-dimensional face shape; M shp is a three-dimensional grid image corresponding to the first facial shape feature; M exp is the three-dimensional grid image corresponding to the first facial expression feature; the updated rotation matrix R′ and translation matrix t′ are calculated according to the updated posture parameter p′(. The updated posture parameter p′ is the facial posture feature output by the encoding network; in addition, the facial feature output by the encoding network also includes the updated shape parameter α′ shp (used to construct the above three-dimensional grid map M shp , i.e. the second facial shape feature mentioned above), the updated expression parameter α′ exp (used to construct the above three-dimensional grid map M exp , which is the second facial expression feature mentioned above).

[0072] In this way, the facial features in the finally reconstructed three-dimensional facial structure conform to the nonlinear change law in the real scene, so that the reconstruction accuracy of the three-dimensional facial structure can be effectively improved. In addition, the image processing device 101 reconstructs the final three-dimensional facial structure using the decoupled facial shape features and facial expression features, which can increase the expression ability of the nonlinear information of the face of the collected object, so that when the collected object has a loaded facial expression, the reconstruction effect of the facial features and facial contour in the finally reconstructed three-dimensional facial structure can also reach a higher level, reducing the influence of facial expressions.

[0073] Alternatively, the image processing device 101 may adjust the three-dimensional facial structure according to the following formula (3) to obtain the final three-dimensional facial structure.

[0074]

[0075] Among them, α shp is the shape parameter extracted based on the 3DMM model; α exp A is the expression parameter extracted based on the 3DMM model; shp is the main component of the face shape obtained by principal component analysis (PCA); A exp are the main components of facial expressions obtained by PCA.

[0076] In this way, based on the reconstruction method shown in formula (2), α is added shp , A shp , α exp , A exp This makes the facial features in the three-dimensional facial structure reconstructed based on formula (3) closer to the facial features of the collected object in the real scene, thereby further improving the reconstruction accuracy of the three-dimensional facial structure.

[0077] It is worth noting that the above various implementation examples are only used as some exemplary explanations and are not used to limit their specific implementation methods to the above examples. For example, in other embodiments, the nonlinearly changing facial features may only include facial expression features, so that the image processing device 101 can adjust the initially reconstructed three-dimensional facial features only with the facial feature expression. Alternatively, the nonlinearly changing facial features can be the superposition of facial shape features and facial expression features, so that the image processing device 101 can use a decoding network to decode the facial features, and adjust the initially reconstructed three-dimensional facial features based on the decoded facial features. In this case, the above M shp +M expcan be replaced by the three-dimensional grid map M corresponding to the decoded facial features shp+exp .

[0078] Furthermore, since the accuracy of the three-dimensional facial structure obtained by preliminary reconstruction may be low, the accuracy of the multiple key points extracted from the three-dimensional facial structure may be low. Therefore, the image processing device 101 can also use the pre-trained second neural network to optimize the positions of the multiple key points to obtain multiple optimized key points. The second neural network, for example, can be a multi-layer perceptron, or can be other types of neural networks. Then, the image processing device 101 uses the first neural network to infer the multiple optimized key points to obtain non-linear facial features. In this way, the accuracy of the extracted key points can be improved by optimizing the key points, so that the accuracy of the non-linear facial features inferred from the optimized multiple key points can be improved, and the accuracy of reconstructing the three-dimensional facial structure of the captured object can be further improved.

[0079] In this embodiment, the following implementation examples of optimizing the positions of multiple key points are provided.

[0080] In the first implementation example, the image processing device 101 may extract the positions of the multiple key points and the shape parameters (such as the above α shp ), expression parameters (such as the above α exp ), which is input to the second neural network, and the second neural network outputs multiple new positions, that is, the positions of multiple key points after optimization. Among them, the facial shape features and facial expression features can be used as supplementary information for the multiple key points, so that the second neural network adjusts the positions of the multiple key points according to the supplementary information.

[0081] In the second implementation example, the supplementary information may include not only shape parameters and expression parameters, but also latent space vectors output by the encoder, so that the image processing device 101 may input the positions of the extracted multiple key points, shape parameters, expression parameters, and latent space vectors into the second neural network, and the second neural network outputs the optimized positions of the multiple key points. In this way, the second neural network can optimize the positions of the multiple key points by integrating more information, and can further improve the accuracy of the determined key points.

[0082] In the third implementation example, the image processing device 101 can input the positions, shape parameters, expression parameters (and latent space vectors) of the multiple key points into the second neural network, and the second neural network outputs the residual values ​​for the positions of the multiple key points, so that the image processing device 101 can calculate the optimized positions of the multiple key points according to the positions of the multiple key points and the residual values. For example, the image processing device 101 can calculate the optimized positions of the multiple key points according to the following formula (4).

[0083]

[0084] Among them, L′ lmk is the position of multiple key points after optimization; L lmk is the position of multiple key points before optimization; is the residual value output by the residual network; λ is the weight coefficient, which is a fixed value. In actual application, it can be determined in advance by technicians, or it can be automatically determined by machine learning.

[0085] In this embodiment, the above-mentioned first neural network (for outputting nonlinear facial features) and the second neural network (for optimizing multiple key points) can be trained in advance based on manually annotated training samples. For example, the training sample can specifically include an image of the face of the collected object and the manually annotated result, and the manually annotated result includes the facial features of the collected object (which can be expressed as shape parameters, expression parameters, posture parameters, or as a corresponding three-dimensional grid) and multiple key points in the three-dimensional facial structure, so that the image processing device 101 (or other devices) can use one or more groups of training samples to iteratively train the first neural network and the second neural network. Specifically, for each image in the training sample, the image processing device 101 can reconstruct the corresponding three-dimensional facial structure based on the image based on the 3DMM model, and use the second neural network to optimize the position of multiple key points extracted from the three-dimensional facial structure, and then input the optimized multiple key points into the first neural network, so that the first neural network can infer and output nonlinear facial features based on the multiple key points. Then, the image processing device 101 can compare the nonlinear facial features with the facial features included in the manual annotation results, and update the parameters in the first neural network and the parameters in the second neural network according to the difference between the two facial features. Then, the image processing device 101 can adjust the three-dimensional facial structure using the nonlinear facial features output by the first neural network, or can re-infer and output new nonlinear facial features using the first neural network after completing the parameter update, and adjust the three-dimensional facial structure using the new nonlinear facial features. Finally, the image processing device 101 can compare multiple key points (such as 68 key points, etc.) in the adjusted three-dimensional facial structure with multiple key points included in the manual annotation results, and update the parameters in the first neural network and the parameters in the second neural network according to the difference between the two groups of key points, thereby completing a joint training for the first neural network and the second neural network. Repeat the above process until the first neural network and the second neural network meet the training termination conditions, such as the first neural network and the second neural network converge, etc., thereby completing the iterative training of the first neural network and the second neural network.

[0086] In other implementation examples of training neural networks, the first neural network and the second neural network may also be trained separately; or, the first neural network and the second neural network may also be jointly trained with other networks (such as a neural network implementing an encoder, etc.).

[0087] In actual application, since the three-dimensional facial structure initially reconstructed by the 3DMM model has a certain accuracy, that is, except for the facial features of the nonlinear change part, the remaining features in the three-dimensional facial structure have a high degree of matching with the actual features of the face of the collected object in the real scene. This makes it possible to train the first neural network and the second neural network based on the three-dimensional facial structure output by the 3DMM model, which can accelerate the convergence of the first neural network and the second neural network and improve the training efficiency, and at the same time effectively reduce the convergence difficulty of the first neural network and the second neural network.

[0088] In a further possible implementation, after reconstructing the 3D facial structure of the captured object, the image processing device 101 may also present the 3D facial structure, such as presenting the 3D facial structure to the user 301 through the client 201 .

[0089] Alternatively, the image processing device 101 may present a target processing result obtained according to the three-dimensional facial structure of the collected object. Specifically, the image processing device 101 may further perform a preset operation on the three-dimensional facial structure to obtain a target processing result. For example, in a game design scene, the image processing device 101 may create a game character having the same facial structure as the real person by adding corresponding texture information or lighting information, etc., based on the three-dimensional facial structure corresponding to the real person, so as to enrich the character image in the game. Alternatively, in a film and television synthesis scene, the image processing device 101 may determine the facial structure of an animated character in a film and television according to the three-dimensional facial structure of the real person object, so that the facial features (including shape, expression, and posture, etc.) of the animated character in the film and television are as similar as possible to the facial features of the real person, so as to present a better film and television effect. Then, the image processing device 101 presents the target processing result, which may be, for example, a game character created in the game design scene, or may be an animation of an animated character generated in a film and television synthesis scene.

[0090] In this embodiment, the image processing device 101 first reconstructs the three-dimensional facial structure of the collected object based on the acquired image, and then uses the first neural network to infer the three-dimensional facial structure to obtain nonlinear facial features, and uses the facial features to adjust the reconstructed three-dimensional facial structure, that is, reconstruct a new three-dimensional facial structure. Since the three-dimensional facial structure obtained by preliminary reconstruction is adjusted by using the nonlinear facial features, the facial features of the final three-dimensional facial structure can be made to conform to the nonlinear variation law of the facial features of the collected object in the real scene, that is, the reconstructed three-dimensional facial structure is more in line with the actual facial structure of the collected object, so that the reconstruction accuracy of the face in the three-dimensional modeling process can be improved.

[0091] It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0092] Combination of the above Figures 1 to 4 The image processing method provided in the embodiment of the present application is introduced, and then the structure of the image processing device and the computing equipment provided in the embodiment of the present application are introduced in conjunction with the accompanying drawings.

[0093] See also Figure 5 , shows a schematic diagram of the structure of an image processing device, the image processing device 500 comprises:

[0094] An acquisition module 501 is used to acquire an image, wherein the image includes the face of the object being acquired;

[0095] A reconstruction module 502, configured to reconstruct a three-dimensional facial structure of the captured object according to the image;

[0096] An inference module 503 is used to perform inference based on the three-dimensional facial structure of the collected object using a first neural network to obtain nonlinear facial features, where the facial features are used to indicate one or more features of the shape, expression, and posture of the face of the collected object;

[0097] The adjustment module 504 is used to adjust the three-dimensional facial structure of the collected object by using the non-linear facial features.

[0098] In a possible implementation, the facial features include a first facial shape feature, a first facial expression feature, and a facial posture feature, and the first neural network includes an encoding network, a first decoding network, and a second decoding network;

[0099] Then, the reasoning module 503 is specifically used for:

[0100] Based on the three-dimensional facial structure of the collected object, using the coding network coding to obtain a second facial shape feature, a second facial expression feature and the facial posture feature;

[0101] Decoding the second facial shape feature using the first decoding network to obtain the first facial shape feature;

[0102] The second facial expression feature is decoded using the second decoding network to obtain the first facial expression feature.

[0103] In a possible implementation, the reasoning module 503 is specifically configured to:

[0104] Extracting a plurality of key points from the three-dimensional facial structure of the captured object, wherein the plurality of key points correspond to a plurality of positions on the face of the captured object;

[0105] The encoding network is used to perform reasoning based on the multiple key points to obtain the second facial shape feature, the second facial expression feature, and the facial posture feature.

[0106] In a possible implementation, the reasoning module 503 is specifically configured to:

[0107] Optimizing the positions of the plurality of key points using a second neural network to obtain a plurality of optimized key points;

[0108] The encoding network is used to perform reasoning based on the optimized multiple key points.

[0109] In a possible implementation, the reasoning module 503 is specifically configured to:

[0110] The multiple key points, the shape parameters in the three-dimensional facial structure, and the expression parameters in the three-dimensional facial structure are input into the second neural network to obtain the optimized multiple key points output by the second neural network, the shape parameters are used to indicate the facial shape features of the collected object, and the expression parameters are used to indicate the facial expression features of the collected object.

[0111] In a possible implementation, the reasoning module 503 is specifically configured to:

[0112] Using the second neural network to infer the multiple key points to obtain residual values ​​corresponding to the multiple key points;

[0113] The positions of the plurality of key points are optimized using the residual values.

[0114] In a possible implementation, the encoding network is a multi-layer perceptron MLP.

[0115] In a possible implementation manner, the image processing device 500 further includes:

[0116] The presentation module 505 is used to present the adjusted three-dimensional facial structure of the collected object; or present a target processing result, which is obtained by performing a preset operation on the adjusted three-dimensional facial structure of the collected object.

[0117] because Figure 5 The image processing device 500 shown corresponds to the above Figure 3 The image processing device 101 in the embodiment shown is Figure 5 For the specific implementation of the image processing device 500 and its technical effects, see the above Figure 3 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0118] Figure 6 The hardware structure diagram of a computing device 600 provided in this application is shown in FIG. 6 . The computing device 600 can implement the above Figure 3 The image processing device 101 etc. in the illustrated embodiment.

[0119] like Figure 6 As shown, the computing device 600 includes a processor 601, a memory 602, and a communication interface 603. The processor 601, the memory 602, and the communication interface 603 communicate via a bus 604, and may also communicate via other means such as wireless transmission. The memory 602 is used to store instructions, and the processor 601 is used to execute the instructions stored in the memory 602. Furthermore, the computing device 600 may also include a memory unit 605, and the memory unit 605 may be connected to the processor 601, the storage medium 602, and the communication interface 603 via a bus 604. The memory 602 stores program code, and the processor 601 may call the program code stored in the memory 602 to perform the following operations:

[0120] Acquire an image, wherein the image includes a face of a subject;

[0121] Reconstructing the three-dimensional facial structure of the captured object according to the image;

[0122] Based on the three-dimensional facial structure of the collected object, the first neural network is used to perform reasoning to obtain nonlinear facial features, where the facial features are used to indicate one or more features of the shape, expression, and posture of the face of the collected object;

[0123] The three-dimensional facial structure of the collected object is adjusted by utilizing the nonlinearly changing facial features.

[0124] It should be understood that in the embodiment of the present application, the processor 601 may be a CPU, and the processor 601 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0125] The memory 602 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. The memory 602 may also include a nonvolatile random access memory.

[0126] The memory 602 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM).

[0127] The communication interface 603 is used to communicate with other devices connected to the computing device 600. The bus 604 may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, various buses are labeled as bus 604 in the figure.

[0128] It should be understood that the computing device 600 according to the embodiment of the present application may correspond to the image processing apparatus 500 in the embodiment of the present application, and may correspond to the computer executing the image processing apparatus 500 in the embodiment of the present application. Figure 3 The method performed by the image processing apparatus 101 in the method shown, and the above and other operations and / or functions implemented by the computing device 600 are respectively to implement Figure 3 For the sake of brevity, the process of the corresponding method in will not be repeated here.

[0129] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-mentioned image processing method.

[0130] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0131] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0132] The computer program product may be a software installation package, and when any of the aforementioned image processing methods needs to be used, the computer program product may be downloaded and executed on a computing device.

[0133] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image processing method, It is characterized in that The method comprises: Acquire an image, wherein the image includes a face of a subject; Reconstructing the three-dimensional facial structure of the captured object according to the image; Based on the three-dimensional facial structure of the collected object, the first neural network is used to perform reasoning to obtain nonlinear facial features, where the facial features are used to indicate one or more features of the shape, expression, and posture of the face of the collected object; The three-dimensional facial structure of the collected object is adjusted by utilizing the nonlinearly changing facial features.

2. The method according to claim 1, It is characterized in that The facial features include a first facial shape feature, a first facial expression feature, and a facial posture feature, and the first neural network includes an encoding network, a first decoding network, and a second decoding network; The method of using a first neural network to perform reasoning based on the three-dimensional facial structure of the collected object to obtain nonlinear facial features includes: Based on the three-dimensional facial structure of the collected object, using the coding network coding to obtain a second facial shape feature, a second facial expression feature and the facial posture feature; Decoding the second facial shape feature using the first decoding network to obtain the first facial shape feature; The second facial expression feature is decoded using the second decoding network to obtain the first facial expression feature.

3. The method according to claim 2, It is characterized in that The method of encoding the three-dimensional facial structure of the collected object using the encoding network to obtain the second facial shape feature, the second facial expression feature and the facial posture feature includes: Extracting a plurality of key points from the three-dimensional facial structure of the captured object, wherein the plurality of key points correspond to a plurality of positions on the face of the captured object; The encoding network is used to perform reasoning based on the multiple key points to obtain the second facial shape feature, the second facial expression feature, and the facial posture feature.

4. The method according to claim 3, It is characterized in that The using the encoding network to perform reasoning according to the multiple key points includes: Optimizing the positions of the plurality of key points using a second neural network to obtain a plurality of optimized key points; The encoding network is used to perform reasoning based on the optimized multiple key points.

5. The method according to claim 4, It is characterized in that The optimizing the positions of the plurality of key points by using the second neural network includes: The multiple key points, the shape parameters in the three-dimensional facial structure, and the expression parameters in the three-dimensional facial structure are input into the second neural network to obtain the optimized multiple key points output by the second neural network, the shape parameters are used to indicate the facial shape features of the collected object, and the expression parameters are used to indicate the facial expression features of the collected object.

6. The method according to claim 4, It is characterized in that The optimizing the positions of the plurality of key points by using the second neural network includes: Using the second neural network to infer the multiple key points to obtain residual values ​​corresponding to the multiple key points; The positions of the plurality of key points are optimized using the residual values.

7. The method according to any one of claims 2 to 6, It is characterized in that The encoding network is a multi-layer perceptron MLP.

8. The method according to any one of claims 1 to 7, It is characterized in that The method further comprises: Presenting the adjusted three-dimensional facial structure of the collected object; Or, presenting a target processing result, wherein the target processing result is obtained by performing a preset operation on the adjusted three-dimensional facial structure of the collected object.

9. An image processing device, It is characterized in that The device comprises: An acquisition module, used for acquiring an image, wherein the image includes a face of a subject; A reconstruction module, used for reconstructing a three-dimensional facial structure of the captured object according to the image; An inference module, configured to perform inference using a first neural network based on the three-dimensional facial structure of the collected object to obtain nonlinear facial features, wherein the facial features are used to indicate one or more features of the shape, expression, and posture of the face of the collected object; The adjustment module is used to adjust the three-dimensional facial structure of the collected object by utilizing the nonlinearly changing facial features.

10. The device according to claim 9, It is characterized in that The facial features include a first facial shape feature, a first facial expression feature, and a facial posture feature, and the first neural network includes an encoding network, a first decoding network, and a second decoding network; The reasoning module is used to: Based on the three-dimensional facial structure of the collected object, using the coding network coding to obtain a second facial shape feature, a second facial expression feature and the facial posture feature; Decoding the second facial shape feature using the first decoding network to obtain the first facial shape feature; The second facial expression feature is decoded using the second decoding network to obtain the first facial expression feature.

11. The device according to claim 10, It is characterized in that The reasoning module is used to: Extracting a plurality of key points from the three-dimensional facial structure of the captured object, wherein the plurality of key points correspond to a plurality of positions on the face of the captured object; The encoding network is used to perform reasoning based on the multiple key points to obtain the second facial shape feature, the second facial expression feature, and the facial posture feature.

12. The device according to any one of claims 9 to 11, It is characterized in that The device also includes: The presenting module is used to present the adjusted three-dimensional facial structure of the collected object; or present a target processing result, which is obtained by performing a preset operation on the adjusted three-dimensional facial structure of the collected object.

13. A computing device, It is characterized in that Including processor and memory; The processor is configured to execute instructions stored in the memory so as to enable the computing device to perform the steps of the method according to any one of claims 1 to 8.

14. A computer-readable storage medium, It is characterized in that The method comprises instructions which, when executed on a computing device, cause the computing device to perform the steps of the method according to any one of claims 1 to 8.

Images