Model training method, control method, device, electronic equipment and computer readable storage medium

Abstract

The application provides a model training method, a control method, a device, electronic equipment and a computer readable storage medium; the method comprises: using a prediction model to perform morphing prediction on a plurality of first face sample image frames, and correspondingly obtain prediction morphing information of each first face sample image frame; performing rendering prediction based on the first face sample image frame to obtain a predicted rendering image; determining a first loss value based on the prediction morphing information and target morphing information, and determining a second loss value based on the predicted rendering image and a target rendering image; determining a third loss value based on the prediction morphing information of the plurality of first face sample image frames and the target morphing information of the plurality of first face sample image frames; updating model parameters of the prediction model based on the first loss value, the second loss value and the third loss value to obtain a trained prediction model. Through the application, face morphing prediction can be accurately performed on a face image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, control method, device, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of computer vision, graphics, and deep learning, the demand for accurate prediction of facial modeling and dynamic deformation has increased dramatically. Traditional methods rely on manual modeling or physical simulation, which are inefficient and difficult to adapt to complex scenes. Modern technologies (such as deep learning and 3D reconstruction) offer new possibilities for achieving automated and high-precision facial deformation prediction. Driven by both technological innovation and industrial needs, research on facial deformation prediction is becoming a core direction in computer vision and graphics. Its significance lies not only in improving the realism and interactivity of virtual content, but also in creating profound value in fields such as medicine, security, and culture.

[0003] In related technologies, the three-dimensional representation of human faces is usually based on vertex or three-dimensional deformable model methods, which cannot accurately predict facial deformation in face images, resulting in low accuracy of facial deformation prediction. Summary of the Invention

[0004] This application provides a model training method, control method, device, electronic device, and computer-readable storage medium that can accurately predict facial deformations in face images.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method, the method comprising:

[0007] Using a prediction model, deformation prediction is performed on multiple first facial sample image frames to obtain the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the face of the target object moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image.

[0008] Rendering prediction is performed based on the first facial sample image frame to obtain a predicted rendered image;

[0009] Based on the predicted deformation information and the target deformation information, a first loss value is determined, and based on the predicted rendered image and the target rendered image, a second loss value is determined.

[0010] A third loss value is determined based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames.

[0011] Based on the first loss value, the second loss value, and the third loss value, the model parameters of the prediction model are updated to obtain the trained prediction model.

[0012] This application provides a control method, the method comprising:

[0013] Receive multiple facial image frames to be processed, wherein the multiple facial image frames to be processed are acquired when the target object’s face moves;

[0014] The trained prediction model is invoked to predict the deformation of each of the facial image frames to be processed, thereby obtaining the predicted deformation information of each of the facial image frames to be processed. The trained prediction model is trained using the model training method provided in the embodiments of this application. The predicted deformation information includes displacement data of multiple facial vertices.

[0015] For each of the facial image frames to be processed, the bionic device is controlled to drive multiple facial vertices to move based on multiple displacement data.

[0016] This application provides a model training apparatus, including:

[0017] The deformation prediction module is used to predict the deformation of multiple first facial sample image frames using a prediction model, and obtain the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the face of the target object moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image.

[0018] The rendering prediction module is used to perform rendering prediction based on the first facial sample image frame to obtain the predicted rendering image.

[0019] The first determining module is used to determine a first loss value based on the predicted deformation information and the target deformation information, and to determine a second loss value based on the predicted rendered image and the target rendered image.

[0020] The second determining module is used to determine a third loss value based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames.

[0021] The model update module is used to update the model parameters of the prediction model based on the first loss value, the second loss value, and the third loss value, so as to obtain the trained prediction model.

[0022] This application provides a control device, including:

[0023] An image acquisition module is used to receive multiple facial image frames to be processed, wherein the multiple facial image frames to be processed are acquired when the target object's face moves;

[0024] The deformation prediction module is used to call the trained prediction model to perform deformation prediction on each of the facial image frames to be processed, and obtain the predicted deformation information of each of the facial image frames to be processed. The trained prediction model is trained using the model training method described above, and the predicted deformation information includes displacement data of multiple facial vertices.

[0025] A facial motion module is used to control the bionic device to drive multiple facial vertices to move based on multiple displacement data for each of the facial image frames to be processed.

[0026] This application provides an electronic device, the electronic device comprising:

[0027] Memory is used to store executable instructions or computer programs.

[0028] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method or control method provided in the embodiments of this application.

[0029] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the model training method or control method provided in this application.

[0030] This application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the model training method or control method provided in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] By applying the embodiments of this application, a prediction model is used to predict the deformation of multiple first facial sample image frames, thereby obtaining the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the target object's face moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image. Then, rendering prediction is performed based on the first facial sample image frames to obtain the predicted rendered image. Based on the predicted deformation information and target deformation information, a first loss value is determined, and based on the predicted rendered image and target rendered image, a second loss value is determined. This allows the prediction model to be trained by combining the geometric deformation prediction dimension and the visual rendering dimension. Based on the predicted deformation information and target deformation information of multiple first facial sample image frames, a third loss value is determined, thereby achieving joint supervision of temporal dynamic deformation and optimizing the prediction model's ability to predict continuous facial movements or expression changes. Finally, based on the first loss value, the second loss value, and the third loss value, the model parameters of the prediction model are updated to obtain the trained prediction model. In this way, by comprehensively utilizing the loss values ​​between predicted deformation information and target deformation information, the loss values ​​between predicted rendered image and target rendered image, and the loss values ​​between predicted deformation information and target deformation information of multiple first facial sample image frames, the model parameters of the prediction model are updated, thereby improving the accuracy of facial deformation prediction using the prediction model for face images. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the application mode of the model training method provided in the embodiments of this application;

[0034] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0035] Figure 3A This is a schematic diagram of the first process of the model training method provided in the embodiments of this application;

[0036] Figure 3B This is a schematic diagram of the second process of the model training method provided in the embodiments of this application;

[0037] Figure 3C This is a schematic diagram of the third process of the model training method provided in the embodiments of this application;

[0038] Figure 3D This is a schematic diagram of the training process structure of the prediction model provided in the embodiments of this application;

[0039] Figure 3E This is a flowchart illustrating the control method provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the prediction model structure provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the training process of the prediction model guided by the rendering graph provided in the embodiments of this application.

[0042] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0046] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0048] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0049] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0050] 1) Deformation information (BlendShape): also known as the deformable target or target shape, including displacement data of multiple face vertices.

[0051] 2) Rendered image: refers to the process of converting an abstract three-dimensional human face model into a visualized two-dimensional image through computer algorithms or physical simulation technology.

[0052] This application provides a model training method, control method, device, electronic device, and computer-readable storage medium that can accurately predict facial deformations in face images.

[0053] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the device is implemented as a server.

[0054] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the model training method provided in the embodiments of this application, for example. Figure 1 The system involves server 200, network 300, and terminal 400. Terminal 400 connects to server 200 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0055] During model training, the user triggers a model training command for the prediction model on terminal 400. Terminal 400 responds to the user's command by sending a model training request to server 200. Server 200 responds to the request by acquiring multiple first facial sample image frames collected by terminal 400 and using the prediction model to predict deformation in these frames, obtaining the predicted deformation information for each frame. These first facial sample image frames are acquired during facial movement of the target object. The sample labels for each frame include target deformation information and a target rendered image. Rendering prediction is then performed based on the first facial sample image frames to obtain a predicted rendered image. A first loss value is determined based on the predicted deformation information and target deformation information, and a second loss value is determined based on the predicted rendered image and the target rendered image. A third loss value is determined based on the predicted deformation information and target deformation information of the multiple first facial sample image frames. Finally, the model parameters of the prediction model are updated based on the first, second, and third loss values ​​to obtain the trained prediction model.

[0056] When performing deformation prediction, server 200 receives multiple facial image frames to be processed, which are acquired during the movement of the target object's face; it calls the trained prediction model to perform deformation prediction on each facial image frame to obtain the predicted deformation information of each facial image frame, which includes the displacement data of multiple facial vertices; and sends the predicted deformation information of each facial image frame to terminal 400, so that terminal 400 generates control commands based on the predicted deformation information, and executes the control commands, thereby driving multiple facial vertices to move based on multiple displacement data. Terminal 400 can be a bionic device.

[0057] In some embodiments, the server (e.g., server 200) can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0058] See Figure 2 , Figure 2This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, and at least one network interface 420. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0059] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0061] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0062] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0063] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0064] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0065] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A model training device 455 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: deformation prediction module 4551, rendering prediction module 4552, first determination module 4553, second determination module 4554, and model update module 4555. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0066] The model training method provided in this application will be described in conjunction with exemplary applications and implementations of the server devices provided in the embodiments of this application.

[0067] The following describes the model training method provided in the embodiments of this application. For example, in order to facilitate understanding of the model training method provided in the embodiments of this application, this embodiment is described using the application in the scenario of predicting facial deformation of human face images as an example. In practical applications, the model training method and control method provided in the embodiments of this application can also be applied to facial deformation prediction scenarios such as animal face images and cartoon face images.

[0068] As mentioned above, the electronic device implementing the model training method of this application embodiment can be a terminal, a server, or a combination of both. The following explanation uses an electronic device as a server as an example to illustrate the model training method provided in this application embodiment. See also... Figure 3A , Figure 3A This is a schematic diagram of the first process of the model training method provided in the embodiments of this application, which will be combined with Figure 3A The steps shown are explained.

[0069] In step 301, the prediction model is used to predict the deformation of multiple first facial sample image frames, and the predicted deformation information of each first facial sample image frame is obtained.

[0070] Here, the prediction model can be built based on a lightweight convolutional neural network (ShuffleNetv2). Multiple first facial sample image frames are acquired during the facial movement of the target object. Multiple facial motion images of the target object can be acquired using the eye camera of a bionic device. Each facial motion image is then cropped to obtain multiple first facial sample image frames. Deformation prediction involves predicting facial deformation (BlendShape) from these multiple first facial sample image frames. The predicted deformation information is 52-dimensional deformation information, which consists of 52 independent facial deformation information segments. Each facial deformation information segment corresponds to a specific facial muscle movement or expression unit, and each segment contains displacement data for multiple facial vertices. The sample labels for each first facial sample image frame include target deformation information and a target rendered image. The target deformation information is the actual 52-dimensional deformation information corresponding to the first facial sample image frame, and the target rendered image is obtained by rendering a 3D face model using the target deformation information.

[0071] In some embodiments, a prediction model is used to predict the deformation of multiple first facial sample image frames, and the predicted deformation information of each first facial sample image frame is obtained. This can be achieved through the following steps: For each first facial sample image frame, the first facial sample image frame is convolved to obtain a second facial sample image frame; the second facial sample image frame is convolved and feature-enhanced to obtain a third facial sample image frame; the third facial sample image frame is convolved and feature-enhanced to obtain a fourth facial sample image frame; the fourth facial sample image frame is convolved to obtain a fifth facial sample image frame; the fifth facial sample image frame is pooled to obtain the predicted features of the first facial sample image frame; and the predicted features of the first facial sample image frame are linearly processed to obtain the predicted deformation information of the first facial sample image frame.

[0072] Here, for each first facial sample image frame, the convolution module (conv3*3) in the prediction model performs one convolution operation on the first facial sample image frame to obtain the second facial sample image frame. Using the convolution module (conv3*3) and the feature enhancement module (bottleneck) in the prediction model, the second facial sample image frame undergoes one convolution operation and five feature enhancement operations respectively to obtain the third facial sample image frame. Using the convolution module (conv3*3) and the feature enhancement module (bottleneck) in the prediction model, the third facial sample image frame undergoes one convolution operation and seven feature enhancement operations respectively to obtain the fourth facial sample image frame. Using the convolution module (conv3*3) in the prediction model, the fourth facial sample image frame undergoes one convolution operation to obtain the fifth facial sample image frame. Using the global mean pooling module in the prediction model, the fifth facial sample image frame undergoes one pooling operation to obtain the predicted features of the first facial sample image frame. Using the fully connected module in the prediction model, the predicted features of the first facial sample image frame are processed linearly twice to obtain the predicted deformation information of the first facial sample image frame, that is, the predicted 52-dimensional deformation information of the first facial sample image frame.

[0073] For example, taking a first facial sample image frame of 112*112*3 (width 112, height 112, number of channels 3), convolution processing is performed on the first facial sample image frame to obtain a second facial sample image frame, which is 56*56*64. Convolution processing and feature enhancement processing are performed on the second facial sample image frame to obtain a third facial sample image frame, which is 28*28*64. Convolution processing and feature enhancement processing are performed on the third facial sample image frame to obtain a fourth facial sample image frame, which is 14*14*16. Convolution processing is performed on the fourth facial sample image frame to obtain a fifth facial sample image frame, which is 7*7*32. Pooling processing is performed on the fifth facial sample image frame to obtain the predicted features of the first facial sample image frame, which are represented as 32-channel features. The predicted features of the first facial sample image frame are linearly processed to obtain the predicted deformation information of the first facial sample image frame. The predicted deformation information of the first facial sample image frame is represented as features of 52 channels.

[0074] In this embodiment, a series of convolutional processing, feature enhancement processing, pooling processing, and linear processing are performed on the first facial sample image frame to obtain the predicted features of the first facial sample image frame. The predicted features of the first facial sample image frame are then linearly processed to obtain the predicted deformation information of the first facial sample image frame. This allows for the gradual extraction of key feature information from the original facial sample image frame, thereby achieving high-precision facial deformation prediction of the prediction model.

[0075] Continue to refer to Figure 3A In step 302, rendering prediction is performed based on the first facial sample image frame to obtain the predicted rendering image.

[0076] Here, through the aforementioned processes of convolution, feature enhancement, pooling, and linear processing on the first facial sample image frame, the second, third, fourth, and fifth facial sample image frames are obtained sequentially. Based on these five facial sample image frames, a prediction is made for the rendered image of the first facial sample image frame, resulting in a predicted rendered image.

[0077] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the model training method provided in the embodiments of this application. Figure 3A Step 302 shown can be achieved through... Figure 3B Steps 3021 to 3026 are implemented, and will be explained in detail below.

[0078] In step 3021, feature extraction is performed on the fifth facial sample image frame to obtain the first sample feature, and feature extraction is performed on the fourth facial sample image frame to obtain the second sample feature.

[0079] Here, the convolutional module (conv3*3) in the prediction model is used to extract features from the fifth facial sample image frame, obtaining the first sample feature. The convolutional module (conv3*3) in the prediction model is then used to extract features from the fourth facial sample image frame, obtaining the second sample feature. For example, the fifth facial sample image frame is 7*7*32, and feature extraction from it yields the first sample feature, which is 7*7*16. The fourth facial sample image frame is 14*14*16, and feature extraction from it yields the second sample feature, which is 14*14*8.

[0080] In step 3022, the first sample features and the second sample features are concatenated to obtain the first concatenated features, and the first concatenated features are convolved to obtain the first convolution result.

[0081] Here, the first sample features and the second sample features are concatenated to obtain the first concatenated feature. The first concatenated feature is then convolved using the convolution module (conv3*3) in the prediction model to obtain the first convolution result, ensuring that the dimension of the first convolution result is the same as the dimension of the fourth facial sample image frame. Continuing with the above embodiment, the first sample feature is 7*7*16, and the second sample feature is 14*14*8. The first sample feature and the second sample feature are concatenated to obtain the first concatenated feature, which is 14*14*32. The first concatenated feature is then convolved to obtain the first convolution result, which is 14*14*16. The dimension of the first convolution result is the same as the dimension of the fourth facial sample image frame (14*14*16).

[0082] In step 3023, feature extraction is performed on the third facial sample image frame to obtain the third sample feature, and the third sample feature and the first convolution result are concatenated to obtain the second concatenated feature. The second concatenated feature is then convolved to obtain the second convolution result.

[0083] Here, the convolutional module (conv3*3) in the prediction model is used to extract features from the third facial sample image frame, obtaining the third sample features. The third sample features and the first convolution result are concatenated to obtain the second concatenated features. Then, the convolutional module (conv3*3) in the prediction model is used to perform convolution processing on the second concatenated features to obtain the second convolution result, ensuring that the dimension of the second convolution result is the same as the dimension of the third facial sample image frame. Continuing with the above embodiment, the third facial sample image frame is 28*28*64. Feature extraction from the third facial sample image frame yields the third sample features as 28*28*32. The first convolution result is 14*14*16. The third sample features and the first convolution result are concatenated to obtain the second concatenated features as 28*28*128. Convolution processing on the second concatenated features yields the second convolution result as 28*28*64.

[0084] In step 3024, feature extraction is performed on the second facial sample image frame to obtain the fourth sample feature, and the fourth sample feature and the second convolution result are concatenated to obtain the third concatenated feature. The third concatenated feature is then convolved to obtain the third convolution result.

[0085] Here, the convolutional module (conv3*3) in the prediction model is used to extract features from the second facial sample image frame, obtaining the fourth sample feature. The fourth sample feature and the second convolution result are concatenated to obtain the third concatenated feature. Then, the convolutional module (conv3*3) in the prediction model is used to convolve the third concatenated feature to obtain the third convolution result, ensuring that the dimension of the third convolution result is the same as the dimension of the second facial sample image frame. Continuing with the above embodiment, the second facial sample image frame is 56*56*64. Feature extraction from the second facial sample image frame yields a fourth sample feature of 56*56*32. The second convolution result is 28*28*64. The fourth sample feature and the second convolution result are concatenated to obtain a third concatenated feature of 56*56*128. Convolution is then performed on the third concatenated feature to obtain a third convolution result of 56*56*64.

[0086] In step 3025, feature extraction is performed on the first facial sample image frame to obtain the fifth sample feature, and the fifth sample feature and the third convolution result are concatenated to obtain the fourth concatenated feature.

[0087] Here, the convolutional module (conv3*3) in the prediction model is used to extract features from the first facial sample image frame, resulting in the fifth sample feature. The fifth sample feature and the result of the third convolution are then concatenated to obtain the fourth concatenated feature. Continuing with the above embodiment, the first facial sample image frame is 112*112*3. Feature extraction from the first facial sample image frame yields the fifth sample feature, which is 112*112*64. The result of the third convolution is 56*56*64. The fifth sample feature and the result of the third convolution are then concatenated to obtain the fourth concatenated feature, which is 112*112*128.

[0088] In step 3026, rendering prediction is performed based on the fourth stitching feature to obtain the predicted rendering image.

[0089] Here, a linear transformation is performed on the fourth concatenated feature, and the channel dimension is adjusted using the convolution module (conv1*1) in the prediction model to obtain the processed feature. The processed feature is then mapped to the target image space (such as RGB channels) to obtain the predicted rendering image.

[0090] In this embodiment, the sample features of the first, second, third, fourth, and fifth facial sample image frames are stitched together to obtain a fourth stitched feature. Based on the fourth stitched feature, rendering prediction is performed to obtain a predicted rendered image, which can improve the detail richness and noise robustness of the predicted rendered image.

[0091] Continue to refer to Figure 3AIn step 303, a first loss value is determined based on the predicted deformation information and the target deformation information, and a second loss value is determined based on the predicted rendered image and the target rendered image.

[0092] Here, the predicted deformation information and the target deformation information of the first sample image frame are used to calculate the loss value, resulting in the first loss value. The loss value between the predicted rendered image and the real rendered image is calculated using the smoothed absolute value loss function (L1_Smooth), resulting in the second loss value.

[0093] In some embodiments, determining a first loss value based on predicted deformation information and target deformation information can be achieved through the following steps: determining a first error value between the predicted deformation information and the target deformation information; when the absolute value of the first error value is less than a first preset value, performing a logarithmic transformation on the first error value to obtain a first logarithmic value; and determining the product of the first preset value and the first logarithmic value as the first loss value; when the absolute value of the first error value is greater than or equal to the first preset value, determining the difference between the absolute value of the first error value and a second preset value as the first loss value.

[0094] Here, the positional error of the corresponding facial vertices in the predicted deformation information and the target deformation information is determined as the first error value between the predicted deformation information and the target deformation information. When the absolute value of the first error value is less than the first preset value, the first error value is divided by the preset parameter, and the quotient is logarithmically transformed to obtain the first logarithmic value. The product of the first preset value and the first logarithmic value is determined as the first loss value. At this time, the first loss value can be expressed as wrn(1+|x| / ∈), where x represents the first error value, w represents the first preset value, and ∈ represents the preset parameter.

[0095] When the absolute value of the first error value is greater than or equal to the first preset value, the absolute value of the first error value is subtracted from the second preset value, and the difference is determined as the first loss value. The first loss value can then be expressed as |x|-C, where x represents the first error value and C represents the second preset value. For example, the first loss value can be determined using the following formula (1):

[0096]

[0097] Where WingLoss represents the first loss value, x represents the first error value, w represents the first preset value, ∈ represents the preset parameter, and C represents the second preset value.

[0098] In this embodiment, a first error value between the predicted deformation information and the target deformation information is determined, and a first loss value of the prediction model is determined based on the absolute value of the first error value and the magnitude of a first preset value. By combining the absolute value of the error and the preset threshold in a segmented loss design, the prediction model can ensure efficient optimization of small errors while suppressing the negative impact of large errors.

[0099] In some embodiments, determining the second preset value can be achieved through the following steps: performing a logarithmic transformation on the first preset value to obtain a second logarithmic value, and determining the product of the second logarithmic value and the first preset value as the first parameter value; and determining the difference between the first parameter value and the first preset value as the second preset value.

[0100] Here, the first preset value is divided by the preset parameter, and the quotient is logarithmically transformed to obtain the second logarithmic value. The second logarithmic value is then multiplied by the first preset value, and the product is determined as the first parameter value. The first parameter value is then subtracted from the first preset value, and the difference is determined as the second preset value. For example, the second preset value can be represented as w - wln(1 + w / ∈), where w represents the first preset value and ∈ represents the preset parameter.

[0101] In some embodiments, determining a second loss value based on a predicted rendered image and a target rendered image can be achieved through the following steps: determining a second error value between the predicted rendered image and the target rendered image; determining the second error value as the input value of a preset smooth absolute value loss function; determining the output value of the smooth absolute value loss function; and determining the output value as the second loss value.

[0102] Here, the difference between the pixel values ​​in the predicted rendered image and the target rendered image is determined as the second error value, and the second error value is input into the preset smooth absolute value loss function (L1_Smooth), and the output value of the smooth absolute value loss function is determined as the second loss value.

[0103] In this embodiment, a second error value between the predicted rendered image and the target rendered image is determined, and the second error value is input into a preset smooth absolute value loss function to obtain the second loss value of the prediction model. This can simultaneously take into account accuracy and noise resistance, thereby improving the stability and quality of the prediction model in rendering image prediction.

[0104] Continue to refer to Figure 3A In step 304, a third loss value is determined based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames.

[0105] In some embodiments, see Figure 3C , Figure 3CThis is a schematic diagram of the third process of the model training method provided in the embodiments of this application. Figure 3A Step 304 shown can be achieved through Figure 3C Steps 3041 to 3044 are implemented, and will be explained in detail below.

[0106] In step 3041, a difference constraint is performed based on the predicted deformation information of the (i-1)th first face sample image frame, the predicted deformation information of the ith first face sample image frame, the target deformation information of the (i-1)th first face sample image frame, and the target deformation information of the ith first face sample image frame to obtain the ith first constraint value.

[0107] Here, i takes the value of an integer from 1 to N-1, where N is the total number of first facial sample image frames. Two adjacent first facial sample image frames are determined from the multiple first facial sample image frames, namely the (i-1)th and the ith first facial sample image frame. The predicted deformation information and target deformation information corresponding to the (i-1)th and ith first facial sample image frames are subjected to difference constraints to obtain the ith first constraint value.

[0108] In some embodiments, the i-th first constraint value is obtained by performing a difference constraint on the predicted deformation information of the (i-1)th first facial sample image frame, the predicted deformation information of the i-th first facial sample image frame, the target deformation information of the (i-1)th first facial sample image frame, and the target deformation information of the i-th first facial sample image frame. This can be achieved through the following steps: performing a difference processing on the predicted deformation information of the (i-1)th first facial sample image frame and the predicted deformation information of the i-th first facial sample image frame to obtain a first difference value; performing a difference processing on the target deformation information of the (i-1)th first facial sample image frame and the target deformation information of the i-th first facial sample image frame to obtain a second difference value; and performing a difference processing on the first difference value and the second difference value to obtain the i-th first constraint value.

[0109] Here, the predicted deformation information of the (i-1)th first facial sample image frame is subtracted from the predicted deformation information of the ith first facial sample image frame, and the resulting difference is determined as the first difference. The target deformation information of the (i-1)th first facial sample image frame is subtracted from the target deformation information of the ith first facial sample image frame, and the resulting difference is determined as the second difference. The first difference and the second difference are subtracted, and the resulting difference is determined as the ith first constraint value. For example, the ith first constraint value can be represented as... in, This represents the predicted deformation information of the (i-1)th first facial sample image frame. y represents the predicted deformation information of the i-th first facial sample image frame. ti-1 y represents the target deformation information of the (i-1)th first facial sample image frame. t i This represents the target deformation information of the i-th first facial sample image frame.

[0110] Continue to refer to Figure 3C In step 3042, a difference constraint is performed based on the predicted deformation information of the (j-2)th first face sample image frame, the predicted deformation information of the jth first face sample image frame, the target deformation information of the (j-2)th first face sample image frame, and the target deformation information of the jth first face sample image frame to obtain the jth second constraint value.

[0111] Here, j takes the value of an integer from 2 to N-1. The predicted deformation information of the (j-2)th first facial sample image frame is subtracted from the predicted deformation information of the j-th first facial sample image frame, and the resulting difference is determined as the third difference. The target deformation information of the (j-2)th first facial sample image frame is subtracted from the target deformation information of the j-th first facial sample image frame, and the resulting difference is determined as the fourth difference. The third difference and the fourth difference are subtracted, and the resulting difference is determined as the i-th second constraint value. For example, the i-th second constraint value can be represented as... in, This represents the predicted deformation information of the j-th first facial sample image frame. y represents the predicted deformation information of the (j-2)th first facial sample image frame. t j y represents the target deformation information of the j-th first facial sample image frame. t j-2 This represents the target deformation information of the (j-2)th first facial sample image frame.

[0112] Continue to refer to Figure 3C In step 3043, the first constraint mean of N-1 first constraint values ​​is determined, and the second constraint mean of N-2 second constraint values ​​is determined.

[0113] Here, the N-1 first constraint values ​​are summed and averaged to obtain the mean of the first constraint. Similarly, the N-2 second constraint values ​​are summed and averaged to obtain the mean of the second constraint. For example, taking N = 128, the mean of the first constraint can be expressed as... The second constraint mean can be expressed as:

[0114] Continue to refer to Figure 3C In step 3044, the sum of the first constraint mean and the second constraint mean is determined as the third loss value.

[0115] Here, the mean of the first constraint and the mean of the second constraint are added together, and the sum is determined as the third loss value. For example, the third loss value can be determined using the following formula (2):

[0116]

[0117] Among them, Loss t This represents the third loss value. This represents the predicted deformation information of the (i-1)th first facial sample image frame. y represents the predicted deformation information of the i-th first facial sample image frame. t i-1 y represents the target deformation information of the (i-1)th first facial sample image frame. t i This represents the target deformation information of the i-th first facial sample image frame. This represents the predicted deformation information of the j-th first facial sample image frame. y represents the predicted deformation information of the (j-2)th first facial sample image frame. t j y represents the target deformation information of the j-th first facial sample image frame. t j-2 This represents the target deformation information of the (j-2)th first facial sample image frame.

[0118] In this embodiment, a third loss value is determined based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames, thereby achieving joint supervision of temporal dynamic deformation and optimizing the prediction model's ability to predict continuous facial movements or expression changes.

[0119] Continue to refer to Figure 3A In step 305, the model parameters of the prediction model are updated based on the first loss value, the second loss value, and the third loss value to obtain the trained prediction model.

[0120] Here, the model parameters of the prediction model are updated based on the first loss value, the second loss value, and the third loss value to train the prediction model and obtain the trained prediction model.

[0121] In some embodiments, updating the model parameters of the prediction model based on the first loss value, the second loss value, and the third loss value can be achieved through the following steps: determining the first weight value, the second weight value, and the third weight value corresponding to each of the first loss value, the second loss value, and the third loss value; using the first weight value, the second weight value, and the third weight value to perform weighted fusion of the first loss value, the second loss value, and the third loss value to obtain the total loss value of the prediction model; and updating the model parameters of the prediction model based on the total loss value.

[0122] Here, weights are assigned to the first, second, and third loss values ​​to obtain their respective first, second, and third weight values. Using these weights, the first, second, and third loss values ​​are weighted and summed to obtain the total loss value of the prediction model. Based on this total loss value, the model parameters are updated.

[0123] In this embodiment, by utilizing the first weight value, the second weight value, and the third weight value, the first loss value, the second loss value, and the third loss value are fused into the total loss value of the prediction model. Based on the total loss value, the model parameters of the prediction model are updated, thereby improving the accuracy of predicting facial deformation of face images using the prediction model.

[0124] Example, reference Figure 3D , Figure 3D This is a schematic diagram of the training process structure of the prediction model provided in this application embodiment. Using the prediction model 310, deformation prediction is performed on multiple first facial sample image frames 311, corresponding to the predicted deformation information of each first facial sample image frame. The sample labels of the first facial sample image frames include target deformation information 312 and a target rendered image 313. Rendering prediction is performed based on the multiple first facial sample image frames 311 to obtain a predicted rendered image. Based on the predicted deformation information and target deformation information 312, a first loss value is determined, and based on the predicted rendered image and the target rendered image 313, a second loss value is determined. Based on the predicted deformation information and target deformation information of the multiple first facial sample image frames 311, a third loss value is determined. Based on the first loss value, the second loss value, and the third loss value, the model parameters of the prediction model are updated to obtain the trained prediction model.

[0125] The control method provided in the embodiments of this application is described below. See also: Figure 3E , Figure 3E This is a flowchart illustrating the control method provided in the embodiments of this application, which will be combined with... Figure 3E The steps shown are explained.

[0126] In step 320, multiple facial image frames to be processed are received.

[0127] Here, multiple facial image frames to be processed are captured during the facial movement of the target object. Multiple facial motion images of the target object can be captured using the eye camera of a bionic device, and each facial motion image can be cropped to obtain multiple facial image frames to be processed.

[0128] In step 321, the trained prediction model is invoked to predict the deformation of each face image frame to be processed, thereby obtaining the predicted deformation information of each face image frame to be processed.

[0129] Here, the trained prediction model is obtained using the model training method provided in the embodiments of this application, and the predicted deformation information includes displacement data of multiple facial vertices. For each facial image frame to be processed, convolution processing is performed on the facial image frame to be processed to obtain a first facial image frame. Convolution processing and feature enhancement processing are performed on the first facial image frame to obtain a second facial image frame. Convolution processing and feature enhancement processing are performed on the second facial image frame to obtain a third facial image frame. Convolution processing is performed on the third facial image frame to obtain a fourth facial image frame. Pooling processing is performed on the fourth facial image frame to obtain the predicted features of the facial image frame to be processed. Linear processing is performed on the predicted features of the facial image frame to be processed to obtain the predicted deformation information of the facial image frame to be processed.

[0130] In step 322, for each facial image frame to be processed, the bionic device is controlled to drive multiple facial vertices to move based on multiple displacement data.

[0131] Here, the bionic device can be a bionic robot. It obtains displacement data of multiple facial vertices from the predicted deformation information of each facial image frame to be processed, and uses the multiple displacement data to control the bionic device to move multiple facial vertices, so as to drive the bionic device to perform real-time facial imitation motion.

[0132] Based on steps 320 to 322, by calling the trained prediction model, deformation prediction is performed on multiple facial image frames to be processed, and the predicted deformation information of each facial image frame to be processed is obtained. Based on the predicted deformation information of multiple facial image frames to be processed, the bionic device is controlled to move multiple facial vertices. This makes it easier to control the bionic device to perform facial imitation through motors, thereby improving the efficiency of the bionic device to perform real-time facial imitation.

[0133] In some embodiments, the model training method provided in this application can be applied in the field of cloud technology. On a cloud platform, a prediction model is used to predict deformation of multiple first facial sample image frames, obtaining predicted deformation information for each first facial sample image frame. These multiple first facial sample image frames are acquired during facial movement of the target object. The sample labels of the first facial sample image frames include target deformation information and a target rendered image. Then, rendering prediction is performed based on the first facial sample image frames to obtain a predicted rendered image. A first loss value is determined based on the predicted deformation information and the target deformation information, and a second loss value is determined based on the predicted rendered image and the target rendered image. This allows for training of the prediction model by combining geometric deformation prediction dimensions and visual rendering dimensions. A third loss value is determined based on the predicted deformation information and target deformation information of multiple first facial sample image frames, achieving joint supervision of temporal dynamic deformation and optimizing the prediction model's ability to predict continuous facial movements or expression changes. Finally, based on the first, second, and third loss values, the model parameters of the prediction model are updated to obtain the trained prediction model. In this way, by comprehensively utilizing the loss values ​​between predicted deformation information and target deformation information, the loss values ​​between predicted rendered image and target rendered image, and the loss values ​​between predicted deformation information and target deformation information of multiple first facial sample image frames, the model parameters of the prediction model are updated, thereby improving the accuracy of facial deformation prediction using the prediction model for face images.

[0134] The following will describe an exemplary application of the model training method provided in the embodiments of this application in the scenario of predicting facial deformation in face images.

[0135] In related technologies, the 3D representation of human faces is typically based on vertex or 3D deformable model methods, which cannot be used to mimic faces by controlling bionic robots with motors. This application addresses the problems of related technologies by proposing a model training method, which includes the following improvements compared to related technologies:

[0136] This application proposes a facial deformation (BlendShape) prediction model. Based on facial motion images, it predicts facial deformation information and uses this information to control motors to drive a bionic robot's facial mimicry. During the training of the prediction model, a rendered image is used as an auxiliary constraint (the second loss value in the above embodiment), and multi-frame motion constraints are introduced (the third loss value in the above embodiment) to make the bionic robot's facial mimicry process smoother.

[0137] In this embodiment, the predicted deformation information is based on 52-dimensional blended shape information predicted from a human face image, i.e., 52 independent facial deformation information. Each facial deformation information corresponds to a specific facial muscle movement or expression unit, and each facial deformation information contains displacement data of multiple vertices. For example, see reference... Figure 4 , Figure 4 This is a schematic diagram of the prediction model structure provided in the embodiments of this application.

[0138] A facial image and its corresponding real deformation information (target deformation information in the above embodiment) are acquired using a motion capture device. The facial image is preprocessed to obtain a first facial sample image frame 401. For example, the first facial sample image frame is 112*112*3. The first facial sample image frame 401 is input into a prediction model, which is built on a lightweight convolutional neural network (ShuffleNetv2). The prediction model performs a series of convolutional processing, feature enhancement processing, pooling processing, and linear processing on the first facial sample image frame 401, sequentially obtaining a second facial sample image frame 402, a third facial sample image frame 403, a fourth facial sample image frame 404, and a fifth facial sample image frame 405. The output of the prediction model is the predicted 52-dimensional deformation information. The processing procedure for the first facial sample image frame is shown in Table 1.

[0139] Table 1. Processing procedure for the first facial sample image frame.

[0140]

[0141]

[0142] To reduce the error in predicted deformation information, such as optimizing lip shape changes in facial deformation, a loss value is calculated between the predicted deformation information 406 and the actual deformation information 407 to obtain a first loss value 408. For example, the first loss value can be determined using the following formula (1):

[0143]

[0144] Where WingLoss represents the first loss value, ∈ represents the set hyperparameter, w represents the control amplification factor, x represents the positional error of the corresponding facial vertex in the predicted deformation information and the actual deformation information, and C is represented as w-wln(1+w / ∈). Since the ln(1+x) function responds much better to subtle differences than a quadratic function... Therefore, the above formula (1) can be used to achieve refined prediction of deformation information.

[0145] Assigning corresponding physical meaning to the predicted deformation information involves introducing an auxiliary branch into the prediction model to predict the rendered image of the facial sample frame, guiding the prediction model to better learn and refine the prediction of deformation information. For example, see reference... Figure 5 , Figure 5 This is a schematic diagram of the training process of the prediction model guided by the rendering graph provided in the embodiments of this application.

[0146] The training of the auxiliary branch follows the design of a U-shaped network. First, a 3x3 convolutional module is used to extract features from the last-level facial sample image frame. The extracted features are then concatenated with the features from the previous-level facial sample image frame to obtain concatenated features. Next, a 3x3 convolutional module is used to adjust the dimensions of the concatenated features, and the above steps are repeated to obtain the final predicted rendering image 501. Face rendering is performed using real deformation information to obtain the real rendering image 502. The loss value between the predicted rendering image and the real rendering image (the target rendering image in the above embodiment) is calculated using the L1_Smooth absolute value loss function to obtain the second loss value 503, thus reconstructing the rendering image. This helps the prediction model better understand and learn facial deformation information. The auxiliary branch is only applied during the training process of the prediction model. Figure 1 The predictive model in the model is used for facial deformation inference.

[0147] When training the prediction model, 128 facial sample image frames are set. Multiple facial sample image frames are input into the prediction model sequentially according to their chronological order. The loss values ​​between two adjacent facial sample image frames and between two facial sample image frames spanning multiple frames are calculated to obtain the third loss value. For example, the third loss value can be determined using the following formula (2):

[0148]

[0149] Among them, Loss t This represents the third loss value. Represents predicted deformation information, y t The true deformation information is represented by the above formula (2) to establish the multi-frame motion loss, which realizes the difference between the predicted deformation information difference between two adjacent frames, the difference between the true deformation information difference between two adjacent frames, and the difference between the predicted deformation information difference across frames and the difference between the true deformation information difference across frames.

[0150] Finally, based on the first loss value, the second loss value, and the third loss value obtained in the above process, the prediction model is trained to obtain the trained prediction model.

[0151] In the above scenario of predicting facial deformation in human face images, the eye camera of the bionic robot can be used to capture facial motion images. Based on the prediction model, facial deformation can be predicted from the facial motion images to obtain predicted deformation information. The predicted deformation information can then be used to control the motor to further drive the bionic robot to mimic the face in real time.

[0152] The following description continues to illustrate the exemplary structure of the model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the model training device 455 in the memory 450 may include: a deformation prediction module 4551, used to predict the deformation of multiple first facial sample image frames using a prediction model, and obtain the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the face of the target object moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image. A rendering prediction module 4552 is used to perform rendering prediction based on the first facial sample image frames to obtain a predicted rendered image. A first determination module 4553 is used to determine a first loss value based on the predicted deformation information and target deformation information, and to determine a second loss value based on the predicted rendered image and target rendered image. A second determination module 4554 is used to determine a third loss value based on the predicted deformation information and target deformation information of the multiple first facial sample image frames. A model update module 4555 is used to update the model parameters of the prediction model based on the first loss value, the second loss value and the third loss value to obtain the trained prediction model.

[0153] In some embodiments, the deformation prediction module 4551 is further configured to, for each first facial sample image frame, perform convolution processing on the first facial sample image frame to obtain a second facial sample image frame; perform convolution processing and feature enhancement processing on the second facial sample image frame to obtain a third facial sample image frame; perform convolution processing and feature enhancement processing on the third facial sample image frame to obtain a fourth facial sample image frame; perform convolution processing on the fourth facial sample image frame to obtain a fifth facial sample image frame; perform pooling processing on the fifth facial sample image frame to obtain the predicted features of the first facial sample image frame; and perform linear processing on the predicted features of the first facial sample image frame to obtain the predicted deformation information of the first facial sample image frame.

[0154] In some embodiments, the rendering prediction module 4552 is further configured to: extract features from a fifth facial sample image frame to obtain a first sample feature; extract features from a fourth facial sample image frame to obtain a second sample feature; concatenate the first sample feature and the second sample feature to obtain a first concatenated feature; convolve the first concatenated feature to obtain a first convolution result; extract features from a third facial sample image frame to obtain a third sample feature; concatenate the third sample feature and the first convolution result to obtain a second concatenated feature; convolve the second concatenated feature to obtain a second convolution result; extract features from a second facial sample image frame to obtain a fourth sample feature; concatenate the fourth sample feature and the second convolution result to obtain a third concatenated feature; convolve the third concatenated feature to obtain a third convolution result; extract features from a first facial sample image frame to obtain a fifth sample feature; concatenate the fifth sample feature and the third convolution result to obtain a fourth concatenated feature; and perform rendering prediction based on the fourth concatenated feature to obtain a predicted rendered image.

[0155] In some embodiments, the first determining module 4553 is further configured to determine a first error value between the predicted deformation information and the target deformation information; when the absolute value of the first error value is less than a first preset value, the first error value is logarithmically transformed to obtain a first logarithmic value, and the product of the first preset value and the first logarithmic value is determined as a first loss value; when the absolute value of the first error value is greater than or equal to the first preset value, the difference between the absolute value of the first error value and a second preset value is determined as a first loss value.

[0156] In some embodiments, the first determining module 4553 is further configured to perform a logarithmic transformation on the first preset value to obtain a second logarithmic value, and to determine the product of the second logarithmic value and the first preset value as a first parameter value; and to determine the difference between the first parameter value and the first preset value as a second preset value.

[0157] In some embodiments, the first determining module 4553 is further configured to determine a second error value between the predicted rendered image and the target rendered image; determine the second error value as the input value of a preset smooth absolute value loss function; determine the output value of the smooth absolute value loss function; and determine the output value as the second loss value.

[0158] In some embodiments, the second determining module 4554 is further configured to perform difference constraints based on the predicted deformation information of the (i-1)th first facial sample image frame, the predicted deformation information of the ith first facial sample image frame, the target deformation information of the (i-1)th first facial sample image frame, and the target deformation information of the ith first facial sample image frame to obtain the ith first constraint value, where i is an integer from 1 to N-1, and N is the total number of first facial sample image frames; perform difference constraints based on the predicted deformation information of the (j-2)th first facial sample image frame, the predicted deformation information of the jth first facial sample image frame, the target deformation information of the (j-2)th first facial sample image frame, and the target deformation information of the jth first facial sample image frame to obtain the jth second constraint value, where j is an integer from 2 to N-1; determine the first constraint mean of the N-1 first constraint values, and determine the second constraint mean of the N-2 second constraint values; and determine the sum of the first constraint mean and the second constraint mean as the third loss value.

[0159] In some embodiments, the model update module 4555 is further configured to determine the first weight value, the second weight value, and the third weight value corresponding to the first loss value, the second loss value, and the third loss value respectively; to perform weighted fusion of the first loss value, the second loss value, and the third loss value using the first weight value, the second weight value, and the third weight value to obtain the total loss value of the prediction model; and to update the model parameters of the prediction model based on the total loss value.

[0160] The following describes the exemplary structure of the control device provided in the embodiments of this application as a software module. In some embodiments, the software module in the control device may include: an image acquisition module, used to receive multiple facial image frames to be processed, which are acquired when the target object's face moves; a deformation prediction module, used to call a trained prediction model to perform deformation prediction on each facial image frame to be processed, and obtain the predicted deformation information of each facial image frame to be processed. The trained prediction model is trained using the model training method provided in the embodiments of this application, and the predicted deformation information includes displacement data of multiple facial vertices; and a facial movement module, used to control the bionic device to drive multiple facial vertices to move based on multiple displacement data for each facial image frame to be processed.

[0161] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the model training method or control method provided in this application.

[0162] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method or control method provided in this application. For example, ... Figure 3A The model training method is shown.

[0163] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0164] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0165] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0166] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0167] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Using a prediction model, deformation prediction is performed on multiple first facial sample image frames to obtain the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the face of the target object moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image. Rendering prediction is performed based on the first facial sample image frame to obtain a predicted rendered image; Based on the predicted deformation information and the target deformation information, a first loss value is determined, and based on the predicted rendered image and the target rendered image, a second loss value is determined. A third loss value is determined based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames. Based on the first loss value, the second loss value, and the third loss value, the model parameters of the prediction model are updated to obtain the trained prediction model.

2. The method according to claim 1, characterized in that, The method of using a prediction model to predict deformation in multiple first facial sample image frames, and obtaining the predicted deformation information for each first facial sample image frame, includes: For each of the first facial sample image frames, the first facial sample image frame is convolved to obtain the second facial sample image frame. The second facial sample image frame is subjected to convolution and feature enhancement processing to obtain the third facial sample image frame. The third facial sample image frame is subjected to convolution and feature enhancement processing to obtain the fourth facial sample image frame; The fourth facial sample image frame is convolved to obtain the fifth facial sample image frame; The fifth facial sample image frame is subjected to pooling processing to obtain the predicted features of the first facial sample image frame; Linear processing is performed on the predicted features of the first facial sample image frame to obtain the predicted deformation information of the first facial sample image frame.

3. The method according to claim 2, characterized in that, The step of rendering prediction based on the first facial sample image frame to obtain the predicted rendered image includes: Feature extraction is performed on the fifth facial sample image frame to obtain the first sample feature, and feature extraction is performed on the fourth facial sample image frame to obtain the second sample feature; The first sample feature and the second sample feature are concatenated to obtain a first concatenated feature, and the first concatenated feature is convolved to obtain a first convolution result; Feature extraction is performed on the third facial sample image frame to obtain the third sample feature. The third sample feature and the first convolution result are then concatenated to obtain the second concatenated feature. The second concatenated feature is then convolved to obtain the second convolution result. Feature extraction is performed on the second facial sample image frame to obtain the fourth sample feature, and the fourth sample feature and the second convolution result are concatenated to obtain the third concatenated feature. The third concatenated feature is then convolved to obtain the third convolution result. Feature extraction is performed on the first facial sample image frame to obtain the fifth sample feature, and the fifth sample feature and the third convolution result are concatenated to obtain the fourth concatenated feature; Based on the fourth stitching feature, a rendering prediction is performed to obtain the predicted rendering image.

4. The method according to claim 1, characterized in that, Determining the first loss value based on the predicted deformation information and the target deformation information includes: A first error value is determined between the predicted deformation information and the target deformation information. When the absolute value of the first error value is less than a first preset value, the first error value is logarithmically transformed to obtain a first logarithmic value. The product of the first preset value and the first logarithmic value is determined as the first loss value. When the absolute value of the first error value is greater than or equal to the first preset value, the difference between the absolute value of the first error value and the second preset value is determined as the first loss value.

5. The method according to claim 1, characterized in that, Determining the second loss value based on the predicted rendered image and the target rendered image includes: Determine a second error value between the predicted rendered image and the target rendered image; The second error value is determined as the input value of the preset smooth absolute value loss function, and the output value of the smooth absolute value loss function is determined. The output value is determined as the second loss value.

6. The method according to claim 4, characterized in that, The method further includes: Perform a logarithmic transformation on the first preset value to obtain a second logarithmic value, and determine the product of the second logarithmic value and the first preset value as the first parameter value; The difference between the first parameter value and the first preset value is determined as the second preset value.

7. The method according to any one of claims 1 to 6, characterized in that, The determination of the third loss value based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames includes: Based on the predicted deformation information of the (i-1)th first facial sample image frame, the predicted deformation information of the ith first facial sample image frame, the target deformation information of the (i-1)th first facial sample image frame, and the target deformation information of the ith first facial sample image frame, a difference constraint is performed to obtain the ith first constraint value, where i takes the value of an integer from 1 to N-1, and N is the total number of first facial sample image frames; Based on the predicted deformation information of the (j-2)th first facial sample image frame, the predicted deformation information of the jth first facial sample image frame, the target deformation information of the (j-2)th first facial sample image frame, and the target deformation information of the jth first facial sample image frame, a difference constraint is performed to obtain the jth second constraint value, where j takes the value of an integer from 2 to N-1; Determine the mean of the first constraints for N-1 first constraint values, and determine the mean of the second constraints for N-2 second constraint values; The sum of the first constraint mean and the second constraint mean is determined as the third loss value.

8. The method according to claim 7, characterized in that, The method of performing a difference constraint on the predicted deformation information of the (i-1)th first facial sample image frame, the predicted deformation information of the ith first facial sample image frame, the target deformation information of the (i-1)th first facial sample image frame, and the target deformation information of the ith first facial sample image frame to obtain the ith first constraint value includes: The predicted deformation information of the (i-1)th first facial sample image frame and the predicted deformation information of the ith first facial sample image frame are subjected to difference processing to obtain a first difference; The target deformation information of the (i-1)th first facial sample image frame and the target deformation information of the ith first facial sample image frame are subjected to difference processing to obtain a second difference; The first difference and the second difference are processed to obtain the i-th first constraint value.

9. The method according to any one of claims 1 to 5, characterized in that, The step of updating the model parameters of the prediction model based on the first loss value, the second loss value, and the third loss value includes: Determine the first weight value, the second weight value, and the third weight value corresponding to each of the first loss value, the second loss value, and the third loss value; The first loss value, the second loss value, and the third loss value are weighted and fused using the first weight value, the second weight value, and the third weight value to obtain the total loss value of the prediction model; Based on the total loss value, the model parameters of the prediction model are updated.

10. A control method, characterized in that, The method includes: Receive multiple facial image frames to be processed, wherein the multiple facial image frames to be processed are acquired when the target object’s face moves; The trained prediction model is invoked to predict the deformation of each of the facial image frames to be processed, thereby obtaining the predicted deformation information of each of the facial image frames to be processed. The trained prediction model is trained using the model training method described in any one of claims 1 to 9, and the predicted deformation information includes displacement data of multiple facial vertices. For each of the facial image frames to be processed, the bionic device is controlled to drive multiple facial vertices to move based on multiple displacement data.

11. A model training device, characterized in that, The device includes: The deformation prediction module is used to predict the deformation of multiple first facial sample image frames using a prediction model, and obtain the predicted deformation information of each first facial sample image frame. The multiple first facial sample image frames are collected when the face of the target object moves. The sample labels of the first facial sample image frames include target deformation information and target rendered image. The rendering prediction module is used to perform rendering prediction based on the first facial sample image frame to obtain the predicted rendering image. The first determining module is used to determine a first loss value based on the predicted deformation information and the target deformation information, and to determine a second loss value based on the predicted rendered image and the target rendered image; The second determining module is used to determine a third loss value based on the predicted deformation information of multiple first facial sample image frames and the target deformation information of multiple first facial sample image frames. The model update module is used to update the model parameters of the prediction model based on the first loss value, the second loss value, and the third loss value, so as to obtain the trained prediction model.

12. A control device, characterized in that, The device includes: An image acquisition module is used to receive multiple facial image frames to be processed, wherein the multiple facial image frames to be processed are acquired when the target object's face moves; The deformation prediction module is used to call the trained prediction model to perform deformation prediction on each of the facial image frames to be processed, and obtain the predicted deformation information of each of the facial image frames to be processed. The trained prediction model is trained using the model training method according to any one of claims 1 to 9. The predicted deformation information includes displacement data of multiple facial vertices. A facial motion module is used to control the bionic device to drive multiple facial vertices to move based on multiple displacement data for each of the facial image frames to be processed.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method according to any one of claims 1 to 9 or the control method according to claim 10.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 9 or the control method according to claim 10.

Images