Expression generation method, neural network training method, device, equipment and medium

Abstract

The present disclosure provides an expression generation method, a neural network training method, an apparatus, a device and a medium, relating to the technical field of virtual digital people, augmented reality, virtual reality, mixed reality, extended reality, metaverse and the like. The expression generation method comprises: obtaining face image data; performing feature extraction on the face image data to obtain a to-be-matched face feature; determining a target face feature matched with the to-be-matched face feature from a plurality of pre-determined template face features; determining a face driving parameter based on the target face feature; and driving generation of a face expression corresponding to the face image data according to the face driving parameter.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the technical fields of virtual digital humans, augmented reality, virtual reality, mixed reality, extended reality, metaverse, etc., and particularly to an expression generation method, a neural network training method, an expression generation device, a neural network training device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] Facial expression driving is a key technology in digital human applications, and how to accurately and naturally drive facial expressions has always been a technical challenge. Currently, mainstream facial expression driving is usually based on optical solutions, that is, using monocular / multi-view cameras to extract facial features from facial images, and then driving the virtual face based on the extracted features.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides an expression generation method, a neural network training method, an expression generation device, a neural network training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, an expression generation method is provided. The method includes: acquiring face image data; extracting features from the face image data to obtain face features to be matched; determining a target face feature that matches the face features to be matched from a plurality of pre-determined template face features; determining face driving parameters based on the target face feature; and driving the generation of a face expression corresponding to the face image data according to the face driving parameters.

[0007] According to one aspect of this disclosure, a method for training a neural network is provided. The method includes: acquiring multiple template face image data and multiple real face driving parameters corresponding to the multiple template face image data; extracting features from the multiple template face image data to obtain multiple template face features; inputting each of the multiple template face features into an initial parameter generation neural network to obtain multiple predicted face driving parameters corresponding to the multiple template face image data; and adjusting the parameters of the initial parameter generation neural network based on the multiple real face driving parameters and the multiple predicted face driving parameters to obtain a trained parameter generation neural network.

[0008] According to another aspect of this disclosure, an expression generation apparatus is provided. The apparatus includes: a first acquisition unit configured to acquire face image data; a first feature extraction unit configured to extract features from the face image data to obtain face features to be matched; a matching unit configured to determine a target face feature that matches the face features to be matched from a plurality of pre-determined template face features; a determining unit configured to determine face driving parameters based on the target face feature; and a driving unit configured to drive the generation of a face expression corresponding to the face image data according to the face driving parameters.

[0009] According to another aspect of this disclosure, a training apparatus for a neural network is provided. The apparatus includes: a second acquisition unit configured to acquire multiple template face image data and multiple real face driving parameters corresponding to the multiple template face image data; a second feature extraction unit configured to extract features from the multiple template face image data to obtain multiple template face features; a parameter generation unit configured to input each of the multiple template face features into an initial parameter generation neural network to obtain multiple predicted face driving parameters corresponding to the multiple template face images; and a first parameter tuning unit configured to adjust the parameters of the initial parameter generation neural network based on the multiple real face driving parameters and the multiple predicted face driving parameters to obtain a trained parameter generation neural network.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described above.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0013] According to one or more embodiments of this disclosure, by matching the facial features to be matched extracted from facial image data with a plurality of pre-determined template facial features, it is possible to obtain target facial features that are similar to the facial features to be matched but are more standard, easier to identify, and more aesthetically pleasing. In turn, facial driving parameters can be determined based on the target facial features to obtain more accurate, natural, and aesthetically pleasing facial expressions.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0017] Figure 2 A flowchart of an expression generation method according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 3 A flowchart illustrating feature extraction from facial image data according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 4 A flowchart of an expression generation method according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 5 A flowchart illustrating a method for training a neural network according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 6 A structural block diagram of an expression generation apparatus according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 7 A structural block diagram of a training apparatus for a neural network according to exemplary embodiments of the present disclosure is shown; and

[0023] Figure 8A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0026] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0027] In the three-dimensional facial expression driving methods of related technologies, because the facial expressions of different drivers vary greatly, and the size and distribution of face shapes / features cannot be completely consistent, a series of unnatural expression problems will occur during actual driving.

[0028] To address the aforementioned issues, this disclosure involves matching the facial features to be matched extracted from facial image data with multiple pre-determined template facial features. This results in target facial features that are similar to the facial features to be matched but are more standard, easier to identify, and more aesthetically pleasing. Furthermore, facial driving parameters can be determined based on the target facial features to obtain more accurate, natural, and aesthetically pleasing facial expressions.

[0029] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0031] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of emoji generation methods.

[0032] In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) network.

[0033] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0034] Users can use client devices 101, 102, 103, 104, 105, and / or 106 for human-computer interaction. The client devices can provide interfaces that allow users to interact with them; for example, a camera on the client device can capture images of the user in real time. The client devices can also output information to the user through this interface; for example, the client can output facial expressions generated by an expression generation method running on the server. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0035] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0036] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0037] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0038] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0039] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0040] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0041] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a data repository used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data from and from the database in response to commands.

[0042] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0043] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0044] According to one aspect of this disclosure, a method for generating facial expressions is provided. For example... Figure 2 As shown, the expression generation method includes: step S201, acquiring face image data; step S202, extracting features from the face image data to obtain face features to be matched; step S203, determining the target face feature that matches the face feature to be matched from a plurality of pre-determined template face features; step S204, determining face driving parameters based on the target face feature; and step S205, driving the generation of a face expression corresponding to the face image data according to the face driving parameters.

[0045] Therefore, by matching the facial features to be matched extracted from facial image data with multiple pre-determined template facial features, it is possible to obtain target facial features that are similar to the facial features to be matched but are more standard, easier to recognize, and more aesthetically pleasing. Then, facial driving parameters can be determined based on the target facial features to obtain more accurate, natural, and beautiful facial expressions.

[0046] In step S201, the face image data may be face image data acquired by a monocular camera or a multi-camera, or it may be image data including a face obtained by other means, which is not limited here.

[0047] In step S202, facial image data can be used to extract features to obtain facial features. In some embodiments, traditional non-deep learning feature extraction methods (e.g., facial landmark detection methods) can be used for feature extraction, or deep learning neural networks (e.g., deep convolutional neural networks, DCNNs) can be used for feature extraction. The facial features to be matched can be facial landmark features or feature maps.

[0048] In some embodiments, if feature extraction and 3D face driving are performed on face image data frame by frame, the lack of consideration for the continuity between consecutive frames can lead to jittering or abrupt changes in facial expressions, thus affecting the user experience. Therefore, this disclosure proposes a method for face driving based on facial motion features.

[0049] According to some embodiments, facial image data may include multiple consecutively acquired facial image frames, and the facial features to be matched may characterize the movement of the face in the facial image data between the multiple facial image frames. For example, the facial features to be matched may include motion features such as rotation and displacement of facial key points in the facial image data between the multiple facial image frames. Therefore, by acquiring multiple consecutive frames of facial images and extracting facial features that characterize the motion information of the face, the final generated facial expressions can be more coherent and natural.

[0050] According to some embodiments, such as Figure 3 As shown, step S202, extracting features from the face image data to obtain the face features to be matched, may include: step S301, extracting features from each face image frame in multiple face image frames to obtain multiple static face features corresponding to the multiple face image frames; and step S302, determining the face features to be matched based on the multiple static face features. By extracting the static features of each frame, and then extracting face features representing the motion information of the face based on these static features, it is possible to obtain more refined face features that better reflect the changes in face motion between frames, thereby improving the quality of the final generated face expression.

[0051] According to some embodiments, multiple face image frames can be acquired within a preset time interval, and each of the multiple static face features can include a preset number of keypoint sequences. Step S302, determining the face features to be matched based on the multiple static face features, can include: inputting the multiple static face features into a trained motion feature extraction neural network to obtain the face features to be matched. The face features to be matched can include feature vectors of a preset dimension. In this way, more effective face features to be matched can be obtained, thereby improving the quality of the final generated facial expression.

[0052] In one exemplary embodiment, the motion feature extraction neural network can employ a sequence model, such as a Long Short-Term Memory (LSTM) network. The preset time interval can be 200ms, the preset number of keypoint sequences can be a 68-keypoint sequence of a face, and the output face features to be matched can be 256-dimensional low-dimensional motion features. It is understood that the above settings are merely exemplary, and can be adjusted according to actual needs when implementing the method of this disclosure; no limitation is made herein.

[0053] In some embodiments, in step S202, multiple face image frames can also be processed directly to obtain the face features to be matched, representing the motion information of the face. For example, multiple face image frames can be directly input into a trained neural network to obtain the feature vector output by the neural network.

[0054] Before executing the expression generation method, a facial action feature library including multiple template facial features can be pre-constructed. According to some embodiments, these multiple template facial features can be determined by feature extraction from multiple template facial image data. As described above, due to differences in the shape of different drivers' faces, the distribution of face shapes / features cannot be completely consistent. Furthermore, when different drivers make the same expression, the amplitude and detail of facial movements are not entirely consistent. Therefore, directly using image features extracted from the driver's image data to obtain the corresponding facial driving parameters may result in unnatural facial expressions. In addition, the driver's image data and corresponding image features may be "new" to the downstream unit or model generating facial driving parameters. Therefore, the parameter generation unit / model may not accurately capture the driver's expression information or details, resulting in the generated facial driving parameters failing to accurately reflect the driver's expression.

[0055] By collecting a large number of template facial image data of standard facial expressions and movements made by models, and extracting features from these image data to obtain multiple corresponding template facial features, the user's facial expressions and movements are matched with a large number of standard facial expressions and movements of the model when generating facial driving parameters. The template facial features of the matched standard facial expressions and movements are then used to obtain facial driving parameters. This allows the facial driving parameter generation unit / model to better understand the expression that the driver wants to express, and makes the generated facial expressions more standard, more beautiful, and more natural.

[0056] It is understandable that the method for extracting template face features can be consistent with the method for extracting the face features to be matched. In an exemplary embodiment, the template face features can be 256-dimensional motion features obtained by processing the 68 key point sequences of each face in multiple model face image frames acquired within 200ms using an LSTM network.

[0057] In some embodiments, image data of a model making different facial movements or expressions can be collected. Exemplary facial movements or expressions may include: raising the inner corner of the eyebrow, raising the outer corner of the eyebrow, lowering the eyebrow, raising the cheek, stretching the corner of the lip, raising the chin, closing the eyes, blinking, etc. In some embodiments, these labeling information may not be required when using template facial features corresponding to these facial movements or expressions. In step S203, these target facial features can be directly matched with the facial features to be matched extracted from the driver's facial image data to determine the target facial features; in step S204, the corresponding facial driving parameters can be directly determined using the target facial features, as will be described below.

[0058] According to some embodiments, step S203, determining the target face feature that matches the face feature to be matched from a predetermined plurality of template face features, may include: performing similarity matching between the plurality of template face features and the face feature to be matched, so as to determine the target face feature from the plurality of template face features. By using similarity matching, the template face feature that is closest to the face feature to be matched can be obtained.

[0059] In some embodiments, cosine distance can be used as a measure of similarity. In some embodiments, methods such as kd-trees can be used to accelerate the matching process.

[0060] According to some embodiments, step S204, determining face driving parameters based on target face features, may include: inputting the target face features into a trained parameter generation neural network to obtain face driving parameters.

[0061] Because neural networks are highly sensitive to input, poor-quality input data (or training data) can prevent them from producing accurate results. Therefore, using more standardized, easily recognizable, and aesthetically pleasing target facial features can yield face-driving parameters that generate more accurate, natural, and attractive facial expressions.

[0062] In some embodiments, template face features can be used to train the parameter generation neural network. Since the input to the parameter generation neural network in practical applications is a pre-determined template face feature and does not include face features from different drivers, there is no generalization requirement for the neural network, and template face features can be used to train it. Furthermore, using a neural network to model the mapping relationship between template face features and face driving parameters avoids directly storing these mapping relationships, thus greatly saving system storage resources.

[0063] According to some embodiments, face driving parameters may include at least one of hybrid deformation weights and face bone parameters. It is understood that face driving parameters may also include other parameters, which are not limited herein. In some embodiments, the driven face may be a two-dimensional face or a three-dimensional face, and the face driving parameters may accordingly include two-dimensional face driving parameters or three-dimensional face driving parameters.

[0064] In some embodiments, such as Figure 4 As shown, the expression generation method may include the following steps.

[0065] Preliminary step S400: Facial motion feature library acquisition. In this preliminary step, a large number of template facial image data of standard facial expressions and movements made by the model can be acquired in advance. These image data are then input into a motion feature extraction neural network to obtain multiple corresponding template facial features (i.e., a facial motion feature library). It is understood that the preliminary step can be performed before the expression generation method is executed, and does not necessarily need to be performed before each expression generation method execution.

[0066] Step S401: Monocular / Multi-view Camera Image Acquisition. A monocular or multi-view camera can be used to acquire face image data for the current frame.

[0067] Step S402: Facial Feature Extraction. Facial features can be extracted from the facial image data acquired in step S401.

[0068] Step S403: Facial Feature Caching. Facial features within a preset time period (e.g., 200ms) can be cached.

[0069] Step S404: Motion Feature Extraction. Facial features within a preset time period can be input into a motion feature extraction neural network for motion feature extraction.

[0070] Step S405: Motion Feature Matching. The motion features obtained in step S404 can be input into the facial action feature library collected in the previous step S400 for matching to obtain the closest template facial features.

[0071] Step S406: Obtaining Face-Driven Parameters. The template face feature input parameters matched in step S405 can be used to generate a neural network to obtain the face-driven parameters.

[0072] Step S407, Facial Expression Driving. The facial driving parameters obtained in step S406 can be input into the facial model to generate facial expressions for the virtual digital human.

[0073] Therefore, by performing facial feature caching and motion feature matching, the jitter and jump problems caused by single-frame input and single-frame output can be solved. Furthermore, by using a facial motion feature library for collection and matching, the facial expressions of the generated virtual digital human can be made more natural.

[0074] According to another aspect of this disclosure, a method for training a neural network is provided. For example... Figure 5As shown, the method includes: step S501, acquiring multiple template face image data and multiple real face driving parameters corresponding to the multiple template face image data; step S502, extracting features from the multiple template face image data to obtain multiple template face features; step S503, inputting each template face feature from the multiple template face features into an initial parameter generation neural network to obtain multiple predicted face driving parameters corresponding to the multiple template face image data; and step S504, adjusting the parameters of the initial parameter generation neural network based on the multiple real face driving parameters and the multiple predicted face driving parameters to obtain a trained parameter generation neural network. It is understood that... Figure 5 Some of the operations in steps S501-S503 have been described above and will not be repeated here.

[0075] In some embodiments, multiple real face driving parameters corresponding to multiple template face image data can be determined by annotation.

[0076] According to some embodiments, each template face image data in a plurality of template face image data may include a plurality of consecutive template face image frames. Template face features can characterize the movement of the face in the corresponding template face image data between the plurality of template face image frames included in the template face image data. Template face features may, for example, include motion features such as rotation and displacement of facial key points in the template face image data between the plurality of template face image frames.

[0077] According to some embodiments, feature extraction of multiple template face image data to obtain multiple template face features may include: for each template face image data in the multiple template face image data, feature extraction of each template face image frame in the multiple template face image frames included in the template face image data to obtain multiple static template features corresponding to the multiple template face image frames; and determining the template face features corresponding to the template face image data based on the multiple static template features.

[0078] According to some embodiments, each of the multiple static template face features may include a preset number of keypoint sequences. Determining the template face feature corresponding to the template face image data based on the multiple static template features may include: inputting the multiple static template features into an initial motion feature extraction neural network to obtain the template face feature corresponding to the template face image data. It is understood that feature extraction of the template face image data can refer to... Figure 2 Step S202 in the process will not be described in detail here.

[0079] In some embodiments, the training method may further include: adjusting the parameters of an initial motion feature extraction neural network based on multiple real face driving parameters and multiple predicted face driving parameters to obtain a trained motion feature extraction neural network.

[0080] In some embodiments, a loss function can be determined to evaluate the difference between the real face-driving parameters and the predicted face-driving parameters, and a loss value with respect to the real face-driving parameters and the predicted face-driving parameters can be calculated based on the loss function. The parameters of the initial parameter generation neural network (and optionally, the parameters of the initial motion feature extraction neural network) can then be adjusted based on the loss value. It is understood that those skilled in the art can also adjust the parameters of the above-described neural network in other ways, which are not limited here.

[0081] In some embodiments, the initial parameter generating neural network (and optionally, the initial motion feature extraction neural network) may be obtained through random initialization or through prior training, without limitation.

[0082] According to another aspect of this disclosure, an expression generation apparatus is provided. For example... Figure 6 As shown, the device 600 includes: a first acquisition unit 610 configured to acquire face image data; a first feature extraction unit 620 configured to extract features from the face image data to obtain face features to be matched; a matching unit 630 configured to determine a target face feature that matches the face features to be matched from a plurality of pre-determined template face features; a determining unit 640 configured to determine face driving parameters based on the target face feature; and a driving unit 650 configured to drive the generation of a face expression corresponding to the face image data according to the face driving parameters. It can be understood that the operations of units 610-650 in the device 600 are respectively related to... Figure 2 The operations of steps S201-S205 are similar and will not be described in detail here.

[0083] According to some embodiments, multiple template face features can be determined by feature extraction from multiple template face image data.

[0084] According to some embodiments, the determining unit may include: a parameter generating neural network configured to receive target facial features to obtain face driving parameters.

[0085] According to some embodiments, the matching unit can be configured to perform similarity matching between multiple template face features and face features to be matched in order to determine the target face feature among the multiple template face features.

[0086] According to some embodiments, face image data may include multiple consecutive face image frames, and the face features to be matched may characterize the movement of the face in the face image data between multiple face image frames.

[0087] According to some embodiments, the feature extraction unit may include: a first static feature extraction subunit, configured to extract features from each of the multiple face image frames to obtain multiple static face features corresponding to the multiple face image frames; and a first motion feature extraction subunit, configured to determine the face features to be matched based on the multiple static face features.

[0088] According to some embodiments, multiple face image frames may be acquired within a preset time interval, and each of the multiple static face features may include a preset number of keypoint sequences. The motion feature extraction subunit may include a motion feature extraction neural network configured to receive multiple static face features to obtain face features to be matched. The face features to be matched may include feature vectors of a preset dimension.

[0089] According to some embodiments, the 3D face driving parameters may include at least one of hybrid deformation weights and face bone parameters.

[0090] According to another aspect of this disclosure, an expression generation apparatus is provided. For example... Figure 7 As shown, the device 700 includes: a second acquisition unit 710 configured to acquire multiple template face image data and multiple real face driving parameters corresponding to the multiple template face image data; a second feature extraction unit 720 configured to extract features from the multiple template face image data to obtain multiple template face features; a parameter generation unit 730 configured to input each of the multiple template face features into an initial parameter generation neural network to obtain multiple predicted face driving parameters corresponding to the multiple template face features; and a first parameter tuning unit 740 configured to adjust the parameters of the initial parameter generation neural network based on the multiple real face driving parameters and the multiple predicted face driving parameters to obtain a trained parameter generation neural network. It can be understood that the operations of units 710-740 in the device 700 are respectively related to... Figure 5 The operations of steps S501-S504 are similar and will not be described in detail here.

[0091] According to some embodiments, each of the multiple template face image data may include multiple consecutive template face image frames, and the template face features may characterize the movement of the face in the corresponding template face image data between the multiple template face image frames included in the template face image data.

[0092] According to some embodiments, the second feature extraction unit may include: a second static feature extraction subunit, configured to perform feature extraction on each template face image frame in the plurality of template face image data for each template face image data in the plurality of template face image data, so as to obtain a plurality of static template features corresponding to the plurality of template face image frames; and a second motion feature extraction subunit, configured to determine the template face features corresponding to the template face image data based on the plurality of static template features.

[0093] According to some embodiments, each of the multiple static template face features may include a preset number of keypoint sequences. A second motion feature extraction subunit may be configured to input the multiple static template features into an initial motion feature extraction neural network to obtain template face features corresponding to the template face image data. The training device may further include a second parameter tuning unit (not shown in the figure), configured to adjust the parameters of the initial motion feature extraction neural network based on multiple real face driving parameters and multiple predicted face driving parameters to obtain a trained motion feature extraction neural network.

[0094] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0095] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0096] refer to Figure 8 The present invention describes a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0097] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0098] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to device 800. Input unit 806 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 807 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 can include, but is not limited to, a hard disk and an optical disk. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0099] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the expression generation method. For example, in some embodiments, the expression generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the expression generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the expression generation method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0105] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0107] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for generating facial expressions, comprising: Acquire facial image data; Feature extraction is performed on the facial image data to obtain the facial features to be matched; Among a plurality of pre-determined template face features, a target face feature that matches the face feature to be matched is determined, wherein the plurality of template face features are determined by feature extraction from a plurality of template face image data of a plurality of standard facial expression actions. Based on the target facial features, determine the face driving parameters, including: The target facial features are input into a trained parameter generation neural network to obtain the face-driving parameters. The parameter generation neural network is trained using the multiple template facial features to model the mapping relationship between the multiple template facial features and their corresponding face-driving parameters. Based on the face driving parameters, a facial expression corresponding to the face image data is generated.

2. The method according to claim 1, wherein, Determining the target face feature that matches the face feature to be matched from a predetermined set of template face features includes: The multiple template face features and the face features to be matched are similar to determine the target face feature from the multiple template face features.

3. The method according to claim 1, wherein, The face image data includes multiple consecutive face image frames, and the face features to be matched characterize the movement of the face in the face image data between the multiple face image frames.

4. The method according to claim 3, wherein, Feature extraction of the facial image data to obtain the facial features to be matched includes: Feature extraction is performed on each of the plurality of face image frames to obtain multiple static face features corresponding to the plurality of face image frames; and Based on the multiple static facial features, the facial features to be matched are determined.

5. The method according to claim 4, wherein, Each of the plurality of static facial features includes a preset number of key point sequences. Among them, the facial features to be matched are determined based on the plurality of static facial features, including: The multiple static facial features are input into a trained motion feature extraction neural network to obtain the facial features to be matched.

6. The method according to claim 1, wherein, The face driving parameters include at least one of hybrid deformation weights and face bone parameters.

7. A method for training a neural network, comprising: Acquire multiple template face image data that perform multiple standard facial expression actions and multiple real face driving parameters corresponding to the multiple template face image data; Feature extraction is performed on the multiple template face image data to obtain multiple template face features; Each of the multiple template face features is input into the initial parameters to generate a neural network, so as to obtain multiple predicted face driving parameters corresponding to the multiple template face image data; as well as Based on the multiple real face driving parameters and the multiple predicted face driving parameters, the parameters of the initial parameter generation neural network are adjusted to obtain a trained parameter generation neural network.

8. The method according to claim 7, wherein, Each of the multiple template face image data includes multiple consecutive template face image frames, and the template face features characterize the movement of the face in the corresponding template face image data between the multiple template face image frames included in the template face image data.

9. The method according to claim 8, wherein, Feature extraction is performed on the multiple template face image data to obtain multiple template face features, including: For each template face image data in the plurality of template face image data, feature extraction is performed on each template face image frame in the plurality of template face image frames included in the template face image data to obtain a plurality of static template features corresponding to the plurality of template face image frames; and Based on the multiple static template features, the template face features corresponding to the template face image data are determined.

10. The method according to claim 9, wherein, Each of the plurality of static template features includes a preset number of key point sequences. Among them, determining the template face features corresponding to the template face image data based on the multiple static template features includes: The multiple static template features are input into an initial motion feature extraction neural network to obtain template face features corresponding to the template face image data. The method further includes: Based on the multiple real face driving parameters and the multiple predicted face driving parameters, the parameters of the initial motion feature extraction neural network are adjusted to obtain a trained motion feature extraction neural network.

11. An expression generation device, comprising: The first acquisition unit is configured to acquire face image data; The first feature extraction unit is configured to extract features from the face image data to obtain the face features to be matched. The matching unit is configured to determine a target face feature that matches the face feature to be matched from a plurality of pre-determined template face features, wherein the plurality of template face features are determined by feature extraction from a plurality of template face image data of acquired face making a plurality of standard facial expression actions. The determining unit is configured to determine face driving parameters based on the target face features; and The driving unit is configured to drive the generation of facial expressions corresponding to the facial image data based on the facial driving parameters. The determining unit includes: A parameter generation neural network is configured to receive the target facial features to obtain the face driving parameters, wherein the parameter generation neural network is trained using the plurality of template facial features to model the mapping relationship between the plurality of template facial features and the corresponding face driving parameters.

12. The apparatus according to claim 11, wherein, The matching unit is configured to perform similarity matching between the plurality of template face features and the face features to be matched, so as to determine the target face feature among the plurality of template face features.

13. The apparatus according to claim 11, wherein, The face image data includes multiple consecutive face image frames, and the face features to be matched characterize the movement of the face in the face image data between the multiple face image frames.

14. The apparatus according to claim 13, wherein, The first feature extraction unit includes: The first static feature extraction subunit is configured to extract features from each of the plurality of face image frames to obtain a plurality of static face features corresponding to the plurality of face image frames; and The first motion feature extraction subunit is configured to determine the face features to be matched based on the plurality of static face features.

15. The apparatus according to claim 14, wherein, Each of the plurality of static facial features includes a preset number of key point sequences. The first motion feature extraction subunit includes: A motion feature extraction neural network is configured to receive the plurality of static face features to obtain the face features to be matched.

16. The apparatus according to claim 11, wherein, The face driving parameters include at least one of hybrid deformation weights and face bone parameters.

17. A training device for a neural network, comprising: The second acquisition unit is configured to acquire multiple template face image data of making multiple standard facial expression actions and multiple real face driving parameters corresponding to the multiple template face image data; The second feature extraction unit is configured to extract features from the plurality of template face image data to obtain plurality of template face features; The parameter generation unit is configured to input each of the multiple template face features into the initial parameter generation neural network to obtain multiple predicted face driving parameters corresponding to the multiple template face image data. as well as The first parameter tuning unit is configured to adjust the parameters of the initial parameter generation neural network based on the plurality of real face driving parameters and the plurality of predicted face driving parameters, so as to obtain a trained parameter generation neural network.

18. The apparatus according to claim 17, wherein, Each of the multiple template face image data includes multiple consecutive template face image frames, and the template face features characterize the movement of the face in the corresponding template face image data between the multiple template face image frames included in the template face image data.

19. The apparatus according to claim 18, wherein, The second feature extraction unit includes: The second static feature extraction subunit is configured to perform feature extraction on each template face image frame in the plurality of template face image data for each template face image data, thereby obtaining a plurality of static template features corresponding to the plurality of template face image frames; and The second motion feature extraction subunit is configured to determine the template face features corresponding to the template face image data based on the plurality of static template features.

20. The apparatus according to claim 19, wherein, Each of the plurality of static template features includes a preset number of key point sequences. The second motion feature extraction subunit is configured to input the plurality of static template features into an initial motion feature extraction neural network to obtain template face features corresponding to the template face image data. The device further includes: The second parameter tuning unit is configured to adjust the parameters of the initial motion feature extraction neural network based on the plurality of real face driving parameters and the plurality of predicted face driving parameters, so as to obtain a trained motion feature extraction neural network.

21. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-10.

Images