Expression synchronization method and device of virtual character, and electronic equipment

Abstract

The invention discloses a virtual character expression synchronization method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a facial picture sequence of an expression providing object from a video to be processed; the face picture sequence and role information of the virtual role to be synchronized are input into a pre-trained expression migration model, a multi-dimensional expression parameter sequence for the virtual role to be synchronized output by the expression migration model is obtained, and each dimension in the multi-dimensional expression parameter sequence is used for representing the driving amount of one expression; and driving the to-be-synchronized virtual character based on the multi-dimensional expression parameter sequence, so that the to-be-synchronized virtual character displays the facial expression of the expression providing object in the to-be-processed video. According to the method, the technical problems of time and labor consumption, insufficient expressive force, high hardware cost, high calculation complexity and high resource consumption of synchronous virtual character expressions in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable storage medium for synchronizing expressions of a virtual character. Background Art

[0002] With the rapid advancement of computer technology and graphics processing technology, the game industry has entered an unprecedented stage of development. Game developers are no longer only concerned with the richness of game plots and gameplay, but are also paying more and more attention to the authenticity and immersion of the game experience. Among them, the expressiveness of virtual characters is particularly important for enhancing players' immersion.

[0003] At present, for the expression control of virtual characters, developers usually create multiple sets of expression animation templates in advance, and play them based on corresponding trigger conditions or player choices during the game to drive the facial expressions of virtual characters. However, manually creating expression animations is time-consuming and laborious, which undoubtedly increases the development cost of the game. In addition, the limitations of manually created expression animation templates also greatly limit the expressiveness of virtual characters. In response to the above defects, a face capture and expression transfer method is proposed. Based on high-precision cameras and specific capture devices, the facial movements of players are captured and converted into the expressions of virtual characters. Although this method can make the virtual character show the changes in the player's expression and improve the expressiveness of the virtual character, it relies on expensive hardware equipment, has high computational complexity and high resource consumption, resulting in high implementation costs and is difficult to popularize in most application scenarios.

[0004] Therefore, there is an urgent need for a simple, efficient, and low-cost method for synchronizing virtual character expressions, which can improve the expressiveness of virtual characters while reducing the manpower cost, hardware cost, and computing power cost consumed by synchronizing virtual character expressions. Summary of the invention

[0005] The present application provides a method, device, electronic device, and computer-readable storage medium for synchronizing the expressions of virtual characters to solve the technical problems existing in the prior art that synchronizing the expressions of virtual characters is time-consuming and labor-intensive, lacks expressiveness, and has high hardware costs, high computational complexity, and high resource consumption.

[0006] In a first aspect, an embodiment of the present application provides a method for synchronizing expressions of a virtual character, the method comprising: obtaining a facial picture sequence of an expression providing object from a video to be processed, the video to be processed at least presenting the facial expressions of the expression providing object, the facial picture sequence comprising a plurality of facial pictures, the facial pictures comprising facial images of the expression providing object; inputting the facial picture sequence and character information of the virtual character to be synchronized into a pre-trained expression transfer model to obtain a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, each dimension in the multi-dimensional expression parameter sequence being used to characterize the driving amount of an expression; and driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence so that the virtual character to be synchronized displays the facial expressions of the expression providing object in the video to be processed.

[0007] In a second aspect, an embodiment of the present application provides an expression synchronization device for a virtual character, the device comprising: a facial image acquisition unit, an expression parameter generation unit, and an expression synchronization unit; the facial image acquisition unit is used to acquire a facial image sequence of an expression providing object from a video to be processed, the video to be processed at least presents the facial expression of the expression providing object, the facial image sequence comprises a plurality of facial images, and the facial images comprise facial images of the expression providing object; the expression parameter generation unit is used to input the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, and obtain a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, each dimension in the multi-dimensional expression parameter sequence being used to characterize the driving amount of an expression; the expression synchronization unit is used to drive the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized displays the facial expression of the expression providing object in the video to be processed.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to implement the above method.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having one or more computer instructions stored thereon, which, when executed by a processor, executes the above method.

[0010] Compared with the prior art, the expression synchronization method of the virtual character provided by the present application includes: obtaining a facial image sequence of an expression providing object from a video to be processed, inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, obtaining a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression, and driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed. The method can obtain a multi-dimensional expression parameter sequence that can drive the expression synchronization of the virtual character to be synchronized based on the video to be processed, thereby realizing that the virtual character to be synchronized shows a facial expression consistent with the expression providing object in the video to be processed. First, the method synchronizes the expression of the virtual character to be synchronized based on a video (the video to be processed) and a pre-trained expression transfer model, so there are no problems such as time-consuming and labor-intensive methods of manually creating expression animation templates, and limited virtual character expressiveness. Second, this method can obtain a multi-dimensional expression parameter sequence that drives the synchronization of the expression of the virtual character to be synchronized based only on a video (the video to be processed), and does not rely on other specific data, that is, this method does not require the support of specific hardware equipment, so there is no problem of relying on expensive hardware equipment in the face capture and expression transfer methods. Third, since the original data processed by this method is only a video (the video to be processed), the data calculation complexity of this method is significantly reduced, and the resource consumption is significantly reduced compared to the face capture and expression transfer methods. In summary, the virtual character expression synchronization method provided in this application solves the technical problems of the prior art that synchronizing virtual character expressions is time-consuming and labor-intensive, lacks expressiveness, and has high hardware cost, high computational complexity, and high resource consumption. Therefore, this method is a simple, efficient, and low-cost virtual character expression synchronization method. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is an application system diagram of the expression synchronization method of a virtual character provided in an embodiment of the present application;

[0012] Figure 2 is a flow chart of a method for synchronizing expressions of a virtual character provided in the first embodiment of the present application;

[0013] Figure 3 It is a flowchart of a method for synchronizing the expression of a virtual character provided in the first embodiment of the present application;

[0014] Figure 4 is a structural diagram of a virtual character expression synchronization device provided in a second embodiment of the present application;

[0015] Figure 5 It is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present application. DETAILED DESCRIPTION

[0016] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0017] With the rapid advancement of computer technology and game development technology, the game industry has entered an unprecedented stage of development. Game developers are no longer only concerned with the richness of game plots and gameplay, but are also paying more and more attention to the authenticity and immersion of the game experience. Among them, the expressiveness of virtual characters is particularly important for enhancing the player's immersion. Synchronizing the player's expression with the virtual character's expression in the game can greatly enhance the player's immersion and game interaction experience.

[0018] At present, for the expression control of virtual characters, developers usually create multiple sets of expression animation templates in advance, and play them based on corresponding trigger conditions or player choices during the game to drive the facial expressions of virtual characters. Manually creating expression animations is a very time-consuming and labor-intensive process. Especially for virtual characters that require a large number of different expressions, the workload of developers will increase exponentially, which undoubtedly increases the manpower development cost of the game. And because the manually created expression animation templates have limitations, the expressiveness of virtual characters is greatly limited. Whether playing expression animations based on trigger conditions or based on player choices, only the expression animations in the expression animation templates can be played, and the actual synchronization effect with the player's expression cannot be achieved.

[0019] With the development of virtual reality and augmented reality technology, face capture and expression transfer methods have been proposed to overcome the above defects to a certain extent. Face capture and expression transfer methods are based on high-precision cameras and specific capture devices (such as depth sensors, markers and non-marker devices, etc.) to capture the player's facial movements and convert them into the expressions of virtual characters. Although this method can make the virtual character show the player's expression changes and improve the expressiveness and synchronization of the virtual character, it relies on specific hardware support, and the hardware cost is often too high, making it difficult to popularize in most application scenarios. In addition, face capture and expression transfer methods capture a lot of facial data, which requires a complex post-processing process, with high computational complexity, high resource consumption, and high computing power cost.

[0020] In view of this, the present application provides a simple, efficient and low-cost method for synchronizing the expression of a virtual character. The method can obtain a multi-dimensional expression parameter sequence that can drive the expression synchronization of the virtual character to be synchronized based on the video to be processed, thereby realizing that the virtual character to be synchronized shows a facial expression consistent with the expression providing object in the video to be processed. First, the method synchronizes the expression of the virtual character to be synchronized based on a video (the video to be processed) and a pre-trained expression transfer model. Therefore, there are no problems such as time-consuming and labor-intensive methods of manually creating expression animation templates, and limited virtual character expressiveness. Second, the method can obtain a multi-dimensional expression parameter sequence that drives the expression synchronization of the virtual character to be synchronized based only on a video (the video to be processed), and does not rely on other specific data, that is, the method does not require the support of specific hardware equipment, so there is no problem of relying on expensive hardware equipment in the face capture and expression transfer methods. Third, since the original data processed by the method is only a video (the video to be processed), compared with the face capture and expression transfer methods, the data calculation complexity of the method is significantly reduced, and the resource consumption is significantly reduced.

[0021] The virtual character expression synchronization method, device, electronic device, and computer-readable storage medium described in the present application are further described in detail below in conjunction with specific embodiments and drawings.

[0022] Figure 1 : is an application system diagram of the virtual character expression synchronization method provided in the embodiment of the present application. Figure 1 As shown, the system includes a user terminal 101 and a server terminal 102. The user terminal 101 can be any device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant (PDA), etc. The server terminal 102 can be a processing module inside the user terminal 101, or a processing device electrically connected to the user terminal 101, or a cloud server connected to multiple user terminals 101 in communication. The server terminal 102 is deployed with a virtual character's expression synchronization method. When the user triggers a virtual character expression synchronization instruction through the user terminal 101, the server terminal 102 can synchronize the user's facial expression with the virtual character's facial expression based on the method.

[0023] The first embodiment of the present application provides a method for synchronizing the expression of a virtual character, which is deployed in Figure 1 The server 102 shown is used to drive the facial expressions of the virtual characters in the virtual scene.

[0024] Figure 2 : is a flow chart of the expression synchronization method of the virtual character provided in this embodiment. Figure 2The virtual character expression synchronization method provided by this embodiment is described in detail. The embodiments described below are used to explain the technical solution of this application and are not intended to be used as limitations for actual use.

[0025] like Figure 2 As shown, the virtual character expression synchronization method provided in this embodiment includes the following steps S210 to S230:

[0026] Step S210, obtaining a facial picture sequence of an expression providing object from a video to be processed, wherein the video to be processed at least presents the facial expression of the expression providing object, and the facial picture sequence includes a plurality of facial pictures, and the facial pictures include facial images of the expression providing object.

[0027] The video to be processed may be a video including facial expressions of the user himself and / or other users shot by a camera of a terminal device such as a mobile phone or computer, or a video including facial expressions of a virtual object (such as an animated character, a game character, etc.) produced by the user himself or obtained from other channels.

[0028] In an optional implementation, the video to be processed is uploaded by the user to the server (e.g., a server) through the user end (e.g., a mobile phone or other terminal device). The user end sending the video to be processed to the server means that an expression synchronization instruction for synchronizing the expressions of the virtual characters in the virtual scene is simultaneously initiated to the server. The server can process the video to be processed upon receiving the video to be processed to synchronize the expressions of the virtual characters in the virtual scene.

[0029] In another optional implementation, the video to be processed is uploaded to the storage end (such as a cloud storage) by the user through the user end (such as a mobile phone or other terminal device), and then the storage end sends the link address of the video to be processed to the server end (such as a cloud server), and then the server end downloads it from the storage end. The user end sends the video to be processed to the storage end, and the storage end sends the link address of the video to be processed to the server end, which means that an expression synchronization instruction for synchronizing the expression of the virtual character in the virtual scene is initiated to the server end. The server end can process the video to be processed after downloading it from the storage end to synchronize the expression of the virtual character in the virtual scene. Based on this, before the step of obtaining the facial picture sequence of the expression providing object from the video to be processed, the method provided in this embodiment can also include the following steps: downloading the video to be processed from the storage end according to the link address corresponding to the video to be processed, and the link address is generated by the storage end in response to receiving the video to be processed uploaded by the user end and sent to the server end. On the one hand, this implementation method reduces the amount of data transmission between the server end and the user end through storage at the storage end, and improves the throughput of the server end. On the other hand, it ensures efficient management and fast access of video data through the link address.

[0030] In a specific implementation, the video to be processed is stored in the storage end in the form of object storage. The object storage is a distributed storage method for storing and managing unstructured data. It uses a flat data model, each storage object has a globally unique identifier, and provides a network application program interface, allowing developers to obtain data from the storage end through HTTP (HyperText Transfer Protocol). Storing the video to be processed in the storage end in the form of object storage can achieve efficient management and access of massive data and reduce the load on the server end.

[0031] The expression providing object refers to the object whose expression is to be imitated, which may be a user or another virtual object. The expression providing object is presented in the video to be processed in the form of an image. In this embodiment, the video to be processed may include only the image of the expression providing object, or may also include images of other objects. There is no limit on the number of object images that can be presented in the video to be processed.

[0032] The facial picture sequence can be understood as a plurality of facial pictures arranged in a certain order. In the present embodiment, the facial pictures only include the facial image of the object providing the expression, that is, the facial pictures are obtained by separating the area which only presents the facial image of the object providing the expression from the video frame picture of the video to be processed. The plurality of facial pictures are arranged in the time sequence of the corresponding video frame pictures in the video to be processed to form a facial picture sequence.

[0033] Step S220, input the facial image sequence and the character information of the virtual character to be synchronized into the pre-trained expression transfer model, obtain the multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, and each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression.

[0034] In this embodiment, a virtual character located in a virtual scene and pre-synchronized with the facial expression of the expression providing object presented in the video to be processed is defined as a virtual character to be synchronized. The character information of the virtual character to be synchronized can be understood as information that can describe at least part of the characteristics of the virtual character to be synchronized, such as the character type (such as type, gender, etc.) of the virtual character to be synchronized, the facial shape (such as long face, square face, round face, etc.) of the virtual character to be synchronized, and the facial details (such as eye size, mouth size, etc.) of the virtual character to be synchronized. It can be understood that the richer the character information input into the expression transfer model together with the facial picture sequence, the more the expression parameters output by the expression transfer model according to the facial picture sequence fit the virtual character to be synchronized. Exemplarily, for the open mouth expression of the expression providing object presented in the video to be processed, if the character information of the virtual character to be synchronized is male, usually males have larger mouths, and the driving amount of the open mouth expression output based on the character information (male) will be larger, and if the character information of the virtual character to be synchronized is female, usually females have smaller mouths, and the driving amount of the open mouth expression output based on the character information (female) will be smaller. The richer the character information input into the expression transfer model, for example, the more character information such as the face shape, mouth size, eye size, etc. of the virtual character to be synchronized is input into the expression transfer model, the more the expression transfer model will output expression parameters that are more in line with the virtual character to be synchronized. Of course, the character information input into the expression transfer model needs to be consistent with the training data when training the expression transfer model, that is, the expression transfer model can output expression parameters that are in line with the character information included in the training data. Exemplarily, when training the expression transfer model, the character information included in the training data is the character type. Then, in the application stage, the character type of the virtual character to be synchronized is input into the expression transfer model, and the expression parameters output by the expression transfer model that are in line with the character type of the virtual character to be synchronized will be obtained. If character information such as face shape is input into the expression transfer model, expression parameters that are in line with the character information such as face shape will not be obtained.

[0035] The expression transfer model is a pre-trained model that can use facial images and character information of the virtual character to be synchronized as input data to perform expression parameter inference and output accurate expression parameters for the virtual character to be synchronized.

[0036] In a specific implementation, the expression transfer model is a convolutional neural network, which is obtained by training based on a deep learning method. Based on this, the method provided in this embodiment also includes the following steps: obtaining a first training sample set; wherein the first training sample set includes multiple first training samples, and the first training samples include facial images, role information of the virtual character to be synchronized, and expression parameters for the virtual character to be synchronized that are annotated on the facial images; using the facial images and role information of the virtual character to be synchronized included in each first training sample as training data, inputting the initial expression transfer model, and obtaining the predicted expression parameters for the virtual character to be synchronized output by the initial expression transfer model; based on the predicted expression parameters and the expression parameters included in the first training sample, optimizing the model parameters of the initial expression transfer model until the loss function between the predicted expression parameters and the expression parameters included in the first training sample reaches a preset threshold; obtaining a pre-trained expression transfer model. The expression parameters in the first training sample can be obtained through expert / user labeling. Specifically, a facial picture and character information of the virtual character to be synchronized are provided to the expert / user (provided to the expert / user in the form of text or pictures to help the expert / user understand the characteristics of the virtual character to be synchronized), and the expert / user is allowed to manually set or select expression parameters that are more suitable for the virtual character to be synchronized (such as the degree of opening and closing of the eyes, the degree of curvature of the mouth corners, etc.) according to the expression of the object in the facial picture.

[0037] When the first training sample is used to train the initial expression transfer model, facial images and character information of the virtual character to be synchronized are used as training data, and the annotated expression parameters are used as training labels. The reverse gradient optimization algorithm is used to optimize the model parameters of the initial expression transfer model until the loss function value between the expression parameters predicted by the expression transfer model and the annotated expression parameters reaches a preset threshold (e.g., less than 0.05), and the pre-trained expression transfer model can be obtained.

[0038] In the method provided in this embodiment, the expression parameters output by the expression transfer model based on the facial image are multi-dimensional expression parameters, and the multi-dimensional expression parameters output based on each facial image are combined into a multi-dimensional expression parameter sequence. Each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression. Specifically, a set of facial deformation editors can be used to represent the output multi-dimensional expression parameter sequence, and each dimension can drive a single expression action of the face of the virtual character to be synchronized.

[0039] Step S230, driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed.

[0040] Since each dimension in the multi-dimensional expression parameter sequence represents the driving amount of an expression, applying the multi-dimensional expression parameter sequence to the virtual character to be synchronized can drive the virtual character to be synchronized to perform the expression actions represented by the expression parameter sequence of each dimension. Since the multi-dimensional expression parameter sequence comes from the video to be processed, the facial expression of the expression providing object in the video to be processed can be presented after the virtual character to be synchronized performs the expression actions represented by the expression parameter sequence of each dimension.

[0041] Exemplarily, the expression parameter sequence of the first dimension output by the expression transfer model is [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], and the first dimension is used to characterize the driving amount of the "open mouth" expression. Driving the virtual character to be synchronized based on the expression parameter sequence of the first dimension can make the virtual character to be synchronized show the mouth movement from closing the mouth to opening the mouth. Exemplarily, the expression parameter sequence of the second dimension output by the expression transfer model is [0.5, 0.6, 0.7, 0.8, 0.9, 1.0], and the second dimension is used to characterize the driving amount of the "open eyes" expression. Driving the virtual character to be synchronized based on the expression parameter sequence of the second dimension can make the virtual character to be synchronized show the eye movement from normal open eyes to wide eyes.

[0042] The following describes in detail various optional implementations provided in this embodiment:

[0043] In an optional implementation, obtaining a facial image sequence of an expression providing object from a video to be processed may specifically include the following steps S211 to S213:

[0044] Step S211, input each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, and the face detection frame is used to locate the position of the facial image of the expression providing object in the video frame image.

[0045] Step S212, based on the facial detection frame corresponding to each video frame image, each video frame image is cropped to obtain a facial image corresponding to each video frame image.

[0046] Step S213, based on the time sequence of each video frame image in the video to be processed, the facial image corresponding to each video frame image is combined into a facial image sequence.

[0047] Specifically, the face detection model is used to identify the position and area of ​​the face of the expression providing object in each video frame image, and the area including only the facial image of the expression providing object is cropped out to form a facial image corresponding to each video frame image. It can be understood that the facial image is actually a partial area in the video frame image. By cleaning the video frame images and only retaining the facial image including the facial image of the expression providing object, it can not only save server memory, but also reduce interference with later model reasoning, and ensure the stability of model reasoning.

[0048] The facial detection model is a face detection algorithm based on a convolutional neural network. The model uses an RGB (red, blue, green) picture as input data and can output the coordinates of the detection frame of the face in the RBG picture. The facial detection model is trained by a deep learning algorithm and can show good robustness for pictures under various light and background conditions, thereby accurately obtaining the facial area and its detection frame in the RGB picture. Therefore, in this embodiment, the production or shooting process of the video to be processed does not require a specific environment or specific equipment, and the method provided in this embodiment can be applied in various scenarios.

[0049] After the facial detection frame in each video frame is output based on the facial detection model, each video frame can be cropped based on the facial detection frame. Specifically, the video frame is cropped along the edge of the facial detection frame, the video frame outside the facial detection frame is removed, and the video frame within the facial detection frame is retained. In this embodiment, the video frame within the retained facial detection frame is defined as a facial image. Then, each facial image is sorted according to the time sequence of the corresponding video frame in the video to be processed, so as to form a facial image sequence.

[0050] In another optional implementation, the video to be processed presents facial expressions of multiple objects, including an expression providing object. Usually, the expression providing object is the main object among the multiple objects, that is, the object with the largest facial image area, but since the positions and image sizes of the multiple objects may change during the video process, in order not to limit the limitations of the method provided in this embodiment on the video to be processed, the facial image of the main object in the video to be processed can be tracked to determine a unified facial detection frame of the expression providing object in all video frame images.

[0051] Based on this, each video frame image in the video to be processed is input into the face detection model to obtain the face detection frame corresponding to each video frame image, which may specifically include the following steps S211-1 to S211-4:

[0052] Step S211 - 1 , input each video frame image into the face detection model to obtain multiple candidate face detection frames corresponding to each video frame image, each candidate face detection frame is used to locate the position of a facial image of an object in the video frame image.

[0053] Step S211-2, determining the face detection frame corresponding to the first video frame picture according to the sizes of multiple candidate face detection frames corresponding to the first video frame picture, where the first video frame picture is the first video frame picture in time sequence among the multiple video frame pictures included in the video to be processed.

[0054] Step S211-3, determining the facial detection frame corresponding to the second video frame picture according to the intersection-and-europey ratio of multiple candidate facial detection frames corresponding to the second video frame picture and the facial detection frame corresponding to the first video frame picture, where the second video frame picture is the second video frame picture in time sequence among the multiple video frame pictures included in the video to be processed.

[0055] Step S211 - 4 , sequentially determining a face detection frame corresponding to each of the multiple video frame images included in the video to be processed.

[0056] Since the video to be processed presents facial expressions of multiple objects, the video frame image is input into the face detection model to obtain a face detection frame for each object. In this implementation, the face detection frame for each object is defined as a candidate face detection frame, and a candidate face detection frame locates the position of an object in the video frame image.

[0057] Generally, in the front section of the video, the facial image of the main object occupies the largest area. As the video progresses, the change in the position of the main object and other objects may cause the facial image of the main object to occupy a smaller area. Therefore, in order to more accurately track the only main object in the video to be processed, in this embodiment, the facial detection frame corresponding to the first video frame image is determined based on the size of the multiple candidate facial detection frames corresponding to the first video frame image (the video frame image that is arranged first in the time sequence among multiple video frame images). Specifically, the candidate facial detection frame with the largest area among the multiple candidate facial detection frames is used as the facial detection frame corresponding to the first video frame image. Then, according to the intersection over union (IoU) of the multiple candidate facial detection frames corresponding to the second video frame image (the video frame image that is arranged second in the time sequence among multiple video frame images) and the facial detection frame corresponding to the first video frame image, the facial detection frame corresponding to the second video frame image is determined. Specifically, the candidate facial detection frame with the highest IoU is used as the facial detection frame. According to the IoU of the multiple candidate facial detection frames corresponding to each video frame image and the facial detection frame corresponding to the previous video frame image (determined in the previous step), the facial detection frame corresponding to each video frame image is determined.

[0058] The intersection-over-union ratio is a measurement method for evaluating the degree of overlap between two areas, and can be used to track the facial movement of the main object in continuous video frame images. Exemplarily, calculating the intersection-over-union ratio of face detection frame A and face detection frame B includes: calculating the intersection, which can calculate the overlapping area between the two according to the coordinates of face detection frame A and the coordinates of face detection frame B, and the overlapping area is the intersection of face detection frame A and face detection frame B; calculating the union, which can be obtained by calculating the area of ​​face detection frame A plus the area of ​​face detection frame B, and then subtracting the overlapping area to obtain the combined area of ​​the two, and the combined area is the union of face detection frame A and face detection frame B; calculating IoU, and taking the quotient of the intersection divided by the union as the intersection-over-union ratio of face detection frame A and face detection frame B. It can be understood that the value range of IoU is [0,1]. When IoU is 0, it means that face detection frame A and face detection frame B do not overlap. When IoU is close to 1, it means that face detection frame A and face detection frame B almost completely overlap. The larger the calculated IoU is, the larger the overlap area between face detection frame A and face detection frame B is, and the greater the possibility that they are the face detection frames of the same object is; the smaller the IoU is, the smaller the overlap area between face detection frame A and face detection frame B is, and the less likely they are the face detection frames of the same object is. Based on this method, the main object located by the face detection frame in the first video frame can always be tracked.

[0059] In an optional implementation, after inputting each video frame image in the video to be processed into the face detection model and obtaining the face detection frame corresponding to each video frame image, the method provided in this embodiment may further include the following steps: smoothing the face detection frame corresponding to each video frame image to obtain the processed face detection frame corresponding to each video frame image. Based on this, based on the face detection frame corresponding to each video frame image, cropping each video frame image to obtain the face image corresponding to each video frame image may specifically include: based on the processed face detection frame corresponding to each video frame image, cropping each video frame image to obtain the face image corresponding to each video frame image.

[0060] Specifically, in order to reduce the jitter of the facial detection frame between video frames and the impact on the subsequent expression synchronization effect, a Kalman filter can be used to smooth the facial detection frame corresponding to each video frame to improve the stability of the facial detection frame between video frames.

[0061] The Kalman filter is an efficient recursive Bayesian filter, which is mainly used for state estimation of linear dynamic systems and can give the optimal state estimation in the presence of noise. Specifically, the Kalman filter can predict the position of the face detection frame in the next video frame picture, use the actual position of the face detection frame in the current video frame picture to update the state of the Kalman filter, and use the predicted value of the Kalman filter as the final face detection frame position.

[0062] Based on this, the cropping process for each video frame is actually performed based on the smoothed facial detection frame, so that the jitter between the facial images corresponding to each video frame obtained is smaller and more stable.

[0063] In another optional implementation, after the step of cropping each video frame image based on the facial detection frame corresponding to each video frame image to obtain the facial image corresponding to each video frame image, the method provided in this embodiment may further include the following step: scaling the facial image corresponding to each video frame image to a first preset size. Based on this, based on the timing of each video frame image in the video to be processed, the facial images corresponding to each video frame image are combined into a facial image sequence, which may specifically include: based on the timing of each video frame image in the video to be processed, the facial images scaled to the first preset size corresponding to each video frame image are combined into a facial image sequence.

[0064] Since the position and size of the facial image of the expression providing object may change during the video process, the size of the facial image obtained by cropping each video frame image based on the detected facial detection frame may be different. In order to facilitate the subsequent expression transfer model to perform expression parameter inference, each cropped facial image can be scaled to a uniform size. In this embodiment, the uniform size of the facial image scaled is defined as a first preset size, which is an adjustable value and can be set according to the requirements of the expression transfer model for input data, without specific restrictions.

[0065] Based on this, the formed facial image sequence actually includes a plurality of facial images of the first preset size arranged in time sequence.

[0066] In an optional implementation, in order to improve the efficiency of expression parameter reasoning and improve the performance of the method provided in this embodiment, when inputting a facial image sequence into an expression transfer model, multiple facial images may be packaged into a batch, and then multiple facial images of a batch may be simultaneously input into the expression transfer model for reasoning. Specifically, inputting a facial image sequence and character information of a virtual character to be synchronized into a pre-trained expression transfer model may include the following steps S221 to S222:

[0067] Step S221 , packing the multiple facial images included in the facial image sequence into multiple batches, each batch including a preset number of facial images.

[0068] Step S222, inputting the facial image sequence into the expression transfer model in batches, so that the expression transfer model synchronously processes a preset number of facial images included in each batch.

[0069] The number of facial images included in a batch is a preset value and can be adjusted according to the video memory size of the server. There is no specific limit.

[0070] For example, there are 8 facial pictures, and it takes 10 ms for the expression transfer model to infer one facial picture. Then, if only one facial picture is input into the expression transfer model at a time, it will take 80 ms to process all 8 facial pictures. However, if the 8 facial pictures are packaged into 2 batches, each batch includes 4 facial pictures, and the 4 facial pictures included in each batch are input into the expression transfer model at one time for parallel calculation, it will only take 20 ms to process all 8 facial pictures, which is 4 times more efficient.

[0071] In an optional implementation, after the step of inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model to obtain the multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, the method provided in this embodiment may further include the following steps: inputting the facial image sequence into a pre-trained head posture estimation model to obtain the head posture parameter sequence output by the head posture estimation model.

[0072] The head posture estimation model is a pre-trained model that can use facial images as input data to infer head posture parameters and output a head posture parameter sequence corresponding to a facial image sequence.

[0073] In a specific implementation, the head posture estimation model is a convolutional neural network, which is obtained by training based on a deep learning method. Based on this, the method provided in this embodiment also includes the following steps: obtaining a second training sample set; wherein the second training sample set includes multiple second training samples, and the second training samples include facial images and head posture parameters annotated for the facial images; using the facial images included in each second training sample as training data, inputting the initial head posture estimation model, and obtaining the predicted head posture parameters output by the initial head posture estimation model; based on the predicted head posture parameters and the head posture parameters included in the second training sample, optimizing the model parameters of the initial head posture estimation model until the loss function between the predicted head posture parameters and the head posture parameters included in the second training sample reaches a preset threshold; obtaining a pre-trained head posture estimation model.

[0074] The second training sample is actually a pairing data of a facial image and head posture parameters, wherein the head posture parameters can be obtained by expert / user labeling. Specifically, a facial image is provided to the expert / user, and the expert / user is asked to manually set or select head posture parameters (such as the degree of lowering the head, the degree of raising the head, the degree of turning, etc.) for the head posture of the object in the facial image. When the second training sample is used to train the initial head posture estimation model, the facial image is used as the training data, and the labeled head posture parameters are used as the training labels. The model parameters of the initial head posture estimation model are optimized using the reverse gradient optimization algorithm until the loss function value between the head posture parameters predicted by the head posture estimation model and the labeled head posture parameters reaches a preset threshold (such as less than 0.05), and the pre-trained head posture estimation model can be obtained.

[0075] In the method provided in this embodiment, the head posture estimation model infers head posture parameters for each facial image, and the head posture parameters inferred based on each facial image are combined into a head posture parameter sequence, which can drive the head posture of the virtual character to be synchronized to keep it consistent with the head posture of the expression providing object in the video to be processed. The animation performed by the virtual character to be synchronized can be made more realistic and natural, and the dynamic expressiveness of the virtual character to be synchronized can be enhanced.

[0076] Based on this, driving the virtual character to be synchronized based on a multi-dimensional expression parameter sequence can specifically include: driving the virtual character to be synchronized based on a multi-dimensional expression parameter sequence and a head posture parameter sequence, so that the virtual character to be synchronized displays the facial expressions and head postures of the expression providing object in the video to be processed, thereby further enriching the synchronization effect of the virtual character to be synchronized.

[0077] In an optional implementation, after the step of inputting the facial image sequence into the pre-trained head pose estimation model to obtain the head pose parameter sequence output by the head pose estimation model, the method provided in this embodiment may further include the following steps S11 to S12:

[0078] Step S11, obtaining an eye picture sequence corresponding to the expression provided from the facial picture sequence, wherein the eye picture sequence includes a plurality of eye pictures, and the eye pictures include eye images of the object providing the expression.

[0079] The eye picture sequence can be understood as a plurality of eye pictures arranged in a certain order. In the present embodiment, the eye picture only includes the eye image of the expression providing object, that is, the eye picture is obtained by separating the area presenting the eye image of the expression providing object from the facial picture. A plurality of eye pictures are arranged in the time sequence of the corresponding facial pictures to form an eye picture sequence.

[0080] In a specific implementation, obtaining expressions from a facial image sequence to provide a corresponding eye image sequence may specifically include the following steps S11-1 to S11-4:

[0081] Step S11-1, input each facial image in the facial image sequence into the facial key point detection model to obtain the eye detection frame corresponding to each facial image, and the eye detection frame is used to locate the position of the eye image of the expression providing object in the facial image.

[0082] Step S11 - 2 , based on the eye detection frame corresponding to each facial image, each facial image is cropped to obtain an eye image corresponding to each facial image.

[0083] Step S11-3, scaling the eye image corresponding to each facial image to a second preset size.

[0084] Step S11 - 4 , based on the time sequence of each facial image in the facial image sequence, combining the eye images corresponding to each facial image scaled to a second preset size into an eye image sequence.

[0085] Specifically, the position and area of ​​the eyes in each facial image are identified based on the facial key point detection model, and the area including only the eye image is cropped out to form an eye image corresponding to each facial image. It can be understood that the eye image is actually a part of the facial image. By cleaning the facial images and only retaining the eye images including the eye images, it can not only save server memory, but also reduce interference with later model reasoning, and ensure the stability of model reasoning.

[0086] The facial key point detection model is a facial key point detection algorithm based on a convolutional neural network. The model locates facial key points, such as eyes, nose, mouth, etc., by identifying important facial features.

[0087] After outputting the eye detection frame in each facial image based on the facial key point detection model, each facial image can be cropped based on the eye detection frame. Specifically, the facial image is cropped along the edge of the eye detection frame, the facial image outside the eye detection frame is cleared, and the facial image within the eye detection frame is retained. In this embodiment, the facial image within the retained eye detection frame is defined as an eye image. Since the sizes of the eye images obtained by cropping may vary, in order to facilitate the subsequent line of sight estimation model to perform line of sight parameter inference, each cropped eye image can be scaled to a uniform size. In this embodiment, the uniform size of the eye image scaled is defined as a second preset size, and the second preset size is an adjustable value, which can be set according to the requirements of the line of sight estimation model for input data, and is not specifically limited.

[0088] After each eye image corresponding to the facial image is scaled to the second preset size, each eye image that has been scaled to the second preset size can be sorted according to the time sequence of the corresponding facial image to form an eye image sequence.

[0089] Step S12: input the eye image sequence into a pre-trained gaze estimation model to obtain a gaze direction parameter sequence output by the gaze estimation model.

[0090] The gaze estimation model is a pre-trained model that can use eye images as input data to infer gaze direction parameters and output a gaze direction parameter sequence corresponding to an eye image sequence.

[0091] In a specific implementation, the sight estimation model is a convolutional neural network, which is obtained by training based on a deep learning method. Based on this, the method provided in this embodiment also includes the following steps: obtaining a third training sample set; wherein the third training sample set includes multiple third training samples, and the third training samples include eye pictures and sight direction parameters annotated for the eye pictures; using the eye pictures included in each third training sample as training data, inputting the initial sight estimation model, and obtaining the predicted sight direction parameters output by the initial sight estimation model; based on the predicted sight direction parameters and the sight direction parameters included in the third training sample, optimizing the model parameters of the initial sight estimation model until the loss function between the predicted sight direction parameters and the sight direction parameters included in the third training sample reaches a preset threshold; obtaining a pre-trained sight estimation model.

[0092] The third training sample is actually a pairing data of an eye picture and a sight direction parameter, wherein the sight direction parameter can be obtained by expert / user labeling. Specifically, an eye picture is provided to the expert / user, and the expert / user is asked to manually set or select the sight direction parameter (e.g., straight ahead, squint degree, etc.) for the sight direction of the object in the eye picture. When the third training sample is used to train the initial sight estimation model, the eye picture is used as training data, and the labeled sight direction parameter is used as a training label. The model parameters of the initial sight estimation model are optimized using a reverse gradient optimization algorithm until the loss function value between the sight direction parameter predicted by the sight estimation model and the labeled sight direction parameter reaches a preset threshold (e.g., less than 0.05), and the pre-trained sight estimation model can be obtained.

[0093] In the method provided in this embodiment, the gaze estimation model infers gaze direction parameters for each eye image, and the gaze direction parameters inferred based on each eye image are combined into a gaze direction parameter sequence. The gaze direction parameter sequence can drive the gaze direction of the virtual character to be synchronized so that it is consistent with the gaze direction of the expression providing object in the video to be processed, thereby enhancing the eye interactivity of the virtual character to be synchronized.

[0094] Based on this, driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence and the head posture parameter sequence can specifically include: driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the line of sight direction parameter sequence, so that the virtual character to be synchronized displays the facial expression, head posture, and line of sight direction of the expression providing object in the video to be processed, thereby enhancing the expression synchronization effect of the virtual character to be synchronized.

[0095] In an optional implementation, driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence may specifically include the following steps S21 to S23:

[0096] Step S21 , integrating the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence to generate animation data for driving the virtual character to be synchronized.

[0097] Step S22, performing time series smoothing processing on the animation data to obtain processed animation data.

[0098] Step S23, applying the processed animation data to the virtual character to be synchronized, so that the virtual character to be synchronized displays the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

[0099] Specifically, the integration processing of the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the line of sight direction parameter sequence may include uniformly converting the data format of the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the line of sight direction parameter sequence, so that the data format conforms to the data type that drives the virtual character to be synchronized to perform facial actions, and then the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the line of sight direction parameter sequence after the format conversion are merged one by one in time sequence to form animation data. After the animation data is formed, the animation data can be time-smoothed to reduce the jitter between the data sequences and ensure that the processed animation data has stability and continuity. Using the processed animation data to drive the virtual character to be synchronized can achieve high-quality synchronization of the expression of the virtual character to be synchronized and enhance the expressiveness of the virtual character to be synchronized.

[0100] The above steps provide an optional method for synchronizing the expression of a virtual character, which can enable the virtual character to be synchronized in the virtual scene to express the facial expressions of the user, other virtual objects, etc. in the video to be processed.

[0101] The method provided in this embodiment is further described in detail below by taking synchronization of user expressions as an example. Figure 3 It is a flowchart of the method for synchronizing the expression of a virtual character provided in this embodiment.

[0102] like Figure 3 As shown, the virtual character expression synchronization method provided in this embodiment may include the following steps S301 to S309:

[0103] Step S301, user video (video to be processed) is uploaded and obtained.

[0104] The user uses the camera on the user terminal device to shoot a user video showing his or her facial expression, and uploads the user video to the cloud storage, which is stored in the cloud storage in the form of object storage. The cloud storage sends the link address of the user video to the cloud server. The cloud server downloads the user video from the cloud storage according to the link address and decodes it, and extracts the user video into a video frame image sequence.

[0105] In this step, the user video is stored in the cloud storage, which reduces the amount of data transmission between the cloud server and the user end and improves the throughput of the cloud server. At the same time, sending the link address to the cloud server ensures efficient management and fast access to the user video.

[0106] Step S302: face detection.

[0107] The cloud server uses a convolutional neural network-based facial detection model to perform face detection on each video frame in the user's video, identify the location and area of ​​the face, and obtain the coordinates of the facial detection frame.

[0108] In this step, face detection is performed using a face detection model trained based on a deep learning algorithm. This model has good robustness for images under various lighting and background conditions, and can accurately obtain the coordinates of the face area and its detection frame.

[0109] Step S303: smoothing the detection frame.

[0110] First, the cloud server determines the main character in the user's video and tracks his face based on the IoU (intersection over union) of the face detection frame corresponding to each video frame and the face detection frame of the previous video frame, filters the face detection frames of other non-main characters, and retains one face detection frame for each video frame. Secondly, in order to reduce the jitter of the face detection frame between video frames and its impact on the subsequent expression synchronization effect, the cloud server uses a Kalman filter to smooth the face detection frame of each video frame to improve the stability of the face detection frame in each video frame.

[0111] Step S304: face cropping.

[0112] The cloud server crops the facial area in each video frame according to the processed facial detection frame, extracts the facial image corresponding to each video frame, and scales the facial image to the same size (a first preset size) to facilitate the reasoning of the subsequent expression transfer model.

[0113] In this step, cropping the facial image can remove redundant areas of the video frame image and retain only the facial area, saving memory and increasing the stability of subsequent expression transfer model reasoning.

[0114] Step S305, expression parameter inference.

[0115] The cloud server inputs the facial image sequence and the character information of the virtual character to be synchronized into the pre-trained expression transfer model to infer the expression parameters. Specifically, the expression parameters are represented by a set of facial deformation editors, and each dimension can drive a single expression action (such as closing eyes and opening mouth) of the face of the virtual character to be synchronized.

[0116] In this step, expression parameters can be accurately extracted according to the virtual character to be synchronized. At the same time, in order to improve the efficiency of this step and ensure the performance of the expression synchronization method, multiple images in the facial image sequence can be packaged into a batch, and multiple facial images in a batch can be simultaneously input into the expression transfer model for synchronous inference of expression parameters.

[0117] Step S306: head posture estimation.

[0118] The cloud server inputs the facial image sequence into the head pose estimation model to obtain the head pose parameter sequence. The head pose estimation model is obtained based on the training of the convolutional neural network. During the training process, a large amount of facial image-head pose pairing data is used to optimize the model parameters using the reverse gradient optimization method.

[0119] Using the head posture parameter sequence obtained in this step to drive the head posture of the virtual character to be synchronized can make the virtual character to be synchronized show a head posture consistent with the user in the user video, make the expression performed by the virtual character to be synchronized more real and natural, and enhance the dynamic expression ability of the virtual character to be synchronized.

[0120] Step S307: eye cropping and gaze direction estimation.

[0121] The cloud server inputs the facial image sequence into the facial key point detection model, extracts the facial key points, and then calculates the coordinates of the detection frame corresponding to the eye area based on the key point detection results. The eye area is cropped from the facial image according to the coordinates of the eye detection frame and scaled to a uniform size (a second preset size) to form an eye image sequence. The eye image sequence is then input into the pre-trained line of sight estimation model to estimate the line of sight direction and obtain the gaze direction of the user in the user video.

[0122] Using the sight direction parameter sequence obtained in this step to drive the sight direction of the virtual character to be synchronized can enhance the eye contact interaction of the virtual character to be synchronized.

[0123] Step S308: post-processing and data integration.

[0124] The cloud server post-processes and integrates the expression parameter sequence, head posture parameter sequence, and sight direction parameter sequence to generate animation data that drives the virtual character to be synchronized. The generated animation data is smoothed in time to ensure the stability and continuity of the results.

[0125] In this step, through post-processing and timing smoothing of each parameter, the continuity and consistency of the animation data is ensured, and abrupt and unnatural changes are reduced. At the same time, the integration of multiple parameters (expression parameters, head posture parameters, and line of sight direction parameters) can drive the virtual character to be synchronized to perform more comprehensive and comprehensive expression actions.

[0126] Step S309: driving the virtual character to be synchronized.

[0127] The cloud server applies the processed and integrated animation data to the virtual character to be synchronized, driving the character to be synchronized to perform facial expressions consistent with the user in the user video, achieving efficient expression synchronization and improving the user's interactive experience.

[0128] In summary, the above-mentioned first embodiment provides a method for synchronizing the expressions of virtual characters. This solution does not require expensive hardware equipment and software support, which lowers the threshold for use and can be popularized to most application projects. In addition, the method has good decoupling properties for irrelevant information such as different light, background, and character identity, and is universal for users and virtual characters. This method is a low-threshold, low-cost, high-quality, and high-efficiency method for synchronizing the expressions of virtual characters. It should be noted that the examples in the first embodiment are only for explaining the method described in this application, and are not intended to be a limitation for actual use. The method for synchronizing the expressions of virtual characters provided in this application includes but is not limited to the method described in the first embodiment.

[0129] The second embodiment of the present application provides a virtual character expression synchronization device. Figure 4 Schematic diagram of the structure of the virtual character expression synchronization device provided in this embodiment.

[0130] like Figure 4 As shown, the expression synchronization device for a virtual character provided in this embodiment includes: a facial image acquisition unit 401, an expression parameter generation unit 402, and an expression synchronization unit 403.

[0131] The facial image acquisition unit 401 is used to acquire a facial image sequence of an expression providing object from a video to be processed, wherein the video to be processed at least presents the facial expression of the expression providing object, and the facial image sequence includes multiple facial images, and the facial images include facial images of the expression providing object.

[0132] Optionally, before the step of acquiring a facial picture sequence of an expression providing object from the video to be processed, the method is further used to:

[0133] Download the video to be processed from the storage end according to the link address corresponding to the video to be processed; wherein the link address is generated and sent by the storage end in response to receiving the video to be processed uploaded by the user end, and the video to be processed is stored in the storage end in the form of object storage.

[0134] Optionally, the step of obtaining a facial image sequence of an expression providing object from the video to be processed includes:

[0135] Input each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, wherein the face detection frame is used to locate the position of the facial image of the expression providing object in the video frame image;

[0136] Based on the face detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain a face image corresponding to each of the video frame images;

[0137] Based on the time sequence of each of the video frame images in the video to be processed, the facial images corresponding to each of the video frame images are combined into the facial image sequence.

[0138] Optionally, the video to be processed presents facial expressions of multiple objects;

[0139] The step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image includes:

[0140] Input each of the video frame images into the face detection model to obtain a plurality of candidate face detection frames corresponding to each of the video frame images, each of the candidate face detection frames being used to locate a position of a face image of the object in the video frame image;

[0141] Determine a face detection frame corresponding to a first video frame picture according to sizes of multiple candidate face detection frames corresponding to the first video frame picture, wherein the first video frame picture is a video frame picture that is first in time sequence among multiple video frame pictures included in the video to be processed;

[0142] Determine a face detection frame corresponding to a second video frame picture according to an intersection-and-union ratio of a plurality of candidate face detection frames corresponding to a second video frame picture and a face detection frame corresponding to the first video frame picture, wherein the second video frame picture is a video frame picture that is second in time sequence among a plurality of video frame pictures included in the video to be processed;

[0143] The face detection frame corresponding to each of the multiple video frame pictures included in the video to be processed is determined in sequence.

[0144] Optionally, after the step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, the method is further used to:

[0145] Performing smoothing processing on the face detection frame corresponding to each of the video frame images to obtain a processed face detection frame corresponding to each of the video frame images;

[0146] The step of cropping each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images includes:

[0147] Based on the processed facial detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain the facial image corresponding to each of the video frame images.

[0148] Optionally, after the step of performing cropping processing on each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images, the method is further used to:

[0149] Scaling the facial image corresponding to each of the video frame images to a first preset size;

[0150] The combining the facial pictures corresponding to each of the video frame pictures into the facial picture sequence based on the time sequence of each of the video frame pictures in the video to be processed includes:

[0151] Based on the timing of each of the video frame images in the video to be processed, the facial images scaled to the first preset size corresponding to each of the video frame images are combined into the facial image sequence.

[0152] The expression parameter generating unit 402 is used to input the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, and obtain a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, wherein each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression.

[0153] Optionally, inputting the facial image sequence into a pre-trained expression transfer model comprises:

[0154] Packing the multiple facial images included in the facial image sequence into multiple batches, each batch including a preset number of facial images;

[0155] The facial image sequences are input into the expression transfer model in batches, so that the expression transfer model synchronously processes the preset number of facial images included in each batch.

[0156] Optionally, also used for:

[0157] Acquire a first training sample set; wherein the first training sample set includes a plurality of first training samples, the first training samples including a facial image, character information of a virtual character to be synchronized, and expression parameters for the virtual character to be synchronized annotated on the facial image;

[0158] Using the facial image and the character information of the virtual character to be synchronized included in each of the first training samples as training data, inputting an initial expression transfer model, and obtaining predicted expression parameters for the virtual character to be synchronized output by the initial expression transfer model;

[0159] Based on the predicted expression parameters and the expression parameters included in the first training sample, optimizing the model parameters of the initial expression migration model until a loss function between the predicted expression parameters and the expression parameters included in the first training sample reaches a preset threshold;

[0160] Get the pre-trained expression transfer model.

[0161] The expression synchronization unit 403 is used to drive the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed.

[0162] Optionally, after the step of inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model to obtain a multi-dimensional expression parameter sequence output by the expression transfer model for the virtual character to be synchronized, the method is further used to:

[0163] Inputting the facial image sequence into a pre-trained head posture estimation model to obtain a head posture parameter sequence output by the head posture estimation model;

[0164] The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence comprises:

[0165] The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence and the head posture parameter sequence, so that the virtual character to be synchronized shows the facial expression and head posture of the expression providing object in the video to be processed.

[0166] Optionally, also used for:

[0167] Acquire a second training sample set; wherein the second training sample set includes a plurality of second training samples, and the second training samples include facial images and head posture parameters annotated for the facial images;

[0168] Using the facial image included in each of the second training samples as training data, inputting the facial image into an initial head pose estimation model, and obtaining predicted head pose parameters output by the initial head pose estimation model;

[0169] Based on the predicted head posture parameters and the head posture parameters included in the second training sample, optimizing the model parameters of the initial head posture estimation model until a loss function between the predicted head posture parameters and the head posture parameters included in the second training sample reaches a preset threshold;

[0170] Get a pre-trained head pose estimation model.

[0171] Optionally, after the step of inputting the facial image sequence into a pre-trained head pose estimation model to obtain a head pose parameter sequence output by the head pose estimation model, the method is further used to:

[0172] Acquire an eye picture sequence corresponding to the expression provided from the facial picture sequence, wherein the eye picture sequence includes a plurality of eye pictures, and the eye pictures include eye images of the object providing the expression;

[0173] Inputting the eye image sequence into a pre-trained gaze estimation model to obtain a gaze direction parameter sequence output by the gaze estimation model;

[0174] The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence and the head posture parameter sequence comprises:

[0175] The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence, so that the virtual character to be synchronized displays the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

[0176] Optionally, obtaining the expression from the facial image sequence to provide a corresponding eye image sequence includes:

[0177] Input each of the facial images in the facial image sequence into a facial key point detection model to obtain an eye detection frame corresponding to each of the facial images, wherein the eye detection frame is used to locate the position of the eye image of the expression providing object in the facial image;

[0178] Based on the eye detection frame corresponding to each of the facial images, cropping each of the facial images to obtain an eye image corresponding to each of the facial images;

[0179] Scaling the eye image corresponding to each of the facial images to a second preset size;

[0180] Based on the time sequence of each of the facial images in the facial image sequence, the eye images corresponding to each of the facial images and scaled to a second preset size are combined into the eye image sequence.

[0181] Optionally, the driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence includes:

[0182] Integrate the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence to generate animation data for driving the virtual character to be synchronized;

[0183] Performing time series smoothing processing on the animation data to obtain processed animation data;

[0184] The processed animation data is applied to the virtual character to be synchronized, so that the virtual character to be synchronized shows the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

[0185] Optionally, also used for:

[0186] Acquire a third training sample set; wherein the third training sample set includes a plurality of third training samples, and the third training samples include eye pictures and sight direction parameters annotated for the eye pictures;

[0187] Using the eye image included in each of the third training samples as training data, inputting the data into an initial sight line estimation model, and obtaining a predicted sight line direction parameter output by the initial sight line estimation model;

[0188] Based on the predicted sight line direction parameter and the sight line direction parameter included in the third training sample, optimizing the model parameters of the initial sight line estimation model until a loss function between the predicted sight line direction parameter and the sight line direction parameter included in the third training sample reaches a preset threshold;

[0189] Get a pre-trained gaze estimation model.

[0190] A third embodiment of the present application provides an electronic device, Figure 5 It is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0191] like Figure 5 As shown, the electronic device provided in this embodiment includes: a memory 501 and a processor 502;

[0192] The memory 501 is used to store computer instructions for executing the method for synchronizing the expression of a virtual character;

[0193] The processor 502 is configured to execute computer instructions stored in the memory 501 to perform the following operations:

[0194] Acquire a facial picture sequence of an expression providing object from a video to be processed, wherein the video to be processed at least presents a facial expression of the expression providing object, and the facial picture sequence includes a plurality of facial pictures, and the facial pictures include a facial image of the expression providing object;

[0195] Inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, obtaining a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, wherein each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression;

[0196] The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed.

[0197] Optionally, before the step of obtaining a facial picture sequence of an expression providing object from the video to be processed, the following steps are further performed:

[0198] Download the video to be processed from the storage end according to the link address corresponding to the video to be processed; wherein the link address is generated and sent by the storage end in response to receiving the video to be processed uploaded by the user end, and the video to be processed is stored in the storage end in the form of object storage.

[0199] Optionally, the step of obtaining a facial image sequence of an expression providing object from the video to be processed includes:

[0200] Input each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, wherein the face detection frame is used to locate the position of the facial image of the expression providing object in the video frame image;

[0201] Based on the face detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain a face image corresponding to each of the video frame images;

[0202] Based on the time sequence of each of the video frame images in the video to be processed, the facial images corresponding to each of the video frame images are combined into the facial image sequence.

[0203] Optionally, the video to be processed presents facial expressions of multiple objects;

[0204] The step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image includes:

[0205] Input each of the video frame images into the face detection model to obtain a plurality of candidate face detection frames corresponding to each of the video frame images, each of the candidate face detection frames being used to locate a position of a face image of the object in the video frame image;

[0206] Determine a face detection frame corresponding to a first video frame picture according to sizes of multiple candidate face detection frames corresponding to the first video frame picture, wherein the first video frame picture is a video frame picture that is first in time sequence among multiple video frame pictures included in the video to be processed;

[0207] Determine a face detection frame corresponding to a second video frame picture according to an intersection-and-union ratio of a plurality of candidate face detection frames corresponding to a second video frame picture and a face detection frame corresponding to the first video frame picture, wherein the second video frame picture is a video frame picture that is second in time sequence among a plurality of video frame pictures included in the video to be processed;

[0208] The face detection frame corresponding to each of the multiple video frame pictures included in the video to be processed is determined in sequence.

[0209] Optionally, after the step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, the following is further performed:

[0210] Performing smoothing processing on the face detection frame corresponding to each of the video frame images to obtain a processed face detection frame corresponding to each of the video frame images;

[0211] The step of cropping each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images includes:

[0212] Based on the processed facial detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain the facial image corresponding to each of the video frame images.

[0213] Optionally, after the step of performing cropping processing on each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images, the following is further performed:

[0214] Scaling the facial image corresponding to each of the video frame images to a first preset size;

[0215] The combining the facial pictures corresponding to each of the video frame pictures into the facial picture sequence based on the time sequence of each of the video frame pictures in the video to be processed includes:

[0216] Based on the timing of each of the video frame images in the video to be processed, the facial images scaled to the first preset size corresponding to each of the video frame images are combined into the facial image sequence.

[0217] Optionally, inputting the facial image sequence into a pre-trained expression transfer model comprises:

[0218] Packing the multiple facial images included in the facial image sequence into multiple batches, each batch including a preset number of facial images;

[0219] The facial image sequences are input into the expression transfer model in batches, so that the expression transfer model synchronously processes the preset number of facial images included in each batch.

[0220] Optionally, also execute:

[0221] Acquire a first training sample set; wherein the first training sample set includes a plurality of first training samples, the first training samples including a facial image, character information of a virtual character to be synchronized, and expression parameters for the virtual character to be synchronized annotated on the facial image;

[0222] Using the facial image and the character information of the virtual character to be synchronized included in each of the first training samples as training data, inputting an initial expression transfer model, and obtaining predicted expression parameters for the virtual character to be synchronized output by the initial expression transfer model;

[0223] Based on the predicted expression parameters and the expression parameters included in the first training sample, optimizing the model parameters of the initial expression migration model until a loss function between the predicted expression parameters and the expression parameters included in the first training sample reaches a preset threshold;

[0224] Get the pre-trained expression transfer model.

[0225] Optionally, after the step of inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model to obtain a multi-dimensional expression parameter sequence output by the expression transfer model for the virtual character to be synchronized, the following steps are further performed:

[0226] Inputting the facial image sequence into a pre-trained head posture estimation model to obtain a head posture parameter sequence output by the head posture estimation model;

[0227] The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence comprises:

[0228] The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence and the head posture parameter sequence, so that the virtual character to be synchronized shows the facial expression and head posture of the expression providing object in the video to be processed.

[0229] Optionally, also execute:

[0230] Acquire a second training sample set; wherein the second training sample set includes a plurality of second training samples, and the second training samples include facial images and head posture parameters annotated for the facial images;

[0231] Using the facial image included in each of the second training samples as training data, inputting the facial image into an initial head pose estimation model, and obtaining predicted head pose parameters output by the initial head pose estimation model;

[0232] Based on the predicted head posture parameters and the head posture parameters included in the second training sample, optimizing the model parameters of the initial head posture estimation model until a loss function between the predicted head posture parameters and the head posture parameters included in the second training sample reaches a preset threshold;

[0233] Get a pre-trained head pose estimation model.

[0234] Optionally, after the step of inputting the facial image sequence into a pre-trained head pose estimation model to obtain a head pose parameter sequence output by the head pose estimation model, the following steps are further performed:

[0235] Acquire an eye picture sequence corresponding to the expression provided from the facial picture sequence, wherein the eye picture sequence includes a plurality of eye pictures, and the eye pictures include eye images of the object providing the expression;

[0236] Inputting the eye image sequence into a pre-trained gaze estimation model to obtain a gaze direction parameter sequence output by the gaze estimation model;

[0237] The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence and the head posture parameter sequence comprises:

[0238] The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence, so that the virtual character to be synchronized displays the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

[0239] Optionally, obtaining the expression from the facial image sequence to provide a corresponding eye image sequence includes:

[0240] Input each of the facial images in the facial image sequence into a facial key point detection model to obtain an eye detection frame corresponding to each of the facial images, wherein the eye detection frame is used to locate the position of the eye image of the expression providing object in the facial image;

[0241] Based on the eye detection frame corresponding to each of the facial images, cropping each of the facial images to obtain an eye image corresponding to each of the facial images;

[0242] Scaling the eye image corresponding to each of the facial images to a second preset size;

[0243] Based on the time sequence of each of the facial images in the facial image sequence, the eye images corresponding to each of the facial images and scaled to a second preset size are combined into the eye image sequence.

[0244] Optionally, the driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence includes:

[0245] Integrate the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence to generate animation data for driving the virtual character to be synchronized;

[0246] Performing time series smoothing processing on the animation data to obtain processed animation data;

[0247] The processed animation data is applied to the virtual character to be synchronized, so that the virtual character to be synchronized shows the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

[0248] Optionally, also execute:

[0249] Acquire a third training sample set; wherein the third training sample set includes a plurality of third training samples, and the third training samples include eye pictures and sight direction parameters annotated for the eye pictures;

[0250] Using the eye image included in each of the third training samples as training data, inputting the data into an initial sight line estimation model, and obtaining a predicted sight line direction parameter output by the initial sight line estimation model;

[0251] Based on the predicted sight line direction parameter and the sight line direction parameter included in the third training sample, optimizing the model parameters of the initial sight line estimation model until a loss function between the predicted sight line direction parameter and the sight line direction parameter included in the third training sample reaches a preset threshold;

[0252] Get a pre-trained gaze estimation model.

[0253] A fourth embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the methods described in the embodiments of the present application.

[0254] It should be noted that the relational terms such as "first" and "second" in this article are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "includes" and other similar forms are the same in meaning, and the end of any one or more items after any of the above words is open-ended, and any of the above nouns does not mean that the one or more items have been exhaustively listed or are limited to the one or more items listed.

[0255] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are not feasible. For example, if it is expressed that a database may include A or B, then unless otherwise specifically stated or not feasible, it may include database A, or B, or A and B. As a second example, if it is expressed that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0256] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in the present disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0257] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary from implementation to implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments may be readily apparent from the specific embodiments disclosed in the present application. This specification and examples are for exemplary purposes only, and the true scope and essence of the present application are described by the claims. The sequence of steps shown in the diagrams is also for the purpose of explanation only and is not intended to be limited to any particular steps or sequence. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0258] In the drawings and detailed description of the present application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are only general and descriptive and not for the purpose of limitation.

Claims

1. A method for synchronizing expression of a virtual character, characterized in that: The method comprises: Acquire a facial picture sequence of an expression providing object from a video to be processed, wherein the video to be processed at least presents a facial expression of the expression providing object, and the facial picture sequence includes a plurality of facial pictures, and the facial pictures include a facial image of the expression providing object; Inputting the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, obtaining a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, wherein each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression; The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed.

2. The method according to claim 1, characterized in that Before the step of acquiring a facial picture sequence of an expression providing object from the video to be processed, the method further comprises: Download the video to be processed from the storage end according to the link address corresponding to the video to be processed; wherein the link address is generated and sent by the storage end in response to receiving the video to be processed uploaded by the user end, and the video to be processed is stored in the storage end in the form of object storage.

3. The method according to claim 1, characterized in that: The step of obtaining a facial image sequence of an expression providing object from a video to be processed includes: Input each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, wherein the face detection frame is used to locate the position of the facial image of the expression providing object in the video frame image; Based on the face detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain a face image corresponding to each of the video frame images; Based on the time sequence of each of the video frame images in the video to be processed, the facial images corresponding to each of the video frame images are combined into the facial image sequence.

4. The method according to claim 3, characterized in that The video to be processed presents facial expressions of multiple objects; The step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image includes: Input each of the video frame images into the face detection model to obtain a plurality of candidate face detection frames corresponding to each of the video frame images, each of the candidate face detection frames being used to locate a position of a face image of the object in the video frame image; Determine a face detection frame corresponding to a first video frame picture according to sizes of multiple candidate face detection frames corresponding to the first video frame picture, wherein the first video frame picture is a video frame picture that is first in time sequence among multiple video frame pictures included in the video to be processed; Determine a face detection frame corresponding to a second video frame picture according to an intersection-and-union ratio of multiple candidate face detection frames corresponding to the second video frame picture and the face detection frame corresponding to the first video frame picture, wherein the second video frame picture is a video frame picture that is second in time sequence among multiple video frame pictures included in the video to be processed; The face detection frame corresponding to each of the multiple video frame pictures included in the video to be processed is determined in sequence.

5. The method according to claim 3, characterized in that: After the step of inputting each video frame image in the video to be processed into a face detection model to obtain a face detection frame corresponding to each video frame image, the method further includes: Performing smoothing processing on the face detection frame corresponding to each of the video frame images to obtain a processed face detection frame corresponding to each of the video frame images; The step of cropping each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images includes: Based on the processed facial detection frame corresponding to each of the video frame images, each of the video frame images is cropped to obtain the facial image corresponding to each of the video frame images.

6. The method according to claim 3, characterized in that After the step of cropping each of the video frame images based on the face detection frame corresponding to each of the video frame images to obtain a face image corresponding to each of the video frame images, the method further includes: Scaling the facial image corresponding to each of the video frame images to a first preset size; The combining the facial pictures corresponding to each of the video frame pictures into the facial picture sequence based on the time sequence of each of the video frame pictures in the video to be processed includes: Based on the timing of each of the video frame images in the video to be processed, the facial images scaled to the first preset size corresponding to each of the video frame images are combined into the facial image sequence.

7. The method according to claim 1, characterized in that The step of inputting the facial image sequence into a pre-trained expression transfer model comprises: Packing the multiple facial images included in the facial image sequence into multiple batches, each batch including a preset number of facial images; The facial image sequences are input into the expression transfer model in batches, so that the expression transfer model synchronously processes the preset number of facial images included in each batch.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a first training sample set; wherein the first training sample set includes a plurality of first training samples, the first training samples including a facial image, character information of a virtual character to be synchronized, and expression parameters for the virtual character to be synchronized annotated on the facial image; Using the facial image and the character information of the virtual character to be synchronized included in each of the first training samples as training data, inputting an initial expression transfer model, and obtaining predicted expression parameters for the virtual character to be synchronized output by the initial expression transfer model; Based on the predicted expression parameters and the expression parameters included in the first training sample, optimizing the model parameters of the initial expression migration model until a loss function between the predicted expression parameters and the expression parameters included in the first training sample reaches a preset threshold; Get the pre-trained expression transfer model.

9. The method according to claim 1, characterized in that: After the step of inputting the facial image sequence and the character information of the virtual character to be synchronized into the pre-trained expression transfer model to obtain the multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, the method further includes: Inputting the facial image sequence into a pre-trained head posture estimation model to obtain a head posture parameter sequence output by the head posture estimation model; The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence comprises: The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence and the head posture parameter sequence, so that the virtual character to be synchronized shows the facial expression and head posture of the expression providing object in the video to be processed.

10. The method according to claim 9, characterized in that The method further comprises: Acquire a second training sample set; wherein the second training sample set includes a plurality of second training samples, and the second training samples include facial images and head posture parameters annotated for the facial images; Using the facial image included in each of the second training samples as training data, inputting the facial image into an initial head pose estimation model, and obtaining predicted head pose parameters output by the initial head pose estimation model; Based on the predicted head posture parameters and the head posture parameters included in the second training sample, optimizing the model parameters of the initial head posture estimation model until a loss function between the predicted head posture parameters and the head posture parameters included in the second training sample reaches a preset threshold; Get a pre-trained head pose estimation model.

11. The method according to claim 9, characterized in that After the step of inputting the facial image sequence into a pre-trained head posture estimation model to obtain a head posture parameter sequence output by the head posture estimation model, the method further comprises: Acquire an eye picture sequence corresponding to the expression provided from the facial picture sequence, wherein the eye picture sequence includes a plurality of eye pictures, and the eye pictures include eye images of the object providing the expression; Inputting the eye image sequence into a pre-trained gaze estimation model to obtain a gaze direction parameter sequence output by the gaze estimation model; The step of driving the virtual character to be synchronized based on the multi-dimensional expression parameter sequence and the head posture parameter sequence comprises: The virtual character to be synchronized is driven based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence, so that the virtual character to be synchronized displays the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

12. The method according to claim 11, characterized in that The step of obtaining the expression from the facial image sequence and providing a corresponding eye image sequence comprises: Input each of the facial images in the facial image sequence into a facial key point detection model to obtain an eye detection frame corresponding to each of the facial images, wherein the eye detection frame is used to locate the position of the eye image of the expression providing object in the facial image; Based on the eye detection frame corresponding to each of the facial images, cropping each of the facial images to obtain an eye image corresponding to each of the facial images; Scaling the eye image corresponding to each of the facial images to a second preset size; Based on the time sequence of each of the facial images in the facial image sequence, the eye images corresponding to each of the facial images and scaled to a second preset size are combined into the eye image sequence.

13. The method according to claim 11, characterized in that The driving of the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence comprises: Integrate the multi-dimensional expression parameter sequence, the head posture parameter sequence, and the sight direction parameter sequence to generate animation data for driving the virtual character to be synchronized; Performing time series smoothing processing on the animation data to obtain processed animation data; The processed animation data is applied to the virtual character to be synchronized, so that the virtual character to be synchronized shows the facial expression, head posture, and sight direction of the expression providing object in the video to be processed.

14. The method according to claim 11, characterized in that The method further comprises: Acquire a third training sample set; wherein the third training sample set includes a plurality of third training samples, and the third training samples include eye pictures and sight direction parameters annotated for the eye pictures; Using the eye image included in each of the third training samples as training data, inputting the data into an initial sight line estimation model, and obtaining a predicted sight line direction parameter output by the initial sight line estimation model; Based on the predicted sight line direction parameter and the sight line direction parameter included in the third training sample, optimizing the model parameters of the initial sight line estimation model until a loss function between the predicted sight line direction parameter and the sight line direction parameter included in the third training sample reaches a preset threshold; Get a pre-trained gaze estimation model.

15. A virtual character expression synchronization device, characterized in that: The device comprises: a facial picture acquisition unit, an expression parameter generation unit, and an expression synchronization unit; The facial image acquisition unit is used to acquire a facial image sequence of an expression providing object from a video to be processed, wherein the video to be processed at least presents the facial expression of the expression providing object, and the facial image sequence includes a plurality of facial images, and the facial images include a facial image of the expression providing object; The expression parameter generating unit is used to input the facial image sequence and the character information of the virtual character to be synchronized into a pre-trained expression transfer model, and obtain a multi-dimensional expression parameter sequence for the virtual character to be synchronized output by the expression transfer model, wherein each dimension in the multi-dimensional expression parameter sequence is used to characterize the driving amount of an expression; The expression synchronization unit is used to drive the virtual character to be synchronized based on the multi-dimensional expression parameter sequence, so that the virtual character to be synchronized shows the facial expression of the expression providing object in the video to be processed.

16. An electronic device, characterized in that: include: Memory, processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the method according to any one of claims 1-14.

17. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 14 is performed.

Images