Virtual expression migration method and device, storage medium and electronic device

Abstract

The application discloses a virtual expression migration method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a target video frame sequence, wherein a face of a target object with a to-be-migrated expression is displayed in a video frame of the target video frame sequence; extracting a target object face key point set of the target object from the target video frame sequence; determining a reference expression base matched with the face of the target object and a key frame in the target video frame sequence based on the target object face key point set, wherein a key expression of the target object is displayed in the key frame; determining a target expression base of a target virtual character subjected to expression migration by using the reference expression base and the key expression in the key frame; and controlling the target virtual character to generate a virtual expression migrated from the face of the target object based on the target expression base. The application solves the technical problem that the accuracy of a virtual expression obtained by using an existing expression migration method is low.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a method and apparatus for transferring virtual facial expressions, a storage medium, and an electronic device. Background Technology

[0002] Nowadays, in many video sharing scenarios, in order to enrich the user's product experience, users often choose to transfer their real facial expressions onto the face of a virtual character, thereby enabling them to synchronously display their facial expression changes to their friends or fans in the image of the virtual character, thus increasing the interactive fun.

[0003] The most common migration method currently is to capture the user's 3D facial expressions in real time within the video frame, and then migrate these real 3D expressions to the corresponding virtual character's face according to the definition of parallel parameters. However, this virtual expression migration method is limited by fixed parameter definition standards, making it difficult to achieve refined expressions in the migrated virtual character, thus resulting in low accuracy of the migration results.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, storage medium, and electronic device for transferring virtual facial expressions, in order to at least solve the technical problem of low accuracy of virtual facial expressions obtained by existing facial expression transfer methods.

[0006] According to one aspect of the present invention, a method for transferring virtual expressions is provided, comprising: acquiring a target video frame sequence, wherein the video frames of the target video frame sequence display the face of a target object whose expression is to be transferred; extracting a set of object facial key points of the target object from the target video frame sequence; determining a reference expression base that matches the face of the target object and key frames in the target video frame sequence based on the set of object facial key points, wherein the key frames display key expressions of the target object; determining a target expression base for a target virtual character to receive expression transfer using the reference expression base and the key expressions in the key frames; and controlling the target virtual character to generate a virtual expression transferred from the face of the target object based on the target expression base.

[0007] According to another aspect of the present invention, a virtual expression transfer apparatus is also provided, comprising: an acquisition unit, configured to acquire a target video frame sequence, wherein the video frames of the target video frame sequence display the face of a target object whose expression is to be transferred; an extraction unit, configured to extract a set of key points of the target object's face from the target video frame sequence; a first determination unit, configured to determine a reference expression base matching the face of the target object and key frames in the target video frame sequence based on the set of key points of the target object's face, wherein the key frames display key expressions of the target object; a second determination unit, configured to determine a target expression base of a target virtual character receiving expression transfer using the reference expression base and the key expressions in the key frames; and a generation unit, configured to control the target virtual character to generate a virtual expression transferred from the face of the target object based on the target expression base.

[0008] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described virtual expression migration method at runtime.

[0009] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the virtual facial expression transfer method as described above.

[0010] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described virtual expression migration method through the computer program.

[0011] In this embodiment of the invention, a target video frame sequence is acquired, wherein the video frames of the target video frame sequence display the face of the target object whose expression is to be transferred; a set of key points of the target object's face is extracted from the target video frame sequence; a reference expression base matching the face of the target object is determined based on the set of key points of the target object's face, and key frames in the target video frame sequence are determined, wherein the key frames display the key expressions of the target object; using the reference expression base and the key expressions in the key frames, a target expression base for the target virtual character to receive expression transfer is determined; the target virtual character is controlled to generate a virtual expression transferred from the face of the target object based on the target expression base, thereby determining the target expression base for generating virtual expressions based on the video frame sequence of the target object, and generating virtual expressions transferred from the face of the target object based on the target expression base, which improves the accuracy of the generated virtual expressions and solves the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0013] Figure 1 This is a schematic diagram of the hardware environment for an optional virtual facial expression transfer method according to an embodiment of the present invention;

[0014] Figure 2 This is a flowchart of an optional virtual facial expression transfer method according to an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of an optional virtual facial expression transfer method according to an embodiment of the present invention;

[0016] Figure 4 This is a schematic diagram of another optional virtual facial expression transfer method according to an embodiment of the present invention;

[0017] Figure 5 This is a schematic diagram of an optional virtual facial expression transfer method according to an embodiment of the present invention;

[0018] Figure 6 This is a schematic diagram of another optional virtual facial expression transfer method according to an embodiment of the present invention;

[0019] Figure 7 This is a schematic diagram of another optional virtual facial expression transfer method according to an embodiment of the present invention;

[0020] Figure 8This is a flowchart of an optional virtual facial expression transfer method according to an embodiment of the present invention;

[0021] Figure 9 This is a schematic diagram of another optional virtual expression transfer method according to an embodiment of the present invention;

[0022] Figure 10 This is a schematic diagram of the structure of an optional virtual facial expression transfer device according to an embodiment of the present invention;

[0023] Figure 11 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The following explains the terminology used in this application:

[0027] Facial expression tracking: A task that involves acquiring 2D video input from the user and converting it into 3D facial expression information;

[0028] Expression transfer: The task of acquiring the user's 3D facial expression information and transferring it to the game character's face.

[0029] ROM: Range of Motion, refers to the maximum range of facial expressions (maximum facial expression range);

[0030] AutoROM: Automatic Range of Motion, which calculates the maximum range of motion automatically. Note: The original range of motion needs to be defined manually to determine the types of extreme facial expressions.

[0031] Parallelization parameters: This refers to ensuring that the semantics, number, and other parameters of the game character's facial expressions are consistent with those of the user, so that the same set of parameters can be used to control the game character's facial expressions to be consistent with the user's facial expressions;

[0032] Blendshapes: Blend shape; indicates the difference of each key point between each 3D facial expression and a 3D neutral face;

[0033] According to one aspect of the present invention, a method for transferring virtual facial expressions is provided. As an optional implementation, the above-described method for transferring virtual facial expressions can be applied, but is not limited to, to applications such as... Figure 1 The virtual facial expression migration system shown consists of server 102 and terminal device 104. For example... Figure 1As shown, server 102 is connected to terminal device 104 via network 110. This network may include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless network includes Bluetooth, Wi-Fi, and other networks enabling wireless communication. The terminal device may include, but is not limited to, at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, PDAs, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, in-vehicle devices, etc. The terminal device may have a client installed, such as an emoji generation client or a game client. The terminal device also includes a display, a processor, and a memory. The display can be used to show the program interface of the emoji generation client or game client, as well as the video content uploaded to the server. The processor can be used to preprocess the video files to be uploaded before transmission, for example, by compressing the acquired video files. The memory is used to store the videos to be uploaded. It is understood that after obtaining the target video to be uploaded in the aforementioned terminal device 104, the terminal device 104 can send the target video to the server 102 via network 110. Upon receiving the target video, the server 102 generates a target emoji base that matches the target video based on the video uploaded by the terminal device 104, and generates a corresponding virtual emoji based on the target emoji base. The terminal device 104 can receive the virtual emoji returned by the server 102 via network 110. The server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud server. The aforementioned server includes a database and a processing engine. The database may include a set of basic emoji bases for matching corresponding emoji bases for user objects; the processing engine is used to generate the corresponding emoji base based on the acquired target video.

[0034] According to one aspect of the present invention, the virtual expression migration system described above may further perform the following steps: Terminal device 104 performs step S102 to acquire a target video to be identified; then performs step S104, whereby terminal device 104 sends the target video to server 102 via network 110; server 102 performs steps S106 to S114 to acquire a target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression is to be migrated; extracts a set of key points of the target object's face from the target video frame sequence; determines a reference expression base that matches the target object's face based on the set of key points of the target object's face, and key frames in the target video frame sequence, wherein the key frames display the key expressions of the target object; uses the reference expression base and the key expressions in the key frames to determine the target expression base of the target virtual character receiving the expression migration; controls the target virtual character to generate a virtual expression migrated from the target object's face based on the target expression base; then performs step S116, whereby server 102 sends the virtual expression to terminal device 104 via network 110. It is understandable that if the terminal device 104 is a device with sufficient computing power, the above steps S106 to S114 can also be performed in the terminal device 104.

[0035] In this embodiment of the invention, a target video frame sequence is acquired, wherein the video frames of the target video frame sequence display the face of the target object whose expression is to be transferred; a set of key points of the target object's face is extracted from the target video frame sequence; a reference expression base matching the face of the target object is determined based on the set of key points of the target object's face, and key frames in the target video frame sequence are determined, wherein the key frames display the key expressions of the target object; using the reference expression base and the key expressions in the key frames, a target expression base for the target virtual character to receive expression transfer is determined; the target virtual character is controlled to generate a virtual expression transferred from the face of the target object based on the target expression base, thereby determining the target expression base for generating virtual expressions based on the video frame sequence of the target object, and generating virtual expressions transferred from the face of the target object based on the target expression base, which improves the accuracy of the generated virtual expressions and solves the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0036] The above is merely an example, and no limitation is made in this embodiment.

[0037] As an optional implementation method, such as Figure 2 As shown, the above-mentioned method for transferring virtual expressions includes the following steps:

[0038] S202, Obtain the target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred;

[0039] Understandably, a target video segment can be acquired before acquiring the target video frame sequence. This target video includes the target object's face. In an optional implementation, the target object can be instructed to activate the terminal camera to acquire the target video via text or voice prompts on the terminal interface, or instructed to upload the target video via touch operation. When instructing the target object to activate the terminal camera to acquire the target video via text or voice prompts on the terminal interface, the target object can be further instructed to make a specified facial expression via voice or text, such as by playing voice prompts like "Please open your mouth as wide as possible," "Please open your eyes as wide as possible," "Please close your eyes," or "Please laugh," to instruct the target object to make the corresponding expression.

[0040] Understandably, the aforementioned expressions can be more exaggerated than the target subject's normal expressions. By analyzing frame-by-frame videos of the target subject making these exaggerated expressions in response to voice prompts, we can obtain the changes in the target subject's face from normal to exaggerated expressions, thereby accurately determining the subtle facial changes during the process of making different expressions.

[0041] S204, Extract the set of facial key points of the target object from the target video frame sequence;

[0042] It is understandable that the aforementioned set of facial key points can be a set of locations of key parts of the target object's face. For example, key parts could be: pupils, outer corners of the eyes, inner corners of the eyes, inner and outer corners of the eyebrows, left corner of the mouth, right corner of the mouth, etc. The selection methods for the key parts and key point sets mentioned above are merely examples, and no specific restrictions are placed on the extraction methods here.

[0043] In one alternative approach, the aforementioned set of facial key points can be a two-dimensional point set, treating the target object's face as a plane and determining the position coordinates of each point on that plane. For example, assuming the area containing the target object's face is a 500px * 800px rectangle, the two-dimensional coordinates of the left pupil are (200px, 600px), and the corresponding coordinates of the right pupil are (400px, 600px). Based on this method, the position coordinates of each key point on the target object's face are determined, thus extracting the set of facial key points.

[0044] In another alternative approach, the aforementioned set of facial key points can be a three-dimensional point set, that is, reconstructing the facial spatial structure of the target object using the three-dimensional coordinates of different points. By determining the three-dimensional spatial coordinates of each key point on the target object's face, the set of facial key points of the target object is extracted.

[0045] Understandably, for each video frame, the set of facial key points for that frame can be extracted. By extracting the set of facial key points for all frames, the positional changes of each key point on the face can be determined.

[0046] As a specific method, such as Figure 3 As shown, the facial key points in this embodiment are defined using a 68-key-point definition. Figure 3 In the center, points 1 to 17 indicate the position of the facial contour; points 18 to 22 indicate the position of the left eyebrow; points 23 to 27 indicate the position of the right eyebrow; points 28 to 31 indicate the position of the bridge of the nose; points 32 to 36 indicate the position of the tip of the nose; points 37 to 42 and points 43 to 48 indicate the position of the eyes; and points 49 to 68 indicate the position of the mouth.

[0047] In this embodiment, the facial expression capture algorithm uses the RGBD driver output of ARKit. Other alternative methods include dynamic xyz, faceware, etc., and no specific facial expression capture algorithm is limited here.

[0048] S206, Based on the set of facial key points of the object, determine the reference expression base that matches the face of the target object, and the key frames in the target video frame sequence, wherein the key frames display the key expressions of the target object;

[0049] It should be noted that the above-mentioned reference expression base can be the initial Blendshapes of the video character. Blendshapes is a set of references that make up the overall expression. Given a set of references for the overall expression, multiple expressions can be determined by linear combination of these references.

[0050] As an alternative approach, the aforementioned key expressions can be extreme expressions, such as: a person's facial expression with their mouth wide open; a person's facial expression with their eyes wide open; or a person's facial expression with their mouth twisted to the far right. Video frames displaying these facial expressions are the key frames in the target video frame sequence.

[0051] It should be noted that, as an optional approach, when the set of facial key points of the object is obtained, an average set of key points can be determined based on the set of facial key points of each frame, so as to determine the deformation of the average expression of the current target object relative to each key point of the facial model, and thus determine the initial Blendshapes.

[0052] S208, using the reference expression base and key expressions in the keyframes, determine the target expression base of the target virtual character receiving expression transfer;

[0053] It is understandable that after obtaining the initialized Blendshapes, the target expression base of the target virtual character receiving the expression transfer, i.e., the target Blendshapes, can be determined based on the key expressions in the keyframes.

[0054] The aforementioned target virtual character can be a virtual game character, a virtual game animal character, a virtual anime character, etc. In other words, using the method described in this application, a virtual avatar can be generated based on the target character's facial expressions by acquiring a video clip of the target character. This virtual avatar can make the same facial expressions as the target character.

[0055] For example, upon receiving a video of facial expressions sent by user A, a set of blendshapes corresponding to a virtual avatar (let's say a game character B) selected by user A can be generated by parsing the video. While controlling the virtual avatar to make various facial expressions, the generated blendshapes drive the virtual avatar to make the corresponding expressions. It's understandable that since the virtual avatar's blendshapes are generated based on the facial expression video of the target user A, the expressions made by the virtual avatar are identical to those made by user A. In another alternative approach, the difference between the expressions made by the virtual avatar (game character B) and those made by user A under the same circumstances is less than a target threshold.

[0056] S210, control the target virtual character to generate virtual expressions transferred from the target object's face based on the target expression base.

[0057] The following combination Figure 4 One specific implementation of the above method will be described.

[0058] First, users need to input a video containing as many facial expressions as possible. For example... Figure 4 The example shows four key expressions from a user-input facial video: the first is "eyes closed," the second is "eyes wide open and lips tightly closed," the third is "eyes open and mouth turned to the left," and the fourth is "eyes open and mouth open." Figure 4 As shown, the range of motion for each expression is maximized;

[0059] Subsequently, this video was used to track changes in the 3D Face mesh. It can be seen that the expressions of the four facial models determined by the Face mesh also correspond to the four expressions mentioned above. The first model's expression is "eyes closed", the second model's expression is "eyes wide open and lips closed", the third model's expression is "eyes open and mouth turned to the left", and the fourth model's expression is "eyes open and mouth open".

[0060] Next, key points are extracted from each frame of the acquired video, such as... Figure 4 The image shows four sets of key point location information corresponding to the four expressions mentioned above;

[0061] It should be noted that the target virtual character in this embodiment is a game character. Accordingly, it is also necessary to extract the corresponding key points for each Blendshape of the game character;

[0062] Finally, the user's 3D key points and the game character's 3D key points are used together as input to the AutoROM algorithm, which then generates corresponding personalized blendshapes. These personalized blendshapes then drive the game character to make corresponding virtual expressions based on the user's facial expressions.

[0063] In this embodiment of the invention, a target video frame sequence is acquired, wherein the video frames of the target video frame sequence display the face of the target object whose expression is to be transferred; a set of key points of the target object's face is extracted from the target video frame sequence; a reference expression base matching the face of the target object is determined based on the set of key points of the target object's face, and key frames in the target video frame sequence are determined, wherein the key frames display the key expressions of the target object; using the reference expression base and the key expressions in the key frames, a target expression base for the target virtual character to receive expression transfer is determined; the target virtual character is controlled to generate a virtual expression transferred from the face of the target object based on the target expression base, thereby determining the target expression base for generating virtual expressions based on the video frame sequence of the target object, and generating virtual expressions transferred from the face of the target object based on the target expression base, which improves the accuracy of the generated virtual expressions and solves the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0064] As an optional implementation, the reference expression base for matching the target object's face, determined based on the object's facial key point set, and the keyframes in the target video frame sequence include:

[0065] S1, obtain the template expression base, wherein the template expression base includes the basic facial expressions corresponding to each facial part used to combine different expressions;

[0066] It is understood that the aforementioned template expression base can be the template Blendshapes, which can be a set of facial template data composed of 52 baseline facial features. For example, for the eyeBlinkLeft template in Blendshapes, it indicates the difference between the position coordinates of each key point of the left eye when the left eye blinks and the position coordinates of each key point of the left eye in the neutral expression. In other words, by adding the difference data from the eyeBlinkLeft template to the left eye coordinates in the neutral expression, the position and shape of the left eye in the "blinking left eye" expression can be determined. Similarly, for the eyeLookDownLeft template in Blendshapes, it indicates the difference between the position coordinates of each key point of the left eye when the left eye looks down and the position coordinates of each key point of the left eye in the neutral expression. In other words, by adding the difference data from the eyeLookDownLeft template to the left eye coordinates in the neutral expression, the position and shape of the left eye in the "looking down" expression can be determined. The above are merely illustrative examples, and the specific data format in the template Blendshapes is limited.

[0067] S2, according to the key point positions indicated by the key point set of the object's face, transfer the template expression base to the face of the target object to obtain a reference expression base that matches the face of the target object;

[0068] As an alternative approach, the above steps can use the RBF (Radial Basis Function) algorithm to transfer the template blendshapes to the 3D neutral face of the person in the video, thereby obtaining the initial blendshapes of the video character.

[0069] S3, based on the temporal relationship of the facial key points appearing in each video frame, fit the distribution probability that matches each video frame.

[0070] Taking the key points of the mouth as an example, suppose that in the video uploaded by the target object, the two-dimensional coordinates of the first key point of the mouth in the first frame of the video are (200px, 300px); the two-dimensional coordinates of the first key point of the mouth in the second frame of the video are (210px, 320px); the two-dimensional coordinates of the first key point of the mouth in the third frame of the video are (205px, 315px)... and so on. The coordinates of the first key point of the mouth in all video frames are obtained (assuming there are 2400 frames). Based on the obtained 2400 sets of two-dimensional coordinate data, the position information of the first key point of the mouth is fitted to determine the probability distribution of the first key point of the mouth in different positions. Similarly, a similar fitting method is used for each of the above facial key points to determine the probability distribution of each facial key point appearing in different positions.

[0071] S4 uses probability distribution to determine the keyframes in the target video frame sequence.

[0072] Based on the probability distribution of various facial key points appearing at different locations, the video frames corresponding to key point locations with probability distributions below a certain threshold are identified as keyframes. This means that the locations of key points with probability distributions below a certain threshold represent the most exaggerated facial expressions made by the target object, and thus the corresponding video frames can be determined as keyframes based on their probability distributions.

[0073] The following combination Figure 5 , Figure 6 The above methods will be explained. For example... Figure 5 As shown, the RBF algorithm is first used to transfer the template Blendshapes to the 3D neutral face of the person in the video, obtaining the initial Blendshapes of the video character; then as... Figure 6 As shown, 2D keypoints are extracted from each frame of the video sequence, and for each keypoint i of the mouth, based on its position (x... i y i Fit a Gaussian distribution G(x) i y i Then, the 20 frames with the lowest probability of occurrence were selected as keyframes. This means that the frames with the lowest frequency of occurrence are generally when the mouth shape is at its limit, such as when the mouth is wide open or pouting.

[0074] Through the above-described embodiments of this application, by obtaining a template expression base, wherein the template expression base includes basic facial expressions corresponding to each facial part for combining different expressions; according to the key point positions indicated by the object's facial key point set, the template expression base is transferred to the target object's face to obtain a reference expression base that matches the target object's face; based on the temporal relationship of the facial key points appearing in each video frame of the object's facial key point set, a distribution curve probability matching the facial key points of each video frame is fitted; the key frames in the target video frame sequence are determined using the distribution probability curve, thereby generating a virtual expression transferred from the target object's face based on the target expression base, improving the accuracy of the generated virtual expression and solving the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0075] As an optional implementation, fitting the distribution probability matching each video frame to the temporal relationship of the facial keypoints in the above-mentioned object facial keypoint set in each video frame includes:

[0076] S1, determine the subset of facial key points located in each video frame from the set of facial key points of the object;

[0077] S2, iterate through each facial keypoint subset, and sequentially use each facial keypoint subset as the current facial keypoint subset, and perform the following operations:

[0078] S3, obtain the probability distribution of each facial key point in the current facial key point subset;

[0079] S4, fuse the probability distributions of each facial key point to obtain the current distribution probability that matches the current video frame corresponding to the current subset of facial key points;

[0080] S5, if the current video frame is not the last video frame in the target video frame sequence, obtain the next facial key point subset as the current facial key point subset.

[0081] Combination Figure 3 The above methods are explained as follows: Figure 3 As shown in the figure, a total of 68 keypoints are displayed. Assuming that in the first frame of the video, the position of keypoint 1 is (110px, 320px); in the second frame, the position of keypoint 1 is (115px, 325px); in the third frame, the position of keypoint 1 is (120px, 300px)... and in the 1000th frame, the position of keypoint 1 is (130px, 350px). Then, the set of position information of keypoint 1 in the above 1000 video frames is obtained, and the probability distribution of keypoint 1's position is fitted based on the 1000 position information points to determine the probability distribution of keypoint 1 at different positions.

[0082] Similarly, continue to obtain the position information of key point 2 in the 1st, 2nd, 3rd...1000th frames of the video, that is, obtain the set of position information of key point 2 in the above 1000 video frames, and fit the position probability distribution of key point 2 based on the 1000 position information to determine the probability distribution of key point 2 in different positions.

[0083] Similarly, obtain the position information of all remaining key points in different frames, and calculate the probability distribution of different key point frames at different positions.

[0084] Assuming that the positional distributions of the aforementioned key points are uncorrelated, the positional probabilities of the aforementioned key points are cumulatively calculated in a distributed manner to obtain the current distribution probability that matches the current video frame corresponding to the current subset of facial key points.

[0085] Through the above-described embodiments of this application, by determining a subset of facial key points located in each video frame from the set of facial key points of the target; traversing each subset of facial key points, and sequentially using each subset as the current subset of facial key points, the following operations are performed: obtaining the probability distribution of each facial key point in the current subset of facial key points; fusing the probability distributions of each facial key point to obtain the current distribution probability that matches the current video frame corresponding to the current subset of facial key points; and, if the current video frame is not the last video frame in the target video frame sequence, obtaining the next subset of facial key points as the current subset of facial key points, thereby accurately selecting key frames based on the positional distribution probability of each key point, thus improving the facial expression transfer effect.

[0086] As an optional implementation, obtaining the probability distribution of each facial key point in the current subset of facial key points includes:

[0087] Iterate through each facial keypoint in the current set of facial keypoints and perform the following operations:

[0088] S1, obtain the position information of the i-th facial key point in the current facial key point subset, where i is a natural number greater than or equal to 1 and less than or equal to Q, and Q is the number of facial key points contained in the current facial key point subset.

[0089] S2, based on the position information of the i-th facial key point, fit a Gaussian probability distribution that matches the i-th facial key point.

[0090] As an optional implementation, the above-mentioned fusion of the probability distributions of each facial key point to obtain the current distribution probability that matches the current video frame corresponding to the current subset of facial key points includes: after obtaining the distribution probability that matches each video frame in the target video frame sequence, sorting the distribution probabilities in ascending order, and determining the first p video frames as key frames, where p is a natural number greater than or equal to 1 and less than or equal to N, and N is the number of video frames in the target video frame sequence.

[0091] Specifically, such as Figure 6 As shown, 2D keypoints are extracted from each frame of the video sequence, and for each keypoint i of the mouth, based on its position (x... i y i Fit a Gaussian distribution G(x) i y i The probability distribution of this keypoint can then be represented by the following formula:

[0092] G(x1, x2, ..., x) n y1, y2, ..., y n )=G1(x, y)×G2(x, y)×...×G n (x, y)

[0093] As shown in the above formula, the assumption is that the probability of each key point appearing is independent. Therefore, the 20 frames with the lowest probability of occurrence are selected as key frames, meaning that the frames with the lowest frequency of occurrence are generally when the mouth shape is at its limit, such as when the mouth is wide open or pouting.

[0094] Through the above-described embodiments of this application, by traversing each facial keypoint in the current facial keypoint subset, the following operations are performed: obtaining the position information of the i-th facial keypoint in the current facial keypoint subset, where i is a natural number greater than or equal to 1 and less than or equal to Q, and Q is the number of facial keypoints contained in the current facial keypoint subset; based on the position information of the i-th facial keypoint, fitting a Gaussian probability distribution that matches the i-th facial keypoint, thereby accurately selecting keyframes according to the position distribution probability of each keypoint, thus improving the facial expression transfer effect.

[0095] As an optional implementation, the above-described method of determining the target expression base of the target virtual character receiving expression transfer using a reference expression base and key expressions in keyframes includes:

[0096] S1, Construct an expression base description function that matches the key expressions in each keyframe, wherein the expression base description function includes the position of the expression base associated with the key expressions;

[0097] S2, using the expression base description function to construct the adaptation function and smoothing function to match the key frame. The adaptation function is used to represent the relationship between the expression base associated with the key expression and the key points in the key frame, and the smoothing function is used to represent the weight relationship between the expression bases associated with each key expression.

[0098] S3, perform joint iteration on the fitting function and the smoothing function until the target convergence condition is met;

[0099] S4, the expression basis associated with the key expressions when the target convergence condition is met is determined as the target expression basis.

[0100] Specifically, such as Figure 7 The method for solving personalized facial expressions is illustrated below. The above-described method of constructing the facial expression base description function that matches the key facial expressions in each keyframe can be achieved by representing the positions of each key point of the virtual character using the following function:

[0101] S k = a1×B1 + a2×B2 + ... + a m ×B m

[0102] It should be noted that the above S k Used to represent the expression in the k-th keyframe, the above a i The combination coefficients, which can be considered constants in this step, are B as mentioned above. i For each expression base blendshape, blendshape can form expression bases blendshapes, where k is the corresponding keyframe number.

[0103] The above adaptation function can be expressed by the following formula:

[0104] E fit =Σ k w k ×(s k -M k ) 2

[0105] Where M k Key points corresponding to keyframes captured by data; w k The corresponding weights are set here to the result of the previous iteration as exp((s k -M k ) 2 In the first iteration, w k =1.

[0106] The above smoothing function can be expressed by the following formula:

[0107] E smooth =∑k w k ×(a i -a q ) 2

[0108] Where a q These are the combination coefficients obtained using the previous iteration of Blendshapes. That is, E fit B at its minimum i The corresponding combination coefficients.

[0109] The above method of jointly iterating over the fit function and the smoothing function can be implemented using the following formula:

[0110] E α =E fit +E smooth

[0111] In the process of iterating through the above formula, we first iterate to make E fit Reaching the minimum value, then continuing the iteration to make E α The minimum value is reached, thus determining whether the convergence condition has been met.

[0112] If the convergence condition is met through the above method, the final a is obtained. i And B i This allows us to determine the target facial expression base.

[0113] The above method constructs an expression base description function that matches the key expressions in each keyframe, where the expression base description function includes the position of the expression base associated with the key expressions. Using the expression base description function, an adaptation function and a smoothing function are constructed to match the keyframes. The adaptation function represents the relationship between the expression base associated with the key expressions and the key points in the keyframes, and the smoothing function represents the weight relationship between the expression bases associated with each key expression. The adaptation function and the smoothing function are jointly iterated until the target convergence condition is reached. The expression base associated with the key expressions when the target convergence condition is met is determined as the target expression base. Then, a virtual expression transferred from the face of the target object is generated based on the target expression base, improving the accuracy of the generated virtual expression and solving the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0114] As an optional implementation, the above-described adaptation function and smoothing function for keyframe matching using the expression basis description function include:

[0115] S1, obtain the first difference between the k-th expression base description function and the key points in the k-th keyframe; construct the adaptation function based on the weighted sum of the squares of the first difference;

[0116] S2, obtain the description weights that match each expression base description function and the second difference between them and the historical description weights obtained in the previous iteration; construct a smoothing function based on the weighted sum of the squares of the second difference.

[0117] It should be noted that the first difference between the k-th expression basis description function and the key points in the k-th keyframe can be expressed as s. k -M k Therefore, the weighted sum of the squares of the first difference is determined to be ∑ k w k ×(s k -M k ) 2 That is, determine the adaptor function:

[0118] E fit =∑ k w k ×(s k -M k ) 2

[0119] Furthermore, the aforementioned descriptive weights can be a i The historical description weights obtained from the previous iteration can be represented as a. q The second difference between the two can be expressed as a i -a q Furthermore, the smoothing function can be determined as:

[0120] E smooth =∑ k w k ×(a i -a q ) 2

[0121] The method described in this application obtains the first difference between the k-th expression base description function and the key points in the k-th keyframe; an adaptation function is constructed based on the weighted sum of the squares of the first difference; a second difference is obtained between the description weights matched with each expression base description function and the historical description weights obtained in the previous iteration; a smoothing function is constructed based on the weighted sum of the squares of the second difference, and a suitable constraint function is constructed. This allows the parameter set corresponding to the target expression base to be determined through iterative convergence, improving the accuracy of the generated virtual expressions and solving the technical problem of low accuracy in virtual expressions obtained by existing expression transfer methods.

[0122] As an optional implementation, the above-mentioned joint iteration of the fit function and the smoothing function until the target convergence condition is met includes:

[0123] S1, combine the fit function and the smoothing function to obtain the target function;

[0124] S2, when the output of the fitting function reaches its minimum value in multiple joint iterations, and the output of the objective function reaches its minimum value, the objective convergence condition is determined to be met.

[0125] Understandably, the above method of jointly iterating the fit function and the smoothing function can be implemented using the following formula:

[0126] E α =E fit +E smooth

[0127] In the process of iterating through the above formula, we first iterate to make E fit Reaching the minimum value, then continuing the iteration to make E α The minimum value is reached, thus determining whether the convergence condition has been met.

[0128] Through the above-described embodiments of this application, the target function is obtained by combining the adaptation function and the smoothing function; when the output of the adaptation function reaches its minimum value in multiple joint iterations, and the output of the target function reaches its minimum value, the target convergence condition is determined to be met. Then, the parameter set corresponding to the target expression basis can be determined by iterative convergence, which improves the accuracy of the generated virtual expression and solves the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0129] As an optional implementation, the above-mentioned control target virtual character generates virtual expressions transferred from the target object's face based on the target expression base, including:

[0130] S1, Determine the set of facial key points for the target virtual character;

[0131] S2 uses the key point positions indicated by the key point set of the target expression base on the character's face to drive the target virtual character to adjust the facial parts in order to generate a virtual expression transferred from the target object's face.

[0132] In this embodiment, the facial expression capture algorithm uses the RGBD driver output of arkit. Other alternative methods include dynamic xyz (dynamic axis method), faceware (facial software method), etc. No specific facial expression capture algorithm is limited here.

[0133] Through the above implementation method, a set of key points on the face of the target virtual character is determined; using the key point positions indicated by the key point set on the face of the target expression, the target virtual character is driven to adjust its facial features to generate a virtual expression transferred from the face of the target object. This generates a precisely corresponding blendshape based on the user object's expression video, and drives the virtual object to present the corresponding expression based on the precise blendshape, thereby improving the accuracy of the generated virtual expression and solving the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0134] The following combination Figure 8 A specific embodiment of this application will be described.

[0135] like Figure 8 As shown, S802, Blendshapes initialization;

[0136] Specifically, it can be like Figure 5 As shown, this step can use the RBF algorithm to transfer the template Blendshapes to the 3D neutral face of the person in the video, and obtain the initial Blendshapes of the video character.

[0137] Next, as in step S804, keyframe filtering is performed.

[0138] Specifically, such as Figure 6 As shown, 2D keypoints are extracted from each frame of the video sequence, and for each keypoint i of the mouth, based on its position (x... i y i Fit a Gaussian distribution G(x) i y i The probability distribution of this keypoint can then be represented by the following formula:

[0139] G(x1, x2, ..., x) n y1, y2, ..., y n )=G1(x, y)×G2(x, y)×...×G n (x, y)

[0140] As shown in the above formula, the assumption is that the probability of each key point appearing is independent. Therefore, the 20 frames with the lowest probability of occurrence are selected as key frames, meaning that the frames with the lowest frequency of occurrence are generally when the mouth shape is at its limit, such as when the mouth is wide open or pouting.

[0141] Next, as in step S806, solve for the personalized expression Blendshapes;

[0142] Input the character's initial Blendshapes and video keyframes. Then, using the keyframes as examples and the keypoints of the initial Blendshapes as templates, use the example-based facial rigging algorithm to obtain the character's personalized Blendshapes.

[0143] Specifically, in this embodiment, the loss function of keyframe fitting is used to adjust the weights obtained in each step. Through each iteration, the effectiveness of the keyframe fitting is continuously adjusted. The specific steps are as follows:

[0144] First, solve for B corresponding to Blendshapes. i , i = 1, ..., M, where M is the number of Blendshapes.

[0145] The key elements of a virtual avatar can be represented by the following formula:

[0146] S k = a1×B1 + a2×B2 + ... + a m ×B m

[0147] The above a i The combination coefficients, which can be considered constants in this step, are B as mentioned above. i The Blendshape corresponds to an expression base, where k is the corresponding keyframe number.

[0148] E fit =Σ k w k ×(s k -M k ) 2

[0149] Where M k Key points corresponding to keyframes captured by data; w k The corresponding weights are set here to the result of the previous iteration as exp((s k -M k ) 2 In the first iteration, w k =1. Corresponding to E above. fit The optimization goal is: Even if E fit Minimize, and thus obtain the corresponding B i .

[0150] Next, solve for the coefficient a. i , i = 1, ..., M.

[0151] First, construct the smoothing function.

[0152] E smooth =∑ k w k ×(a i -a q ) 2

[0153] Among them, a q These are the combination coefficients obtained using the previous iteration of Blendshapes. That is, E fit B at its minimum i The corresponding combination coefficients.

[0154] The optimization goal is to address this as follows:

[0155] E α =E fit +E smooth

[0156] And satisfy

[0157] That is, to obtain E through the above objectives. α The minimum value of a corresponds to i .

[0158] In this embodiment, approximately 5 iterations (i.e., the combined iterations of the two steps mentioned above) are typically required to obtain a stable and reliable B corresponding to Blendshapes. i , i = 1, ..., M.

[0159] The following combination Figure 9 Another alternative implementation will be described.

[0160] like Figure 9 As shown, the user inputs a video of an expression, and then the facemesh vertices of the user are obtained through an expression tracking algorithm. Next, the blendshape parameters for each frame are calculated using personalized expression blendshapes. Finally, the blendshape parameters for each frame are used to drive the game character. In this embodiment, a parallel parameter-driven approach is adopted, meaning that the user's blendshapes and the game character's blendshapes are defined identically, but their behavior varies depending on the neutral face.

[0161] In this embodiment of the invention, a target video frame sequence is acquired, wherein the video frames of the target video frame sequence display the face of the target object whose expression is to be transferred; a set of key points of the target object's face is extracted from the target video frame sequence; a reference expression base matching the face of the target object is determined based on the set of key points of the target object's face, and key frames in the target video frame sequence are determined, wherein the key frames display the key expressions of the target object; using the reference expression base and the key expressions in the key frames, a target expression base for the target virtual character receiving expression transfer is determined; the target virtual character is controlled to generate a virtual expression transferred from the face of the target object based on the target expression base, thereby determining the target expression base for generating virtual expressions based on the video frame sequence of the target object, and generating virtual expressions transferred from the face of the target object based on the target expression base. This can achieve expression transfer that is more consistent with range-of-motion, and can more accurately align the semantics of the expression input by the user with the expression output to the new character, thereby improving the accuracy of the generated virtual expressions and solving the technical problem of low accuracy of virtual expressions obtained by existing expression transfer methods.

[0162] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0163] According to another aspect of the present invention, a virtual expression migration apparatus for implementing the above-described virtual expression migration method is also provided. For example... Figure 10 As shown, the device includes:

[0164] The acquisition unit 1002 is used to acquire a target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred;

[0165] Extraction unit 1004 is used to extract the set of facial key points of the target object from the target video frame sequence;

[0166] The first determining unit 1006 is used to determine a reference expression base that matches the face of the target object based on the set of facial key points of the object, and key frames in the target video frame sequence, wherein the key frames display the key expressions of the target object.

[0167] The second determining unit 1008 is used to determine the target expression base of the target virtual character receiving expression transfer by using the reference expression base and the key expressions in the key frame;

[0168] The generation unit 1010 is used to control the target virtual character to generate virtual expressions transferred from the face of the target object based on the target expression base.

[0169] Optionally, in this embodiment, the implementation of each of the above-mentioned unit modules can be referred to the above-mentioned method embodiments, which will not be repeated here.

[0170] According to another aspect of the present invention, an electronic device for implementing the above-described virtual expression transfer method is also provided, the electronic device being... Figure 11 The terminal device or server shown. This embodiment uses this electronic device as an example for illustration. Figure 11 As shown, the electronic device includes a memory 1102 and a processor 1104. The memory 1102 stores a computer program, and the processor 1104 is configured to execute the steps of any of the above method embodiments via the computer program.

[0171] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0172] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0173] S1, Obtain the target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred;

[0174] S2, Extract the set of facial key points of the target object from the target video frame sequence;

[0175] S3, based on the set of facial key points of the object, determine the reference expression base that matches the face of the target object, and the key frames in the target video frame sequence, wherein the key frames display the key expressions of the target object;

[0176] S4. Using the reference facial expression base and key facial expressions in the keyframes, determine the target facial expression base of the target virtual character receiving the facial expression transfer;

[0177] S5 controls the target virtual character to generate virtual expressions transferred from the target object's face based on the target expression base.

[0178] Alternatively, as those skilled in the art will understand, Figure 11The structure shown is for illustrative purposes only. Electronic devices can also be in-vehicle terminals, smartphones (such as Android phones, iOS phones, etc.), tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 11 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 11 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 11 The different configurations shown.

[0179] The memory 1102 can be used to store software programs and modules, such as the program instructions / modules corresponding to the virtual expression migration method and device in this embodiment of the invention. The processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102, thereby realizing the aforementioned virtual expression migration method. The memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1102 may further include memory remotely located relative to the processor 1104, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1102 may be used, but is not limited to, to store information such as various elements in the viewing perspective image and virtual expression migration information. As an example, such as Figure 11 As shown, the memory 1102 may include, but is not limited to, the acquisition unit 1002, extraction unit 1004, first determination unit 1006, second determination unit 1008, and generation unit 1010 of the virtual expression migration device. Furthermore, it may include, but is not limited to, other module units of the virtual expression migration device, which will not be described in detail in this example.

[0180] Optionally, the transmission device 1106 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1106 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1106 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0181] In addition, the above-mentioned electronic device also includes a display 1108 and a connection bus 1110 for connecting the various module components in the above-mentioned electronic device.

[0182] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0183] According to one aspect of this application, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in embodiments of this application.

[0184] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0185] According to one aspect of this application, a computer-readable storage medium is provided, wherein a processor of a computer device reads computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the above-described virtual expression migration method.

[0186] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0187] S1, Obtain the target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred;

[0188] S2, Extract the set of facial key points of the target object from the target video frame sequence;

[0189] S3, based on the set of facial key points of the object, determine the reference expression base that matches the face of the target object, and the key frames in the target video frame sequence, wherein the key frames display the key expressions of the target object;

[0190] S4. Using the reference facial expression base and key facial expressions in the keyframes, determine the target facial expression base of the target virtual character receiving the facial expression transfer;

[0191] S5 controls the target virtual character to generate virtual expressions transferred from the target object's face based on the target expression base.

[0192] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0193] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0194] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0196] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0198] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for transferring virtual facial expressions, characterized in that, include: Obtain a target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred; Extract the set of facial key points of the target object from the target video frame sequence; Based on the set of facial key points of the object, key frames in the target video frame sequence are determined, wherein the key frames display the key expressions of the target object; According to the key point positions indicated by the set of key points on the object's face, the template expression base is transferred to the face of the target object to obtain a reference expression base that matches the face of the target object. The template expression base includes the basic facial expressions corresponding to each facial part for combining different expressions. An adaptation function and a smoothing function matching the keyframe are constructed, and the adaptation function and the smoothing function are jointly iterated until the target convergence condition is reached to obtain the target expression base of the target virtual character that accepts expression transfer. The adaptation function is used to represent the relationship between the expression base associated with the key expression and the key points in the keyframe, and the smoothing function is used to represent the weight relationship between the expression bases associated with each key expression. The target expression base is the expression base associated with the key expression when the target convergence condition is reached. The target virtual character is controlled to generate virtual expressions transferred from the face of the target object based on the target expression base.

2. The method according to claim 1, characterized in that, The process of determining keyframes in the target video frame sequence based on the set of facial key points of the object includes: Based on the temporal relationship of the facial key points of the object's facial key point set appearing in each video frame, a distribution probability matching each video frame is fitted. The keyframes in the target video frame sequence are determined using the probability distribution.

3. The method according to claim 2, characterized in that, The step of fitting a probability distribution matching each video frame based on the temporal relationship of facial key points appearing in each video frame of the object's facial key point set includes: A subset of facial key points located within each video frame is determined from the set of facial key points of the object. Iterate through each subset of facial key points, and sequentially use each subset of facial key points as the current subset of facial key points, and perform the following operations: Obtain the probability distribution of each facial key point in the current facial key point subset. The probability distributions of each facial key point are fused to obtain the current distribution probability that matches the current video frame corresponding to the current subset of facial key points. If the current video frame is not the last video frame in the target video frame sequence, the next subset of facial key points is obtained as the current subset of facial key points.

4. The method according to claim 3, characterized in that, The step of obtaining the probability distribution of each facial key point in the current subset of facial key points includes: Iterate through each facial keypoint in the current subset of facial keypoints and perform the following operations: Obtain the position information of the i-th facial key point in the current facial key point subset, where i is a natural number greater than or equal to 1 and less than or equal to Q, and Q is the number of facial key points contained in the current facial key point subset. Based on the location information of the i-th facial key point, a Gaussian probability distribution matching the i-th facial key point is fitted.

5. The method according to claim 3, characterized in that, The process of fusing the probability distributions of each facial key point to obtain the current probability distribution that matches the current video frame corresponding to the current subset of facial key points includes: Given the distribution probability of each video frame in the target video frame sequence, the distribution probabilities are sorted in ascending order, and the first p video frames are determined as the keyframes, where p is a natural number greater than or equal to 1 and less than or equal to N, and N is the number of video frames in the target video frame sequence.

6. The method according to claim 1, characterized in that, The process of constructing an adaptation function and a smoothing function that match the keyframes, and jointly iterating the adaptation function and the smoothing function until the target convergence condition is met, to obtain the target facial expression base of the target virtual character that accepts facial expression transfer, includes: Construct an expression base description function that matches the key expression in each keyframe, wherein the expression base description function includes the position of the expression base associated with the key expression; The adaptation function and the smoothing function that match the keyframe are constructed using the expression base description function; The fitting function and the smoothing function are jointly iterated until the target convergence condition is reached. The expression base associated with the key expression when the target convergence condition is met is determined as the target expression base.

7. The method according to claim 6, characterized in that, The step of constructing the adaptation function and smoothing function that match the keyframe using the expression base description function includes: Obtain the first difference between the k-th expression base description function and the key points in the k-th keyframe; construct the adaptation function based on the weighted sum of the squares of the first difference; Obtain the description weights that match each of the expression base description functions, and the second difference between them and the historical description weights obtained in the previous iteration; construct the smoothing function based on the weighted sum of the squares of the second difference.

8. The method according to claim 6, characterized in that, The joint iteration of the fitting function and the smoothing function until the target convergence condition is met includes: The target function is obtained by combining the adaptation function and the smoothing function. When the output of the adaptation function reaches its minimum value during multiple joint iterations, and the output of the objective function reaches its minimum value, the objective convergence condition is determined to have been met.

9. The method according to any one of claims 1 to 8, characterized in that, The step of controlling the target virtual character to generate virtual expressions transferred from the target object's face based on the target expression base includes: Determine the set of facial key points for the target virtual character; Using the target expression base at the key point locations indicated by the character's facial key point set, the target virtual character is driven to adjust its facial features to generate a virtual expression transferred from the target object's face.

10. A method for transferring virtual facial expressions, characterized in that, include: An acquisition unit is used to acquire a target video frame sequence, wherein the video frames of the target video frame sequence display the face of the target object whose expression needs to be transferred; Extraction unit, used to extract the set of facial key points of the target object from the target video frame sequence; The first determining unit is configured to determine keyframes in the target video frame sequence based on the set of facial key points of the object, wherein the keyframes display the key expressions of the target object; and to transfer the template expression base to the face of the target object according to the key point positions indicated by the set of facial key points of the object, so as to obtain a reference expression base that matches the face of the target object and the keyframes in the target video frame sequence, wherein the template expression base includes the basic facial expressions corresponding to each facial part for combining different expressions; The second determining unit is used to construct an adaptation function and a smoothing function that match the key frame, and to jointly iterate the adaptation function and the smoothing function until the target convergence condition is reached, so as to obtain the target expression base of the target virtual character that accepts expression transfer. The adaptation function is used to represent the relationship between the expression base associated with the key expression and the key points in the key frame, the smoothing function is used to represent the weight relationship between the expression bases associated with each key expression, and the target expression base is the expression base associated with the key expression when the target convergence condition is reached. The generation unit is used to control the target virtual character to generate virtual expressions transferred from the face of the target object based on the target expression base.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method according to any one of claims 1 to 9.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 9 through the computer program.

Images