Data processing method and device, equipment, medium and program product

By obtaining face images for face pinching, and driving benchmark virtual characters based on expression data, fusion processing generates target virtual characters, solving the problem of custom separation of virtual characters, realizing the coherent execution of face pinching and expression-driven, reducing virtual characters Role customization cost.

CN119091089APending Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310657012.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, virtual characters' face-pinching and expression-driven links are independent, resulting in the separation of virtual characters' customization and separation, increasing the cost and user experience difficulty.

Method used

By obtaining face images for face pinching, and driving benchmark virtual characters based on expression data, fusion processing generates target virtual characters to ensure that the face after face pinching has facial features and virtual facial expressions.

Benefits of technology

It realizes coherent execution of face-pinching and expression-driven, reduces the cost of virtual character customization, improves user experience, and solves the problem of virtual character customization separation caused by the independence of face-pinching and expression-driven links.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091089A_ABST
    Figure CN119091089A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a medium and a program product, and the method comprises the steps: obtaining a human face image, and carrying out the face pinching processing of a human face in the human face image; obtaining expression data, and performing expression driving on the reference virtual character based on the expression data to obtain a reference virtual character with a virtual facial expression; and performing fusion processing on the face after face pinching processing and the reference virtual character with the virtual facial expression to generate a target virtual character corresponding to the face image. According to the embodiment of the invention, face pinching and expression driving are integrated into one solution, and the continuity of customization of the virtual character is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of image processing, and specifically to a data processing method, a data processing device, a computer equipment, a computer-readable storage medium and a computer program product. Background Art

[0002] The Metaverse is a virtual world that is constructed by humans using digital technology. It is a reflection of the real world or transcends the real world and can interact with the real world. In the virtual world of the Metaverse, people need to have a brand new virtual character in order to achieve the same things as real things (such as conversation or making friends) in the virtual world.

[0003] Currently, the facial features and facial expressions of virtual characters are customized independently; for example, users can customize the facial features of virtual characters into a puppy, and drive the facial expressions of virtual characters into a smiling expression, etc. Therefore, how to combine facial feature pinching and facial expression driving to achieve virtual character customization has become a research hotspot. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, medium and program product, which effectively combine face pinching and expression drive to ensure the consistency of customization for virtual characters.

[0005] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Acquire a face image, and perform face pinching processing on the face in the face image;

[0007] Acquire expression data, and based on the expression data, perform expression driving on the reference virtual character to obtain a reference virtual character with a virtual facial expression; the expression data is used to describe the virtual facial expression;

[0008] The face after the face-pinching process and the reference virtual character with virtual facial expressions are fused to generate a target virtual character corresponding to the face image; the target virtual character has the facial features of the face after the face-pinching process and has virtual facial expressions.

[0009] On the other hand, an embodiment of the present application provides a data processing device, the device comprising:

[0010] The acquisition unit is used to acquire a face image, and the processing unit is used to perform face pinching processing on the face in the face image;

[0011] The acquisition unit is further used to acquire expression data, and the processing unit is further used to drive the reference virtual character based on the expression data to obtain the reference virtual character with virtual facial expressions; the expression data is used to describe the virtual facial expressions;

[0012] The processing unit is also used to fuse the face after the face-pinching process and the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image; the target virtual character has the facial features of the face after the face-pinching process and has virtual facial expressions.

[0013] In one implementation, the processing unit, when performing face pinching processing on a face in a face image, is specifically used to:

[0014] An automatic face pinching algorithm is obtained, and the face in the face image is automatically pinched using the automatic face pinching algorithm to generate a first skeleton skin; the skeleton skin includes skeletons and skins, and the skin is composed of mesh vertices and mesh edges; the mesh vertices are bound to the skeletons, so that the positions of the mesh vertices can be changed by controlling the position changes of the skeletons; each face part in the skeleton skin is associated with one or more bones; one face part corresponds to one adjustment control, and the adjustment control is used to adjust the position of the bones associated with the face part;

[0015] If there is a trigger operation for the adjustment control corresponding to the target face part in the first skeleton skin, then determining the second adjustment value of the adjustment control according to the trigger operation; the target face part is any face part in the first skeleton skin;

[0016] Determining second position change information of a bone associated with the target facial part according to a second adjustment value of the adjustment control;

[0017] Based on the second position change information of the bone, the position of the bone in the first skeleton skin is adjusted, and the adjusted first skeleton skin is used as the second skeleton skin.

[0018] In one implementation, the adjustment range of the adjustment control has a mapping relationship with the position change information of the bone; the processing unit is further used for:

[0019] Determine first position information of mesh vertices bound to the bones associated with the target face part from the first bone skin, and calculate first position change information of the bones according to the first position information of the mesh vertices;

[0020] Mapping the first position change information to the adjustment range of the adjustment control to obtain a first adjustment value of the adjustment control;

[0021] Subtracting the maximum limit value in the adjustment range of the adjustment control from the first adjustment value to obtain a first remaining adjustable range; the first remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during manual face pinching;

[0022] The second adjustment value of the adjustment control belongs to the first remaining adjustable range.

[0023] In one implementation, the processing unit is further configured to:

[0024] Obtaining a first adjustment value and a second adjustment value of an adjustment control corresponding to a target facial part; the target facial part is any facial part of a face after the face pinching process;

[0025] Adding the first adjustment value and the second adjustment value of the adjustment control to obtain an addition result;

[0026] The maximum limit value in the adjustment range of the adjustment control is subtracted from the addition result to obtain a second remaining adjustable range; the second remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during the expression driving process.

[0027] In one implementation, the processing unit is used to drive the reference virtual character based on the expression data to obtain the reference virtual character with the virtual facial expression, specifically to:

[0028] Decoding the expression data to obtain decoded expression data;

[0029] Performing data format conversion on the decoded expression data to obtain converted expression data;

[0030] Based on the converted expression data and the second remaining adjustable range, the reference virtual character is expression-driven to obtain a reference virtual character with a virtual facial expression.

[0031] In one implementation, if the logical thread is occupied during the data format conversion process, the processing unit is used to convert the data format of the decoded expression data, and when the converted expression data is obtained, it is used to:

[0032] Get the accelerated conversion algorithm;

[0033] The decoded expression data is converted into a data format using an accelerated conversion algorithm to obtain converted expression data.

[0034] In one implementation, the accelerated conversion algorithm includes executing data format conversion by starting a target subthread; a processing unit is used to perform data format conversion on the decoded expression data using the accelerated conversion algorithm, and when the converted expression data is obtained, it is specifically used to:

[0035] When starting data format conversion, start the target sub-thread;

[0036] Use the target sub-thread to convert the data format of the expression data;

[0037] After the target child thread completes the data format conversion, it notifies the logic thread to obtain the converted expression data.

[0038] In one implementation, the accelerated conversion algorithm includes performing data format conversion by a graphics processor; a processing unit is used to perform data format conversion on the decoded expression data using the accelerated conversion algorithm, and when the converted expression data is obtained, it is specifically used to:

[0039] Calling the shader in the graphics processor to convert the data format of the expression data to obtain the converted expression data;

[0040] The converted expression data is sent back to the central processing unit.

[0041] In one implementation, the processing unit is used to drive the reference virtual character's expression based on the converted expression data and the second remaining adjustable range to obtain the reference virtual character with a virtual facial expression, specifically for:

[0042] Obtain one or more preset expression poses; each preset expression pose presents a facial expression through a reference virtual character;

[0043] Determine an expression weight corresponding to each preset expression posture in one or more preset expression postures according to the virtual facial expression described by the converted expression data and the second remaining adjustable range;

[0044] Based on the expression weight corresponding to each preset expression posture, one or more preset expression postures are weighted to obtain a reference virtual character with a virtual facial expression;

[0045] Among them, the third adjustment value of the adjustment control corresponding to the target facial part in the benchmark virtual character with virtual facial expression belongs to the second remaining adjustable range.

[0046] In one implementation, the face after the face-pinching process is skinned by the second skeleton; the processing unit is used to fuse the face after the face-pinching process with the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image, specifically for:

[0047] Replace the baseline virtual character with virtual facial expressions with the second skeleton skin;

[0048] The second skeleton skin with virtual facial expression after replacement is used as the third skeleton skin;

[0049] The third skeleton skin is rendered to generate a target virtual character corresponding to the face image.

[0050] In one implementation, the application scenario to which the data processing method is applicable includes at least one of the following: a game scenario, an audio and video scenario, and a conversation scenario; when the application scenario is a game scenario, the data processing method is integrated into an Unreal Engine that provides the game scenario;

[0051] The expression data is any video frame containing a virtual facial expression in the video; the video is obtained by shooting with a camera or received from an external device; and the number of expression data is multiple.

[0052] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0053] A processor for loading and executing computer programs;

[0054] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method is implemented.

[0055] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned data processing method.

[0056] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0057] The embodiment of the present application supports face pinching processing of a face in a face image provided by a user, so as to obtain a face with a face pinching effect by face pinching processing; then, it also supports expression driving of a reference virtual character based on expression data describing a virtual facial expression, so that the reference virtual character can present the virtual facial expression. Furthermore, after obtaining a face after face pinching processing and a reference virtual character with a virtual facial expression, the face after face pinching processing and the reference virtual character with a virtual facial expression can be fused to generate a target virtual character corresponding to the face image; at this time, the target virtual character has the facial features of the face after face pinching processing and has a virtual facial expression. It can be seen that the embodiment of the present application proposes a solution that combines face pinching and expression driving. This solution can not only pinch the face in the face image according to the user's face pinching needs, and obtain the face after face pinching whose facial features match the facial features of the face in the face image, but also make the face after face pinching present virtual facial expressions through expression driving; by integrating face pinching and expression driving into one solution, the problem of virtual character customization separation caused by the independence of face pinching link and expression driving link is effectively solved; for users, it can be used out of the box, realize the coherent execution of face pinching and expression driving, and reduce the cost of virtual character customization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 is a schematic diagram of a skeleton skin provided by an exemplary embodiment of the present application;

[0060] Figure 2 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application;

[0061] Figure 3 It is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0062] Figure 4 This is a flowchart of a combination of automatic face pinching and manual face pinching provided by an exemplary embodiment of the present application;

[0063] Figure 5a This is a schematic diagram of a process of automatic face pinching provided by an exemplary embodiment of the present application;

[0064] Figure 5bis a schematic diagram of facial key points in a human face provided by an exemplary embodiment of the present application;

[0065] Figure 6 This is a schematic diagram of a manual face pinching process provided by an exemplary embodiment of the present application;

[0066] Figure 7a This is a schematic diagram of a manual face pinching interface provided by an exemplary embodiment of the present application;

[0067] Figure 7b This is another schematic diagram of a manual face pinching interface provided by an exemplary embodiment of the present application;

[0068] Figure 8 is a schematic diagram of determining the remaining adjustable range of manual face pinching provided by an exemplary embodiment of the present application;

[0069] Fig. 9 is a schematic diagram of determining a remaining adjustable range that can be driven during expression driving provided by an exemplary embodiment of the present application;

[0070] Fig.10 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0071] Fig.11a is a schematic diagram of a real-time expression drive provided by an exemplary embodiment of the present application;

[0072] Fig.11b is a schematic diagram of an algorithm detection provided by an exemplary embodiment of the present application;

[0073] Fig.12 is a schematic diagram of a data format conversion provided by an exemplary embodiment of the present application;

[0074] Fig.13 is a schematic diagram of an expression drive provided by an exemplary embodiment of the present application;

[0075] Fig.14 is a schematic diagram of determining a preset expression posture provided by an exemplary embodiment of the present application;

[0076] Fig.15 is a structural schematic diagram of a data processing device provided by an exemplary embodiment of the present application;

[0077] Fig.16 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0079] In the embodiment of the present application, a data processing solution is proposed, specifically a processing solution integrating face pinching and expression driving is proposed. The following is a brief introduction to the relevant terms and concepts involved in the embodiment of the present application, including:

[0080] (1) Face pinching is a data operation that customizes the appearance (or look) of a virtual character in a virtual world; it includes the setting of the facial features, makeup, hair accessories, and clothing of the virtual character. By pinching the face of a virtual character in the virtual world according to their own wishes, the user can obtain a personalized virtual character, thereby helping the user to gain a better sense of immersion and experience in the virtual world. In an actual face pinching scenario, the user can preset a face pinching target (such as an object in the real world, or a cartoon-style object, etc.), and then pinch the virtual character in the virtual world into a look that matches (has a high degree of similarity to) the face pinching target by setting a series of modeling values ​​(i.e., face pinching parameters). It should be understood that face pinching is not limited to face pinching of a virtual character, but can also be performed on other identity parts of a virtual character other than the face; for the sake of ease of explanation, in the embodiment of the present application, face pinching of a virtual character is taken as an example of face pinching of a virtual character, which is specifically explained here.

[0081] On the one hand, according to the different dimensions of the spatial coordinate system of the virtual character after face pinching, face pinching can be divided into two-dimensional face pinching (abbreviated as 2D (two-dimensional) face pinching) and three-dimensional face pinching (abbreviated as 3D (three-dimensional) face pinching). Among them: ① 2D face pinching is to pinch a plane face (or two-dimensional face); specifically, the face pinching effect can be achieved by replacing the face map in the plane face (such as the part shape map of the face part) or rendering the material. ② 3D face pinching is to pinch a three-dimensional face; compared with a two-dimensional face, a three-dimensional face contains 360-degree information of the face. On the other hand, according to different face pinching methods, face pinching can be divided into automatic face pinching and manual face pinching. Among them: ① Manual face pinching refers to the user setting a set of face pinching parameters according to his own face pinching needs, so as to generate a virtual image through the user-defined face pinching parameter group. ② Automatic face pinching means that through the automatic face pinching algorithm, the program automatically calculates a set of face pinching parameters for the expected face input by the user, so as to obtain a three-dimensional face model after face pinching according to the face pinching parameter group; the facial features of the three-dimensional face model match the facial features of the expected face, and from the visual effect, the appearance of the three-dimensional face model is similar to that of the expected face.

[0082] It should be noted that the embodiments of the present application mainly relate to the above-mentioned 3D face pinching, which can be obtained by two parts of face pinching: automatic face pinching and / or manual face pinching.

[0083] Further, the three-dimensional face model mentioned in the embodiment of the present application specifically refers to 3D animation skeleton skinning (or simply skeleton skinning). Skeleton skinning includes skeletons and skinning. Among them: skeleton is a concept abstracted from the animation algorithm, which can be compared to human skeletons, imitating human skeletons to control model driving (or movement); specifically, the skeletons in the skeleton skinning form a hierarchical structure according to the characteristics of the virtual character; adjacent bones are connected by joints and can move relative to each other; by changing the angle between adjacent bones, the displacement of bones, etc., the skeletons that make up the virtual character can make different actions, thereby driving the skin attached to the skeleton to run, and the visual effect is that the virtual character performs different action effects. The skin is spliced ​​by one or more polygons; and a complex polygon is actually spliced ​​by multiple triangular faces, so the skin is composed of multiple triangular faces connected to each other, specifically composed of mesh vertices and mesh edges that make up the triangular face (which can be regarded as a mesh Mesh, which is composed of mesh vertices and mesh edges of triangular faces). Skinning can be regarded as skin, which can be used as a mesh to cover the skeleton to define the appearance of the virtual character; it should be understood that the skin here is not a fixed rigid mesh, but a deformable mesh that can change under the influence of the skeleton. In addition, when the skin is covered on the skeleton, the mesh vertices of each mesh that makes up the skin are bound to one or more bones, so that the position transformation (or change) of the control bone can change the position of the bound mesh vertices, that is, the position of each mesh vertex that makes up the skin can be affected by the position change of one or more bones. In this way, the position change of the mesh vertex is indirectly driven by controlling the position change of the bones, thereby controlling the change of the skin covered on the skeleton, and the visual effect is that the virtual character changes its action (such as facial expression changes, etc.).

[0084] A schematic diagram of an exemplary arm skeleton skin can be found in Figure 1 ;like Figure 1 As shown, the skeleton skin of the arm is composed of a skeleton structure 101 and a skin 102. Among them, the skeleton structure 101 is a skeleton chain composed of multiple joints and bones; the skin 102 is a skin composed of multiple meshes, such as mesh 103. When it is necessary to control the movement of the arm, such as controlling the forearm to move downward, it is necessary to control the bone 1011 in the skeleton structure 101 to move downward with the joint 1012 as the center point; in this way, as the position of the bone 1011 changes, the position of the mesh vertices (such as mesh vertex 1031) affected by the bone 1011 is controlled to change, thereby realizing the movement of the forearm. In addition, by Figure 1It is not difficult to know that a mesh vertex may be affected by multiple bones; for example, mesh vertex 1031 on the upper arm is affected not only by bone 1011 but also by bone 1013. When a mesh vertex is affected by multiple bones, it actually means that the mesh vertex is affected by different weights of each bone in the multiple bones; that is, the weight of each bone's influence on the position of the mesh vertex can be determined based on the geometric or physical relationship between different bones and the mesh vertex, so as to calculate the weighted sum of the influence of different bones on the mesh vertex, and then the correct position of the mesh vertex in the coordinate system can be obtained.

[0085] (2) Expression drive refers to the process of using multimedia data carrying facial expressions to drive the three-dimensional face model so that the three-dimensional face model can imitate the facial expressions in the multimedia data. The multimedia data may include video or image. When the multimedia data is a video, expression drive can be understood as the process of video driving the three-dimensional face model; in this case, the three-dimensional face model can be driven to imitate the facial expressions in each video frame according to the multiple video frames contained in the video, thereby generating a video. The playback effect of the generated video is: the three-dimensional face model presents each facial expression in sequence according to the time sequence of the facial expressions in the video. When the multimedia data is an image, expression drive can be understood as the process of image driving the three-dimensional face model; in this case, the three-dimensional face model can be directly driven to imitate the facial expression according to the facial expression in the image. It can be seen that the image-driven three-dimensional face model is actually a step in the video-driven three-dimensional face model; for the sake of convenience, the image-driven three-dimensional face model is used as an example to introduce the expression drive process. When the multimedia data is a video, the specific implementation process of each video frame drive can refer to the specific implementation process of the image-driven three-dimensional face model, which is specially explained here.

[0086] In actual applications, the two functions of face pinching and expression driving often exist independently in different applications; that is, the implementation link of the face pinching function and the implementation link of the expression driving function are different, which increases the cost of the virtual character customization process to a certain extent and reduces the user's customization experience. In order to improve the efficiency of customizing virtual characters by face pinching and expression driving and save customization costs, the data processing solution provided in the embodiment of the present application supports: on the one hand, the face in the face image uploaded by the user is pinched to obtain the face after the face pinching process (the face can be expressed as a skeleton skin at this time), and the facial features of the face after the face pinching process are similar to the facial features of the face in the face image, so that the face is pinched to obtain a face whose facial features meet the user's needs. On the other hand, based on the expression data to be imitated, the base virtual character is expression driven to obtain a base virtual character with a virtual facial expression described by the expression data; the base virtual character here is a pre-set virtual character, and the embodiment of the present application does not limit the facial features of the base virtual character. In this way, the face after the face-pinching process and the baseline virtual character with virtual facial expressions can be fused to generate a target virtual character corresponding to the face image; the target virtual character has both the facial features of the face after the face-pinching process (i.e., similar to the appearance of the face in the face image) and the virtual facial expressions described by the expression data (i.e., the facial expressions that you want to imitate).

[0087] It can be seen that the embodiment of the present application proposes a solution that combines face pinching and expression driving. This solution can not only pinch the face in the face image according to the user's face pinching needs to obtain a face after pinching whose facial features match the facial features of the face in the face image, but also make the face after pinching present virtual facial expressions through expression driving. By integrating face pinching and expression driving into one solution, the problem of virtual character customization separation caused by the independence of face pinching link and expression driving link is effectively solved; for users, it can be used out of the box, reducing the cost of virtual character customization.

[0088] The data processing scheme proposed in the embodiment of the present application can be executed by a computer device; the embodiment of the present application does not limit the type and quantity of the computer device. Specifically, the computer device may include a terminal and / or a server; in other words, the data processing scheme proposed in the embodiment of the present application can be executed by a terminal, by a server, or by a terminal and a server together. Among them:

[0089] The terminal may include, but is not limited to: smart phones (such as smart phones with different operating systems, such as smart phones with Android operating systems, and smart phones with Internet working operating systems (IOS), etc., to achieve cross-platform), tablet computers, personal computers, portable personal computers, smart TVs, vehicle-mounted devices, head-mounted devices, smart TVs or smart homes and other smart devices. The embodiments of the present application do not limit the types of terminals, which are explained here. The server is a background server corresponding to the terminal, which is used to interact with the terminal, specifically to interact with the audio and video platform deployed in the terminal, so as to provide computing and application service support for the audio and video platform deployed in the terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Conte nt Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0090] Based on the above introduction to the terminal and the server, in order to facilitate understanding of the data processing scheme involved in the embodiment of the present application, Figure 2 The following is a schematic diagram of the architecture of the data processing system corresponding to the data processing solution when the computer equipment includes terminals and servers. Figure 2 As shown, the user can send the facial image that he wants to get by pinching his face to the server 202 in the form of a facial image through the terminal 201 he holds; similarly, the facial expression that he wants to present can be sent to the server 202 in the form of expression data (such as the expression data is a video or an image). Then, after receiving the facial image and expression data sent by the terminal 201, the server 202 can perform face pinching processing on the face in the face image, and perform expression driving on the reference virtual character based on the expression data; and merge the face pinching effect after the face pinching processing and the expression effect after the expression driving, and obtain the target virtual character whose facial features are similar to those of the face in the face image after the face pinching processing, and whose facial expression is the virtual facial expression described by the expression data. Finally, the server 202 can return the relevant data of the target virtual character to the terminal 201, so that the terminal 201 renders and displays the target virtual character after the face pinching and expression driving on the display screen of the terminal based on the relevant data.

[0091] It should be understood that the above briefly introduces the process of the data processing solution by taking the computer device as a terminal and a server as an example; however, when the computer device is a terminal or a server, the idea of ​​the computer device executing the data processing solution is similar to the process described above, except that the execution subject is different, which will not be repeated here.

[0092] The data processing solution provided in the embodiment of the present application can be applied to any application scenario with face pinching and expression driving requirements; the application scenarios may include but are not limited to at least one of the following: game scenarios, audio and video scenarios, and conversation scenarios, etc.

[0093] Optionally, the application scenario is a game scenario, in which game players often have image settings for virtual characters in the game scenario; in this way, game players can only provide facial images and expression data (which can also be provided by game applications), and the game application can automatically implement personalized customization of virtual characters for game players using the data processing scheme provided by the embodiment of the present application; not only personalized customization of facial features of virtual characters is implemented, but also personalized customization of expressions is implemented, enriching the character image customization function of game players. In addition, when the application scenario is a game scenario, the data processing scheme provided by the embodiment of the present application can be integrated into the game application running in the computer device, and specifically can be integrated into the game engine deployed in the game application in the form of a plug-in; in this way, the plug-in can be called by the game engine to implement the data processing scheme. Among them, the plug-in is a collection of code and data that developers can enable or disable project by project in the editor; the plug-in can add the gameplay functions required at runtime, including but not limited to modifying the built-in engine functions (or adding new functions), creating new file types, using new menus, using toolbar commands and sub-mode extension editors and other functions; by introducing plug-ins in the game engine, functional extensions for the game engine can be implemented. The embodiments of the present application do not limit the specific type of game engine deployed in the game application. The game engine may include but is not limited to: unity engine, cocoa2d engine, unreal engine or unreal engine, etc.

[0094] Optionally, the application scenario is an audio and video scenario, such as an audio and video scenario being a live broadcast scenario. In a live broadcast scenario, the host only needs to provide facial images and expression data (which can be provided by the host or automatically collected by a camera) to achieve personalized customization of the facial features and facial expressions of the virtual characters in the live broadcast scene; it not only enriches the live broadcast function, but also effectively protects the host's personal privacy to a certain extent. Specifically, the data processing solution provided in the embodiment of the present application can be integrated into an application with a live broadcast function (such as integration using an SDK (Software Development Kit)); in this way, when the host uses the application for live broadcast, he can quickly call the data processing solution to achieve the corresponding functions of face pinching and expression driving.

[0095] Optionally, the application scenario is a conversation scenario, such as a video chat scenario (such as a single chat or a group chat). In the video chat scenario, a conversation participant can upload a face image, so that a virtual character with similar facial features can be pinched for the conversation participant based on the facial features of the face in the face image. In addition, the computer device used by the conversation participant for video chat can also collect the facial expressions of the conversation participant in real time through a camera, so as to drive the virtual character to imitate the facial expressions of the conversation participant in real time through the collected facial expressions. Specifically, the data processing scheme provided in the embodiment of the present application can be integrated into an application with a video chat function, so that when the conversation participant uses the application for video chat, he only needs to upload a face image to realize face pinching and expression driving, which not only increases the fun of video chat, but also effectively protects the personal privacy of the conversation participant.

[0096] It is worth noting that the application scenarios to which the data processing solution provided in the embodiments of the present application is applicable are not limited to the above-mentioned ones; and as the application scenarios vary, the applications or platforms that carry the data processing solution may vary.

[0097] It should also be noted that the collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of the laws and regulations of relevant countries. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as when obtaining a facial image, it is necessary to obtain the permission or consent of the user holding the facial image, and the collection, use and processing of relevant data (such as the collection and release of the bullet screen posted by the object, etc.) need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0098] Based on the data processing scheme described above, the embodiment of the present application proposes a more detailed data processing method. The data processing method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0099] Figure 3 A flow chart of a data processing method provided by an exemplary embodiment of the present application is shown; the data processing method may be executed by a computer device in the aforementioned system, such as a terminal and / or a server; the data processing method may include but is not limited to steps S301-S303:

[0100] S301: Acquire a face image, and perform face pinching processing on the face in the face image.

[0101] A face image is an image containing a face; the face can be a real face in the real world or a cartoon face in anime. In a specific implementation, when a user has a face-pinching requirement, a face image containing an expected face can be uploaded; that is, the face appearance of the face in the face image is the face appearance of the target virtual character that the user wants to obtain by pinching the face.

[0102] Among them, the source of the facial image uploaded by the user may include but is not limited to: obtained from the local storage space or cloud storage space held by the user; or, a frame of video captured from the video; or, downloaded from the network; or, sent by other devices; and so on. The embodiment of the present application does not limit the source of the facial image, which is conducive to the user uploading any facial image that he wants to pinch to the computer device, enriching the selection path of the facial image. For example, the source of the facial image uploaded by the user is sent by other devices; under this implementation method, a communication connection has been established between the terminal held by the user and the other device, so the terminal can directly receive the facial image from the other device. Among them, the embodiment of the present application does not limit the communication connection method established between the terminal and the other devices, including but not limited to: near field communication method (technology for short-distance contactless data exchange; such as near field communication may include but is not limited to: near field communication (NFC), Bluetooth communication or wifi communication, etc.), data communication (such as email, conversation, SMS, etc.) and wired connection communication.

[0103] As described above, 3D face pinching can be divided into automatic face pinching and manual face pinching according to the face pinching method. For users, automatic face pinching is similar to a black box. Users only need to provide the face image to be pinched, and the automatic face pinching algorithm (or program) can automatically pinch the face according to the facial features of the face in the face image, and obtain a three-dimensional face model whose facial features match (i.e. are similar to) the facial features of the face in the face image. On the contrary, manual face pinching requires users to manually adjust the face pinching parameters of each face part in the three-dimensional face model according to their own face pinching needs, so that the facial features of the three-dimensional face model meet the face pinching needs.

[0104] The face pinching process mentioned in the embodiments of the present application may include automatic face pinching and / or manual face pinching; in other words, it supports automatic face pinching only for the face in the face image, or manual face pinching only for the face in the face image, or supports automatic face pinching and manual face pinching for the face in the face image. Considering that in actual applications, if the face in the face image is first automatically pinched to obtain a three-dimensional face model whose facial features roughly match the facial features in the face image, and then the user manually pinches the face on the three-dimensional face model; compared with only automatically pinching the face image, the accuracy of face pinching can be effectively improved; compared with only manually pinching the face image, the user's face pinching workload can be reduced. Therefore, the embodiments of the present application will subsequently explain this solution by taking the face pinching process including automatic face pinching and manual face pinching as an example; of course, the situation where the face pinching process only includes automatic face pinching or manual face pinching is also applicable to this solution, which is specially explained here.

[0105] Among them, the exemplary process of face pinching processing of the face in the face image by automatic face pinching and manual face pinching can be seen in Figure 4 ;like Figure 4 As shown, the face pinching process may include but is not limited to:

[0106] (1) Obtain an automatic face pinching algorithm (or automatic face pinching program), and use the automatic face pinching algorithm to automatically pinch the face in the face image to obtain the face after automatic pinching. The face after automatic pinching is a three-dimensional face model. For the convenience of description, the three-dimensional face model after automatic pinching can be called a first skeleton skin. The first skeleton skin is a 3D face model, which is specifically composed of a skeleton and a skin, and the facial features of the face shown by the skin match the facial features of the face in the face image. Among them, the facial features can be features used to distinguish different facial appearances, including but not limited to: the shape of the face part (such as single eyelids or double eyelids), the position of the face part, and the color of the face part (such as whether the eyebrows are brown or black, etc.).

[0107] Among them, the three-dimensional face model obtained by automatic face pinching can include a non-parametric model and a parametric model. The face pinching algorithm that supports face pinching to obtain a non-parametric model directly deforms the mesh vertices (which can be called Mesh points) of the face to make it tend to the expected face. The face pinching algorithm that supports face pinching to obtain a parametric model can directly estimate the face pinching parameters of the three-dimensional face model; specifically, it supports expressing the three-dimensional geometric structure of the face as a combination of a high-dimensional basis vector and an average face, and the coefficient combination corresponding to different basis vectors can control the face shape, facial features and expression of the three-dimensional face model. One parametric model is a morphable face model (3D Morphable FaceModel, 3DMM), which is a general three-dimensional face model that supports the use of a fixed number of mesh vertices to represent the face; its core idea is that the face can be matched one by one in three-dimensional space, and can be obtained by weighted linear addition of orthogonal bases of multiple other faces.

[0108] It should be noted that there are many automatic face pinching algorithms, and the present application embodiment does not limit which specific algorithm is used to implement automatic face pinching for human faces in face images. The following takes the automatic face pinching algorithm based on the batch sample library as an example to give an exemplary face pinching process of automatic face pinching. The schematic diagram of the face pinching process can be found in Figure 5a ① Use the face in the face image to perform face matching and face pinching in the sample library to match and obtain a three-dimensional face model. The sample library may contain multiple three-dimensional face models with different facial features, or the sample library may contain face pinching parameters for different face parts, etc. The embodiment of the present application does not limit the data format of the data contained in the sample library. In detail, there may be multiple ways to use the face in the face image to perform face matching and face pinching in the sample library. Optionally, face recognition features are used for matching and face pinching; specifically, image recognition or face recognition and other technologies are used to perform face recognition processing on the face in the face image to obtain face recognition features of the face, and a three-dimensional face model whose face recognition features are the same as the face recognition features of the face in the face image is matched from the sample library according to the face recognition features. Optionally, facial key points are used for matching and face pinching; specifically, facial key points are identified from the face in the face image, and then a combination of facial key points in different three-dimensional face models in the sample library is used to generate a three-dimensional face model with facial key points roughly the same as the facial key points of the face in the face image. A schematic diagram of an exemplary facial key point in a face can be seen in Figure 5b ;like Figure 5b As shown, the facial key points in the face are mainly concentrated in the cheeks, mouth, eyes, nose and eyebrow areas of the face. By connecting the facial key points, the geometric features of the face can be roughly described.

[0109] ② Fine-tune the parameters of the 3D face model obtained by matching the face pinching. That is to say, after matching the 3D face model from the sample library, it also supports the face pinching parameters (such as bone parameters (such as bone position information), skinning information (such as mesh vertex position information) and the binding relationship between mesh vertices and bones) based on the existing 3D face model to further improve the face pinching result. In the specific parameter fine-tuning process, the actual physical meaning of the face pinching parameters will be considered to adjust the different face pinching parameters in a targeted manner. For example, if the actual physical meaning of the face pinching parameter 1 indicates the position and shape of the facial part "upper lip" in the human face; if the position and shape of the facial part "upper lip" indicated by the face pinching parameter 1 in the three-dimensional face model are significantly different from the position and shape of the facial part "upper lip" in the face image, then the facial key points of the facial part "upper lip" in the three-dimensional face model are fine-tuned with reference to the facial key points of the facial part "upper lip" in the face image (in essence, fine-tuning the facial key points is achieved by fine-tuning the face pinching parameter 1) to achieve a position and shape of the facial part "upper lip" in the three-dimensional face model that is similar to the position and shape of the facial part "upper lip" in the face image.

[0110] It can be seen that the above automatic face pinching algorithm obtains a parameterized model that conforms to the product style by pinching the face, and fine-tunes the face pinching parameters considering the actual physical meaning of the face pinching parameters to obtain a personalized three-dimensional face model. This method of first pinching the face to match to obtain a rough three-dimensional face model with similar facial features, and then fine-tuning it with reference to the physical meaning of the face pinching parameters improves the accuracy of automatic face pinching. In practical applications, the embodiments of the present application also support the use of other automatic face pinching algorithms to achieve automatic face pinching for faces in face images, and the specific implementation process of the automatic face pinching algorithm is not limited.

[0111] It should be understood that facial parts cannot be infinitely adjusted, such as the facial part "upper lip" cannot be infinitely adjusted upward or downward; therefore, the embodiment of the present application supports setting a corresponding adjustment control for each facial part in the face, so as to limit the degree of face pinching of the facial part through the adjustment control. Among them, the adjustment control is used to adjust the position of the bones associated with the corresponding facial part, that is, the adjustment control is used as animation data that can generate and drive the bones to move; specifically, the user adjusts the position of the bones associated with the corresponding facial part through the adjustment control, thereby adjusting the position and shape of the facial part through the adjustment control. In addition, the adjustment control corresponding to the facial part is a control with an adjustment range; the adjustment range of the adjustment control is pre-set, and is used to limit the adjustment limit of the corresponding facial part in the face, so as to ensure the rationality and naturalness of the adjusted facial part and improve the aesthetics of the facial part. As described above, the adjustment control adjusts the position of the bones associated with the facial parts, so the adjustment range of the adjustment control has a mapping relationship with the position transformation information of the bones; that is, the adjustment value of the adjustment control each time can be mapped to the position transformation information of the corresponding bones, so that the position change information of the corresponding bones can be directly determined based on the adjustment value of the adjustment control.

[0112] Based on this, when automatically pinching the face in the face image, it is necessary to ensure that the first adjustment value of the adjustment control corresponding to each facial part in the face belongs to the adjustment range of the adjustment control, so as to ensure that each facial part in the face will not be through the model after automatic face pinching (such as the phenomenon that two facial parts are completely or partially overlapped), etc. For the sake of convenience, the first adjustment value of the adjustment control corresponding to the target facial part in the face during automatic face pinching is represented as value1, and the adjustment range of the adjustment control is represented as [Min, Max] as an example, then the following conditions need to be guaranteed during automatic face pinching: Min≤value1≤Max; for example, Min=0, Max=100.

[0113] (2) Manually adjust the face pinching parameters of the first skeleton skin generated by automatic face pinching. As described above, the embodiment of the present application sets different adjustment controls for different facial parts, so that when the user wants to manually pinch one or more facial parts, the user can control the adjustment controls corresponding to the facial parts to adjust the position of one or more bones associated with (or controlled by) the facial parts, and then drive the position transformation of the mesh vertices bound to the bones through the bones after the position transformation. In terms of visual effect, the position change of the mesh vertex causes the position and / or shape of the corresponding facial part to change, thereby achieving the purpose of the user's adjustment of the facial part.

[0114] In a specific implementation, the second position change information of the bone associated with the target facial part is determined according to the second adjustment value generated by the manual face pinching adjustment control; then, based on the second position change information of the bone, the position of the bone in the first bone skin is adjusted, and the adjusted first bone skin is used as the second bone skin generated by combining manual face pinching and automatic face pinching.

[0115] Combine the following Figure 6 The implementation process of manual face pinching given above is introduced; Figure 6 As shown, the process diagram of the user manually pinching the face can be seen in Figure 6 ;like Figure 6 As shown, when the user has a need to manually pinch the face of the target face part in the first skeleton skin (such as any face part in the first skeleton skin), a trigger operation can be performed on the adjustment control corresponding to the target face part (such as when the adjustment control is a slider, the trigger operation here is a sliding operation or a dragging operation of dragging the slider). Then, if the computer device detects that there is a trigger operation for the adjustment control corresponding to the target face part in the first skeleton skin, the second adjustment value of the adjustment control is determined according to the trigger operation. Then, the computer device can determine the second adjustment value of the adjustment control according to the trigger operation, and determine the second position change information (or bone parameter) of the bone according to the mapping relationship between the adjustment range of the adjustment control and the position transformation information of the bone associated with the target face part. Finally, the position of the bone in the first skeleton skin is adjusted based on the second position change information of the bone, so that the adjusted first skeleton skin is used as the second skeleton skin.

[0116] It is not difficult to understand that when manually pinching the face on the basis of the first skeleton skin generated by automatic face pinching, if the adjustment for the first skeleton skin is simply superimposed on the first skeleton skin, it is likely to cause the face after manual face pinching to be unnatural; for example, the face after manual face pinching will appear to be through the model, causing the face to look awkward. Therefore, in order to overcome the problem of facial through the model or unnaturalness when automatic face pinching and manual face pinching are combined, the embodiment of the present application supports ensuring that the two face pinchings will not appear unnatural or through the model by stipulating that the sum of the adjustment values ​​of the adjustment controls during the two face pinchings must be less than the maximum limit in the adjustment range, ensuring the matching of the two face pinchings, and making the face after manual face pinching more natural and beautiful.

[0117] Based on this, the embodiment of the present application supports determining the first remaining adjustable range that allows the user to manually pinch the target face part during manual face pinching according to the first position change information of the bone associated with the target face part in the first bone skinning after automatic face pinching and before manual face pinching, so as to ensure that there will be no model penetration during manual face pinching. For the sake of convenience, taking the first adjustment value of the adjustment control corresponding to the target face part in the face during automatic face pinching as value1, the second adjustment value of the adjustment control corresponding to the target face part in the face during manual face pinching as value2, and the adjustment range of the adjustment control as [Min, Max] as an example, then when automatic face pinching and manual face pinching are combined, the following conditions need to be guaranteed: Min≤value2≤Max, and Min≤value1+value2≤Max.

[0118] For example, if Figure 7a As shown, it is assumed that the target face part in the first skeleton skin after automatic face pinching is the face outline 701, and the adjustment range of the adjustment control 702 of the face outline 701 is [0, 100]. When the face is pinched automatically, the first adjustment value value1 for the face outline 701 is 20, then the first remaining adjustable range of the face outline 701 when the face is pinched manually is determined to be [20, 100], that is, when the user manually pinches the face outline 701 through the adjustment control 702, the second adjustment value should fall within [20, 100] to avoid the penetration phenomenon caused by manual and automatic face pinching. It should be noted that Figure 7a The adjustment control 702 is described as a sliding bar (or simply a sliding bar) as an example; however, the expression form of the adjustment control 702 is not limited to the sliding bar form. Figure 7b As shown, the adjustment control 702 may also be a disk that can change the center angle of a circle, etc. In addition, the embodiment of the present application does not limit the shape and style of the sliding rod and the disk.

[0119] Combine the following Figure 8 The flowchart shown in the figure introduces the connection process between automatic face pinching and manual face pinching in more detail. Figure 8 As shown:

[0120] First, after obtaining the first skeleton skin after automatic face pinching, the first position information of the mesh vertex bound to the skeleton associated with the target face part (i.e., the coordinate information of the mesh vertex in the world coordinate system) can be determined from the first skeleton skin. Then, based on the binding relationship between the skeleton and the mesh vertex, the first position transformation information (or first position change information) of the corresponding skeleton is reversely calculated based on the first position information of the mesh vertex; the first position transformation information is used to characterize the position adjustment of the skeleton during the automatic face pinching process. Secondly, the first position change information of the skeleton is mapped to the adjustment range of the adjustment control corresponding to the target face part, and the first adjustment value value1 of the adjustment control can be obtained; the first adjustment value value1 indicates that the value1 value of the adjustment control has been adjusted during the automatic face pinching process. Finally, the maximum limit Max in the adjustment range of the adjustment control is subtracted from the first adjustment value value1 to obtain the first remaining adjustable range; the first remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during the manual face pinching process, that is, the range supported by the user during the manual face pinching process. That is to say, during the manual face pinching process, the user needs to ensure that the second adjustment data for the adjustment control belongs to (or falls within) the first remaining adjustable range to avoid the phenomenon of pinching the face out of range.

[0121] It can be seen that the embodiment of the present application comprehensively considers the relationship between the first adjustment value of the adjustment control during automatic face pinching and the second adjustment value of the adjustment control during manual face pinching, and the maximum limit of the adjustment range when manually pinching the face, so as to avoid problems such as penetration or unnaturalness caused by the direct addition of the bone position change information during two face pinchings, and effectively ensure the aesthetics of the face after face pinching.

[0122] It should be noted that ① the above Figure 8 Taking the manual face pinching for a target facial part in the face as an example, the combined process of manual face pinching and automatic face pinching is summarized; it should be understood that when manually pinching the face, multiple facial parts in the face can be pinched, so the above process needs to be executed for each facial part of the face to ensure the naturalness of each facial part after the face pinching process, thereby ensuring the authenticity and aesthetics of the entire face. ② The face of the embodiment of the present application can be rendered and displayed after each face pinching, so that the user can intuitively perceive the success of each face pinching, and it is convenient for the user to continue the subsequent face pinching. For example, after the automatic face pinching, the first skeleton skin can be rendered and displayed on the display screen of the terminal held by the user, and the virtual character corresponding to the first skeleton skin is displayed on the terminal screen; similarly, after the manual face pinching, the second skeleton skin can be rendered and displayed on the display screen of the terminal held by the user, and the virtual character corresponding to the second skeleton skin is displayed on the terminal screen.

[0123] S302: Acquire expression data, and perform expression driving on the reference virtual character based on the expression data to obtain a reference virtual character with a virtual facial expression.

[0124] Expression data is data used to describe virtual facial expressions; the expression data can be an image, or any video frame in a video that contains a virtual facial expression. For example, if a user wants to set the facial expression of a face after face pinching processing to a virtual facial expression, the user needs to provide expression data containing the virtual facial expression, so as to drive the face after face pinching processing to present the virtual facial expression based on the expression data. Among them, the embodiment of the present application does not limit the expression type of the virtual facial expression contained in the expression data, including but not limited to: smiling, pursed lips, laughing, squinting, crying, sadness, and single eye blinking, etc. In addition, the embodiment of the present application does not limit the human face presenting the virtual facial expression in the expression data; that is, the embodiment of the present application only focuses on the virtual facial expression presented by the human face in the expression data, and does not focus on the facial features of the human face.

[0125] As described above, facial parts cannot be infinitely adjusted. In order to overcome the model penetration phenomenon caused by the direct combination of face pinching and expression driving; before executing expression driving based on expression data, the embodiment of the present application also needs to determine the adjustment value range that can be driven during expression driving, so as to perform expression driving within the drivable adjustment data range, and ensure that the expression driving after face pinching will not cause the face model penetration phenomenon. For the convenience of explanation, the first adjustment value of the adjustment control corresponding to the target facial part in the face during automatic face pinching is represented as value1, the second adjustment value of the adjustment control corresponding to the target facial part in the face during manual face pinching is represented as value2, the second adjustment value of the adjustment control corresponding to the target facial part in the face during expression driving is represented as value3, and the adjustment range of the adjustment control is represented as [Min, Max] as an example, then when automatic face pinching, manual face pinching and expression driving are combined, the following conditions need to be guaranteed: Min≤value3≤Max, and Min≤value1+value2+value3≤Max.

[0126] The process of determining the adjustment data range that can be driven when driving by expression can be found in Fig. 9 ;like Fig. 9As shown: First, after the face pinching process, obtain the first adjustment value value1 and the second adjustment value value2 of the adjustment control corresponding to the target facial part; the target facial part is any facial part in the face after the face pinching process. Then, add the first adjustment value value1 and the second adjustment value value2 of the adjustment control to obtain the addition result value1+value2. Finally, subtract the maximum limit Max in the adjustment range of the adjustment control from the addition result to obtain the second remaining adjustable range Max-(value1+value2); the second remaining adjustable range Max-(value1+value2) is used to indicate: during the expression driving process, the remaining adjustable range of the adjustment control. In this way, during the expression driving, it is necessary to ensure that the third adjustment value of the adjustment control corresponding to the target facial part after the expression driving falls into the second remaining adjustable range to ensure the naturalness of the face generated by the combination of face pinching and expression driving.

[0127] Based on this, during expression driving, it is specifically supported to drive the base virtual character with expression based on the expression data and the second remaining adjustable range, so as to obtain a base virtual character with virtual facial expressions. This ensures that the target virtual character after the face pinching and expression driving are combined will not have the phenomenon of model penetration, improves the naturalness and aesthetics of the target virtual character, and improves the user's role customization experience. Among them, the above-mentioned base virtual character can be a pre-set base or standard three-dimensional face model, and the embodiment of the present application does not limit the facial features of the base virtual character.

[0128] S303: Fusing the face after the face pinching process with the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image.

[0129] After obtaining the face pinching effect (i.e., the face after face pinching processing (such as the second skeleton skin generated by combining automatic face pinching and manual face pinching)) based on the aforementioned step S301, and obtaining the expression effect (i.e., the baseline virtual character with virtual facial expressions) based on the aforementioned step S302, the embodiment of the present application supports fusing the face pinching effect and the expression effect to generate a target virtual character corresponding to the face image. The target virtual character has the facial features of the face after face pinching processing and has the virtual facial expression described by the expression data, that is, the target virtual character has both the facial expression of the face in the face image that the user wants to pinch, and the virtual facial expression described by the expression data that the user wants to present.

[0130] Among them, when the face after the face-pinching process and the base virtual character with virtual facial expressions are fused, the fusion of the face-pinching effect and the expression effect is mainly achieved by replacing the base virtual character with the virtual facial expression with the face after the face-pinching process. Specifically, the face after the face-pinching process is the second skeleton skin; at this time, the base virtual character with virtual facial expressions is replaced with the second skeleton skin; then, the second skeleton skin with virtual facial expressions after replacement is used as the third skeleton skin, and the third skeleton skin has the facial features of the face after the face-pinching process and also has the virtual facial expression described by the expression data; finally, the third skeleton skin is rendered and displayed to generate the target virtual character corresponding to the face image.

[0131] In summary, the embodiment of the present application proposes a solution that combines face pinching and expression driving, which simultaneously realizes automatic face pinching, manual face pinching and expression driving, avoids interruptions caused by independent execution of each function, reduces the workload of users, and realizes the automatic execution of face pinching and expression driving. In addition, by calculating the remaining adjustable range of the next stage after each stage (such as the automatic face pinching stage and the means face pinching stage) is completed, the naturalness of the target virtual character generated after the combination of each stage can be ensured, and the penetration phenomenon caused by two face pinching and expression driving can be avoided. In addition, the automatic face pinching, manual face pinching and expression driving are integrated into a solution, which is convenient for users to use out of the box and reduces the cost of virtual character customization.

[0132] See also Fig.10 , Fig.10 A flow chart of another data processing method provided by an exemplary embodiment of the present application is shown; the data processing method may be executed by a computer device in the aforementioned system, such as a terminal and / or a server; the data processing method may include but is not limited to steps S1001-S1006:

[0133] S1001: Acquire a face image, and perform face pinching processing on the face in the face image.

[0134] It should be noted that the specific implementation process shown in step S1001 can refer to the aforementioned Figure 3 The relevant description of the specific implementation process shown in step S301 in the illustrated embodiment is not repeated here.

[0135] S1002: Obtain expression data.

[0136] As described above, the expression data can be an independent image or a video frame captured from a video frame. Among them, the method of obtaining the expression data may include but is not limited to: shooting through a camera deployed in a computer device, or obtaining from an external device. Among them: ① The expression data can be shot by a camera deployed in a computer device. Specifically, an upper-level unified interface is deployed in the computer device (specifically, an Unreal Engine with a data processing method deployed in the computer device), and the upper-level unified interface encapsulates the camera and microphone call related paths of different underlying platform systems (such as Android operating system or IOS operating system); in this way, the system permission can be initiated through the upper-level unified interface, and the camera and microphone logic of the underlying platform system can be directly called to realize the shooting of the physical environment in which the user is located; for the user, the bottom layer is not perceived, and only the upper-level unified interface needs to be simply called to complete the permission acquisition and arouse the relevant hardware to receive the data captured by the local camera (such as images or videos). Among them, the camera involved in the embodiment of the present application can be a monocular camera or a binocular camera; a monocular camera has only one camera, and the face image taken by the monocular camera is a two-dimensional image; compared with a monocular camera, a binocular camera includes two cameras, and the pixel space position is obtained by the distance between the two cameras. ② The expression data can be obtained from an external device; the embodiment of the present application does not limit the device type of the external device, such as the external device can be a server that stores audio and video, so that the user can download the expression data from the server through the terminal he holds.

[0137] It should be noted that, ① in addition to the expression data being uploaded by the user when the user has an expression-driven demand as described above, the expression data can also be cached in a computer device; with this implementation, the user only needs to provide a face image without providing expression data, and can also realize the facial expression of the face after face pinching processing driven by expression data.

[0138] ② The embodiment of the present application takes the expression data as a single image or a video frame as an example to introduce the entire process of combining automatic face pinching, manual face pinching and expression drive. However, it is understandable that the number of expression data can also be multiple; if the user wants to be able to drive the facial expression of the face after face pinching in real time according to the virtual face described by each video frame in the video, the data processing method provided by the embodiment of the present application can be executed for each video frame in the video. For example, Fig.11aAs shown, assuming that the video contains video frame 1, video frame 2 and video frame 3 played sequentially; and video frame 1 describes virtual facial expression 1 "tilted head smile", video frame 2 describes virtual facial expression 2 "blinking smile" and video frame 3 describes virtual facial expression 3 "open mouth surprise"; then expression driving can be performed for video frame 1, video frame 2 and video frame 3 respectively, to obtain a target virtual character with virtual facial expression 1 corresponding to video frame 1, a target virtual character with virtual facial expression 2 corresponding to video frame 2, and a target virtual character with virtual facial expression 3 corresponding to video frame 3. Then, the second skeleton skin after face pinching processing (such as automatic face pinching and manual face pinching) for the face in the face image can be fused with the reference virtual character with virtual facial expression 1 to obtain the target virtual character with virtual facial expression 1, fused with the reference virtual character with virtual facial expression 2 to obtain the target virtual character with virtual facial expression 2, and fused with the reference virtual character with virtual facial expression 3 to obtain the target virtual character with virtual facial expression 3. It can be seen that after the base virtual character driven by the expression of each video frame in the video is fused with the second skeleton skin after the face pinching process, a target virtual character showing three different virtual facial expressions can be obtained. In this way, the target virtual character can be driven to perform virtual facial expression 1, virtual facial expression 2 and virtual facial expression 3 in sequence according to the playback order between the video frames in the video.

[0139] S1003: Decode the expression data to obtain decoded expression data.

[0140] S1004: Convert the decoded expression data into a data format to obtain converted expression data.

[0141] In steps S1003-S1004, after acquiring the expression data, the computer device may perform algorithm detection on the expression data. When the expression data is a video, the algorithm detection here is video algorithm detection. An exemplary process of algorithm detection can be found in Fig.11b The process may include: preprocessing of expression data, 3D face reconstruction and expression coefficient determination. The preprocessing of expression data can make the data format of the preprocessed expression data conform to the data format required by the subsequent algorithm detection; 3D face reconstruction and expression coefficient determination are mainly implemented to determine the expression coefficient of the virtual facial expression described by the expression data.

[0142] The specific implementation method of the preprocessing of expression data is introduced below, wherein: the expression data is decoded to obtain decoded expression data. In addition, considering that the sources of expression data are not the same, the data format of expression data is not unified. For example, some platforms use yuv (y is the brightness component, u and v are the chrominance components, U is the blue projection, and V is the red projection) as the encoding method, while some platforms use RGB (Red, Green and Blue) as the encoding method; and the data format required by the expression driving algorithm is unified; therefore, after obtaining the decoded expression data, it is necessary to convert the data format of the expression data so that the data format of the converted expression data meets the data format requirements of the expression driving algorithm. For example, assuming that the data format required by the expression driving algorithm is RGB format, then when the data format of the received expression data is non-RGB (such as yuv, MP4 or WAV, etc.) format, the data format of the expression data needs to be converted into RGB format. The exemplary process of the above data format conversion can be found in Fig.12 ;like Fig.12 As shown, the data format of the expression data received by the computer device is not uniform, so the data format conversion module in the data preprocessing module can be used to convert the data format of the expression data to obtain expression data whose converted data format meets the expression driving algorithm.

[0143] It is worth noting that the traditional data format conversion performance is poor; for example, when converting the YUV format into the RGB format, the traditional data format conversion usually accesses the data source at the CPU (Central Processing Unit) logic layer, reads pixel by pixel, and inputs the source encoding format pixels through the conversion logic function to achieve data format conversion. This data format conversion process will block the current logical thread; only when the data format conversion is completed can the current logical thread be restored to execute subsequent logic. In order to achieve fast data format conversion and avoid interrupting the current logical thread task, if the logical thread is occupied during the data format conversion process, the embodiment of the present application supports obtaining an accelerated conversion algorithm, and uses the accelerated conversion algorithm to perform data format conversion on the decoded expression data to obtain the converted expression data. This accelerated conversion algorithm can be implemented by an acceleration module (such as Fig.12 As shown in the figure, the acceleration module can be used as a submodule of the data preprocessing module to help the data format conversion module improve the conversion performance. Two exemplary acceleration conversion algorithms provided in the embodiments of the present application are given below, where:

[0144] In one implementation, the accelerated conversion algorithm includes executing data format conversion by starting a target sub-thread. In this case, the specific process of using the accelerated conversion algorithm to convert the decoded expression data into a data format may include but is not limited to: when starting the data format conversion, starting the target sub-thread, which runs independently; then, while keeping the logical thread running, using the target sub-thread to convert the expression data into a data format; finally, after the target sub-thread completes the data format conversion, notifying the logical thread to obtain the converted expression data. It can be seen that by starting the target sub-thread to realize data format conversion, the data format conversion can be accelerated by asynchronous threads, the rate of data format conversion can be increased, and the main thread (i.e., logical thread) task can be ensured not to be interrupted.

[0145] In other implementations, the accelerated conversion algorithm includes performing data format conversion through a graphics processor. The specific process of using the accelerated conversion algorithm to convert the decoded expression data into a data format may include but is not limited to: calling the shader in the graphics processor to convert the expression data into a data format to obtain the converted expression data; and returning the converted expression data to the central processing unit CPU. In detail, each platform (such as the Android operating system or the IOS operating system) calls the drawing function through the supported graphics interface to input the expression data into the graphics processing unit GPU (Graphics Processing Unit); then, write the shader code to implement the data format conversion for the expression data in the graphics processing unit GPU; finally, by reading the graphics processing unit GPU texture, the converted expression data in the graphics processing unit GPU is returned to the central processing unit GPU. In this way, the data conversion efficiency is greatly increased by virtue of the advantages of batch parallel computing and high computing speed of the graphics processing unit GPU.

[0146] S1005: Based on the converted expression data and the second remaining adjustable range, the reference virtual character is expression-driven to obtain a reference virtual character with a virtual facial expression.

[0147] Among them, the second remaining adjustable range is used to indicate the remaining adjustable range of the adjustment control corresponding to the target facial part during the expression driving process; the determination process of the second remaining adjustable range can refer to the relevant description of the specific implementation process shown in the aforementioned step S302, which will not be repeated here. During the expression driving process, in order to ensure that the face obtained by the expression driving and the face pinching processing will not be penetrated, it is necessary to ensure that the position change information of the bones associated with the target facial part is mapped to the third adjustment value when the adjustment range is adjusted during the expression driving, and falls into the second remaining adjustable range.

[0148] The specific implementation process of driving the expression of the reference virtual character based on the converted expression data and the second remaining adjustable range to obtain the reference virtual character with virtual facial expressions can be found in Fig.13 ;like Fig.13 As shown:

[0149] First, one or more preset expression poses are obtained; each preset expression pose presents a facial expression through a base virtual character. One or more preset expression poses are pre-set and stored; specifically, a method for making a deformation animation can be used to generate preset expression poses (or animation poses, where the information of mesh vertices in each animation pose is different) corresponding to various expressions for the base virtual character. Fig.14 As shown, based on the rich expression information, a deformation animation can be formulated to design a deformable mesh for the facial expressions indicated by different expression information, and then the mesh vertices in the deformable mesh corresponding to each facial expression are bound to the reference virtual character to obtain preset expression postures that present different facial expressions.

[0150] Then, according to the virtual facial expression described by the converted expression data and the second remaining adjustable range, the expression weight (or simply referred to as weight, coefficient) corresponding to each preset expression posture in one or more preset expression postures is determined. In other words, it supports analyzing the contribution weight of each preset expression posture to the virtual facial expression described by the expression data; if the weight of preset expression posture 1 is 0.5, and the weight of preset expression posture 2 is 0.1, the weighted processing of preset expression posture 1 and preset expression posture 2 can obtain a virtual facial expression. Of course, when determining the weight of each preset expression posture, it is also necessary to ensure that the third adjustment value associated with the virtual facial expression weighted based on the weight falls within the second remaining adjustable range to ensure the naturalness of the face after expression drive. Specifically, after the weights of each detected preset expression posture are input into the animation system, the positions of each mesh vertex when the benchmark virtual character presents the virtual facial expression can be calculated according to the weight of each preset expression posture, thereby realizing the expression drive effect.

[0151] Finally, based on the expression weight corresponding to each preset expression posture, one or more preset expression postures are weighted to obtain a reference virtual character with a virtual facial expression. Specifically, the expression weight corresponding to each preset expression posture is input into the virtual human character animation system, and the expression drive is realized through the virtual human character animation system. Among them, the third adjustment value of the adjustment control corresponding to the target face part in the reference virtual character with virtual facial expression belongs to the second remaining adjustable range, so as to ensure that the face after expression drive and face pinching will not appear through the model. Among them, the calculation formula 1 for weighted processing of the preset expression posture to determine the position of the grid vertex in the reference virtual character after driving is: the weight of the grid vertex of the reference virtual character + ∑ (the grid vertex of the reference virtual character-the grid vertex of the preset expression posture i) * the weight of the grid vertex of the preset expression posture i; the preset expression posture i in the calculation formula 1 is the i-th preset expression posture among one or more preset expression postures.

[0152] S1006: Fusing the face after the face pinching process with the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image.

[0153] Based on the above-mentioned related description, it can be known that the essence of the fusion processing of the face after face pinching and the benchmark virtual character with virtual facial expressions is to replace the benchmark virtual character with virtual facial expressions with the face after face pinching; then, the replaced face with virtual facial expressions is used as the third skeleton skin, and the third skeleton skin is rendered to generate the target virtual character corresponding to the face image. When the replacement process is reflected by the above-mentioned calculation formula 1, the replacement process can be described as: using the second skeleton skin generated by automatic face pinching and manual face pinching to replace the benchmark virtual character with virtual facial expressions in the calculation formula, the final effect of the fusion of face pinching effect and expression effect is obtained; at this time, the above calculation formula 1 can be changed to calculation formula 2, which is: mesh vertices in the second skeleton skin of face pinching + (baseline virtual character mesh vertices-preset expression posture i mesh vertices) * weights of preset expression posture i mesh vertices.

[0154] In summary, the embodiment of the present application supports face pinching processing of a face in a face image provided by a user, so as to obtain a face with a face pinching effect by face pinching processing; then, it also supports expression driving of a reference virtual character based on expression data describing a virtual facial expression, so that the reference virtual character can present the virtual facial expression. Furthermore, after obtaining a face after face pinching processing and a reference virtual character with a virtual facial expression, the face after face pinching processing and the reference virtual character with a virtual facial expression can be fused to generate a target virtual character corresponding to the face image; at this time, the target virtual character has the facial features of the face after face pinching processing and has a virtual facial expression. It can be seen that the embodiment of the present application proposes a solution that combines face pinching and expression driving. This solution can not only pinch the face in the face image according to the user's face pinching needs, and obtain the face after pinching whose facial features match the facial features of the face in the face image, but also make the face after pinching present virtual facial expressions through expression driving; by integrating face pinching and expression driving into one solution, the problem of virtual character customization separation caused by the independence of face pinching link and expression driving link is effectively solved; for users, it can be used out of the box, reducing the cost of virtual character customization.

[0155] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0156] Fig.15 A schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application is shown; the data processing device can be used to execute Figure 3 or Fig.10 Some or all of the steps in the method embodiment shown. Fig.15 , the data processing device comprises the following units:

[0157] The acquisition unit 1501 is used to acquire a face image, and the processing unit 1502 is used to perform face pinching processing on the face in the face image;

[0158] The acquisition unit 1501 is further used to acquire expression data, and the processing unit 1502 is further used to perform expression driving on the reference virtual character based on the expression data to obtain a reference virtual character with a virtual facial expression; the expression data is used to describe the virtual facial expression;

[0159] The processing unit 1502 is also used to fuse the face after the face-pinching processing and the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image; the target virtual character has the facial features of the face after the face-pinching processing and has virtual facial expressions.

[0160] In one implementation, the processing unit 1502, when performing face pinching processing on a face in a face image, is specifically configured to:

[0161] An automatic face pinching algorithm is obtained, and the face in the face image is automatically pinched using the automatic face pinching algorithm to generate a first skeleton skin; the skeleton skin includes skeletons and skins, and the skin is composed of mesh vertices and mesh edges; the mesh vertices are bound to the skeletons, so that the positions of the mesh vertices can be changed by controlling the position changes of the skeletons; each face part in the skeleton skin is associated with one or more bones; one face part corresponds to one adjustment control, and the adjustment control is used to adjust the position of the bones associated with the face part;

[0162] If there is a trigger operation for the adjustment control corresponding to the target face part in the first skeleton skin, then determining the second adjustment value of the adjustment control according to the trigger operation; the target face part is any face part in the first skeleton skin;

[0163] Determining second position change information of a bone associated with the target facial part according to a second adjustment value of the adjustment control;

[0164] Based on the second position change information of the bone, the position of the bone in the first skeleton skin is adjusted, and the adjusted first skeleton skin is used as the second skeleton skin.

[0165] In one implementation, the adjustment range of the adjustment control has a mapping relationship with the position change information of the bone; the processing unit 1502 is further used to:

[0166] Determine first position information of mesh vertices bound to the bones associated with the target face part from the first bone skin, and calculate first position change information of the bones according to the first position information of the mesh vertices;

[0167] Mapping the first position change information to the adjustment range of the adjustment control to obtain a first adjustment value of the adjustment control;

[0168] Subtracting the maximum limit value in the adjustment range of the adjustment control from the first adjustment value to obtain a first remaining adjustable range; the first remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during manual face pinching;

[0169] The second adjustment value of the adjustment control belongs to the first remaining adjustable range.

[0170] In one implementation, the processing unit 1502 is further configured to:

[0171] Obtaining a first adjustment value and a second adjustment value of an adjustment control corresponding to a target facial part; the target facial part is any facial part of a face after the face pinching process;

[0172] Adding the first adjustment value and the second adjustment value of the adjustment control to obtain an addition result;

[0173] The maximum limit value in the adjustment range of the adjustment control is subtracted from the addition result to obtain a second remaining adjustable range; the second remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during the expression driving process.

[0174] In one implementation, the processing unit 1502 is used to perform expression driving on the reference virtual character based on the expression data to obtain the reference virtual character with a virtual facial expression, specifically for:

[0175] Decoding the expression data to obtain decoded expression data;

[0176] Performing data format conversion on the decoded expression data to obtain converted expression data;

[0177] Based on the converted expression data and the second remaining adjustable range, the reference virtual character is expression-driven to obtain a reference virtual character with a virtual facial expression.

[0178] In one implementation, if the logical thread is occupied during the data format conversion process, the processing unit 1502 is used to perform data format conversion on the decoded expression data, and when the converted expression data is obtained, it is used to:

[0179] Get the accelerated conversion algorithm;

[0180] The decoded expression data is converted into a data format using an accelerated conversion algorithm to obtain converted expression data.

[0181] In one implementation, the accelerated conversion algorithm includes executing data format conversion by starting a target child thread; the processing unit 1502 is used to perform data format conversion on the decoded expression data using the accelerated conversion algorithm, and when the converted expression data is obtained, it is specifically used to:

[0182] When starting data format conversion, start the target sub-thread;

[0183] Use the target sub-thread to convert the data format of the expression data;

[0184] After the target child thread completes the data format conversion, it notifies the logic thread to obtain the converted expression data.

[0185] In one implementation, the accelerated conversion algorithm includes performing data format conversion by a graphics processor; the processing unit 1502 is used to perform data format conversion on the decoded expression data using the accelerated conversion algorithm, and when the converted expression data is obtained, it is specifically used to:

[0186] Calling the shader in the graphics processor to convert the data format of the expression data to obtain the converted expression data;

[0187] The converted expression data is sent back to the central processing unit.

[0188] In one implementation, the processing unit 1502 is used to drive the reference virtual character for expression based on the converted expression data and the second remaining adjustable range to obtain the reference virtual character with a virtual facial expression, specifically for:

[0189] Obtain one or more preset expression poses; each preset expression pose presents a facial expression through a reference virtual character;

[0190] Determine an expression weight corresponding to each preset expression posture in one or more preset expression postures according to the virtual facial expression described by the converted expression data and the second remaining adjustable range;

[0191] Based on the expression weight corresponding to each preset expression posture, one or more preset expression postures are weighted to obtain a reference virtual character with a virtual facial expression;

[0192] Among them, the third adjustment value of the adjustment control corresponding to the target facial part in the benchmark virtual character with virtual facial expression belongs to the second remaining adjustable range.

[0193] In one implementation, the face after the face-pinching process is skinned by the second skeleton; the processing unit 1502 is used to fuse the face after the face-pinching process with the reference virtual character with virtual facial expressions to generate a target virtual character corresponding to the face image, specifically for:

[0194] Replace the baseline virtual character with virtual facial expressions with the second skeleton skin;

[0195] The second skeleton skin with virtual facial expression after replacement is used as the third skeleton skin;

[0196] The third skeleton skin is rendered to generate a target virtual character corresponding to the face image.

[0197] In one implementation, the application scenario to which the data processing method is applicable includes at least one of the following: a game scenario, an audio and video scenario, and a conversation scenario; when the application scenario is a game scenario, the data processing method is integrated into an Unreal Engine that provides the game scenario;

[0198] The expression data is any video frame containing a virtual facial expression in the video; the video is obtained by shooting with a camera or received from an external device; and the number of expression data is multiple.

[0199] According to one embodiment of the present application, Fig.15 The various units in the data processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the data processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 3 and Fig.10 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.15 The data processing device shown in the embodiment of the present application is implemented by the data processing device. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0200] The embodiment of the present application proposes a solution that combines face pinching and expression driving. This solution can not only pinch the face in the face image according to the user's face pinching needs to obtain a face after pinching whose facial features match the facial features of the face in the face image, but also make the face after pinching present virtual facial expressions through expression driving. By integrating face pinching and expression driving into one solution, the problem of virtual character customization separation caused by the independence of the face pinching link and the expression driving link is effectively solved. For users, it can be used out of the box to achieve the coherent execution of face pinching and expression driving, thereby reducing the cost of virtual character customization.

[0201] Fig.16 FIG. 1 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Fig.16, the computer device includes a processor 1601, a communication interface 1602 and a computer-readable storage medium 1603. The processor 1601, the communication interface 1602 and the computer-readable storage medium 1603 may be connected via a bus or other means. The communication interface 1602 is used to receive and send data. The computer-readable storage medium 1603 may be stored in the memory of the computer device, the computer-readable storage medium 1603 is used to store computer programs, and the processor 1601 is used to execute the computer programs stored in the computer-readable storage medium 1603. The processor 1601 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is specifically suitable for loading and executing one or more computer programs to implement corresponding method flows or corresponding functions.

[0202] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more computer programs suitable for being loaded and executed by the processor 1601 are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0203] In one embodiment, the computer device may be the terminal or server mentioned in the aforementioned embodiment; one or more computer programs are stored in the computer-readable storage medium; the processor 1601 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the aforementioned data processing method embodiment; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded and executed by the processor 1601 to execute the steps of each embodiment of the present application; wherein, the steps of each embodiment of the present application may refer to the relevant descriptions of the aforementioned embodiments, and will not be repeated here.

[0204] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the data processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0205] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0206] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a computer program (one or more). When a computer program is loaded and executed on a computer device, the computer program executes the above-mentioned process or function of the embodiment of the present application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer program can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer program can be transmitted from a website site, a computer device, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer device, server or data center. The computer-readable storage medium can be any available medium that a computer device can access or a data storage device such as a server or a data center that contains one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (Solid State Disk, SSD)) and the like.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: Acquire a face image, and perform face pinching processing on a face in the face image; Acquire expression data, and perform expression driving on a reference virtual character based on the expression data to obtain the reference virtual character with a virtual facial expression; the expression data is used to describe the virtual facial expression; The human face after the face-pinching process and the reference virtual character with the virtual facial expression are fused to generate a target virtual character corresponding to the human face image; the target virtual character has the facial features of the human face after the face-pinching process and has the virtual facial expression.

2. The method according to claim 1, characterized in that The pinching of the face in the face image comprises: An automatic face pinching algorithm is obtained, and the automatic face pinching algorithm is used to automatically pinch the face in the face image to generate a first skeleton skin; the skeleton skin includes a skeleton and a skin, and the skin is composed of mesh vertices and mesh edges; the mesh vertices are bound to the skeleton, so that controlling the position change of the skeleton can change the position of the mesh vertices; each face part in the skeleton skin is associated with one or more bones; one face part corresponds to one adjustment control, and the adjustment control is used to adjust the position of the bone associated with the face part; If there is a trigger operation for the adjustment control corresponding to the target face part in the first skeleton skin, determining a second adjustment value of the adjustment control according to the trigger operation; the target face part is any face part in the first skeleton skin; Determining second position change information of the bone associated with the target facial part according to a second adjustment value of the adjustment control; Based on the second position change information of the bone, the position of the bone in the first skeleton skin is adjusted, and the adjusted first skeleton skin is used as the second skeleton skin.

3. The method according to claim 2, characterized in that The adjustment range of the adjustment control has a mapping relationship with the position change information of the skeleton; after the automatic face pinching algorithm is used to automatically pinch the face in the face image to generate the first skeleton skin, the method further includes: Determining first position information of a mesh vertex bound to a bone associated with the target facial part from the first bone skin, and calculating first position change information of the bone according to the first position information of the mesh vertex; Mapping the first position change information to the adjustment range of the adjustment control to obtain a first adjustment value of the adjustment control; Subtracting the maximum limit value in the adjustment range of the adjustment control from the first adjustment value to obtain a first remaining adjustable range; the first remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during manual face pinching; Among them, the second adjustment value of the adjustment control belongs to the first remaining adjustable range.

4. The method according to claim 1, characterized in that Before the expression-driven processing of the reference virtual character based on the expression data to obtain the reference virtual character with a virtual facial expression is performed, the method further comprises: Obtaining a first adjustment value and a second adjustment value of an adjustment control corresponding to a target facial part; the target facial part is any facial part of the face after the face pinching process; Adding the first adjustment value and the second adjustment value of the adjustment control to obtain an addition result; The maximum limit value in the adjustment range of the adjustment control is subtracted from the addition result to obtain a second remaining adjustable range; the second remaining adjustable range is used to indicate: the remaining adjustable range of the adjustment control during the expression driving process.

5. The method according to claim 4, characterized in that The step of driving the reference virtual character with expression based on the expression data to obtain the reference virtual character with virtual facial expression includes: Decoding the expression data to obtain decoded expression data; Performing data format conversion on the decoded expression data to obtain the converted expression data; Based on the converted expression data and the second remaining adjustable range, the reference virtual character is expression-driven to obtain the reference virtual character with a virtual facial expression.

6. The method according to claim 5, characterized in that If the logical thread is occupied during the data format conversion process, the decoded expression data is converted into a data format to obtain the converted expression data, including: Get the accelerated conversion algorithm; The accelerated conversion algorithm is used to convert the decoded expression data into a data format to obtain the converted expression data.

7. The method according to claim 6, characterized in that The accelerated conversion algorithm includes executing data format conversion by starting a target subthread; The step of converting the decoded expression data into a data format by using the accelerated conversion algorithm to obtain the converted expression data comprises: When starting data format conversion, start the target sub-thread; Using the target sub-thread to convert the expression data into a data format; After the target sub-thread completes the data format conversion, the logic thread is notified to obtain the converted expression data.

8. The method according to claim 6, characterized in that The accelerated conversion algorithm includes performing data format conversion by a graphics processor; the accelerated conversion algorithm is used to convert the decoded expression data into a data format to obtain the converted expression data, including: Calling a shader in the graphics processor to perform data format conversion on the expression data to obtain the converted expression data; The converted expression data is sent back to the central processing unit.

9. The method according to claim 5, characterized in that The step of driving the reference virtual character with expression based on the converted expression data and the second remaining adjustable range to obtain the reference virtual character with virtual facial expression includes: Obtain one or more preset expression poses; each preset expression pose presents a facial expression through a reference virtual character; Determining an expression weight corresponding to each of the one or more preset expression postures according to the virtual facial expression described by the converted expression data and the second remaining adjustable range; Based on the expression weight corresponding to each preset expression posture, weighted processing is performed on the one or more preset expression postures to obtain the reference virtual character with virtual facial expression; Among them, the third adjustment value of the adjustment control corresponding to the target facial part in the reference virtual character with virtual facial expression belongs to the second remaining adjustable range.

10. The method according to claim 1, characterized in that The face after the face pinching process is skinned with a second skeleton; the face after the face pinching process is fused with the reference virtual character having the virtual facial expression to generate a target virtual character corresponding to the face image, including: Replacing the reference virtual character having the virtual facial expression with the second skeleton skin; Using the second skeleton skin having the virtual facial expression after replacement as the third skeleton skin; The third skeleton skin is rendered to generate a target virtual character corresponding to the facial image.

11. The method according to claim 1, characterized in that The method is applicable to at least one of the following application scenarios: a game scenario, an audio and video scenario, and a conversation scenario; when the application scenario is a game scenario, the method is integrated into an Unreal Engine that provides the game scenario; The expression data is any video frame containing virtual facial expressions in the video; the video is captured by a camera or received from an external device; and the number of the expression data is multiple.

12. A data processing device, characterized in that: include: An acquisition unit, used for acquiring a face image, and a processing unit, used for performing face pinching processing on a face in the face image; The acquisition unit is further used to acquire expression data, and the processing unit is further used to perform expression driving on the reference virtual character based on the expression data to obtain the reference virtual character with virtual facial expressions; the expression data is used to describe the virtual facial expressions; The processing unit is further used to fuse the face after the face-pinching processing and the reference virtual character with the virtual facial expression to generate a target virtual character corresponding to the face image; the target virtual character has the facial features of the face after the face-pinching processing and has the virtual facial expression.

13. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the data processing method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the data processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.