Virtual expression driving method and device, electronic device, and storage medium

Abstract

The present application provides a virtual expression driving method and device, electronic device, and computer-readable storage medium, the method comprising: obtaining a basic face model and target animation data corresponding to the basic face model; obtaining the expression offset extreme values ​​corresponding to the basic face model in multiple expression dimensions; writing the expression offset extreme values ​​into a to-be-filled map to obtain a deformation map; obtaining the weights corresponding to the basic face model in each frame of the target animation in the multiple expression dimensions from the target animation data to obtain a weight combination; rendering the expression animation of the basic face model according to the deformation map and the weight combination. The present application solution solves the problem that expressions cannot be rendered normally when there are too many expression dimensions due to the browser's restrictions on the number of attribute variables.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a virtual expression driving method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In order to make the facial expressions of 3D virtual people more abundant, multiple dimensional deformation animations can be generated for 3D virtual people. Deformation animations of different dimensions can correspond to different parts of the face. Deformation animations are called Blendshape or morph target in engines such as Maya, 3DMax and Unity. Each dimensional deformation animation represents the extreme state of a part during deformation. The expression animation generation algorithm can generate a weight for each dimensional deformation animation, and the weight is between 0 and 1. Based on each dimensional deformation animation and the corresponding weight, the expression state of the virtual person in the current image can be determined.

[0003] In the related art, after uploading the deformation animation of each dimension to the vertex shader in the form of WebGL attributes, the deformation animation of each dimension can be weighted and summed in the vertex shader to determine the coordinates of each vertex of the virtual human face, and then render the expression animation. However, due to the browser's limit on the number of attributes, only 8-dimensional deformation animation can be processed at most. In this case, when there are multiple-dimensional deformation animations, it cannot be processed normally. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a virtual expression driving method and device, an electronic device, and a computer-readable storage medium, which are used to solve the problem that the browser cannot normally drive high-dimensional animation.

[0005] On the one hand, the present application provides a virtual expression driving method, comprising:

[0006] Acquire a basic face model and target animation data corresponding to the basic face model;

[0007] Obtaining the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions;

[0008] Writing the expression offset extreme value into the to-be-filled map to obtain a deformation map;

[0009] Obtaining the weights of the basic face model in each frame of the target animation in the expression image from the target animation data corresponding to the multiple expression dimensions, and obtaining a weight combination;

[0010] The expression animation of the basic face model is rendered according to the deformation map and the weight combination.

[0011] In one embodiment, before writing the expression offset extreme value into the to-be-filled map to obtain the deformation map, the method further includes:

[0012] Determining the total amount of the expression deviation extreme values ​​according to the number of vertices of the basic face model and the number of the expression dimensions;

[0013] The width and height of the to-be-filled texture are determined according to the total amount of the expression offset extreme values, and the to-be-filled texture is generated according to the width and the height.

[0014] In one embodiment, determining the width and height of the to-be-filled texture according to the total amount of the expression offset extreme value includes:

[0015] Determine a minimum parameter in a parameter set that is greater than the total amount of the expression deviation extreme value, and use the minimum parameter in the parameter set as a target parameter; wherein all parameters in the parameter set are powers of two;

[0016] Determine a first number corresponding to the height and a second number corresponding to the width based on the target parameter;

[0017] The height is two to the first power and the width is two to the second power.

[0018] In one embodiment, writing the expression offset extreme value into the to-be-filled map to obtain the deformation map includes:

[0019] The expression offset extreme values ​​of multiple vertices in the basic face model in multiple expression dimensions are written into the to-be-filled map in sequence to obtain the deformation map.

[0020] In one embodiment, the method further comprises:

[0021] Acquire vertex attribute data of the basic face model; wherein the vertex attribute data includes basic coordinates of each vertex of the basic face model;

[0022] The rendering to obtain the expression animation of the basic face model according to the deformation map and the weight combination includes:

[0023] Determine the target coordinates of each vertex of the basic face model in each frame of the expression image according to the expression offset extreme value of each vertex in the deformation map, the weight combination and the basic coordinates of each vertex in the vertex attribute data;

[0024] According to the target coordinates of each vertex of the basic face model in each frame of the expression image, the expression animation of the basic face model is rendered.

[0025] In one embodiment, the vertex attribute data includes the vertex order corresponding to each vertex of the basic face model;

[0026] Before determining the target coordinates of each vertex of the basic face model in each frame of the expression image according to the expression offset extreme value of each vertex in the deformation map, the weight combination and the basic coordinates of each vertex in the vertex attribute data, the method includes:

[0027] Determine the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model;

[0028] According to the texture coordinates corresponding to each vertex in the deformation map, the table offset extreme values ​​corresponding to each vertex in the multiple expression dimensions are obtained from the deformation map.

[0029] In one embodiment, determining the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model includes:

[0030] Determine the rectangular coordinates of each vertex in the deformation map corresponding to each expression dimension according to the vertex order corresponding to each vertex of the basic face model and the width and height of the deformation map;

[0031] According to the rectangular coordinates corresponding to each vertex and each expression dimension, the texture coordinates corresponding to each vertex and each expression dimension are determined.

[0032] On the other hand, the present application also provides a virtual expression driving device, comprising:

[0033] A first acquisition module, used to acquire a basic face model and target animation data corresponding to the basic face model;

[0034] A second acquisition module is used to obtain the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions;

[0035] A writing module, used for writing the expression offset extreme value into the to-be-filled map to obtain a deformation map;

[0036] A determination module, used for obtaining the weights of the basic face model in each frame of the target animation in the expression image in the target animation respectively corresponding to the multiple expression dimensions, and obtaining a weight combination;

[0037] A rendering module is used to render the expression animation of the basic face model according to the deformation map and the weight combination.

[0038] Furthermore, the present application also provides an electronic device, the electronic device comprising:

[0039] Processor; wherein the processor includes a central processing unit and a graphics processing unit;

[0040] a memory for storing processor-executable instructions;

[0041] Wherein, the processor is configured to execute the above-mentioned virtual expression driving method.

[0042] In addition, the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned virtual expression driving method.

[0043] The present application scheme can generate a deformation map based on the expression offset extreme values ​​after obtaining the expression offset extreme values ​​corresponding to the basic face model in multiple expression dimensions; obtain the weights corresponding to the basic face model in multiple expression dimensions in each frame of the expression image from the target animation data to obtain the weight combination; render the expression animation of the basic face model according to the deformation map and the weight combination; after constructing the deformation map, transmit the deformation map in the form of a single uniform variable. When there are expression offset extreme values ​​in multiple expression dimensions, there is no need to transmit the expression offset extreme values ​​in each expression dimension separately, which solves the problem of not being able to render expressions normally due to the browser's limitation on the number of attribute variables. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0045] Figure 1 A schematic diagram of an application scenario of a virtual expression driving method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0047] Figure 3 A flowchart of a virtual expression driving method provided in one embodiment of the present application;

[0048] Figure 4 A schematic diagram of a flow chart of a method for determining the size of a texture to be filled provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a deformation map provided by an embodiment of the present application;

[0050] Figure 6 A schematic diagram of an expression animation driver provided in one embodiment of the present application;

[0051] Figure 7A schematic diagram of the architecture of an expression animation driver provided in one embodiment of the present application;

[0052] Figure 8 A block diagram of a virtual expression driving device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0054] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] Figure 1 Schematic diagram of application scenarios of the virtual expression driving method provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes a user terminal 20 and a server 30; the server 30 can be a server, a server cluster or a cloud computing center, which is used to generate a weight combination corresponding to each frame of the expression image based on an expression synthesis algorithm, and send the weight combination to the user terminal 20; the user terminal 20 can be an electronic device such as a mobile phone or a tablet computer, which is used to obtain the weight combination corresponding to each frame of the expression image from the server 30, and render the expression animation based on the weight combination and the local expression offset extreme values ​​in multiple dimensions.

[0056] like Figure 2 As shown, this embodiment provides an electronic device 1, including: a central processing unit (CPU) 11, a graphics processing unit (GPU) 12 and a memory 13. The CPU 11, the GPU 12 and the memory 13 are connected via a bus 10, and the memory 13 stores instructions that can be executed by the CPU 11 and the GPU 12, and the instructions are executed by the CPU 11 and the GPU 12, so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be the above-mentioned user terminal 20, which is used to execute the virtual expression driving method.

[0057] The memory 13 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0058] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the central processing unit 11 and the graphics processing unit 12 to complete the virtual expression driving method provided by the present application.

[0059] See also Figure 3 , is a flow chart of a virtual expression driving method provided by an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 - 350 .

[0060] Step 310: Obtain a basic face model and target animation data corresponding to the basic face model.

[0061] Step 320: Obtain the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions.

[0062] The basic face model is a face model of a 3D virtual human, which is a set of basic image objects composed of triangles, combined with textures and internal animations. The basic face model can be in GLTF (Graphics Language Transmission Format) format, FBX (FilmBoX) format, GLTF compression model format supporting KHR_mesh_quantization extension, or other formats.

[0063] The expression animation of the basic face model can be determined by the expression offset extreme values ​​of multiple expression dimensions, and the expression offset extreme value of each expression dimension can indicate the maximum deformation state of a certain part of the face. The expression offset extreme value of each expression dimension can be written in the basic face model in the form of a blendshape or a morph target. Exemplarily, there is a morph target of one dimension corresponding to the eyelid, and the morph target of this dimension can include the vertices of several triangular meshes corresponding to the eyelid in the basic face model, and the vertex coordinate offset when the eyelid is maximally deformed. The offset at this time can be called the expression offset extreme value.

[0064] The target animation data may be data for controlling the basic face model to display virtual expressions. Exemplarily, the target animation data may include a weight combination corresponding to each frame of expression image, and the weight combination includes a weight corresponding to each expression dimension.

[0065] The user terminal can read the model data from the memory and load it to obtain a basic face model.

[0066] The expression animation of the virtual human is composed of multiple consecutive frames of expression images. The user terminal can obtain the weight combination sent by the server frame by frame, or generate the weight combination of expression images frame by frame through a local expression synthesis algorithm (such as: NetEase Fuxi expression synthesis algorithm). The weight combination includes the weights of the deformation animation of each dimension. Exemplarily, there are a total of 50 expression dimensions with extreme values ​​of expression offset, and the weight combination includes 50 weights. After obtaining the weight combination, the user terminal obtains the target animation data.

[0067] The user terminal can obtain the expression offset extreme values ​​corresponding to each vertex in the basic face model in multiple expression dimensions from the Blendshape or morph target of the basic face model.

[0068] For the same vertex, the extreme value of expression offset in different expression dimensions may be different. For example, for a vertex of the lips in a face model, the extreme value of expression offset may be zero in the blendshape or morph target corresponding to the eyelid, while the extreme value of expression offset is not zero in the blendshape or morph target corresponding to the lips.

[0069] Step 330: Write the expression offset extreme value into the to-be-filled map to obtain a deformation map.

[0070] The to-be-filled map is a map with empty pixel values. After the user terminal writes multiple expression offset extreme values ​​as pixel values ​​of each pixel into the to-be-filled map, a deformation map can be obtained. At this time, the deformation map contains the expression offset extreme values ​​of each vertex of the basic face model in multiple expression dimensions.

[0071] Each expression offset extreme value can include the maximum offset of the vertex in the x, y, and z directions in three-dimensional space. The maximum offset of the three dimensions in the expression offset extreme value can be used as a deformation map. Figure 3 Therefore, after writing the extreme value of the expression offset into the map to be filled, a three-dimensional deformation map can be obtained.

[0072] Step 340: Obtain the weights corresponding to the basic face model in each frame of the target animation in multiple expression dimensions from the target animation data to obtain a weight combination.

[0073] The user terminal can determine the weight combination corresponding to each frame of the expression image from the target animation data, and the weight combination includes the weights of the basic face model in multiple expression dimensions. Exemplarily, the target animation data includes the weight combination corresponding to 200 frames of expression images, and the basic face model has 50 expression dimensions, then the user terminal can obtain 200 weight combinations from the target animation data, each weight combination includes 50 weights.

[0074] Step 350: Rendering to obtain the facial expression animation of the basic face model according to the deformation map and the weight combination.

[0075] After obtaining the weight combination corresponding to the deformation map and each frame of the expression image, the weight combination and the deformation map can be uploaded to the webGL (Web Graphics Library) rendering pipeline as uniform variables, so as to render the expression animation corresponding to the basic face model.

[0076] Through the above measures, after constructing the deformation map with the expression offset extreme values ​​corresponding to each vertex of the basic face model in multiple expression dimensions, the deformation map is transmitted in the form of a single uniform variable. When there are expression offset extreme values ​​in multiple expression dimensions, the problem of not being able to render expressions normally due to the browser's limitation on the number of attribute variables can be solved.

[0077] In one embodiment, before executing step 330 , the user terminal may construct a map to be filled.

[0078] The user terminal can determine the total amount of expression offset extreme values ​​according to the number of vertices of the basic face model and the number of expression dimensions. Each vertex has a corresponding expression offset extreme value in each expression dimension. Therefore, the total amount of expression offset extreme values ​​can be obtained by multiplying the number of vertices by the number of expression dimensions. For example, if the basic face model has 1,000 vertices and 50 expression dimensions, 50,000 expression offset extreme values ​​can be obtained.

[0079] After determining the total amount of expression offset extreme values, the user terminal can determine the width and height of the to-be-filled texture that meets the WelGL requirement based on the total amount. Among them, WelGL limits the width and height of the image to be a power of 2.

[0080] After the width and the height are determined, a map to be filled may be generated, in which the pixel value of each pixel is empty.

[0081] In one embodiment, when the user terminal determines the width and height of the to-be-filled sticker according to the total amount of the expression offset extreme value, refer to Figure 4 , is a flow chart of a method for determining the size of a to-be-filled texture provided in an embodiment of the present application, such as Figure 4 As shown, the method may include steps 410 to 430.

[0082] Step 410: Determine the minimum parameter in the parameter set that is greater than the total amount of the expression deviation extreme value, and use the minimum parameter in the parameter set as the target parameter; wherein the parameters in the parameter set are all powers of two.

[0083] The parameter set may include 2 to the first power, 2 to the second power, 2 to the third power, and so on, 2 to the nth power. The user terminal may start from the first power of 2 and gradually compare the power of 2 with the total amount of the extreme value of the expression offset. When the power of 2 is small, the user terminal may increase the number of exponentiations by one and re-compare the power of 2 with the total amount of the extreme value of the expression offset. The user terminal may repeat the above process until the first target parameter greater than the total amount of the offset is determined. The target parameter is the minimum parameter in the parameter set that is greater than the total amount of the extreme value of the expression offset, thereby determining the target number of exponentiations.

[0084] Step 420: Determine a first number corresponding to the height and a second number corresponding to the width based on the target parameters.

[0085] Step 430: Use a first power of two as the height and a second power of two as the width.

[0086] Among them, the first number and the second number are exponents of 2. The first number is used to calculate the height, and the second number is used to calculate the width.

[0087] The user terminal can determine whether the target number is an odd number or an even number. On the one hand, if the target number is an even number, the target number is divided by two to obtain a first number corresponding to the height and a second number corresponding to the width, and the first number is the same as the second number. On the other hand, if the target number is an odd number, the target number is divided by two and rounded down to obtain a first number corresponding to the height. The first number plus one can obtain a second number corresponding to the width.

[0088] The user terminal may determine the first power of two as the height and the second power of two as the width.

[0089] For example, the target parameter is 2 to the power of 36, and the target number is 36; in this case, the first number is 18, the second number is 18, and the height and width are both 2 to the power of 18. The target parameter is 2 to the power of 35, and the target number is 35; in this case, the first number is 17, the second number is 18, the height is 2 to the power of 17, and the width is 2 to the power of 18.

[0090] In one embodiment, after obtaining the map to be filled, when writing the expression offset extreme values ​​into the map to be filled, the user terminal can write the expression offset extreme values ​​of multiple vertices in the basic face model in multiple expression dimensions into the map to be filled in sequence to obtain a deformation map.

[0091] See also Figure 5 , is a schematic diagram of a deformation map provided by an embodiment of the present application, such as Figure 5 As shown, b1_v1 represents the expression offset extreme value of the first vertex in the first-dimensional deformation animation; b1_vn represents the expression offset extreme value of the nth vertex in the first-dimensional deformation animation; b50_vn represents the expression offset extreme value of the nth vertex in the 50th-dimensional deformation animation, and so on. After determining the width and height of the deformation map (to be filled in the map), the number of pixels in each row and the number of pixels in each column of the deformation map can be determined. The user terminal can fill the expression offset extreme values ​​of multiple vertices in the first expression dimension into each pixel as the pixel value in the order from left to right and from top to bottom according to the vertex sorting order. After the expression offset extreme values ​​of the first expression dimension are filled, the user terminal fills the expression offset extreme values ​​of multiple vertices in the second expression dimension into each pixel. Repeat the above process until all expression offset extreme values ​​of all vertices in multiple expression dimensions are filled.

[0092] When the total amount of expression offset extreme values ​​is different from the total amount of pixels in the deformation map, the CPU can set the pixel values ​​of the remaining pixels to zero after filling all the expression offset extreme values.

[0093] In one embodiment, before rendering the facial expression animation of the basic face model, the user terminal may obtain vertex attribute data of the basic face model. The vertex attribute data may include the basic coordinates of each vertex of the basic face model (the coordinates of each vertex when not offset), vertex sequence array, vertex texture coordinates, color, surface normal vector (usually from the surface outward), etc. It can be seen that each vertex has a set of vertex attribute data. The user terminal can extract the vertex attribute data of each vertex from the basic face model.

[0094] In this embodiment, when rendering expression animation according to the deformation map and weight combination, the user terminal can determine the target coordinates of each vertex of the basic face model in each frame of expression image according to the expression offset extreme value of each vertex in the deformation map, the weight combination and the basic coordinates of each vertex in the vertex attribute data. Here, the target coordinates are the actual coordinates of the vertex in a frame of expression image. When rendering the expression image through the WebGL rendering pipeline, it is first processed by the vertex shader. For multiple vertices in the basic face model, the user terminal can determine the expression offset of each vertex based on the expression offset extreme value of multiple expression dimensions corresponding to each vertex in the deformation map, and the weight corresponding to each expression dimension in the weight combination.

[0095] by Figure 5 For example, for the first vertex in the basic face model, the extreme values ​​of the expression offset of the vertex in 50 expression dimensions can be expressed as b1_v1, b2_v1, b3_v1, ..., b49_v1, b50_v1. The weight combination corresponding to a frame of expression image is w1, w2, w3, ..., w50. The user terminal can determine the expression offset of the vertex in the frame of expression image diff1 = (b1_v1*w1+b2_v1*w2+...+b50_v1*w50). After calculating each vertex, the expression offset of each vertex in the frame of expression image can be determined.

[0096] The user terminal performs processing based on the weight combination of the continuous multiple frames of expression images, and can determine the expression offsets of multiple vertices in the continuous multiple frames of expression images.

[0097] For example, the basic face model has n vertices and a total of 50 expression dimensions. After obtaining the weight combination corresponding to any frame of expression image, the expression offset of n vertices in the frame of expression image can be determined by matrix multiplication of the following formula (1):

[0098]

[0099] Among them, w1, w2, ..., w50 are the weights corresponding to 50 expression dimensions in a frame of expression image; bm_vn represents the extreme value of the expression offset of the nth vertex in the mth expression dimension; diffn represents the expression offset of the nth vertex in a frame of expression image.

[0100] After obtaining the expression offsets of the multiple vertices in a frame of expression image, the user terminal may determine the target coordinates of the multiple vertices in the frame of expression image according to the expression offsets of the multiple vertices and the basic coordinates.

[0101] For example, the face model has n vertices and there are 50 expression dimensions in total. After determining the expression offsets of the n vertices in a frame of expression image, the user terminal can determine the target coordinates of each vertex in the frame of expression image according to the following formula (2).

[0102] (v1,v2,…,v n )+(diff1,diff2,…,diff n )=(t1,t2,…,t n ) (2)

[0103] Among them, v1, v2, ..., vn are the basic coordinates of n vertices in the face model; diffn represents the expression offset of the nth vertex in a frame of expression image; t1, t2, ..., tn are the final target coordinates of n vertices in a frame of expression image.

[0104] The user terminal can render the expression animation of the basic face model according to the target coordinates of each vertex of the basic face model in each frame of the expression image. For any frame of the expression image, after determining the target coordinates of each vertex of the basic face model, the frame of the expression image can be rendered. By rendering multiple consecutive frames of expression images, an expression animation can be obtained.

[0105] In one embodiment, the vertex attribute data includes vertex order corresponding to each vertex of the basic face model. Before determining the target coordinates of each vertex of the basic face model in each frame of expression image, the user terminal can determine the expression offset extreme value corresponding to each vertex of the basic face model.

[0106] The user terminal can determine the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model. The texture coordinates are used to indicate the pixel corresponding to the vertex in the deformation map.

[0107] After determining the texture coordinates corresponding to each vertex, the user terminal can obtain the expression offset values ​​corresponding to each vertex in multiple expression temperatures from the deformation map according to the texture coordinates corresponding to each vertex in the deformation map. For any vertex, after obtaining the texture coordinates corresponding to any expression dimension, the user terminal can determine the pixel corresponding to the texture coordinate from the deformation map, and then sample to obtain the expression offset extreme value of the vertex in the expression dimension. After sampling each vertex in each expression dimension, the rendering pipeline of the user terminal can obtain the expression offset extreme values ​​of each vertex in multiple expression dimensions.

[0108] In one embodiment, when determining the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex, the user terminal may determine the rectangular coordinates corresponding to each expression dimension of each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model and the width and height of the deformation map. Here, the rectangular coordinates may be xy coordinates.

[0109] The user terminal can determine the horizontal coordinate in the rectangular coordinate using the following formula (3):

[0110] x0=(n*b+verticeIndex)%width (3)

[0111] The user terminal can determine the ordinate in the rectangular coordinates by the following formula (4):

[0112]

[0113] Among them, x0 represents the horizontal coordinate; y0 represents the vertical coordinate; n represents the total number of vertices in the basic face model; b is the serial number of the current expression dimension minus one, and b starts from 0; verticeIndex represents the sorting order of the vertex among all vertices, and verticeIndex starts from 0; width represents the width of the deformation map.

[0114] Exemplarily, there are 50-dimensional deformation animations in total. For any vertex, a total of 50 rectangular coordinates corresponding to the vertex and the 50 expression dimensions can be determined.

[0115] The user terminal may determine the texture coordinates corresponding to each vertex and each expression dimension according to the rectangular coordinates corresponding to each vertex and each expression dimension.

[0116] The deformation map is a two-dimensional image composed of pixels one by one, and one grid represents one pixel. The rectangular coordinate corresponding to any expression dimension indicates the pixel position corresponding to the vertex on the deformation map. The horizontal coordinate of the rectangular coordinate ranges from 0 to width, and the vertical coordinate ranges from 0 to height. The user terminal uses the texture2D() function to sample the texture through the WebGL shader language, and it is necessary to determine the position of the pixel in the texture space. The texture space is a rectangular area between (0,0) and (1,1), that is, the texture coordinates in the texture space can be obtained by normalizing the rectangular coordinates.

[0117] The horizontal coordinate of the texture coordinate can be determined by the following formula (5):

[0118]

[0119] The ordinate of the texture coordinate can be determined by the following formula (6):

[0120]

[0121] Among them, u0 represents the horizontal coordinate in the texture coordinate; v0 represents the vertical coordinate in the texture coordinate; x0 represents the horizontal coordinate in the rectangular coordinate; y0 represents the vertical coordinate in the rectangular coordinate; width represents the width of the deformation map; height represents the height of the deformation map.

[0122] For any vertex, the rectangular coordinates corresponding to the deformation animation of the vertex in each dimension can be converted into texture coordinates.

[0123] In one embodiment, both the CPU and GPU of the user terminal can participate in the virtual expression driving process. Figure 6 , is a schematic diagram of an expression animation driver provided by an embodiment of the present application, such as Figure 6 As shown, the CPU extracts vertex attribute data from the vertices of the basic face model and transmits it to the GPU as an attribute variable. The CPU transmits the weight combination of the deformation map and each frame of the expression image to the GPU as a uniform variable. The GPU can process the attribute variables and uniform variables through the vertex shader to determine the expression offset of each vertex in a frame of the expression image, thereby determining the target coordinates of each vertex in the frame of the expression image. After determining each target coordinate, the GPU obtains a frame of expression image through primitive assembly, rasterization, fragment shader, fragment-by-fragment operation, frame buffer and other processing, and renders expression animation based on multiple consecutive frames of expression images.

[0124] Through the above measures, the CPU can construct a deformation map based on the extreme values ​​of expression offset in multiple expression dimensions and upload it to the GPU. The GPU will perform high-frequency calculations of the expression image frame by frame, reducing the CPU's time complexity to O(n), greatly improving the driving efficiency of the expression animation.

[0125] In one embodiment, when transmitting the deformation map to the GPU, the CPU may set the first specified attribute and the second specified attribute for the deformation map, and transmit the deformation map with the first specified attribute and the second specified attribute set to the GPU.

[0126] Among them, the first specified attribute is gl.CLAMP_TO_EDGE corresponding to the texture coordinates, which is used to instruct the GPU to sample from the deformation map according to the texture coordinates (uv coordinates) to obtain the offset extreme value of the vertex. The second specified attribute is gl.NEAREST corresponding to the magnification filter and reduction filter of the deformation map, which is used to indicate that the offset extreme value is determined from the deformation map according to the neighbor filtering method. In other words, when the deformation map is enlarged or reduced, the pixel value of the pixel closest to it is selected for the newly added pixel.

[0127] See also Figure 7 , is a schematic diagram of the architecture of the expression animation driver provided in one embodiment of the present application, such as Figure 7 As shown in the figure, the CPU can process the model data preprocessing stage and the animation driving stage. In the model data processing stage, the CPU obtains attribute variables through vertex extraction when loading the model; the CPU can determine the deformation map based on the expression offset extreme value of each vertex in multiple expression dimensions, thereby obtaining uniform variables. The CPU can transfer attribute variables and uniform variables to the GPU. In the animation driving stage, the CPU can obtain the weight combination of each frame of the expression image through the expression animation generation algorithm, and transfer the weight combination to the GPU frame by frame.

[0128] During the GPU calculation stage, the deformation map can be sampled through the WebGL rendering pipeline, and the expression offset of each vertex in a frame of expression image can be determined by combining the weights, and the final target coordinates can be obtained by adding the base coordinates of the vertex, thereby realizing the expression image display.

[0129] Figure 8 is a block diagram of a virtual expression driving device according to an embodiment of the present invention, Figure 8 As shown, the device may include:

[0130] A first acquisition module 810 is used to acquire a basic face model and target animation data corresponding to the basic face model;

[0131] A second acquisition module 820 is used to obtain the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions;

[0132] A writing module 830, used for writing the expression offset extreme value into the to-be-filled map to obtain a deformation map;

[0133] A determination module 840 is used to obtain the weights of the basic face model in each frame of the target animation in the expression image in the target animation respectively corresponding to the multiple expression dimensions, and obtain a weight combination;

[0134] The rendering module 850 is used to render the expression animation of the basic face model according to the deformation map and the weight combination.

[0135] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned virtual expression driving method, and will not be repeated here.

[0136] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0138] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A virtual expression driving method, characterized in that: include: Acquire a basic face model and target animation data corresponding to the basic face model; Obtaining the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions; The expression deviation extreme value of each expression dimension indicates the maximum deformation state of a certain part of the face; Writing the expression offset extreme value into the to-be-filled map to obtain a deformation map; Obtaining the weights of the basic face model in each frame of the target animation in the expression image from the target animation data corresponding to the multiple expression dimensions, and obtaining a weight combination; According to the deformation map and the weight combination, rendering obtains the expression animation of the basic face model. Before writing the expression offset extreme value into the to-be-filled map to obtain the deformation map, the method further comprises: Determining the total amount of the expression deviation extreme values ​​according to the number of vertices of the basic face model and the number of the expression dimensions; Determining the width and height of the to-be-filled texture according to the total amount of the expression offset extreme values, and generating the to-be-filled texture according to the width and the height; The step of determining the width and height of the to-be-filled map according to the total amount of the expression offset extreme values ​​includes: Determine a minimum parameter in a parameter set that is greater than the total amount of the expression deviation extreme value, and use the minimum parameter in the parameter set as a target parameter; wherein all parameters in the parameter set are powers of two; Determine a first number corresponding to the height and a second number corresponding to the width based on the target parameter; The height is two to the first power and the width is two to the second power.

2. The method according to claim 1, characterized in that The step of writing the expression offset extreme value into the to-be-filled map to obtain the deformation map comprises: The expression offset extreme values ​​of multiple vertices in the basic face model in multiple expression dimensions are written into the to-be-filled map in sequence to obtain the deformation map.

3. The method according to claim 1, characterized in that The method further comprises: Acquire vertex attribute data of the basic face model; wherein the vertex attribute data includes basic coordinates of each vertex of the basic face model; The rendering to obtain the expression animation of the basic face model according to the deformation map and the weight combination includes: Determine the target coordinates of each vertex of the basic face model in each frame of the expression image according to the expression offset extreme value of each vertex in the deformation map, the weight combination and the basic coordinates of each vertex in the vertex attribute data; According to the target coordinates of each vertex of the basic face model in each frame of the expression image, the expression animation of the basic face model is rendered.

4. The method according to claim 3, characterized in that The vertex attribute data includes the vertex order corresponding to each vertex of the basic face model; Before determining the target coordinates of each vertex of the basic face model in each frame of the expression image according to the expression offset extreme value of each vertex in the deformation map, the weight combination and the basic coordinates of each vertex in the vertex attribute data, the method includes: Determine the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model; According to the texture coordinates corresponding to each vertex in the deformation map, the table offset extreme values ​​corresponding to each vertex in the multiple expression dimensions are obtained from the deformation map.

5. The method according to claim 4, characterized in that Determining the texture coordinates corresponding to each vertex in the deformation map according to the vertex order corresponding to each vertex of the basic face model includes: Determine the rectangular coordinates of each vertex in the deformation map corresponding to each expression dimension according to the vertex order corresponding to each vertex of the basic face model and the width and height of the deformation map; According to the rectangular coordinates corresponding to each vertex and each expression dimension, the texture coordinates corresponding to each vertex and each expression dimension are determined.

6. A virtual expression driving device, characterized in that: include: A first acquisition module, used to acquire a basic face model and target animation data corresponding to the basic face model; A second acquisition module is used to obtain the expression deviation extreme values ​​corresponding to the basic face model in multiple expression dimensions; the expression deviation extreme value of each expression dimension indicates the maximum deformation state of a certain part of the face; A writing module, used for writing the expression offset extreme value into the to-be-filled map to obtain a deformation map; A determination module, used for obtaining the weights of the basic face model in each frame of the target animation in the expression image in the target animation respectively corresponding to the multiple expression dimensions, and obtaining a weight combination; A rendering module, used for rendering an expression animation of the basic face model according to the deformation map and the weight combination; Before writing the expression offset extreme values ​​into the to-be-filled map to obtain the deformation map, determining the total amount of the expression offset extreme values ​​according to the number of vertices of the basic face model and the number of expression dimensions; Determining the width and height of the to-be-filled texture according to the total amount of the expression offset extreme values, and generating the to-be-filled texture according to the width and the height; The step of determining the width and height of the to-be-filled map according to the total amount of the expression offset extreme values ​​includes: Determine a minimum parameter in a parameter set that is greater than the total amount of the expression deviation extreme value, and use the minimum parameter in the parameter set as a target parameter; wherein all parameters in the parameter set are powers of two; Determine a first number corresponding to the height and a second number corresponding to the width based on the target parameter; The height is two to the first power and the width is two to the second power.

7. An electronic device, characterized in that: The electronic device comprises: Processor; wherein the processor includes a central processing unit and a graphics processing unit; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the virtual expression driving method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the virtual expression driving method described in any one of claims 1-5.

Images