Adaptive BlendShape Deformation Method and Virtual Reality Devices

Abstract

This application relates to the field of VR technology, providing an adaptive BlendShape deformation method and virtual reality device to improve the versatility of BlendShape data. Utilizing the many-to-one correspondence between the second index of the second vertex of the original 3D model in the rendering engine and the first index of the first vertex of the original 3D model, and the one-to-one correspondence between the initial offset of each BlendShape data point established through the first index and the first vertex, the BlendShape data is adaptively mapped to the second vertex. This allows the initial offset of each BlendShape data point to be applied to the corresponding second vertex, completing the model deformation process. Because BlendShape data can adaptively match the model deformation process of different rendering engines, the utilization rate of BlendShape data is effectively improved, reducing the workload of modeling.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides an adaptive BlendShape deformation method and a virtual reality device. Background Technology

[0002] In the field of 3D reconstruction, there are many methods for deforming 3D models, commonly including skeletal animation-based deformation methods, keyframe-based deformation methods, and BlendShape-based deformation methods. Among these, BlendShape-based deformation methods are often used for facial deformation of virtual characters.

[0003] The BlendShape-based deformation method derives new model vertex data from BlendShape data through a series of calculations, thereby deforming the original model. The BlendShape data and the vertices in the original model correspond one-to-one via indices. However, due to the different processing mechanisms of various rendering engines for 3D model data, during 3D reconstruction, importing a 3D model into a rendering engine may result in the engine automatically generating more vertices and scrambling the vertex indices. This leads to inconsistencies between the BlendShape data and the 3D model data in the rendering engine, resulting in incorrect model deformation.

[0004] Currently, the common practice is to customize the corresponding BlendShape data for the rendering engine to deform the model, but this reduces the utilization rate of BlendShape data and increases the workload of modelers. Summary of the Invention

[0005] This application provides an adaptive BlendShape deformation method and a virtual reality device to improve the versatility of BlendShape data.

[0006] On one hand, embodiments of this application provide an adaptive BlendShape deformation method, including:

[0007] Obtain the index correspondence relationship, wherein the index correspondence relationship is the correspondence between the first index of each first vertex and the second index of at least one second vertex corresponding to the first vertex, the first vertex is the vertex in the original 3D model before inputting into the rendering engine, and the second vertex is the vertex in the model to be processed after inputting into the rendering engine;

[0008] Obtain the pre-stored BlendShape dataset, where each BlendShape dataset contains the first index of each first vertex it controls and the initial offset of the corresponding first vertex;

[0009] Obtain the initial coordinates of each second vertex of the model to be processed in the rendering engine;

[0010] For each first vertex, the target offset of the first vertex is obtained based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data.

[0011] Based on the first index of each first vertex controlled by each BlendShape data and the index correspondence, determine the second index of the second vertex corresponding to the BlendShape data;

[0012] Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex. Combine this with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex.

[0013] Based on the target coordinates of each second vertex, the original 3D model is deformed and displayed.

[0014] On the other hand, embodiments of this application provide a virtual reality device, including a processor, a memory, and a display screen, wherein the display screen, the memory, and the processor are connected via a bus;

[0015] The memory includes a data storage unit and a program storage unit. The data storage unit stores the index correspondence, the BlendShape dataset, and the weights of each BlendShape data. The index correspondence is the correspondence between the first index of each first vertex and the second index of at least one second vertex corresponding to the first vertex. The first vertex is a vertex in the original 3D model before being input into the rendering engine, and the second vertex is a vertex in the model to be processed after being input into the rendering engine. Each BlendShape data includes the first index of each first vertex it controls and the initial offset of the corresponding first vertex.

[0016] The program storage unit stores a computer program, and the processor performs the following operations according to the computer program:

[0017] Obtain the index correspondence from the data storage unit, as well as the weights of the BlendShape dataset and each BlendShape data;

[0018] The initial coordinates of each second vertex in the model to be processed are obtained after the original 3D model is input into the rendering engine;

[0019] For each first vertex, the target offset of the first vertex is obtained based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data.

[0020] Based on the first index of each first vertex controlled by each BlendShape data and the index correspondence, determine the second index of the second vertex corresponding to the BlendShape data;

[0021] Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex, and combine them with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex.

[0022] Based on the target coordinates of each second vertex, the original 3D model is deformed, and the deformed target 3D model is displayed on the display screen.

[0023] Optionally, the processor obtains the target offset of the first vertex based on the initial offset contained in each BlendShape data controlling it and the preset weight of each BlendShape data. Specifically, the operation is as follows:

[0024] The preset weight of each BlendShape data is multiplied by the initial offset of the first vertex contained in the corresponding BlendShape data to obtain the sub-offset of the first vertex.

[0025] The sub-offsets of the first vertex determined by each BlendShape data are added together to obtain the target offset of the first vertex.

[0026] Optionally, the processor determines the index correspondence in the following way:

[0027] For each second vertex of the model to be processed in the rendering engine, the nearest first vertex in the original 3D model is determined to obtain the initial correspondence.

[0028] Traverse each first vertex in the original 3D model, and determine the second index of at least one second vertex corresponding to the first index of the currently traversed first vertex according to the initial correspondence, thereby obtaining the index correspondence.

[0029] Optionally, in the index correspondence, the first index of each first vertex corresponds to at least one second index of a second vertex, which constitutes a sub-relation of the index correspondence, and different sub-relations are separated by a first delimiter;

[0030] In each sub-relation, the first index of the first vertex and the second index of at least one second vertex are separated by a second delimiter, and the second indices of the at least one second vertex are separated by a third delimiter.

[0031] Optionally, when performing real-time dynamic deformation, the processor also performs:

[0032] Continuously acquire the preset weights of each BlendShape data multiple times;

[0033] Based on the preset weights of each BlendShape data obtained this time, and the initial offsets of the first vertices of the original 3D model contained in each BlendShape data, the original 3D model is re-deformed and displayed on the display screen.

[0034] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of an adaptive BlendShape deformation method provided in embodiments of this application.

[0035] In the adaptive BlendShape deformation method and virtual reality device provided in this application embodiment, during the model deformation process, after the original 3D model data is input into the rendering engine, the rendering engine processes it to obtain the second vertices contained in the model to be deformed. The second vertices have a many-to-one correspondence with the first vertices of the original 3D model. The initial offset of each BlendShape data is mapped one-to-one with the corresponding first vertex in the original 3D model through a first index. By obtaining the correspondence between the first index of the first vertex and the second index of at least one second vertex, the initial offset of the first vertex contained in each BlendShape data can be applied to the corresponding second vertex in the rendering engine. Thus, based on the preset weights of each BlendShape data and the initial coordinates of each second vertex, the target coordinates of the corresponding second fixed point are determined, completing the deformation of the original 3D model. Because the BlendShape data can adaptively correspond with the second vertices in the rendering engine, it ensures that the BlendShape data can adaptively match the model deformation process of different rendering engines, effectively improving the versatility of the BlendShape data, thereby increasing the utilization rate of the BlendShape data and reducing the workload of modeling.

[0036] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating the method for establishing index correspondence provided in this application embodiment;

[0039] Figure 2 A flowchart illustrating the method for processing the input original 3D model using the rendering engine provided in this embodiment of the application;

[0040] Figure 3 A schematic diagram illustrating the storage method of the index correspondence provided in the embodiments of this application;

[0041] Figure 4 This application provides an overall flowchart for establishing index correspondences in its embodiments.

[0042] Figure 5 A flowchart illustrating the adaptive BlendShape deformation method provided in this application embodiment;

[0043] Figure 6 A flowchart illustrating the method for calculating the target offset provided in this application embodiment;

[0044] Figure 7 The effect diagram of BlendShape deformation provided in the embodiments of this application;

[0045] Figure 8 A flowchart illustrating the overall method for adaptive BlendShape deformation provided in this application embodiment;

[0046] Figure 9 A flowchart illustrating the overall method for continuous adaptive BlendShape deformation provided in this application embodiment;

[0047] Figure 10 A structural diagram of a virtual reality device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0049] In 3D model reconstruction, traditional facial deformation methods are computationally expensive and have poor representation of facial micro-expressions. BlendShape, as a typical form of facial morphology data, uses each BlendShape data point to represent a facial shape. Multiple BlendShape data points are combined with different weights to present the final facial shape, offering advantages such as small data volume and high reconstruction accuracy.

[0050] When performing model deformation based on BlendShape, after the original 3D model data is imported into the rendering engine, different rendering engines will perform secondary processing on the 3D data according to their own design requirements. These different rendering engines often have different processing mechanisms. This results in the vertices in the original 3D model having their topological structure rebuilt, and the vertex indices being scrambled and significantly changed. Since the BlendShape data and the vertices in the original 3D model are linked one-to-one by index, this can cause inconsistencies in the correspondence between the BlendShape data and the vertices in the rendering engine. For universally formatted 3D data and BlendShape data, rendering engines with BlendShape deformation capabilities will automatically process the BlendShape data when vertex and index changes occur, ensuring that the model vertices in the rendering engine can be correctly deformed under the BlendShape deformation function. However, for many proprietary BlendShape data generated by algorithms, these often lack universality, leading to inconsistencies between the BlendShape data and the 3D model data in the rendering engine, resulting in incorrect model deformation and affecting the realism of the model.

[0051] Currently, the relevant technologies to address the above problems generally involve customizing BlendShape data for model deformation for rendering engines. However, this reduces the utilization rate of BlendShape data and increases the workload of modelers.

[0052] In view of this, embodiments of this application provide an adaptive BlendShape deformation method and a virtual reality device. This method establishes a correspondence between the first index of the first vertex in the original 3D model before inputting it into the rendering engine and the second index of the second vertex in the model to be processed after inputting it into the rendering engine. This ensures that the BlendShape data corresponds to the second vertex contained in the model to be processed in the rendering engine, thereby guaranteeing that the BlendShape data can adaptively match the model deformation process of different rendering engines. This effectively improves the versatility of the BlendShape data, thereby increasing the utilization rate of the BlendShape data and reducing the workload of modeling.

[0053] In this embodiment of the application, before performing BlendShape deformation, a preprocessing operation is performed to establish an index correspondence between the first index of the first vertex in the original 3D model and the second index of the second vertex in the model to be processed. For details of the implementation process, please refer to [link to implementation details]. Figure 1 It mainly includes the following steps:

[0054] S101: For each second vertex of the model to be processed in the rendering engine, determine the first vertex in the original 3D model that is closest to it, and obtain the initial correspondence.

[0055] In the embodiments of this application, the original 3D model, as the base model for deformation, is pre-established and stored locally. During the index correspondence establishment process, firstly, the original 3D model data is obtained locally, and the initial coordinates of the first vertex are saved in the first queue list1 according to the index order of the first vertex; then, the rendering engine is opened, and the original 3D model data is opened through the rendering engine. The processed model data to be processed is read, and the initial coordinates of each second vertex contained in the read model to be processed are saved in the second queue list2 according to the second index order of the second vertex.

[0056] When performing deformation based on BlendShape, the BlendShape data maintains consistency with the first vertex of the original 3D model before inputting it into the rendering engine. However, after importing the original 3D model data into the rendering engine, different rendering engines perform secondary processing based on their own design requirements. This results in a discrepancy between the number and index of the second vertices in the model to be processed and the number and index of the first vertices in the imported original 3D model. Because different rendering engines have different processing mechanisms, they obtain the second vertices in the model to be processed in different ways.

[0057] In an alternative implementation, using the Unity3D engine as an example, the process of obtaining the second vertex in the model to be processed is described below. Figure 2 It mainly includes the following steps:

[0058] S100_1: In the rendering engine, for each first vertex in the original 3D model, the first vertex is copied according to the number of normals or texture coordinates of the first vertex.

[0059] S100_2 uses at least one point obtained by copying as the second vertex corresponding to the first vertex, and assigns a normal or texture coordinate of the corresponding first vertex to each second vertex.

[0060] When the first vertex corresponds to one normal or one texture coordinate, the Unity3D engine directly uses that first vertex as a second vertex in the model to be processed. When the first vertex corresponds to multiple normals or multiple texture coordinates, the Unity3D engine copies the initial coordinates of the first vertex multiple times to obtain multiple second vertices corresponding to that first vertex, and assigns a normal or texture coordinate of the corresponding first vertex to each second vertex. Therefore, the number of first vertices in the original 3D model before being input into the rendering engine is less than the number of second vertices in the model to be processed after being imported into the rendering engine.

[0061] During the generation of the second vertex, although the initial coordinates of each second vertex are consistent with the corresponding first vertex, the second vertex will rebuild the topology, causing changes in the index, resulting in inconsistencies between the second index of the second vertex contained in the BlendShape data and the model to be processed.

[0062] Since the initial coordinates of the second vertex added in the rendering engine are the same as the initial coordinates of the first vertex in the original 3D model, there is a one-to-many correspondence between the first vertex in the original 3D model and the second vertex in the model to be processed in the rendering engine. The BlendShape data is consistent with the first vertex in the original 3D model. In order to synchronize the coordinates of the deformed BlendShape data to the second vertex of the model to be processed in the rendering engine, it is necessary to find the mapping relationship between the BlendShape data and the second vertex.

[0063] In one implementation, mapping can be performed using the minimum distance between vertices. However, since the number of second vertices is greater than the number of first vertices—that is, there is a one-to-many correspondence between the first vertices in the original 3D model and the second vertices in the model to be processed in the rendering engine—determining the corresponding second vertex in the rendering engine for the first vertex in the original 3D model using the minimum distance is cumbersome and the calculation process is complex. To improve the efficiency of preprocessing, this application's embodiments employ reverse thinking.

[0064] In practice, the second queue list2 is traversed. For each second vertex reached in the traversal, the spatial distance between the current second vertex and each first vertex in the first queue list1 is calculated, and the first index corresponding to the first vertex with the smallest spatial distance is recorded.

[0065] After the second queue (list2) is traversed, the first index of the recorded first vertex is stored to obtain the initial correspondence. This initial queue relationship can be represented by the third queue (list3). The index order of the third queue (list3) is the same as the order of the second indices of the second vertices traversed by the second queue (list2), and the index value is the first index of the first vertex in the first queue (list1) that is closest to the second vertex.

[0066] S102: Traverse each first vertex in the original 3D model, and based on the initial correspondence, determine the second index of at least one second vertex corresponding to the first index of the currently traversed first vertex, and obtain the index correspondence.

[0067] Since BlendShape-based deformation ultimately updates the position of the second vertex of the model to be processed in the rendering engine by changing the position of the first vertex in the original 3D model, the correspondence stored in the third queue list3 above cannot be used directly.

[0068] In one example, the first vertices of the original 3D model stored in the first queue list1 can be traversed. For each first vertex reached, the first index is queried in the third queue list3, and at least one second index corresponding to the first index in the third queue list3 is recorded and saved. After the first queue list1 is traversed, a one-to-many index correspondence between the first index of the first vertex in the original 3D model and the second index of at least one second vertex in the model to be processed can be obtained.

[0069] It should be noted that the queue is only one way of storing the index in this application embodiment, and there are no restrictive requirements for it. Other forms (such as arrays) can also be used for storage.

[0070] As can be seen from the process of establishing the index correspondence, for each first index, there are several corresponding second indices. Therefore, when storing the index correspondence, the second index of at least one second vertex corresponding to the first index of each first vertex can be regarded as a sub-relation of the index correspondence. Different sub-relations are separated by a first delimiter, and in each sub-relation, the first index of the first vertex and the second index of at least one second vertex are separated by a second delimiter, and the second indices of at least one second vertex are separated by a third delimiter.

[0071] In one example, when storing the above correspondence between indexes in the form of a file, different sub-relationships can be separated by a newline character. That is, at least one second index corresponding to each first index is saved as a single line, with the line number being the first index.

[0072] For example, such as Figure 3 The diagram shows the storage method of the index correspondence. The first part before the second separator ":" is the first index of the first vertex in the original 3D model, and the second part after the second separator ":" is the second index of the second vertex in the model to be processed in the rendering engine. Each first index can correspond to one second index or multiple second indices. Each first index corresponds to at least one second index in a row, and the second indices in the same row are separated by the third separator ",".

[0073] See Figure 4 The overall flowchart for establishing the index correspondence provided in this application embodiment mainly includes the following steps:

[0074] S401: Read the original 3D model data and save the initial coordinates of the first vertex in the first queue according to the first index of the first vertex.

[0075] S402: Read the model data to be processed in the rendering engine, and save the initial coordinates of the second vertex in the second queue according to the second index of the second vertex.

[0076] S403: First, traverse the second vertex in the second queue, determine the first vertex in the first queue that is closest to it, and record it.

[0077] S404: According to the second index order of the second vertex in the second queue, store the first index of the first vertex that is closest to it as the index value to obtain the third queue.

[0078] S405: Traverse the first vertex in the first queue again, and based on the third queue, determine the second index of at least one second vertex corresponding to the first index of the first vertex, and obtain the index correspondence.

[0079] S406: Using the first index as the row number, write at least one corresponding second index as a row to the local mapping file for storage.

[0080] In the embodiments of this application, the computational load of establishing the index correspondence is large and the processing time required is long, which generally cannot be performed in real time. However, for the rendering engine with the same processing mechanism, the above index correspondence remains unchanged. Therefore, the correspondence between the indexes can be established in advance and saved as a local mapping file so that it can be used directly when using BlendShape data for model deformation in the future, thus saving the computational load.

[0081] It should be noted that the above preprocessing process can be performed by virtual reality devices, or by other electronic devices, such as laptops, desktop computers, servers, etc.

[0082] Based on the established index correspondence, BlendShape-based model deformation is performed. First, the locally stored BlendShape data and the index correspondence in the local mapping file are read. Then, according to the weights of the input BlendShape data, combined with the BlendShape data and the index correspondence, the positions of the second vertices in the model to be processed in the rendering engine are directly calculated, thus completing the deformation of the original 3D model and rendering the deformed result. For detailed implementation procedures, please refer to [link to implementation details]. Figure 5 It mainly includes the following steps:

[0083] S501: Get the index mapping relationship.

[0084] When performing model deformation and rendering, the local mapping file is read to obtain the index correspondence. This index correspondence is the relationship between the first index of each first vertex in the original 3D model and the second index of at least one corresponding second vertex in the rendering engine.

[0085] In one example, each line in the mapping file is read, and by finding the delimiter, at least one second index contained in each line is obtained and saved to a fourth queue, list4. Each of the obtained fourth queues, list4, is treated as a sub-queue to generate a fifth queue, list5, corresponding to the index mapping. Each element in the fifth queue, list5, is a sub-queue containing the index of at least one second vertex corresponding to a first vertex of the original 3D model in the rendering engine.

[0086] S502: Obtain a pre-stored BlendShape dataset, where each BlendShape dataset contains the first index of each first vertex it controls and the initial offset of the corresponding first vertex.

[0087] Generally, BlendShape datasets are pre-generated by artists during modeling. Each BlendShape data point corresponds one-to-one with the first vertex in the original 3D model through a first index. Each BlendShape data point contains the first index of each first vertex it controls in the entire original 3D model, as well as the initial offset of the corresponding first vertex under the shape in which the BlendShape data is applied.

[0088] In one example, the BlendShape dataset is read from the local machine, parsed, and the first index of each first vertex contained in the original 3D model controlled by each BlendShape data is obtained and stored in the sixth queue list6. The initial offset of each first vertex contained in the original 3D model controlled by each BlendShape data is obtained and stored in the seventh queue list7.

[0089] S503: Get the initial coordinates of each second vertex of the model to be processed in the rendering engine.

[0090] When performing BlendShape-based deformation, the original 3D model data is input into the rendering engine. The rendering engine then performs secondary processing on the original 3D model data to obtain the initial coordinates of each second vertex in the model to be processed, and stores them in the eighth queue, list8.

[0091] It should be noted that when the index mapping is generated in advance, the rendering engine also generates a second queue, list2, based on the coordinates of the second vertex. Since this is a preprocessing process, it is only executed once and is offline data. However, model deformation is real-time and needs to be executed every frame. At this time, the eighth queue, list8, is online data. The two are essentially not the same application.

[0092] S504: For each first vertex, obtain the target offset of the first vertex based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data.

[0093] Generally, the preset weights of each BlendShape data are often given by algorithm or manually, depending on the application scenario.

[0094] For example, based on a face image captured by a camera, an algorithm generates several BlendShape data sets with preset weights. By using the preset weights of each BlendShape data set, the original 3D model can be transformed into the facial shape in the face image, thus achieving 3D facial reconstruction based on the face image.

[0095] After obtaining the preset weights of each BlendShape data locally, the initial offsets of each first vertex contained in each BlendShape data are weighted to obtain the target offset of the corresponding first vertex. See [link to details] for further information. Figure 6 It mainly includes the following steps:

[0096] S5041: For each first vertex, multiply the preset weight of each BlendShape data by the initial offset of the first vertex contained in the corresponding BlendShape data to obtain the sub-offset of the first vertex.

[0097] S5042: Add the sub-offsets of the first vertex determined by each BlendShape data to obtain the target offset of the first vertex.

[0098] In practice, the sixth queue (list6) is traversed. For the first index of each traversal, the seventh queue (list7) is queried to obtain the initial offset of the first vertex corresponding to the first index under the action of each BlendShape data. Each initial offset is multiplied by the preset weight of the corresponding BlendShape data and then added together to obtain the target offset of the first vertex under the combined action of each BlendShape data.

[0099] S505: Based on the first index of each first vertex controlled by each BlendShape data and the index correspondence, determine the second index of the second vertex corresponding to each BlendShape data.

[0100] In one example, for each first index stored in the sixth queue list6, the subqueue list4 corresponding to the first index is determined by the fifth queue list5, which represents the index correspondence, thereby obtaining the second index of at least one second vertex corresponding to the first vertex in the rendering engine, and then obtaining the second index of the second vertex corresponding to each BlendShape data.

[0101] S506: Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex, and combine them with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex.

[0102] In one example, the initial coordinates of the corresponding second vertex are obtained from the eighth queue list8 by using the second index of each sub-queue list4 in the fifth queue list5. Since the initial coordinates of each second vertex are the same as the initial coordinates of the corresponding first vertex, the target offset of each first vertex under the action of each BlendShape data can be used as the target offset of the corresponding second vertex. Adding the initial coordinates of the corresponding second vertex, the target coordinates of the corresponding second vertex can be obtained.

[0103] The transformation formula for BlendShape is expressed as follows:

[0104]

[0105] Where F(ω) is the deformed form of the model, Base is the original 3D model form, and β i ω is the initial offset contained in the i-th BlendShape data. i It is the weight corresponding to the i-th BlendShape data, and n is the number of BlendShape data.

[0106] See Figure 7 The image shows the effect of deforming each BlendShape data provided in the embodiments of this application. Each BlendShape data represents a facial shape, and multiple BlendShape data are combined with different weights to achieve a final facial shape.

[0107] S507: Based on the target coordinates of each second vertex, complete the deformation of the original 3D model and display it.

[0108] After obtaining the target coordinates of each second fixed point under the action of each BlendShape data, the original 3D model is deformed, and the deformed model is rendered and displayed.

[0109] The adaptive BlendShape deformation method provided in this application embodiment can determine the weights of each BlendShape data based on a single frame image, and deform the original 3D model into the shape in the image through these weights, thereby realizing the 3D reconstruction of the human body through the image; or the weights of each BlendShape data can be manually adjusted to achieve different character images, thereby achieving the effect of character face sculpting.

[0110] See Figure 8 The overall process of adaptive BlendShape deformation provided in this application embodiment mainly includes two parts: initialization and model deformation. The specific implementation process includes the following steps:

[0111] S801: Read the local mapping file containing the storage index correspondence.

[0112] S802: For each line in the local mapping file, determine at least one second index corresponding to each first index according to the delimiter, and store it in the fourth queue list4.

[0113] S803: Combine the obtained fourth queues list4 into a sub-queue and merge them into the fifth queue list5.

[0114] S804: Read the pre-stored BlendShape dataset, store the first index of each first vertex contained in the original 3D model controlled by each BlendShape data into the sixth queue list6, and store the initial offset of each first vertex controlled by each BlendShape data into the seventh queue list7.

[0115] S805: Input the original 3D model data into the rendering engine, read the initial coordinates of the second vertex of the model to be processed in the rendering engine, and store them in the eighth queue list8.

[0116] S806: Get the preset weights of each BlendShape data.

[0117] S807: Traverse each first index in the sixth queue list6, and determine the target offset of the first vertex corresponding to the first index based on the preset weight of each BlendShape data and the initial offset of the first index in the seventh queue.

[0118] S808: Traverse the eighth queue list8, determine the target coordinates of the second vertex based on the initial coordinates of the second vertex being traversed and the target offset of the first vertex corresponding to the second vertex being traversed, and update the eighth queue list8.

[0119] S809: Based on the target coordinates of each second vertex in the eighth queue list8, perform deformation of the original 3D model and render and display the deformed model.

[0120] The adaptive BlendShape deformation method provided in this application embodiment can also generate several weights of continuous BlendShape data by inputting continuously acquired face images from a camera. Based on these continuous weights, the original 3D model is continuously deformed to achieve the effect of face-driven deformation by capturing facial shapes.

[0121] See Figure 9 The overall process of continuous adaptive BlendShape deformation provided in this application embodiment mainly includes two parts: initialization and model deformation. The initialization part can be executed once, and the specific implementation process includes the following steps:

[0122] S901: Read the local mapping file containing the storage index correspondence.

[0123] S902: For each line in the local mapping file, determine at least one second index corresponding to each first index according to the delimiter, and store it in the fourth queue list4.

[0124] S903: Combine the obtained fourth queues list4 into a sub-queue and merge them into the fifth queue list5.

[0125] S904: Read the pre-stored BlendShape dataset, store the first index of each first vertex contained in the original 3D model controlled by each BlendShape data into the sixth queue list6, and store the initial offset of each first vertex controlled by each BlendShape data into the seventh queue list7.

[0126] S905: Input the original 3D model data into the rendering engine, read the initial coordinates of the second vertex of the model to be processed in the rendering engine, and store them in the eighth queue list8.

[0127] S906: Obtain the preset weights of each BlendShape data based on the current frame image.

[0128] S907: Traverse each first index in the sixth queue list6, and determine the target offset of the first vertex corresponding to the first index based on the preset weight of each BlendShape data and the initial offset of the first index in the seventh queue.

[0129] S908: Traverse the eighth queue list8, determine the target coordinates of the second vertex based on the initial coordinates of the second vertex being traversed and the target offset of the first vertex corresponding to the second vertex being traversed, and update the eighth queue list8.

[0130] S909: Based on the target coordinates of each second vertex in the eighth queue list8, perform deformation of the original 3D model and render and display the deformed target 3D model.

[0131] S910: Determine whether to perform real-time dynamic deformation. If yes, return to S906; otherwise, end the process.

[0132] The adaptive BlendShape deformation method provided in this application adopts a reverse thinking approach. It pre-establishes a one-to-many index correspondence between the first index of the first vertex of the original 3D model and the second index of the second vertex of the model to be processed in the rendering engine. The vertex offset contained in each BlendShape data is mapped one-to-one with the first vertex in the original 3D model through the first index. Thus, based on the pre-established index correspondence, the vertex offset contained in each BlendShape data can be applied to the second vertex in the rendering engine, achieving deformation of the original 3D model. The method in this application can adaptively map BlendShape data to the second vertices of the model to be processed in the rendering engine, thereby ensuring that the BlendShape data can adaptively match the model deformation process of different rendering engines. This effectively improves the versatility of BlendShape data, thereby increasing its utilization rate and reducing the workload of modeling.

[0133] Based on the same technical concept, this application provides a virtual reality device that can implement the above-described adaptive BlendShape deformation method steps and achieve the same technical effect.

[0134] See Figure 10 The virtual reality device includes a processor 1001, a memory 1002, and a display screen 1003, which are connected via a bus 1004.

[0135] The memory 1002 includes a data storage unit and a program storage unit. The data storage unit stores the index correspondence, the BlendShape dataset, and the weights of each BlendShape data. The index correspondence is the correspondence between the first index of each first vertex and the second index of at least one second vertex corresponding to the first vertex. The first vertex is the vertex in the original 3D model before being input into the rendering engine, and the second vertex is the vertex in the model to be processed after being input into the rendering engine. Each BlendShape data includes the first index of each first vertex it controls and the initial offset of the corresponding first vertex.

[0136] The program storage unit stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0137] Obtain the index correspondence from the data storage unit, as well as the weights of the BlendShape dataset and each BlendShape data;

[0138] Obtain the initial coordinates of each second vertex in the model to be processed after the original 3D model is input into the rendering engine;

[0139] For each first vertex, the target offset of the first vertex is obtained based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data.

[0140] Based on the first index of each first vertex controlled by each BlendShape data, and the index correspondence, determine the second index of the second vertex corresponding to the BlendShape data;

[0141] Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex, and combine them with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex.

[0142] Based on the target coordinates of each second vertex, the original 3D model is deformed, and the deformed target 3D model is displayed on the display screen 1003.

[0143] Optionally, the processor 1001 obtains the target offset of the first vertex based on the initial offset contained in each BlendShape data controlling it and the preset weights of each BlendShape data. Specifically, the operation is as follows:

[0144] The preset weights of each BlendShape data are multiplied by the initial offset of the first vertex contained in the corresponding BlendShape data to obtain the sub-offset of the first vertex;

[0145] The sub-offsets of the first vertex determined by each BlendShape data are added together to obtain the target offset of the first vertex.

[0146] Optionally, the processor 1001 determines the index mapping in the following way:

[0147] For each second vertex of the model to be processed in the rendering engine, determine the first vertex in the original 3D model that is closest to it to obtain the initial correspondence;

[0148] Traverse each first vertex in the original 3D model, and based on the initial correspondence, determine the second index of at least one second vertex corresponding to the first index of the currently traversed first vertex, thus obtaining the index correspondence.

[0149] Optionally, in the index correspondence, the first index of each first vertex corresponds to at least one second index of a second vertex, which constitutes a sub-relation of the index correspondence, and different sub-relations are separated by a first delimiter;

[0150] In each sub-relation, the first index of the first vertex and the second index of at least one second vertex are separated by a second delimiter, and the second indices of at least one second vertex are separated by a third delimiter.

[0151] Optionally, when performing real-time dynamic deformation, processor 1001 also performs:

[0152] Continuously acquire the preset weights of each BlendShape data multiple times;

[0153] Based on the preset weights of each BlendShape data obtained this time, and the initial offsets of the first vertices of the original 3D model contained in each BlendShape data, the original 3D model is re-deformed and displayed on the display screen 1003.

[0154] It should be noted that, Figure 10 This is merely an example illustrating the hardware necessary for a virtual reality device to perform the steps of an adaptive BlendShape deformation method provided in this application embodiment. Not shown, the virtual reality device may also include conventional hardware such as speakers, microphones, communication interfaces, and power supplies.

[0155] Examples of this application Figure 10 The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0156] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform an adaptive BlendShape deformation method as described in the foregoing embodiments.

[0157] This application also provides a computer program product for storing a computer program that executes an adaptive BlendShape deformation method as described in the foregoing embodiments.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An adaptive BlendShape deformation method, characterized in that, include: Obtain the index correspondence relationship, wherein the index correspondence relationship is the correspondence between the first index of each first vertex and the second index of at least one second vertex corresponding to the first vertex, the first vertex is the vertex in the original 3D model before inputting into the rendering engine, and the second vertex is the vertex in the model to be processed after inputting into the rendering engine; Obtain the pre-stored BlendShape dataset, where each BlendShape dataset contains the first index of each first vertex it controls and the initial offset of the corresponding first vertex; Obtain the initial coordinates of each second vertex of the model to be processed in the rendering engine; For each first vertex, the target offset of the first vertex is obtained based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data. Based on the first index of each first vertex controlled by each BlendShape data and the index correspondence, determine the second index of the second vertex corresponding to the BlendShape data; Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex, and combine them with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex. Based on the target coordinates of each second vertex, the original 3D model is deformed and displayed.

2. The method as described in claim 1, characterized in that, The step of obtaining the target offset of the first vertex based on the initial offset contained in each BlendShape data controlling it and the preset weight of each BlendShape data includes: The preset weight of each BlendShape data is multiplied by the initial offset of the first vertex contained in the corresponding BlendShape data to obtain the sub-offset of the first vertex. The sub-offsets of the first vertex determined by each BlendShape data are added together to obtain the target offset of the first vertex.

3. The method as described in claim 1 or 2, characterized in that, The index correspondence is determined in the following way: For each second vertex of the model to be processed in the rendering engine, the nearest first vertex in the original 3D model is determined to obtain the initial correspondence. Traverse each first vertex in the original 3D model, and determine the second index of at least one second vertex corresponding to the first index of the currently traversed first vertex according to the initial correspondence, thereby obtaining the index correspondence.

4. The method as described in claim 1 or 2, characterized in that, In the index correspondence, the first index of each first vertex corresponds to at least one second index of a second vertex, which constitutes a sub-relation of the index correspondence, and different sub-relations are separated by a first delimiter; In each sub-relation, the first index of the first vertex and the second index of at least one second vertex are separated by a second delimiter, and the second indices of the at least one second vertex are separated by a third delimiter.

5. The method as described in claim 1 or 2, characterized in that, When the deformation is real-time and dynamic, the method further includes: Continuously acquire the preset weights of each BlendShape data multiple times; Based on the preset weights of each BlendShape data obtained this time, and the initial offsets of the first vertices of the original 3D model contained in each BlendShape data, the original 3D model is re-deformed and displayed.

6. A virtual reality device, characterized in that, It includes a processor, a memory, and a display screen, wherein the display screen, the memory, and the processor are connected via a bus; The memory includes a data storage unit and a program storage unit. The data storage unit stores the index correspondence, the BlendShape dataset, and the weights of each BlendShape data. The index correspondence is the correspondence between the first index of each first vertex and the second index of at least one second vertex corresponding to the first vertex. The first vertex is a vertex in the original 3D model before being input into the rendering engine, and the second vertex is a vertex in the model to be processed after being input into the rendering engine. Each BlendShape data includes the first index of each first vertex it controls and the initial offset of the corresponding first vertex. The program storage unit stores a computer program, and the processor performs the following operations according to the computer program: Obtain the index correspondence from the data storage unit, as well as the weights of the BlendShape dataset and each BlendShape data; The initial coordinates of each second vertex in the model to be processed are obtained after the original 3D model is input into the rendering engine; For each first vertex, the target offset of the first vertex is obtained based on the initial offset contained in each BlendShape data that controls it and the preset weight of each BlendShape data. Based on the first index of each first vertex controlled by each BlendShape data and the index correspondence, determine the second index of the second vertex corresponding to the BlendShape data; Based on the second index corresponding to each BlendShape data, obtain the initial coordinates of the corresponding second vertex, and combine them with the target offset of the first vertex corresponding to each second vertex to obtain the target coordinates of the corresponding second vertex. Based on the target coordinates of each second vertex, the original 3D model is deformed, and the deformed target 3D model is displayed on the display screen.

7. The virtual reality device as described in claim 6, characterized in that, The processor obtains the target offset of the first vertex based on the initial offset contained in each BlendShape data controlling it and the preset weight of each BlendShape data. The specific operation is as follows: The preset weight of each BlendShape data is multiplied by the initial offset of the first vertex contained in the corresponding BlendShape data to obtain the sub-offset of the first vertex. The sub-offsets of the first vertex determined by each BlendShape data are added together to obtain the target offset of the first vertex.

8. The virtual reality device as described in claim 6 or 7, characterized in that, The processor determines the index correspondence in the following way: For each second vertex of the model to be processed in the rendering engine, the nearest first vertex in the original 3D model is determined to obtain the initial correspondence. Traverse each first vertex in the original 3D model, and determine the second index of at least one second vertex corresponding to the first index of the currently traversed first vertex according to the initial correspondence, thereby obtaining the index correspondence.

9. The virtual reality device as described in claim 6 or 7, characterized in that, In the index correspondence, the first index of each first vertex corresponds to at least one second index of a second vertex, which constitutes a sub-relation of the index correspondence, and different sub-relations are separated by a first delimiter; In each sub-relation, the first index of the first vertex and the second index of at least one second vertex are separated by a second delimiter, and the second indices of the at least one second vertex are separated by a third delimiter.

10. The virtual reality device as described in claim 6 or 7, characterized in that, When performing real-time dynamic deformation, the processor also performs: Continuously acquire the preset weights of each BlendShape data multiple times; Based on the preset weights of each BlendShape data obtained this time, and the initial offsets of the first vertices of the original 3D model contained in each BlendShape data, the original 3D model is re-deformed and displayed.

Images