Method and device for displaying virtual character model

Abstract

The invention discloses a method and device for displaying a virtual character model. The method comprises the steps of obtaining character point location data of a target character object; processing the figure point location data according to a human body posture estimation algorithm to obtain a target human body grid body comprising a plurality of vertexes representing the shape and posture of a target figure object; a target coefficient corresponding to each target area of the target human body grid body is determined, vertexes belonging to the target areas are processed according to the target coefficients, a target surrounding body corresponding to the target areas is obtained, and the target areas correspond to the body parts of the target person object; according to the target bounding volume of the target area, optimizing the figure point location data belonging to the target area to obtain optimized figure point location data; and generating and displaying a virtual character model according to the optimized character point location data. According to the scheme, the target surrounding body corresponding to each body part is combined to optimize the figure point position data used for generating the virtual figure model, and the more accurate virtual figure model can be displayed.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a method and device for displaying a virtual character model. Background Art

[0002] Virtual shooting technology has been widely used in games, augmented reality (AR), film and television production and other fields. Virtual shooting technology can shoot real people, generate and display virtual character models in virtual scenes based on the captured images.

[0003] Existing virtual shooting technologies have low accuracy when generating virtual character models. Even through motion and facial capture technology, it is difficult to accurately present the real movements of the photographed person on the virtual character model. Summary of the invention

[0004] To this end, this application discloses the following technical solutions:

[0005] A first aspect of the present application provides a method for displaying a virtual character model, comprising:

[0006] Obtaining character point data of a target character object, wherein the character point data includes three-dimensional coordinate data and color data of a plurality of points on the surface of the target character object;

[0007] Processing the person point data according to a human posture estimation algorithm to obtain a target human body mesh body, wherein the target human body mesh body includes a plurality of vertices representing the shape and posture of the target person object;

[0008] For each target area of ​​the target human mesh body, determine a target coefficient corresponding to the target area, and process the vertices belonging to the target area according to the target coefficient to obtain a target bounding volume corresponding to the target area, wherein each target area corresponds to a body part of the target person object;

[0009] According to the target bounding volume of the target area, optimizing the person point data belonging to the target area to obtain optimized person point data;

[0010] A virtual character model of the target character object is generated according to the optimized character point data, and the virtual character model is displayed in a target virtual scene.

[0011] Optionally, determining a target coefficient corresponding to the target area includes:

[0012] Obtaining item information of the target area, wherein the item information indicates whether a body part of the target person object corresponding to the target area has an item worn and the type of the item worn;

[0013] A target coefficient corresponding to the target area is determined according to the item information of the target area.

[0014] Optionally, obtaining the person point data of the target person object includes:

[0015] Obtaining a target depth image of the target person object;

[0016] Identifying a foreground portion of the target depth image;

[0017] The person point data of the target person object is determined according to the foreground part.

[0018] Optionally, the optimizing the person point data belonging to the target area includes:

[0019] Optimizing the three-dimensional coordinate data belonging to the target area in the person point data;

[0020] The method for determining the three-dimensional coordinate data belonging to the target area includes:

[0021] Identifying multiple body parts of the target person object in the foreground portion, and dividing the foreground portion into multiple sub-portions corresponding to the multiple body parts based on the identification result;

[0022] For each of the body parts, the three-dimensional coordinate data obtained according to the sub-portion corresponding to the body part is determined as belonging to the target area corresponding to the body part.

[0023] Optionally, optimizing the person point data belonging to the target area according to the target bounding volume of the target area includes:

[0024] Determine whether each three-dimensional coordinate data belonging to the target area is within the target bounding volume corresponding to the target area;

[0025] For the three-dimensional coordinate data that is not within the target bounding volume corresponding to the target area, the three-dimensional coordinate data is updated based on the target bounding volume to obtain updated three-dimensional coordinate data that is within the target bounding volume corresponding to the target area.

[0026] Optionally, generating a virtual character model of the target character object according to the optimized character point data includes:

[0027] Generate a character mesh model according to the optimized character point data;

[0028] Image texture mapping is performed on the character mesh model to obtain a virtual character model of the target character object.

[0029] Optionally, before generating the virtual character model of the target character object according to the optimized character point data, the method further includes:

[0030] Evaluate the optimization effect of the optimized character point data;

[0031] When the optimization effect does not meet the target optimization condition, updating the target bounding volume corresponding to each target area;

[0032] Based on the updated target bounding volume, return to execute the step of optimizing the character point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized character point data.

[0033] A second aspect of the present application provides a device for displaying a virtual character model, comprising:

[0034] An obtaining unit, used for obtaining character point data of a target character object, wherein the character point data includes three-dimensional coordinate data and color data of a plurality of points on the surface of the target character object;

[0035] A processing unit, configured to process the human point data according to a human posture estimation algorithm to obtain a target human mesh body, wherein the target human mesh body includes a plurality of vertices representing a shape and a posture of the target human object;

[0036] a determination unit, for determining, for each target region of the target human mesh body, a target coefficient corresponding to the target region, and processing vertices belonging to the target region according to the target coefficient to obtain a target bounding volume corresponding to the target region, wherein each target region corresponds to a body part of the target human object;

[0037] An optimization unit, configured to optimize the three-dimensional coordinate data belonging to the target area in the character point data according to the target bounding volume of the target area, so as to obtain optimized character point data;

[0038] A display unit is used to generate a virtual character model of the target character object according to the optimized character point data, and display the virtual character model in a target virtual scene.

[0039] Optionally, when the determination unit determines the target coefficient corresponding to the target area, it is used to:

[0040] Obtaining item information of the target area, wherein the item information indicates whether a body part of the target person object corresponding to the target area has an item worn and the type of the item worn;

[0041] A target coefficient corresponding to the target area is determined according to the item information of the target area.

[0042] Optionally, the optimization unit is further used for:

[0043] Evaluate the optimization effect of the optimized character point data;

[0044] When the optimization effect does not meet the target optimization condition, updating the target bounding volume corresponding to each target area;

[0045] Based on the updated target bounding volume, return to execute the step of optimizing the character point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized character point data.

[0046] The beneficial effects of this solution are:

[0047] After obtaining the character point data, this solution determines the target bounding volume corresponding to each body part based on the target human body mesh, and then optimizes the three-dimensional coordinate data corresponding to each body part in the character point data according to each target bounding volume to obtain the optimized character point data, and finally generates and displays the virtual character model based on the optimized character point data. Through the optimization of the target bounding volume, this solution can obtain more accurate character point data, and then can obtain a virtual character model, so that the displayed virtual character model more accurately reflects the real action and posture of the target character object. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 is a flow chart of a method for displaying a virtual character model provided by an embodiment of the present application;

[0050] Figure 2 is a flow chart of a method for generating a character mesh model provided by an embodiment of the present application;

[0051] Figure 3 is a flow chart of a method for performing image texture mapping provided by an embodiment of the present application;

[0052] Figure 4 is a flow chart of a method for obtaining person point data of a target person object provided by an embodiment of the present application;

[0053] Figure 5It is a structural schematic diagram of a device for displaying a virtual character model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] This embodiment provides a method for displaying a virtual character model. Figure 1 , is a flow chart of the method, and the method may include the following steps.

[0056] S101, obtaining person point data of a target person object, where the person point data includes three-dimensional coordinate data and color data of multiple points on the surface of the target person object.

[0057] The target character object may be any person for whom a corresponding virtual character model needs to be generated and displayed.

[0058] As some examples, the method of this embodiment can be applied to the field of augmented reality (AR). When user A uses an AR device, user A can be regarded as a target person object. The AR device obtains the person point data of user A through a camera connected by communication. Based on the method of this embodiment, the person point data of user A is used to generate and display a virtual character model corresponding to user A in a virtual scene displayed by the AR device, so as to present the visual effect of user A's activities in the virtual scene.

[0059] As other examples, the method of this embodiment can be applied to the field of film and television production. In this case, actor B can be regarded as the target character object, and the character point data of actor B can be obtained by using a camera. Based on the method of this embodiment, the character point data of actor B is used to generate and display a virtual character model of actor B in a specific virtual scene, thereby mapping the actions of actor B to the virtual character model, so that the actions of the virtual character model in the video are more consistent with the real actions of the person.

[0060] It can be understood that the method of this embodiment can continuously obtain multiple frames of character point data at multiple moments at certain time intervals, generate and display a frame of virtual character model corresponding to each moment based on a frame of character point data corresponding to the moment, thereby obtaining a video of a virtual character model moving in a virtual scene.

[0061] For ease of explanation, the following description will be made by taking the processing of a frame of character point data at a moment to obtain a frame of a virtual character model at the moment as an example.

[0062] The character point data may include multiple sets of three-dimensional coordinate data and color data corresponding to each other, each set of three-dimensional coordinate data represents the three-dimensional coordinate of a point on the surface of the target character object, and a set of color data (for example, RGB color data) corresponding to the three-dimensional coordinate data represents the color of the point. The character point data may be in various forms. As an example, the character point data at a certain moment may be represented by the following Table 1. The three-dimensional coordinate data and color data in the first row represent the three-dimensional coordinate and color of point P1 on the surface of the target character object at that moment, the second row represents the three-dimensional coordinate and color of point P2, and so on.

[0063] Table 1

[0064] point 3D coordinate data Color Data P1 (x1, y1, z1) (r1, g1, b1) P2 (x2, y2, z2) (r2, g2, b2) …… …… …… Pn (xn, yn, zn) (rn, gn, bn)

[0065] The points on the surface of the target person object may include points on the target person object's skin, points on clothing, points on accessories (such as rings, necklaces) and points on equipment (such as watches).

[0066] S102, processing the person point data according to a human body posture estimation algorithm to obtain a target human body mesh body, wherein the target human body mesh body includes a plurality of vertices representing the shape and posture of the target human object.

[0067] In step S102, a frame of human point data may be processed using a human posture estimation algorithm to obtain a corresponding target human skeleton structure, and then a target human mesh may be generated according to the target human skeleton structure.

[0068] In step S102, a frame of human point data is processed according to a human posture estimation algorithm to obtain a target human skeleton structure, and the principle and specific implementation method of generating a target human mesh body according to the target human skeleton structure can be referred to the technical solutions in the related field, which will not be described in detail.

[0069] The generated target human skeleton structure can represent the position, shape and posture of the main skeleton of the target person object in three-dimensional space at the corresponding moment.

[0070] The target human skeleton structure may include a plurality of key points and a series of line segments connecting every two adjacent key points, each line segment representing a bone.

[0071] The target human mesh may be a three-dimensional model composed of a plurality of vertices, wherein the vertices are connected into a continuous surface through edges and faces, and can realistically represent the position, shape and posture of the target human object at the corresponding moment.

[0072] S103, for each target area of ​​the target human body mesh body, determine the target coefficient corresponding to the target area, and process the vertices belonging to the target area according to the target coefficient to obtain the target bounding volume corresponding to the target area, each target area corresponds to a body part of the target person object.

[0073] In some embodiments, the target coefficient of each target area may be a fixed parameter pre-configured by relevant personnel based on basic human body size and shape, and range of joint motion. In this case, the relevant configuration table may be searched, and the target coefficient of each target area may be read from the configuration table to execute S103.

[0074] The target area can be divided in many ways. For example, the target human mesh body can be divided into 8 target areas, including the head area, left arm area, right arm area, chest and abdomen area, left leg area, right leg area, left foot area and right foot area, which correspond to the corresponding body parts of the target person object in turn; or, the left arm area can be further subdivided into the left palm area, left forearm area, left upper arm area, and the chest and abdomen area can be divided into the chest cavity area and the abdominal cavity area, etc.

[0075] The target bounding volume corresponding to each target area can be a BBody with the maximum coordinates max and minimum coordinate BBody min The cube with the boundary is determined, which determines the reasonable distribution range of the points on the surface of the target person object in space. In other words, if the three-dimensional coordinate data of a point in the person point data exceeds the target bounding volume of the target area to which it belongs, it can be considered that the three-dimensional coordinate data of the point is wrong and needs to be optimized.

[0076] For each target area, the target bounding volume corresponding to the target area can be obtained in S103 in the following manner.

[0077] First, the maximum value of the X coordinates (max(x), the minimum value of the X coordinates (min(x), the maximum value of the Y coordinates (max(y), the minimum value of the Y coordinates (min(y), the maximum value of the Z coordinates (max(z), and the minimum value of the Z coordinates (min(z))) of all vertices in the target area of ​​the target human body mesh can be counted according to formulas (1) to (3).

[0078]

[0079]

[0080]

[0081] Where v represents the total number of vertices in the target area, k represents the kth vertex in the target area, (x k,y k , z k ) represents the three-dimensional coordinates of the kth vertex, that is, the coordinates of the vertex on the x-axis, y-axis, and z-axis;

[0082] Then, the length depth, width width and height height of the target bounding volume of the target area can be determined according to the following formulas (4) to (6), and the coordinates of the center point c of the target bounding volume of the target area can be determined according to formula (7) (c x , c y , c z );

[0083]

[0084]

[0085]

[0086]

[0087] Finally, the maximum coordinates BBody of the target bounding volume can be determined based on the length depth, width width and height height of the target bounding volume, the coordinates of the center point c of the target bounding volume, and the target coefficient of the target area. max and minimum coordinate BBody min , the target coefficients may include a target scaling factor S and a target offset (T x , T y , T z ), the maximum coordinate and the minimum coordinate can be expressed by the following formulas (8) and (9).

[0088]

[0089]

[0090] The above maximum coordinates and minimum coordinates are equivalent to two opposite vertices in the target bounding volume, and the cube determined based on the two vertices is the target bounding volume of the target area.

[0091] In some embodiments, the target region to which each vertex in the target human body mesh belongs may be determined as follows:

[0092] As the position and posture of the target person object change, the vertices of the target human body mesh will be displaced and scaled in space, causing the vertices to be in different positions in space. However, the number of vertices contained in a target human body mesh, the label of each vertex, and the correspondence between each vertex and the human body remain unchanged. Therefore, each vertex can be manually labeled in advance to indicate which part of the human body it corresponds to, and then based on the manually labeled correspondence, it can be determined which target area each vertex belongs to.

[0093] For example, vertex No. 1 corresponds to the head of the human body, and vertex No. 10 corresponds to the chest of the human body. It can be determined that vertex No. 1 belongs to the head area, and vertex No. 10 belongs to the chest and abdomen area.

[0094] In some embodiments, the target region to which each vertex in the target human body mesh belongs may also be determined as follows:

[0095] When using a camera to obtain person point data, the camera will shoot the target person object to obtain at least one frame of its image. At this time, the existing image recognition algorithm can be used to identify the areas where the various body parts of the target person object are located in the image. In this way, when generating person point data, it is possible to determine which body part each set of three-dimensional coordinate data and color data in the person point data corresponds to, and then when determining the target human body mesh body based on the person point data, it can be determined which target area each vertex belongs to.

[0096] Exemplarily, after identifying the head area in the image, it can be determined that the three-dimensional coordinate data and color data generated according to the pixel content of the head area in the image in the character point data corresponds to the head of the target character object, and then the vertices determined in the target human mesh body according to this part of the three-dimensional coordinate data and color data can be determined as belonging to the head area.

[0097] S104, optimizing the person point data belonging to the target area according to the target bounding volume of the target area to obtain optimized person point data.

[0098] In step S104, the character point data may be optimized in the following manner to obtain optimized character point data:

[0099] A1, determining whether each three-dimensional coordinate data belonging to the target area is within the target bounding box corresponding to the target area;

[0100] A2, for the three-dimensional coordinate data that is not within the target bounding volume corresponding to the target area, the three-dimensional coordinate data is updated based on the target bounding volume to obtain updated three-dimensional coordinate data that is within the target bounding volume corresponding to the target area.

[0101] In step A1, for each three-dimensional coordinate data (xi ,y i , z i ), it can be determined whether the three-dimensional coordinate data satisfies the conditions shown in the following formula (10).

[0102]

[0103] Among them, x min ,y min and z min These are the first, second, and third items of the minimum coordinates of the target bounding volume of the target area to which the three-dimensional coordinate data belongs, that is, BBody min =(x min ,y min , z min );x max ,y max and z max These are the first, second, and third items of the minimum coordinates of the target bounding volume of the target area to which the three-dimensional coordinate data belongs, that is, BBody max =(x max ,y max , z max ).

[0104] If a 3D coordinate data does not satisfy the condition of formula (10), A2 is executed on the 3D coordinate data to update the 3D coordinate data. The updating method may be:

[0105] If the x-axis coordinate x of the three-dimensional coordinate data i Less than x min , the x-axis coordinate of the three-dimensional coordinate data is changed to x min , that is, x min Assign to x i , that is, x i =x min ; If the x-axis coordinate x of the three-dimensional coordinate data i Greater than x max , the x-axis coordinate of the three-dimensional coordinate data is changed to x max , that is, x max Assign to x i , that is, x i =x max ;

[0106] The y-axis coordinate and z-axis coordinate of the three-dimensional coordinate data are also calculated in the above manner, depending on whether they are less than y min and z min , and whether it is greater than y max and z max , and correspondingly update the y-axis coordinate and z-axis coordinate of the three-dimensional coordinate data.

[0107] After updating each three-dimensional coordinate data belonging to the target area in the above manner, the optimized character point data of the target area can be obtained. After the optimization is completed for each target area, the optimized character point data of step S104 is obtained.

[0108] As an example, assume that the 3D coordinate data corresponding to point P1 located in the head area in the character point data does not satisfy formula (10). Specifically, the y coordinate data of point P1 is i Smaller than the y corresponding to the head area min , you can set the y of the head area min Assign to y i , to complete the update of the three-dimensional coordinate data of P1.

[0109] S105, generating a virtual character model of the target character object according to the optimized character point data, and displaying the virtual character model in the target virtual scene.

[0110] The implementation of step S105 may include:

[0111] B1, generating a character mesh model based on the optimized character point data;

[0112] B2, performing image texture mapping on the character mesh model to obtain a virtual character model of the target character object.

[0113] In step B1, the optimized character point data may be connected and combined to generate a character mesh model.

[0114] The basic principles for generating a character mesh model are:

[0115] A sphere is set to roll on the surface of the optimized character point data. Once the surface of the ball touches three points, a triangular mesh with these three points as vertices is generated until all reachable edges are tried and as many triangular meshes as possible are generated. These triangular meshes together constitute the character mesh model represented by the optimized character point data.

[0116] In the process of constructing the above-mentioned character mesh model, the calculation of the normal vector is particularly critical, which often affects the precision and even accuracy of the subsequent construction. Therefore, in this embodiment, before optimizing the character point data, or after optimizing and before executing step B1, the normal vector corresponding to each three-dimensional coordinate data in the character point data can be calculated, so as to control the rolling of the sphere according to the normal vector. The method for calculating the normal vector and the method for controlling the rolling of the sphere according to the normal vector can be referred to the relevant prior art, which will not be repeated here.

[0117] See also Figure 2In this embodiment, the method for generating a character mesh model may include the following steps:

[0118] 1. Initialize a sphere in the optimized character point data.

[0119] Step 1 is an initialization step, in which a sphere can be selected and placed on any point included in the optimized character point data, with the point as the center of the sphere. For example, point P1 in Table 1 can be used as the center of the sphere, and the radius of the sphere can be determined based on the points around the center of the sphere. The method for determining the radius can refer to the relevant prior art and will not be described in detail.

[0120] 2. Determine whether there are candidate points around the sphere that meet the conditions.

[0121] In step 2, based on the points around the current sphere center in the optimized character point data, it can be determined whether a new point can be added to the edge of the sphere to expand the sphere. This added point is called a candidate point, and the candidate point must meet the following conditions:

[0122] The spherical center angle formed by the candidate point and the three points in contact with the spherical surface is less than or equal to the set angle threshold, and the candidate point does not belong to the three points in contact with the spherical surface (that is, it does not intersect with the points around the spherical center).

[0123] If there is no candidate point that meets the conditions, execute step 3; if there is a candidate point that meets the conditions, execute step 4.

[0124] 3. Continue to search for candidate points that meet the conditions until they are found.

[0125] In step 3, points farther from the sphere can be continuously searched in the optimized character point data until a candidate point that meets the above conditions is found.

[0126] 4. Connect the three points on the surface of the sphere into a triangular mesh and roll the sphere from the current center to the candidate point.

[0127] In step 4, the current center of the sphere may be changed to the selected candidate point to enable the sphere to roll toward the candidate point.

[0128] Continuing with the previous example, after determining that P1 is the center of the sphere, assuming that P2 is found to be a candidate point that meets the conditions, the three points that touch the surface of the sphere can be connected to form a triangular mesh, and then the center of the sphere can be changed from P1 to P2.

[0129] In step 4, after the center of the sphere is changed, the radius of the sphere may remain unchanged or may be adjusted according to the points around the changed center of the sphere.

[0130] 5. Determine whether all points of the optimized character position data have been traversed.

[0131] If all points of the optimized character point data have been traversed, the method ends; if all points of the optimized character point data have not been traversed, step 2 is continued based on the current sphere center.

[0132] In step 5, it can be determined whether all points of the optimized character point data serve as vertices of at least one triangular mesh. If there are isolated points that are not used as vertices of at least one triangular mesh, it is determined that all points have not been traversed. If every point serves as a vertex of at least one triangular mesh, it is determined that all points have been traversed.

[0133] In the above process, it is important to note the choice of the radius R of the sphere, which will affect the performance and results of the algorithm. Usually, the radius of the rolling ball can be adjusted dynamically according to the characteristics of the data. In general: the larger R is, the farther the next point the ball can roll to, and the sparser the point set contained in the concave hull boundary; the smaller R is, the closer the next point the ball can roll to, and the denser the point set contained in the concave hull boundary.

[0134] In the case of character mesh models, the R value can be adjusted. In some embodiments, you can try to set different R values, observe the performance of the algorithm when generating character mesh models based on different R values, and the degree of fit between the generated model and the application scenario, so as to determine the appropriate R value for a specific application scenario. In this way, in the initialization of step 1, you can first determine in which virtual scene the virtual character model will eventually be displayed, and then select a suitable R value according to the scene.

[0135] The implementation details of the above method for generating a character mesh model can be referred to the relevant existing technology and will not be described in detail.

[0136] In step B2, it is necessary to assign a texture map to the character mesh model, that is, it is necessary to map the captured character color image to the character mesh model. This process is equivalent to pasting a two-dimensional image (texture) onto the surface of a three-dimensional object. In this process, it is necessary to establish a mapping relationship between the pixel points in the texture image (called texture coordinates (u, v)) and the vertices on the surface of the character mesh model. This mapping relationship enables the surface of the character mesh model to present the color and details of the texture image. Generally, this mapping relationship can include a mapping relationship between each pixel in the image and a point or patch on the character mesh model.

[0137] Optionally, if the foreground part containing the target person object has been segmented from the image taken by the camera when obtaining the person point data, then in step B2, only the mapping relationship between the pixels of the person part in the image and the points of the person grid model can be calculated, which can greatly improve the calculation efficiency.

[0138] See also Figure 3 In step B2, image texture mapping may be performed on the character mesh model in the following manner to obtain a virtual character model of the target character object.

[0139] 6. Project the surface vertices of the character mesh model onto a plane for texture coordinate calculation.

[0140] In step 6, perspective projection rules can be used for projection to present the effect of different sizes of objects near and far.

[0141] As an example, suppose the camera position is (Ca x , Ca y , Ca z ), the surface vertex coordinates of the character mesh model are (x, y, z), then the projected coordinates (x p ,y p ) can be determined according to the following formula (11).

[0142]

[0143] 7. Calculate the texture coordinates of each vertex based on the projection position.

[0144] Texture coordinates are coordinate points on a two-dimensional image, and are used to determine the corresponding positions of the vertices on the surface of the character mesh model on the texture image. The texture image in this embodiment may include the image captured in step S101 to obtain the character point data.

[0145] Assume that the projected coordinates are (x p ,y p ), the width and height of the texture image are W and H respectively, then (x p ,y p ) The corresponding texture coordinates (u, v) can be determined using the following formula (12).

[0146]

[0147] The texture coordinates (u, v) here usually range from [0, 1].

[0148] 8. Map the color image into texture coordinates and match it with the color data contained in the optimized character point data.

[0149] Since each three-dimensional coordinate data on the optimized character point data corresponds to a set of color data (as shown in Table 1), in this embodiment, the texture image can be mapped to the texture coordinates (u, v), and the color data of the optimized character point data is aligned and registered with the pixel coordinate points of the texture image, so that the feature points of the texture image are corresponded to the feature points of the character mesh model, so that the connection between the texture image and the character mesh model can be more accurate. The specific methods of alignment and registration can refer to the relevant existing technology and will not be elaborated on.

[0150] 9. According to the calculated texture coordinates, find the pixel points at the corresponding position in the texture image, and map all its color information to the surface of the character mesh model.

[0151] In step 9, the color data of each pixel in the texture image can be read and processed, and the texture coordinates can be accurately matched with the pixels of the texture image to ensure a smooth transition during texture mapping. The specific processing methods and corresponding methods can be found in the relevant existing technologies and will not be described in detail.

[0152] After obtaining the virtual character model, the generated virtual character model can be imported into the target virtual scene for presentation. The target virtual scene can be constructed using a virtual reality engine and add effects such as lighting, shadows, and animation to enhance the sense of reality and immersion.

[0153] Optionally, the target virtual scene may be constructed based on the background portion of the image captured in step S101 , or may be constructed according to an application used by the user, without limitation.

[0154] The beneficial effect of this embodiment is that after obtaining the character point data, this solution determines the target bounding volume corresponding to each body part based on the target human body grid body, and then optimizes the three-dimensional coordinate data corresponding to each body part in the character point data according to each target bounding volume to obtain the optimized character point data, and finally generates and displays the virtual character model based on the optimized character point data. Through the optimization of the target bounding volume, this solution can obtain more accurate character point data, and then can obtain a virtual character model, so that the displayed virtual character model more accurately reflects the real action and posture of the target character object.

[0155] In some optional embodiments, a frame of target human body mesh is obtained, and further fine-tuning can be performed as follows:

[0156] The target human body mesh of the current frame is compared with the target human body mesh of the previous frame, specifically the coordinates of the vertices with the same labels are compared to determine the displacement of the vertices of the current frame relative to the vertices of the previous frame, and the displacement is compared with a preset threshold. If the displacement of a vertex is less than or equal to the threshold, no adjustment is made. If the displacement of a vertex is greater than the threshold, the coordinates of the vertex in the target human body mesh of the current frame are updated so that the displacement relative to the previous frame after the update is less than or equal to the threshold.

[0157] The threshold here may be based on the movement ability of each body part of the target person object and / or the real-time movement speed.

[0158] By using the above method, when generating a virtual character model in real time, the data at the previous moment can be used to correct the data at the next moment to reduce the error.

[0159] See also Figure 4 , the method of obtaining the person point data of the target person object may include:

[0160] C1, obtain the target depth image of the target person object.

[0161] C2, identifies the foreground part of the target depth image.

[0162] C3, determining the person point data of the target person object based on the foreground part.

[0163] In step C1, a depth camera may be used to photograph a target person object to obtain a target depth image of the target person object. Optionally, multiple depth cameras may be set to photograph multiple target depth images from multiple different angles at the same time to obtain more accurate person point data.

[0164] The target depth image may include a grayscale image and a color image. The color image can be used as the texture image of the aforementioned embodiment. The grayscale value of the pixel in the grayscale image can represent the depth of field distance of each point on the surface of the target person object being photographed (that is, the distance from the point to the camera), and the color image represents the color of each point. By obtaining the depth of field distance of each point in the target depth image and combining the coordinates of the point in the two-dimensional image, the three-dimensional spatial coordinates of each point in the target depth image can be calculated. The three-dimensional spatial coordinates are the three-dimensional coordinate data in the person point data. The color data corresponding to the three-dimensional coordinate data in the person point data can be obtained through the color image.

[0165] It is worth mentioning that in the entire data collection process, a single depth camera will be used to collect the surface point data of the character. The purpose is to reduce the amount of calculation of real-time data to achieve the purpose of rapid collection. At the same time, because the camera itself has a certain perspective angle, the generated point data and grid model are complete and three-dimensional from a fixed camera perspective, so it will not affect the overall collection of actual point data.

[0166] Since only the virtual character model needs to be generated in this embodiment, the background content in the target depth image is not needed when generating the virtual character model, and processing this part of the content will affect the efficiency of generating the virtual character model. Therefore, only the foreground part of the character can be selected from the target depth image in step C2, so as to perform subsequent processing in a targeted manner.

[0167] In step C2, the target depth image may be adaptively threshold segmented, and the foreground portion may be identified by maximizing the inter-class variance method.

[0168] This method assumes that the foreground and background parts in the image have different grayscale distributions, and finds the threshold that maximizes the difference between the foreground and background parts by calculating the inter-class variance.

[0169] The threshold is calculated as follows:

[0170] In the grayscale image of the target depth image, assume that there is a threshold k (in the range of 0-255) to divide all pixels of the image into two categories C1 (less than k) and C2 (greater than k), where C1 is the foreground area of ​​the person and C2 is the background area. The mean values ​​of these two categories of pixels are recorded as m1 and m2 respectively, and the mean value of the grayscale values ​​of all pixels in the entire grayscale image is recorded as mG. At the same time, the probability of pixels being classified into C1 and C2 is recorded as p1 and p2 respectively.

[0171] According to the concept of variance, the between-class variance σ² can be calculated using the following formula (13).

[0172]

[0173] Formula (13) can also be simplified to formula (14).

[0174]

[0175] For any given value of k (denoted by K i Indicates), m1, m2, p1 and p2 and K i The relationship can be expressed by the following formula (15), where K i ∈[0, 255].

[0176]

[0177] Combining formulas (14) and (15), traverse all K i , where the K with the largest inter-class variance i This is the threshold k that is determined in this embodiment to maximize the difference between the foreground part and the background part.

[0178] After determining the threshold k, the black and white mask D(x, y) of the target depth image can be determined using the following segmentation formula (16).

[0179] Use the segmentation formula to binarize:

[0180]

[0181] Where S(x, y) is the grayscale value of the pixel (x, y) in the grayscale image of the target depth image, k is the threshold for segmentation determined by the above method, is the grayscale value of the pixel (x, y) in the black and white mask obtained by the final binarization. When S(x, y) is greater than k, D(x, y) takes the value of 255, that is, the pixel is white in the black and white mask, otherwise it is black.

[0182] After obtaining a black-and-white mask that is consistent with the size of the target depth image in the above manner, the pixels with a value of 255 in the black-and-white mask can be determined as pixels of the foreground part in the color image of the target depth image, thereby identifying the foreground part of the target character object in the color image. This can greatly reduce the amount of calculation and redundancy of the system, thereby improving the efficiency of generating virtual character models.

[0183] In step C3, the person point data of the target person object can be determined based on the pixels of the foreground part in the color image of the target depth image and the depth of field distance of the foreground part in the grayscale image. The specific determination scheme can be referred to the relevant prior art and will not be described in detail.

[0184] In some optional embodiments, before executing S104, an overall bounding volume may be determined, and the character point data may be preliminarily optimized based on the overall bounding volume. The preliminarily optimized character point data may then be optimized by region according to the method of step S104.

[0185] The process of determining the overall bounding volume and optimizing based on the overall bounding volume is the same as the process of determining the target bounding volume and optimizing based on the target bounding volume and optimizing based on the target bounding volume above, except for the following differences, which will not be repeated here.

[0186] Difference 1: When determining the overall bounding volume, all vertices contained in the target human body mesh are counted to determine the maximum and minimum values ​​of the X, Y, and Z coordinates, rather than being limited to the vertices belonging to a certain target area.

[0187] Difference 2: The scaling factor and offset used to determine the overall bounding volume may be different from the target scaling factor and target offset corresponding to each target region, as shown in formulas (17) and (18).

[0188]

[0189]

[0190] Among them, BBody0 min and BBody0 max Respectively represent the minimum and maximum coordinates of the overall bounding volume, S0 and (T0 x , T0 y , T0 z ) represent the scaling factor and offset used to determine the overall bounding volume, (c0 x , c0 y , c0 z ) represents the coordinates of the center point of the overall bounding volume, depth0, width0 and height0 represent the length, width and height of the overall bounding volume respectively.

[0191] In some optional embodiments, in addition to manually configuring a fixed target coefficient, the target coefficient corresponding to the target area may also be determined in the following manner:

[0192] Obtaining item information of the target area, the item information indicating whether the body part corresponding to the target area in the target person object is wearing an item and the type of the item;

[0193] According to the item information of the target area, the target coefficient corresponding to the target area is determined.

[0194] For each target area, the item information of the target area can be obtained as follows:

[0195] The body part corresponding to the target area is identified in the target depth image, and then whether there are objects associated with the body part around the body part in the target depth image is detected, and the detection result is used as the object information of the target area.

[0196] For example, if a crown is detected on the head of the person in the target depth image, the item information in the head area is determined to be "wearing a crown"; if it is detected that the left leg is wearing a knee pad, the item information in the left leg area is determined to be "wearing knee pads"; if it is detected that the right hand is not wearing any items, the item information in the right arm area is determined to be "no items worn"; if a backpack is detected on the back, the item information in the chest and abdomen area is determined to be "wearing a backpack".

[0197] For any target area, if the object information of the target area indicates that the body part corresponding to the target area has no worn object, the pre-configured fixed target coefficient can be obtained according to the above method for subsequent determination of the target bounding volume.

[0198] If the object information of the target area indicates that there is a wearable object on the body part corresponding to the target area, the conventional size of the wearable object of this type may be determined by searching in a database or a network;

[0199] For example, searching for multiple crowns on the Internet, taking the average of the heights of these crowns to get the regular size of the crowns, searching for multiple backpacks in the database, taking the average of the thicknesses of these backpacks to get the regular size of the backpacks;

[0200] After obtaining the regular size, the pre-configured fixed target coefficient of the target area can be found, and the target offset in the target coefficient can be adjusted based on the regular size to obtain the adjusted target coefficient, and then the target bounding volume can be determined based on the adjusted target coefficient according to the above method.

[0201] As an example, if the item information of the head area is "wearing a crown", the offset T corresponding to the z-axis in the target offset can be adjusted upward based on the normal size of the crown. z .

[0202] As another example, if the item information of the chest and abdomen area is "backpack", the offset T corresponding to the y-axis in the target offset can be adjusted upward based on the regular size of the backpack. y .

[0203] In the above example, the larger the regular size is, the larger the upward adjustment can be, and the smaller the regular size is, the smaller the upward adjustment can be. Specifically, the regular size can be converted into the upward adjustment according to a preset proportional coefficient.

[0204] Optionally, if the size of the worn item can be directly identified from the target depth image, the identified size can be used instead of the above conventional size to adjust the target coefficient.

[0205] The benefits of determining the target coefficient according to the above method are:

[0206] If the target bounding volume is determined directly based on a fixed target coefficient, the target bounding volume may not accurately enclose the worn items and the body parts of the target person. When the target bounding volume is subsequently optimized, the points on the surface of the item will be moved to a position close to the surface of the human body, resulting in the inability to present the item in the final generated virtual character model or the presented item being severely deformed.

[0207] When the target person object is wearing a specific item on a body part, the above method can enable the determined target enclosing volume to accurately enclose both the worn item and the body part of the target person object, thereby solving the above problem and enabling the generated virtual character model to present an item with the correct shape.

[0208] As mentioned above, optimizing the person point data belonging to the target area specifically refers to optimizing the three-dimensional coordinate data belonging to the target area in the person point data. To this end, it is necessary to determine which three-dimensional coordinate data belongs to the target area. The method for determining the three-dimensional coordinate data belonging to the target area may include:

[0209] Recognize multiple body parts of the target person object in the foreground portion, and divide the foreground portion into multiple sub-portions corresponding to the multiple body parts based on the recognition result;

[0210] For each body part, the three-dimensional coordinate data obtained from the sub-portion corresponding to the body part is determined as belonging to the target area corresponding to the body part.

[0211] Exemplarily, the head, right arm, left arm, chest and abdomen, left leg, right leg, left foot and right foot can be identified and divided into several sub-parts in the foreground part. Then, when generating the character point data, the three-dimensional coordinate data generated according to the pixels of the head in the foreground part in the character point data is determined to belong to the head area, and the three-dimensional coordinate data generated according to the pixels of the right arm in the foreground part in the character point data is determined to belong to the right arm area, and so on.

[0212] Optionally, before generating a virtual character model of a target character object according to the optimized character point data, the method further includes:

[0213] Evaluate the optimization effect of the optimized character point data;

[0214] When the optimization result does not meet the target optimization condition, the target bounding volume corresponding to each target area is updated;

[0215] Based on the updated target bounding volume, return to execute the step of optimizing the person point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized person point data.

[0216] The optimization effect can be evaluated by:

[0217] Based on the three-dimensional coordinate data and color data contained in the optimized character point data, projection is performed onto the imaging plane of the camera used to shoot the target depth image to obtain a reconstructed image corresponding to the optimized character point data. Then the similarity between the reconstructed image and the foreground part identified from the target depth image can be calculated, and the similarity is used as the optimization effect. The higher the similarity, the better the optimization effect, and the lower the similarity, the worse the optimization effect.

[0218] The target optimization condition may be that the similarity is greater than a preset similarity threshold.

[0219] When the optimization effect does not meet the target optimization condition, the target bounding volume can be updated by updating the maximum and minimum values ​​of the X coordinate, Y coordinate and Z coordinate counted when the target bounding volume was determined last time according to a preset update amplitude, specifically, increasing the maximum value of the X coordinate, Y coordinate and Z coordinate by a preset update amplitude on the basis of the previous time, and reducing the minimum value of the X coordinate, Y coordinate and Z coordinate by a preset update amplitude on the basis of the previous time, and then determining the updated target bounding volume according to the aforementioned method based on the maximum and minimum values ​​of the updated X coordinate, Y coordinate and Z coordinate.

[0220] Through the above method, this embodiment can dynamically adjust and optimize the target bounding volume according to the effect of the obtained optimized character point data, so as to obtain more accurate optimized character point data, and then generate a more accurate virtual character model.

[0221] This embodiment also provides a device for displaying a virtual character model, see Figure 5 , the device may include the following units.

[0222] An obtaining unit 501 is used to obtain character point data of a target character object, wherein the character point data includes three-dimensional coordinate data and color data of multiple points on the surface of the target character object;

[0223] A processing unit 502 is used to process the person point data according to a human posture estimation algorithm to obtain a target human body mesh body, wherein the target human body mesh body includes a plurality of vertices representing the shape and posture of the target human object;

[0224] A determination unit 503 is used to determine a target coefficient corresponding to each target region of the target human body mesh body, and process the vertices belonging to the target region according to the target coefficient to obtain a target bounding volume corresponding to the target region, wherein each target region corresponds to a body part of the target human object;

[0225] An optimization unit 504 is used to optimize the three-dimensional coordinate data belonging to the target area in the person point data according to the target bounding volume of the target area to obtain optimized person point data;

[0226] The display unit 505 is used to generate a virtual character model of the target character object according to the optimized character point data, and display the virtual character model in the target virtual scene.

[0227] Optionally, when the determination unit 503 determines the target coefficient corresponding to the target area, it is used to:

[0228] Obtaining item information of the target area, the item information indicating whether the body part corresponding to the target area in the target person object is wearing an item and the type of the item;

[0229] According to the item information of the target area, the target coefficient corresponding to the target area is determined.

[0230] Optionally, the obtaining unit 501 obtains the person point data of the target person object, including:

[0231] Obtain a target depth image of a target person object;

[0232] Identify the foreground part of the target depth image;

[0233] Determine the person point data of the target person object based on the foreground part.

[0234] Optionally, the person point data belonging to the target area is optimized, including:

[0235] Optimize the three-dimensional coordinate data belonging to the target area in the character point data;

[0236] When the optimization unit 504 determines the three-dimensional coordinate data belonging to the target area, it can be used to:

[0237] Recognize multiple body parts of the target person object in the foreground portion, and divide the foreground portion into multiple sub-portions corresponding to the multiple body parts based on the recognition result;

[0238] For each body part, the three-dimensional coordinate data obtained from the sub-portion corresponding to the body part is determined as belonging to the target area corresponding to the body part.

[0239] Optionally, when the optimization unit 504 optimizes the person point data belonging to the target area according to the target bounding volume of the target area, it can be used to:

[0240] Determine whether each three-dimensional coordinate data belonging to the target area is within the target bounding volume corresponding to the target area;

[0241] For the three-dimensional coordinate data that is not within the target bounding volume corresponding to the target area, the three-dimensional coordinate data is updated based on the target bounding volume to obtain updated three-dimensional coordinate data that is within the target bounding volume corresponding to the target area.

[0242] Optionally, when the display unit 505 generates a virtual character model of the target character object according to the optimized character point data, it can be used to:

[0243] Generate a character mesh model based on the optimized character point data;

[0244] Perform image texture mapping on the character mesh model to obtain a virtual character model of the target character object.

[0245] Optionally, the optimization unit 504 is further configured to:

[0246] Evaluate the optimization effect of the optimized character point data;

[0247] When the optimization result does not meet the target optimization condition, the target bounding volume corresponding to each target area is updated;

[0248] Based on the updated target bounding volume, return to execute the step of optimizing the person point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized person point data.

[0249] The working principle of the device for displaying a virtual character model provided in this embodiment can be found in the relevant steps of the method for displaying a virtual character model provided in the embodiment of the present application, and will not be described in detail.

[0250] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0251] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0252] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0253] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0254] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for displaying a virtual character model, characterized in that: include: Obtaining character point data of a target character object, wherein the character point data includes three-dimensional coordinate data and color data of a plurality of points on the surface of the target character object; Processing the person point data according to a human posture estimation algorithm to obtain a target human body mesh body, wherein the target human body mesh body includes a plurality of vertices representing the shape and posture of the target person object; For each target area of ​​the target human mesh body, determine a target coefficient corresponding to the target area, and process the vertices belonging to the target area according to the target coefficient to obtain a target bounding volume corresponding to the target area, wherein each target area corresponds to a body part of the target human object; According to the target bounding volume of the target area, optimizing the person point data belonging to the target area to obtain optimized person point data; A virtual character model of the target character object is generated according to the optimized character point data, and the virtual character model is displayed in a target virtual scene.

2. The method according to claim 1, characterized in that: The determining of the target coefficient corresponding to the target area includes: Obtaining item information of the target area, wherein the item information indicates whether a body part of the target person object corresponding to the target area has an item worn and the type of the item worn; A target coefficient corresponding to the target area is determined according to the item information of the target area.

3. The method according to claim 1, characterized in that The step of obtaining the person point data of the target person object includes: Obtaining a target depth image of the target person object; Identifying a foreground portion of the target depth image; The person point data of the target person object is determined according to the foreground portion.

4. The method according to claim 3, characterized in that The optimizing of the person point data belonging to the target area includes: Optimizing the three-dimensional coordinate data belonging to the target area in the person point data; The method for determining the three-dimensional coordinate data belonging to the target area includes: Identifying multiple body parts of the target person object in the foreground portion, and dividing the foreground portion into multiple sub-portions corresponding to the multiple body parts based on the identification result; For each of the body parts, the three-dimensional coordinate data obtained according to the sub-portion corresponding to the body part is determined as belonging to the target area corresponding to the body part.

5. The method according to claim 1, characterized in that The optimizing the person point data belonging to the target area according to the target bounding volume of the target area includes: Determine whether each three-dimensional coordinate data belonging to the target area is within the target bounding volume corresponding to the target area; For the three-dimensional coordinate data that is not within the target bounding volume corresponding to the target area, the three-dimensional coordinate data is updated based on the target bounding volume to obtain updated three-dimensional coordinate data that is within the target bounding volume corresponding to the target area.

6. The method according to claim 1, characterized in that The step of generating a virtual character model of the target character object according to the optimized character point data comprises: Generate a character mesh model according to the optimized character point data; Image texture mapping is performed on the character mesh model to obtain a virtual character model of the target character object.

7. The method according to claim 1, characterized in that Before generating the virtual character model of the target character object according to the optimized character point data, the method further includes: Evaluate the optimization effect of the optimized character point data; When the optimization effect does not meet the target optimization condition, updating the target bounding volume corresponding to each target area; Based on the updated target bounding volume, return to execute the step of optimizing the character point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized character point data.

8. A device for displaying a virtual character model, characterized in that: include: An obtaining unit, used to obtain character point data of a target character object, wherein the character point data includes three-dimensional coordinate data and color data of a plurality of points on the surface of the target character object; A processing unit, configured to process the human point data according to a human posture estimation algorithm to obtain a target human mesh body, wherein the target human mesh body includes a plurality of vertices representing a shape and a posture of the target human object; a determination unit, for determining, for each target region of the target human mesh body, a target coefficient corresponding to the target region, and processing vertices belonging to the target region according to the target coefficient to obtain a target bounding volume corresponding to the target region, wherein each target region corresponds to a body part of the target human object; An optimization unit, configured to optimize the three-dimensional coordinate data belonging to the target area in the character point data according to the target bounding volume of the target area, so as to obtain optimized character point data; A display unit is used to generate a virtual character model of the target character object according to the optimized character point data, and display the virtual character model in a target virtual scene.

9. The device according to claim 8, characterized in that When the determination unit determines the target coefficient corresponding to the target area, it is used to: Obtaining item information of the target area, wherein the item information indicates whether a body part of the target person object corresponding to the target area has an item worn and the type of the item worn; A target coefficient corresponding to the target area is determined according to the item information of the target area.

10. The device according to claim 8, characterized in that The optimization unit is also used for: Evaluate the optimization effect of the optimized character point data; When the optimization effect does not meet the target optimization condition, updating the target bounding volume corresponding to each target area; Based on the updated target bounding volume, return to execute the step of optimizing the character point data belonging to the target area according to the target bounding volume of the target area to obtain the optimized character point data.

Images