Virtual object expression generation method, application program, device and storage medium

Abstract

The present application provides a method for generating expressions of a virtual object, an application, an electronic device and a computer-readable storage medium, the method comprising: obtaining a standard template group, a neutral expression base of a virtual object for which an expression is to be generated, and a weight sequence; adjusting the eyeball position of the neutral expression base according to the positions of each eyeball in several expression base templates to obtain several reference expression bases; adjusting the reference expression base corresponding to the eyeball position of the expression base template according to several expression base templates to obtain a new expression base group; fusing the new expression base group based on the weight sequence to obtain several initial expressions of the virtual object; adjusting the initial expression based on an interlacing loss to obtain a plurality of expressions in which the eyeball and the facial area do not overlap, the interlacing loss comprising a first parameter characterizing the degree of overlap between the eyeball and the facial area.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating an expression of a virtual object, an application program, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the fields of 3D animation, film production, virtual reality, and augmented reality, there is often a need to generate expressions for virtual characters. Traditionally, expression production requires artists to do it manually, and the expression effect of expressions requires strict skills and high time costs. For example, to generate fifty different expressions for a virtual character, the corresponding production time is as long as two to four weeks. This seriously limits the quantitative production and application scenarios of virtual object expressions.

[0003] In the related art, although there are some methods for automatically generating expressions, these automatic expressions usually only target the facial area, not the eyeballs. When adding eyeballs to the virtual objects with automatically generated expressions, there will be an eyeball interpenetration problem, that is, the eyeballs overlap with the facial area. In order to overcome the eyeball interpenetration problem, manual adjustment is currently required by artists, which is inefficient and cannot guarantee reliability. Summary of the invention

[0004] In order to overcome the problem of eye interlacing in the automatic expression generation method of virtual objects existing in the related art, the technical problems of relying on artistic adjustment, low efficiency and unreliable reliability, the present application provides a method for generating expressions of virtual objects, an application, an electronic device and a computer-readable storage medium.

[0005] According to a first aspect of an embodiment of the present application, a method for generating expressions of a virtual object is provided, the method comprising: obtaining a standard template group, a neutral expression base of a virtual object whose expression is to be generated, and a weight sequence, the standard template group comprising a plurality of expression base templates with different expressions and a neutral expression base template, the expression base templates in the standard template group and the neutral expression base of the virtual object both having eyeballs; adjusting the eyeball positions of the neutral expression bases according to the positions of the respective eyeballs in the plurality of expression base templates to obtain a plurality of reference expression bases; adjusting the reference expression bases corresponding to the eyeball positions of the expression base templates according to the plurality of expression base templates to obtain a new expression base group; fusing the new expression base group based on the weight sequence to obtain a plurality of initial expressions of the virtual object; adjusting the initial expressions based on an interlacing loss to obtain a plurality of expressions in which the eyeballs do not overlap with the facial area, the interlacing loss comprising a first parameter characterizing the degree of overlap between the eyeballs and the facial area.

[0006] According to a second aspect of an embodiment of the present application, an application is provided, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method described in the first aspect of the embodiment of the present application is implemented.

[0007] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect of the embodiment of the present application when executing the program.

[0008] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect of the embodiment of the present application is implemented.

[0009] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0010] In the expression generation method of the virtual object provided in the embodiment of the present application, based on the premise that the position of the eyeball center of different objects is irrelevant to the expression, but the position relative to the skull is approximately unchanged, the eyeball position of the neutral expression base of the virtual object is adjusted according to the eyeball position of multiple expression base templates in the standard template group to obtain several reference expression bases. Then, based on the adjusted eyeball position, the reference expression base is adjusted according to the expression base template in the standard template group to obtain the expression base group of the virtual corresponding. After the expression base group of the virtual object is obtained, the expression base group of the virtual object is fused based on the weight sequence to obtain several initial expressions of the virtual object. Finally, the first parameter containing the degree of overlap between the eyeball and the facial area is used as the interlacing loss, and the initial expression is adjusted until the interlacing loss meets the preset condition. It can be seen that since the interlacing loss is set to adjust the initial expression of the virtual object, it can overcome the eyeball interlacing problem existing in the related art while automatically generating expressions for the virtual object, thereby avoiding the need for artist intervention in the later stage, which is time-consuming and labor-intensive.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0013] Figure 1 It is a schematic diagram of a principle of obtaining expressions for a virtual object according to an exemplary embodiment of the present application.

[0014] Figure 2A It is a schematic diagram of a standard template group shown in the present application according to an exemplary embodiment.

[0015] Figure 2B It is a schematic diagram of a set of expression bases generated based on a standard template group according to an exemplary embodiment of the present application.

[0016] Figure 3 It is a partial schematic diagram showing an interlacing problem between an eyeball and a facial area according to an exemplary embodiment of the present application.

[0017] Figure 4 It is a flowchart of a method for generating an expression of a virtual object according to an exemplary embodiment of the present application.

[0018] Figure 5 It is a schematic diagram of a specific process of obtaining several reference expression bases according to an exemplary embodiment of the present application.

[0019] Figure 6 It is a schematic diagram of a flow chart of determining a rotation amount according to an exemplary embodiment of the present application.

[0020] Figure 7 It is a schematic diagram showing a principle of determining a rotation amount according to an exemplary embodiment of the present application.

[0021] Figure 8 This is a result of adjusting the eye position of a neutral expression base according to an exemplary embodiment of the present application.

[0022] Fig. 9 It is a schematic diagram showing the principle of adjusting a reference expression base according to an exemplary embodiment of the present application.

[0023] Fig.10 It is a schematic diagram of a process of adjusting a reference expression base according to an exemplary embodiment of the present application.

[0024] Fig.11 It is a flow chart of determining a first parameter according to an exemplary embodiment of the present application.

[0025] Fig.12 It is a schematic diagram of a principle of determining a first parameter according to an exemplary embodiment of the present application.

[0026] Fig.13 This is another flowchart of adjusting the initial expression according to an exemplary embodiment of the present application.

[0027] Fig.14It is a structural schematic diagram of an electronic device for generating expressions for a virtual object according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0029] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0031] In the fields of 3D animation, filmmaking, virtual reality, and augmented reality, there is usually a need to generate different expressions for virtual objects. Facial deformation (Blend shape) is a commonly used method for facial expression animation synthesis. The core principle of this method comes from the Facial Action Coding System, which believes that human expressions can be divided into local meta-expressions, that is, each expression can be formed by the linear superposition of several local meta-expressions. Therefore, this method linearly fuses a set of expression bases that can cover most of the expression details, and interpolates to obtain a new expression. It can be expressed in mathematical expressions as:

[0032]

[0033] Among them, f is a new expression; b 0 The neutral expression base is the natural expression state, without any expression such as "happiness", "anger", "sorrow" etc.; bk For predefined expression bases with expressions, different b k Represents different expressions; w k is a weight coefficient used to weight other expression bases b with different expressions based on the neutral expression base. k With the neutral expression base b 0 The difference is interpolated and fused as in the above formula (1), thereby obtaining a new expression for the virtual object.

[0034] Based on the above analysis, it can be seen that for a virtual object, a new expression can be obtained based on multiple expression bases including a neutral expression base and a weight coefficient. The corresponding principle diagram is as follows: Figure 1 As shown, where f, b k and w k However, if multiple expression bases are established for each virtual object to obtain multiple different expressions of the virtual object, then multiple expression bases need to be established for each virtual object by an artist, which is time-consuming and laborious.

[0035] In the related art, there is a method for automatically generating different expressions for a new character based on a standard template group of an existing character and a neutral expression base of another new character, wherein the standard template group includes a plurality of expression base templates, and the plurality of expression base templates include a neutral expression base template and an expression base template with different expressions. The technical means adopted are: the expression actions of the expression base included in the standard template group of the existing character are transferred to the new character, thereby obtaining a plurality of expression bases of the new character, and then, based on the plurality of expression bases of the new character and a weight sequence, a new expression of the new character is obtained.

[0036] Combination Figure 2A and Figure 2B Provide explanation. Figure 2A It is a standard template group for existing characters, including a neutral expression base template 201 and a plurality of expression base templates with different expressions. Figure 2B It is a set of expression bases generated for the new character based on the neutral expression base 203 of a new character and the standard template group of an existing character. 202 and 203 are a pair of expression base templates and expression bases with corresponding expressions. The expression actions of the expression base templates included in the standard template group of the existing character are transferred to the new character in the following way: first determine the deformation amount of the expression base template with expression in the standard template group, such as 202, relative to the neutral expression base template 201, and then perform corresponding deformation on the neutral expression base of the new character, and then obtain the expression base with corresponding expression such as 204.

[0037] In the related art, this method of directly migrating the facial expressions and movements of the facial expression bases contained in the standard template group of the existing characters to the new character, thereby obtaining several facial expression bases of the new character, can obtain the facial expression bases of the new character. However, different characters have different identities and forms. In addition, the amplitude of the same facial expression of different characters is also different. Therefore, the direct migration of facial expressions will cause the facial expression base and eyeballs of the new character to be interlaced to varying degrees, that is, the eyeballs will overlap with the facial area to varying degrees. Even if there is no interlacing between the facial expression base and the eyeballs of the new character, when different expressions are subsequently generated based on formula (1) using the facial expression base and weight coefficient of the new character, there is a high probability that the above-mentioned interlacing problem will exist between the different expressions generated. Figure 3 As shown, a local schematic diagram of the problem of interlacing between the eyeball and the facial area is given.

[0038] In order to overcome the above-mentioned interleaving problem, the related art adopts the method of manual adjustment by artists for expressions with interleaving problems to eliminate the interleaving problem. Since manual intervention by artists is required, the problem of time-consuming and labor-intensive generation of expressions for new characters still exists.

[0039] In order to overcome the above-mentioned defects of the related art, and based on the standard template group, the neutral expression base of the virtual object whose expression is to be generated, and the weight sequence, an expression without eye penetration phenomenon is generated for the virtual object, and an embodiment of the present application provides an expression generation method for a virtual object. The expression generation method is based on such an objective fact: according to physiological knowledge, the position of the center of the eyeball relative to the skull of different objects is approximately unchanged even if they have different postures, different sizes, etc. Therefore, when the expression of one object is transferred to another object, the position of the center of the eyeball can actually be used as a reference to adjust the expressions of different objects, so that there is no eye penetration phenomenon between the eyeball and the facial area of ​​the generated expression.

[0040] like Figure 4 As shown, the expression generation method of the virtual object provided in the embodiment of the present application includes:

[0041] Step 401, obtaining a standard template group, a neutral expression base of a virtual object whose expression is to be generated, and a weight sequence;

[0042] Step 402, adjusting the eyeball position of the neutral expression base according to the positions of each eyeball in the plurality of expression base templates to obtain a plurality of reference expression bases;

[0043] Step 403, adjusting the reference expression base corresponding to the eye position of the expression base template according to the plurality of expression base templates to obtain a new expression base group;

[0044] Step 404, based on the weight sequence, the new expression basis set is merged to obtain a number of initial expressions of the virtual object;

[0045] Step 405: Based on the interlacing loss, the initial expression is adjusted to obtain multiple expressions in which the eyeballs and the facial area do not overlap, and the interlacing loss includes a first parameter that characterizes the degree of overlap between the eyeballs and the facial area.

[0046] The standard template group includes a number of expression base templates with different expressions and a neutral expression base template, and the expression base templates with different expressions and the neutral expression base template all have eyeballs, and the positions of the eyeballs correspond to their respective expressions. Figure 2A The series of expression base templates shown in FIG. 1, when having eyeballs corresponding to respective expressions, are the standard template group described in the embodiment of the present application. The standard template group can be obtained through art production, or can be obtained from other open source websites, or can be obtained from Figure 2A The standard template group without eyeballs shown is obtained by manually adding eyeballs and adjusting them in the later stage by an artist, and the embodiment of the present application does not impose any limitation on this.

[0047] The neutral expression base of the virtual object whose expression is to be generated also has eyeballs. Figure 2B The neutral expression base of the new character shown. The neutral expression base with eyes of the virtual object whose expression is to be generated can be a neutral expression base of the virtual object obtained from other places, obtained by adding eyes later by an artist, or a neutral expression base of the virtual object with eyes directly obtained from other places, or produced by an artist. The embodiment of the present application does not limit the method of obtaining the neutral expression base with eyes of the virtual object whose expression is to be generated.

[0048] The weight sequence can be obtained in a variety of ways. Optionally, the weight sequence can be obtained by capturing the expressions of objects in an animation video based on an animation capture device, such as MotionAnalysis, Polhemus, Sega Interactive, MAC, X-Ist, FilmBox, NOKOV, etc. The weight sequence can also be determined from a series of expressions of existing characters manually created by an artist. It can also be determined by other methods, which are not limited in the embodiments of the present application.

[0049] Through the above embodiments, it can be seen that, based on the premise that the position of the eyeball center of different objects is irrelevant to the expression, but the position relative to the skull is approximately unchanged, the eyeball position of the neutral expression base of the virtual object is adjusted according to the eyeball position of multiple expression base templates in the standard template group to obtain several reference expression bases. Then, based on the adjusted eyeball position, the reference expression base is adjusted according to the expression base template in the standard template group to obtain the expression base group of the virtual corresponding. After the expression base group of the virtual object is obtained, the expression base group of the virtual object is fused based on the weight sequence to obtain several initial expressions of the virtual object. Finally, the first parameter containing the degree of overlap between the eyeball and the facial area is used as the interlacing loss to adjust the initial expression until the interlacing loss meets the preset conditions. It can be seen that since the interlacing loss is set to adjust the initial expression of the virtual object, it can overcome the eyeball interlacing problem existing in the related art while automatically generating expressions for the virtual object, thereby avoiding the need for artist intervention in the later stage, which is time-consuming and labor-intensive.

[0050] In some embodiments, step 402 adjusts the eye positions of the neutral expression base according to the positions of the eyeballs in the expression base templates to obtain a plurality of reference expression bases, such as Figure 5 As shown, including:

[0051] Step 501, determining the rotation amount of the eyeball of each expression base template with expression relative to the eyeball of the neutral expression base template;

[0052] Step 502: Based on the rotation amount, the eye position of the neutral expression base is adjusted to obtain a reference expression base in which the eye positions of the virtual object and the expression base template with expression are consistent.

[0053] The expression generation method of the virtual object provided in the embodiment of the present application essentially uses the eyeballs to constrain the expression migration, thereby adjusting the expression migrated from the standard template group. Therefore, it is first necessary to adjust the eye position of the neutral expression base of the virtual object according to the eye position of the expression base template with different expressions in the standard template group, so as to obtain several benchmark expression bases of the virtual object.

[0054] As mentioned above, based on physiological knowledge, the rotation center of the eyeball is approximately fixed relative to the skull. Therefore, for an object with an immobile or nearly immobile head, the movement of the eyeball can be regarded as only a rotational transformation. On the other hand, based on the theory of the facial action coding system described above, there are only expression differences between all expression bases, and there is no intrinsic skull movement, which is called rigid stablilization. Therefore, it can be considered that the eyeball center in the expression base with different expressions of the virtual object is consistent with the eyeball center in the neutral expression base, so the eyeball of the neutral expression base of the virtual object can be copied to the expression base with different expressions of the virtual object. Then, the rotation amount of the eyeball of the expression base template with different expressions in the standard template group relative to the neutral expression base template is transferred to the neutral expression base of the virtual object, that is, based on the rotation amount of the eyeball of each expression base template with expression relative to the eyeball of the neutral expression base template, the eyeball position of the neutral expression base is adjusted, thereby obtaining multiple reference expression bases of the virtual object. In each reference expression base, the facial expression of the reference expression base is still the same as the neutral expression base of the virtual object, but the eyeball position is consistent with the eyeball position of an expression base template in the standard template. With the reference expression base, and the eyeball position of the reference expression base is a reference, the expression bases with different expressions can be adjusted based on the eyeball position in the reference expression base to avoid the eyeball interlacing phenomenon, which will be described in detail later.

[0055] For step 501, the rotation amount of the eyeball of each expression base template with expression relative to the eyeball of the neutral expression base template can be determined in a variety of ways, and the present application embodiment does not limit this.

[0056] In some embodiments, the determination of the rotation amount can be as follows: Figure 6 As shown, this is achieved by:

[0057] Step 601, determining second key points corresponding to the plurality of first key points on each of the expression base templates having expressions according to the plurality of first key points on the neutral expression base template that are not in the same plane as the eyeball center;

[0058] Step 602, determining the rotation amount based on the rigid transformation relationship between the first matrix and the second matrix, wherein the first matrix includes the vector formed by each of the first key points and the eye center of the neutral expression base template, and the second matrix includes the vector formed by each of the second key points and the eye center of the expression base template with expression.

[0059] Since for the same object, the expression bases with different expressions have only rotational transformations relative to the neutral expression base. To determine the rotation amount corresponding to the rotational transformation, multiple key points that are no longer in the same plane as the eyeball center can be determined, and they and the eyeball center form a polyhedron. During the rotation process, this polyhedron has only rigid transformations, so the rotation amount can be obtained through the vector determined based on the relationship between the edges and angles of this polyhedron.

[0060] Combination Figure 7 Taking the determination of three key points A, B and C on the neutral expression base template that are not in the same plane as the eyeball center as an example, the determination of the rotation amount of the expression base template with expression relative to the neutral expression base template is introduced. Figure 7 As shown, after three key points A, B and C that are not in the same plane as the center of the eyeball are determined on the neutral expression base template, they can form a tetrahedron ABCO with the center of the eyeball. During the rotation of the eyeball due to changes in expression, the tetrahedron ABCO has only rigid transformation. Therefore, the rotation amount can be directly calculated based on the vectors AO, BO, CO, etc. determined by the tetrahedron through formula (2).

[0061] R[v 0 , v 1 , v 2 ]=[u 0 ,u 1 ,u 2 ] (2)

[0062] Among them, v i u is the vector from the center of the eyeball to the three key points on the neutral expression base template; i is the vector from the center of the eyeball to the corresponding three key points on the expression base template with expression; R is the amount of rotation that the eyeball needs to rotate when the expression of the object changes from the neutral expression base template to the expression base template with expression. The multiple vectors v formed by the key points on the neutral expression base template 0 , v 1 , v 2 , forming the first matrix, that is, [v 0 , v 1 , v 2 ], a plurality of vectors u formed by the key points on the expression base template having an expression 0 ,u 1 ,u 2 , forming the second matrix, that is, [u 0 ,u 1 ,u 2 ].

[0063] Therefore, based on the above formula (2), after determining the three key points A, B and C that are not in the same plane as the center of the eyeball on the neutral expression base template, the corresponding key points A', B' and C' can be determined on the expression base template with expression, and then the rotation amount R can be determined based on the vector formed by these key points and the center of the eyeball.

[0064] Of course, the number of the key points may be multiple, and accordingly, the vectors used to determine the rotation amount may be multiple, which is not limited in the embodiment of the present application. In addition, the description of the first matrix and the second matrix is ​​only based on the example that the number of the key points is 3 and the number of the vectors is 3. When the number of the key points is multiple, the first matrix and the second matrix may be composed of multiple vectors, and the composition method is similar to the above embodiment, which is not described in detail in the embodiment of the present application.

[0065] In addition, since eye movement is a rigid transformation and can be determined according to the semantics of the expression base, the corresponding relationship between the expression base and the rotation amount can also be determined by enumeration. Then, the semantics of the expression base of the virtual object is parsed, and then the rotation amount is determined according to the determined corresponding relationship between the expression base and the rotation amount.

[0066] Of course, with the continuous development and progress of deep learning technology, the determination of the rotation amount can also be obtained based on a trained deep learning network, and the embodiments of the present application do not limit this.

[0067] like Figure 8 As shown, step 501 is given, and the result of adjusting the eyeball position of the neutral expression base template based on the rotation amount of the eyeball of each expression base template with expression relative to the eyeball of the neutral expression base template is given. Among them, A is the eyeball of the neutral expression base template in the standard template group, and B is the eyeball of a certain expression base template with expression in the standard template group. Based on the rotation amount of B relative to the eyeball of A, the eyeball position of the neutral expression base C of the virtual object is adjusted, and the eyeball D of the benchmark expression base consistent with the eyeball position of B can be obtained.

[0068] After the reference expression base is obtained, based on the existing standard template group and the neutral expression base of the virtual object, multiple methods can be used to generate expression bases with eyeballs and different expressions for the virtual object.

[0069] In some embodiments, the expression base of the virtual object can be obtained based on the concept of deformation transfer. The concept of deformation transfer is often used to transfer the actions in an animation sequence of a character (source) to a new character (target), which greatly reduces the cost and time of animation production. Usually an animated character has a static state, that is, a state without action, and then each action can be considered as a static state obtained through deformation. Deformation transfer first measures the deformation variables of the state with action and the static state, and then applies this deformation variable to the new character to generate a new action sequence. Fig. 9 The figure shows a schematic diagram of the principle of obtaining a new action sequence of a character horse based on transformation transfer. The deformation of a horse S' with different action forms relative to a horse S in a static state can be transferred to another horse T in a static state, and the action sequence corresponding to the horse in a static state and S' can be obtained.

[0070] Naturally, the above-mentioned idea of ​​deformation transfer can also be directly used to generate facial expressions. Therefore, in some embodiments, step 403 can be performed as follows Fig. 9 The method described is implemented as follows: Fig.10 As shown, the method includes:

[0071] Step 1001, determining the deformation amount of each expression base template relative to the neutral expression base template;

[0072] Step 1002: adjusting the reference expression base corresponding to the eye position of the expression base template according to the deformation amount.

[0073] Of course, in addition to the above-mentioned idea based on deformation migration, the standard template group and the neutral expression base with eyeballs and the benchmark expression base of the virtual object are used to obtain the expression base with corresponding adjustments of expressions. It is also possible to obtain the expression, or the expression base with corresponding adjustments of the facial area, through many methods based on deep learning models, semantic recognition, etc. The embodiments of the present application do not limit this.

[0074] By adjusting the expression of the reference expression base, a new expression base group can be obtained. Then, based on the formula (1) described above and the obtained weight sequence, the new expression base group can be fused to obtain multiple expressions for the virtual object.

[0075] However, since in step 403, the new expression base is obtained by adjusting several expression base templates in the standard template group, there may be an eye interlacing problem when generating expressions for the virtual object based on the new expression base, due to the large differences in form and appearance between the virtual object and the object corresponding to the standard template group.

[0076] As mentioned above, in the reference expression base, the eyeball position is adjusted based on the standard template group, and the rotation amount of the eyeball position is the same for different objects under the same expression. Therefore, the position of the eyeball in the reference expression base can be used as a spatial constraint. In the process of generating the expression of the virtual object, the quality of expression migration and fusion can be detected by using the eyeball to control the final effect, so that the expression finally generated for the virtual object does not have the situation of eyeball interlacing. That is, through step 404, based on the interlacing loss including the first parameter characterizing the degree of overlap between the eyeball and the facial area, the initial expression obtained in step 403 is adjusted to obtain multiple expressions in which the eyeball and the facial area do not overlap. Among them, the first parameter can be determined in a variety of ways, as long as it can characterize the degree of overlap between the eyeball and the facial area, and the embodiment of the present application is not limited here.

[0077] In some embodiments, the first parameter characterizing the degree of overlap between the eyeball and the facial area can be as follows: Fig.11 As shown, it is determined by:

[0078] Step 1101, determining a plurality of intersection points of a line connecting the eyeball center of the initial expression and a plurality of preset points and a facial area of ​​the initial expression, wherein the preset points are located on the eyeball surface;

[0079] Step 1102: taking the distance difference between the distances between the multiple intersection points and the center of the eyeball and the distance between the corresponding preset point and the center of the eyeball as the first parameter.

[0080] Combination Fig.12 The schematic diagram is used to illustrate this. Fig.12 As shown, a partial schematic diagram of a virtual object is given, and the partial part includes the eyeball 1201 of the virtual object and the facial area 1202 around the eyeball. In the expression generated by the virtual object, the degree of overlap between the eyeball and the facial area can be determined by checking whether the points in the facial area appear inside the eyeball. Whether the points in the facial area appear inside the eyeball can be determined by the distance difference. That is, some preset points can be pre-set on the eyeball surface, and the preset points from the eyeball center to the eyeball surface are connected to form a ray emitted from the eyeball center, and the intersection with the facial area is found. Fig.12In the figure, M is a preset point on the eyeball surface. Connecting the eyeball center O to M, the ray OM and the facial area 1202 will have an intersection point N. In this way, the distance between the intersection and the eyeball center and the distance difference between the corresponding preset point and the eyeball center can be judged to determine whether the eyeball is interpenetrating. If the distance difference between ON and OM is greater than 0, it means that the distance between the intersection and the eyeball center is greater than the distance between the preset point and the eyeball center, that is, the intersection is not located inside the eyeball; if the distance difference between ON and OM is less than 0, it means that the distance between the intersection and the eyeball center is less than the distance between the preset point and the eyeball center, that is, the intersection is located inside the eyeball, and there is an eyeball interpenetration phenomenon.

[0081] Fig.12 Only one preset point is preset on the eyeball surface. Of course, those skilled in the art should understand that the case of presetting multiple preset points on the eyeball surface is similar to the case of presetting one preset point. In step 1102, the distance difference between the distances between the multiple intersection points and the eyeball center and the distance between the corresponding preset point and the eyeball center is used as the first parameter. The distance difference corresponding to each intersection point can be summed as the first parameter. That is, a preset point p ij The corresponding eye center O 00 To preset point p ij The distance is l ij , corresponding to the eye center O 00 To the intersection point v ij The distance is l′ ij , let the first parameter be Pen_loss, then the first parameter is:

[0082] Pen_loss = sum{l′ ij -l ij}=sum{v ij -p ij}; (3)

[0083] The first parameter may also be a distance difference corresponding to each intersection point as an element of a matrix, and the first parameter of the entire matrix is ​​[v 11 -p 11 , v 21 -p 21 ,...v ij -p ij ,...];etc.

[0084] In the above embodiment, it can be noted that the v ij is a point on the facial area, so the initial expression is adjusted based on the first parameter. In fact, by adjusting v ijposition, so that there is no interlacing between the eyeball and the facial area of ​​the expression.

[0085] Of course, those skilled in the art should understand that the first parameter characterizing the degree of overlap between the eyeball and the facial area can also be determined by other means. For example, the eyelid area around the eyeball is relatively flat. When there is an interlacing phenomenon between the eyeball and the area around the eyeball, there will be a bulge in the area. Then, based on the topological relationship contained in the expression base, a three-dimensional facial reconstruction can be performed on the area around the eyeball, and based on the three-dimensional reconstruction, the depth difference or height difference of the key points of the area around the eyeball is determined to determine the degree of overlap between the eyeball and the facial area. The embodiment of the present application does not limit the method for determining the first parameter characterizing the degree of overlap between the eyeball and the facial area.

[0086] Through the above embodiments, it can be seen that in the expression generation method of the virtual object provided in the embodiment of the present application, based on the premise that the position of the eyeball center of different objects is irrelevant to the expression, but the position relative to the skull is approximately unchanged, the eyeball position of the neutral expression base of the virtual object is adjusted according to the eyeball position of multiple expression base templates in the standard template group to obtain several reference expression bases. Then, based on the adjusted eyeball position, the reference expression base is adjusted according to the expression base template in the standard template group to obtain the virtual corresponding expression base group. After the expression base group of the virtual object is obtained, the expression base group of the virtual object is fused based on the weight sequence to obtain several initial expressions of the virtual object. Finally, the first parameter containing the degree of overlap between the eyeball and the facial area is used as the interlacing loss to adjust the initial expression until the interlacing loss meets the preset condition. It can be seen that since the interlacing loss is set to adjust the initial expression of the virtual object, it can overcome the eyeball interlacing problem existing in the related art while automatically generating expressions for the virtual object, thereby avoiding the need for artist intervention in the later stage, which is time-consuming and labor-intensive.

[0087] In some embodiments, in step 405, before adjusting the initial expression based on the interlacing loss to obtain multiple expressions in which the eyeballs and the facial area do not overlap, the method may also be as follows: Fig.13 As shown, including:

[0088] Step 1301, based on the numerical range of the first parameter, determining whether the eyeballs and the facial area of ​​the initial expression overlap;

[0089] Step 1302: When the eyeballs and the facial area of ​​the initial expression overlap, the initial expression is adjusted.

[0090] As mentioned above, the first parameter characterizes the degree of overlap between the eyeball and the facial area. The method for determining the first parameter can be as described above, and the embodiments of the present application will not be described in detail here. In step 404, based on the weight sequence, the new expression basis set is fused to obtain several initial expressions of the virtual object. It is also possible that there is no eye interlacing phenomenon itself. In this case, there is no need to adjust the initial expression based on the interlacing loss. Therefore, before step 405, it is possible to first adjust based on the numerical range of the first parameter characterizing the degree of overlap between the eyeball and the facial area, when it is determined that the eyeball and the facial area of ​​the initial expression overlap. It can effectively save computing resources and improve the applicability of the method described in the embodiments of the present application.

[0091] The interlacing loss described in each of the above embodiments includes a first parameter that characterizes the degree of overlap between the eyeball and the facial area. When the interlacing loss only includes the first parameter, then, based on the interlacing loss, the adjustment of the initial expression is essentially to adjust the initial expression only with the degree of interlacing between the eyeball and the facial area as a constraint. However, this adjustment may cause the deformation of the initial expression after adjustment to be particularly large compared with the initial expression before adjustment in order to avoid eyeball interlacing, and the initial expression before adjustment is obtained by fusion of a new expression base group based on the virtual object, and the new expression base group is obtained based on several expression base templates of the standard template group, that is, the initial expression before adjustment is obtained by migrating the standard expression base of the virtual template group. Therefore, the final result of adjusting the initial expression based on the interlacing loss that only includes the first parameter may be: in order to obtain an expression without interlacing, the adjusted expression is severely deformed relative to the initial expression before adjustment, and seriously deviates from the constraints of the standard template group.

[0092] In order to overcome the constraint that the interleaving loss based on only the first parameter may cause the adjusted expression to be severely deformed relative to the initial expression before adjustment, and seriously deviate from the standard template group, in some embodiments, the interleaving loss may include not only the first parameter, but also a second parameter characterizing the degree of deformation of the initial expression before and after adjustment.

[0093] When the interlacing loss includes not only the first parameter characterizing the degree of overlap between the eyeball and the facial area, but also the second parameter characterizing the degree of deformation of the initial expression before and after adjustment, it essentially takes the degree of overlap between the eyeball and the facial area as a constraint condition. At the same time, it also ensures that the adjustment of the initial expression is also constrained by the deformation of the standard template group, thereby ensuring that the expression base group generated for the virtual object has similar semantics to the corresponding expression base template in the standard template group, and will not be over-stretched by the first parameter, resulting in abnormal deformation that deviates from the constraints of the standard template group.

[0094] Optionally, the second parameter of the degree of deformation of the standard initial expression before and after adjustment can be quantified by the coordinate difference of the key point before and after deformation, that is, the second parameter can be expressed as:

[0095] Dt reg =||Tv′-S|| (4)

[0096] Among them, Dt ref It represents the second parameter characterizing the deformation degree of the initial expression before and after adjustment, Tv′ represents the coordinates of the key points after the adjustment of the initial expression, S represents the coordinates of the corresponding key points before the adjustment of the initial expression, and “||x||” represents the norm of x.

[0097] Therefore, when the interpenetration loss is E, the interpenetration loss can be:

[0098] E=Pen loss +Dt_reg (5)

[0099] Of course, corresponding to the interlacing loss including the first parameter and the second parameter as shown in formula (5), different weights can also be added to the first parameter and the second parameter to adjust the emphasis of the interlacing loss and the degree of deformation in the process of adjusting the initial expression.

[0100] Optionally, a weight β may be added before the second parameter, and then the interleaving loss may be expressed as:

[0101] E=Pen loss +βDt_reg (6)

[0102] The weight β can be a fixed constant, and can be set to decrease linearly as the number of adjustments increases, for example:

[0103]

[0104] Among them, β max , Δβ,∈ and∈ max It can be predetermined based on experience or experimental values.

[0105] Of course, a weight may be added before the first parameter, or the same or different weights may be added before the first parameter and the second parameter at the same time, and the embodiment of the present application does not limit this.

[0106] It can be seen from the above embodiments that when the interlacing loss includes not only the first parameter characterizing the degree of overlap between the eyeball and the facial area, but also the second parameter characterizing the degree of deformation of the initial expression before and after adjustment, it is possible to implement multiple constraints on the adjustment of the initial expression, and can further ensure that the expression finally obtained after iterative adjustment is natural and real and does not contain the eye interlacing phenomenon.

[0107] In step 405, the initial expression is adjusted based on the interlacing loss to obtain multiple expressions in which the eyeballs and the facial area do not overlap. The adjustment may be based on the interlacing loss determined in the above embodiments, and the position of the point on the expression is iteratively adjusted. When the position of the point on the expression changes, the value of the interlacing loss also changes. The end point of the adjustment may be when the value of the interlacing loss is less than a given threshold, or when the number of adjustments reaches a preset number, which is not limited in the embodiments of the present application.

[0108] Through the above embodiments, it can be seen that in the expression generation method of the virtual object provided in the embodiment of the present application, based on the premise that the position of the eyeball center of different objects is irrelevant to the expression, but the position relative to the skull is approximately unchanged, the eyeball position of the neutral expression base of the virtual object is adjusted according to the eyeball position of multiple expression base templates in the standard template group to obtain several reference expression bases. Then, based on the adjusted eyeball position, the reference expression base is adjusted according to the expression base template in the standard template group to obtain the virtual corresponding expression base group. After the expression base group of the virtual object is obtained, the expression base group of the virtual object is fused based on the weight sequence to obtain several initial expressions of the virtual object. Finally, the first parameter containing the degree of overlap between the eyeball and the facial area is used as the interlacing loss to adjust the initial expression until the interlacing loss meets the preset condition. It can be seen that since the interlacing loss is set to adjust the initial expression of the virtual object, it can overcome the eyeball interlacing problem existing in the related art while automatically generating expressions for the virtual object, thereby avoiding the need for artist intervention in the later stage, which is time-consuming and labor-intensive.

[0109] In addition, corresponding to the expression generation method of a virtual object provided in the above embodiment of the present application, the embodiment of the present application also provides a software product, that is, an application, the application includes a computer / instruction, and when the computer program / instruction is executed by the processor, the above-mentioned various method embodiments provided in the embodiment of the present application are implemented. For the relevant content of the application, reference can be made to the various method embodiments in the above text, and the present application will not elaborate on them here.

[0110] Corresponding to the above method embodiment, the present application also provides an electronic device, Fig.14A schematic diagram of the structure of an exemplary electronic device provided in this application.

[0111] like Fig.14 As shown, an electronic device 1400 provided in this embodiment includes: a processor 1401, a memory 1402, and a computer program stored in the memory 1402 and executable on the processor 1401, wherein the processor implements the method of the foregoing application when executing the program. The processor 1401 in the electronic device may be one or more, Fig.14 In the example, a processor 1401 is used, and the processor may include a GPU and a CPU. The processor 1401 and the memory 1402 in the electronic device may be connected via a communication bus or other means. Fig.14 The connection via the communication bus 1403 is taken as an example.

[0112] In this embodiment, the processor 1401 of the electronic device integrates the application provided in the above embodiment. In addition, the memory 1402 in the electronic device, as a computer-readable storage medium, can be used to store one or more programs, which can be software programs, computer executable programs and modules. The processor 1401 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 1402, that is, implementing the above-mentioned various method embodiments.

[0113] The memory 1402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 1402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 1402 may further include a memory remotely arranged relative to the processor 1401, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The processor 1401 executes various functional applications and data processing by running the program stored in the memory 1402 to implement the method provided in the embodiment of the present application.

[0115] In addition, the present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0116] The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CDROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0117] The signal medium of the computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The program code included in the computer-readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the application can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0119] Those skilled in the art will readily appreciate other embodiments of the present application after considering the application and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0120] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0121] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating an expression of a virtual object, characterized in that: The method comprises: Obtaining a standard template group, a neutral expression base of a virtual object whose expression is to be generated, and a weight sequence, wherein the standard template group includes a plurality of expression base templates with different expressions and a neutral expression base template, and both the expression base template in the standard template group and the neutral expression base of the virtual object have eyeballs; According to the positions of the eyeballs in the plurality of expression base templates, the eyeball positions of the neutral expression base are adjusted to obtain a plurality of reference expression bases; According to the plurality of expression base templates, adjusting the reference expression base corresponding to the eyeball position of the expression base template to obtain a new expression base group; Based on the weight sequence, the new expression basis set is fused to obtain a number of initial expressions of the virtual object; Based on the interlacing loss, the initial expression is adjusted to obtain multiple expressions in which there is no overlap between the eyeball and the facial area, and the interlacing loss includes a first parameter that characterizes the degree of overlap between the eyeball and the facial area.

2. The method according to claim 1, characterized in that: According to the positions of the eyeballs in the plurality of expression base templates, adjusting the eyeball positions of the neutral expression base template comprises: Determining the rotation amount of the eyeball of the expression base template relative to the eyeball of the neutral expression base template; Based on the rotation amount, the eye position of the neutral expression base is adjusted.

3. The method according to claim 2, characterized in that Determining the rotation amount of the eyeball of the expression base template relative to the eyeball of the neutral expression base template includes: Determine, according to a plurality of first key points on the neutral expression base template that are not in the same plane as the eyeball center, second key points on the expression base template that correspond to the plurality of first key points; The rotation amount is determined based on a rigid transformation relationship between a first matrix and a second matrix, wherein the first matrix includes a vector formed by each of the first key points and the eye center of the neutral expression base template, and the second matrix includes a vector formed by each of the second key points and the eye center of the expression base template.

4. The method according to claim 1, characterized in that The step of adjusting the expression of the reference expression base corresponding to the eye position of the expression base template according to the plurality of expression base templates comprises: Determining the deformation amount of each expression base template relative to the neutral expression base template; According to the deformation amount, the reference expression base corresponding to the eye position of the expression base template is adjusted.

5. The method according to claim 1, characterized in that: The first parameter characterizing the degree of overlap between the eyeball and the facial area is determined in the following manner: Determine the intersection of a line between the eyeball center of the initial expression and a preset point and the facial area of ​​the initial expression, wherein the preset point is located on the eyeball surface; The distance difference between the distances between the multiple intersection points and the center of the eyeball and the distance between the corresponding preset point and the center of the eyeball is used as the first parameter.

6. The method according to claim 5, characterized in that Before adjusting the initial expression, the method further includes: Based on the numerical range of the first parameter, determining whether the eyeballs and the facial area of ​​the initial expression overlap; When the eyeballs and the facial area of ​​the initial expression overlap, the initial expression is adjusted.

7. The method according to claim 1, characterized in that The interlacing loss also includes a second parameter that characterizes the degree of deformation of the initial expression before and after adjustment.

8. A computer program product, characterized in that The method comprises a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images