Control information generation method, device, electronic device and readable storage medium based on personalized expression base

Abstract

The present application provides a method, device, electronic device and readable storage medium for generating control information based on personalized expression bases, by obtaining multiple facial images of a reference object in different states, splitting the facial features contained in each facial image based on multiple preset expression base templates, and generating multiple personalized expression bases. Then, video frames of the reference object in different set expressions are obtained, and control information containing personalized expression bases corresponding to each set expression is generated and saved based on the video frames and personalized expression bases. This scheme can generate a personalized expression base reflecting the characteristics of the reference object based on the expression base template, and can obtain control information represented by the personalized expression base based on the video frames and personalized expression bases of different set expressions, which can be used for the subsequent control of the virtual image. In this way, the complexity of the device can be reduced and excessive interference can be avoided, and the obtained control information can accurately reflect each set expression.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular to a method, device, electronic device and readable storage medium for generating control information based on a personalized expression base. Background Art

[0002] As the live broadcast industry becomes more popular, more and more people are entering the live broadcast industry as anchors. The traditional live broadcast industry mainly relies on the anchor's personal talent performance to attract audiences. The anchor always broadcasts with a single image (self-image), which lacks diversity and limits many interesting live broadcast modes and gameplay. Therefore, a new form of live broadcast has recently emerged, namely virtual live broadcast, which allows anchors to drive a variety of different virtual images behind the scenes to conduct personalized live broadcasts.

[0003] There are two main ways to produce high-precision, vivid expression animations of virtual characters. One is to drive the virtual character based on the information drawn manually by animators for each frame of the image. This method is very inefficient and not suitable for mass production. Another method is to add markers to the face of the object and generate control information for driving the virtual character based on the markers. This method has the problem of complex equipment and excessive interference due to the need to add markers. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a method, device, electronic device and readable storage medium for generating control information based on a personalized expression base, which can obtain control information that can accurately reflect each expression while reducing the complexity of the device and avoiding excessive interference.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a method for generating control information based on a personalized expression base, the method comprising:

[0007] obtaining a plurality of facial images of a reference object in different states;

[0008] Based on a plurality of preset expression base templates and the plurality of facial images, facial features contained in each of the facial images are split to generate a plurality of personalized expression bases;

[0009] Obtaining video frames of the reference object in different set expressions;

[0010] According to the video frame and the personalized expression base, control information including the personalized expression base corresponding to each of the set expressions is generated, and the control information is stored.

[0011] In an optional embodiment, the method further comprises:

[0012] During the live broadcast, a control expression instruction for the constructed virtual image is obtained, and a set expression corresponding to the control expression is obtained;

[0013] Extracting the saved control information corresponding to the set expression;

[0014] The virtual image is controlled according to the control information to display the control expression.

[0015] In an optional implementation, the step of generating control information including a personalized expression base corresponding to the set expression according to the video frame and the personalized expression base includes:

[0016] After assigning base coefficients to each of the personalized expression bases, the bases are combined to obtain a combined expression base;

[0017] Comparing the video frame with the combined expression base, and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met;

[0018] Control information is generated based on the base coefficient combination and the personalized expression base.

[0019] In an optional embodiment, the video frame includes point cloud information of three-dimensional feature points;

[0020] The step of comparing the video frame with the combined expression base and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met includes:

[0021] Aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base;

[0022] The base coefficients are adjusted according to the result of the alignment operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0023] In an optional implementation, the step of aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base includes:

[0024] For each feature point in the combined expression base, determine the three-dimensional feature point corresponding to the feature point in the point cloud information of the three-dimensional feature point, and perform position alignment constraints on the corresponding feature point and the three-dimensional feature point in terms of coordinate values;

[0025] Determine the plane where the three-dimensional feature point corresponding to the feature point is located, and impose a distance constraint between the feature point and the plane;

[0026] The alignment operation result is obtained according to the results of the position alignment constraint and the distance constraint.

[0027] In an optional embodiment, the video frame includes two-dimensional image information;

[0028] The step of comparing the video frame with the combined expression base and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when the preset requirements are met includes:

[0029] Comparing the two-dimensional image information with the two-dimensional information included in the combined expression base;

[0030] The base coefficients are adjusted according to the result of the comparison operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0031] In an optional implementation, the step of comparing the two-dimensional image information with the two-dimensional information included in the combined expression base includes:

[0032] Rendering the combined expression base based on a preset texture map to obtain a comparison image;

[0033] Performing a position comparison operation on the key points in the comparison image and the corresponding key points in the two-dimensional image information in terms of coordinate values ​​and a pixel comparison operation in terms of pixel values;

[0034] Based on the results of the position comparison operation and the pixel comparison operation, a comparison operation result is obtained.

[0035] In an optional implementation manner, the video frame under the set expression includes multiple continuous video frames, and the control information corresponding to the set expression includes multiple control sub-information corresponding to each of the video frames;

[0036] The step of controlling the virtual image to display the control expression according to the control information includes:

[0037] For a plurality of continuous live video frames to be controlled during the live broadcast process, obtaining control sub-information corresponding to each live video frame in the control information;

[0038] Each of the control sub-information is used to control the virtual image in the corresponding live video frame, so that the virtual image in the continuous multiple frames of live video frames executes the display of the control expression.

[0039] In an optional embodiment, the step of splitting the facial features contained in each of the facial images based on the preset multiple expression base templates and the multiple facial images to generate multiple personalized expression bases includes:

[0040] constructing three-dimensional models in different states according to the multiple facial images;

[0041] Based on a plurality of preset expression base templates and a plurality of three-dimensional models, facial features in each of the three-dimensional models are split to generate a plurality of personalized expression bases;

[0042] The multiple personalized expression bases include a baseline expression base and multiple variable expression bases, and each variable expression base has a facial feature that is different from the corresponding facial feature in the baseline expression base compared to the baseline expression base.

[0043] In an optional implementation, the step of constructing three-dimensional models in different states according to the multiple facial images includes:

[0044] For each state, a plurality of target facial images in the state are obtained, wherein the plurality of target facial images are images taken from different viewing angles;

[0045] Three-dimensional reconstruction is performed based on the multiple target facial images, and topological processing is performed on the model obtained by the three-dimensional reconstruction to obtain the three-dimensional model in the state.

[0046] In a second aspect, the present application provides a control information generation device based on a personalized expression base, the device comprising:

[0047] A first acquisition module is used to obtain a plurality of facial images of a reference object in different states;

[0048] A splitting module, used for splitting the facial features contained in each of the facial images based on a plurality of preset expression base templates and the plurality of facial images, to generate a plurality of personalized expression bases;

[0049] A second obtaining module is used to obtain video frames of the reference object in different set expressions;

[0050] The generating module is used for generating control information including the personalized expression base corresponding to each of the set expressions according to the video frame and the personalized expression base, and storing the control information.

[0051] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0053] The beneficial effects of the embodiments of the present application include, for example:

[0054] The present application provides a method, device, electronic device and readable storage medium for generating control information based on personalized expression bases, by obtaining multiple facial images of a reference object in different states, splitting the facial features contained in each facial image based on multiple preset expression base templates, and generating multiple personalized expression bases. Then, video frames of the reference object in different set expressions are obtained, and control information containing personalized expression bases corresponding to each set expression is generated and saved based on the video frames and personalized expression bases. This scheme can generate a personalized expression base reflecting the characteristics of the reference object based on the expression base template, and can obtain control information represented by the personalized expression base based on the video frames and personalized expression bases of different set expressions, which can be used for the subsequent control of the virtual image. In this way, the complexity of the device can be reduced and excessive interference can be avoided, and the obtained control information can accurately reflect each set expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of an application scenario of the control information generation method provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of another application scenario of the control information generation method provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of a method for generating control information provided in an embodiment of the present application;

[0059] Figure 4 for Figure 3 Flow chart of sub-steps included in step S120;

[0060] Figure 5 The facial images taken from multiple perspectives provided in the embodiments of the present application;

[0061] Figure 6 A schematic diagram of a model obtained by three-dimensional reconstruction provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of a three-dimensional model obtained after topological processing provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of an expression base template provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of a personalized expression provided in an embodiment of the present application;

[0065] Fig.10 for Figure 3 Flow chart of sub-steps included in step S140;

[0066] Fig.11 A schematic diagram of a combined expression base obtained in an embodiment of the present application;

[0067] Fig.12 for Fig.10 Flow chart of sub-steps included in step S142;

[0068] Fig.13 for Fig.12 Flowchart of sub-steps included in step S1421A;

[0069] Fig.14 A schematic diagram of an alignment operation based on point cloud information provided in an embodiment of the present application;

[0070] Fig.15 for Fig.10 Another flowchart of the sub-steps included in step S142;

[0071] Fig.16 for Fig.15 Flow chart of the sub-steps included in step S1421B;

[0072] Fig.17 A schematic diagram of a comparison operation based on two-dimensional image information provided in an embodiment of the present application;

[0073] Fig.18 A flow chart of a control method in a control information generating method provided in an embodiment of the present application;

[0074] Fig.19A schematic diagram of controlling a virtual image provided in an embodiment of the present application;

[0075] Fig. 20 for Fig.18 Flow chart of sub-steps included in step S230;

[0076] Fig.21 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0077] Fig. 22 This is a functional module block diagram of a control information generation device based on a personalized expression base provided in an embodiment of the present application.

[0078] Icons: 100-server; 200-image acquisition device; 300-live broadcast provider; 400-live broadcast receiver; 110-storage medium; 120-processor; 130-control information generating device based on personalized expression base; 131-first acquisition module; 132-splitting module; 133-second acquisition module; 134-generation module; 140-communication interface. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0080] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0081] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0082] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0083] See also Figure 1, is a schematic diagram of an application scenario of the control information generation method based on the personalized expression base provided in an embodiment of the present application, wherein the application scenario includes a server 100 and multiple image acquisition devices 200 that are communicatively connected to the server 100. The multiple image acquisition devices 200 may include a device for acquiring two-dimensional images, such as a camera, and may also include a device for acquiring depth images, such as a depth camera.

[0084] In this embodiment, each image acquisition device 200 can send the acquired image information or video information to the server 100, and the server 100 analyzes and processes the received image information or video information to obtain control information that can be used to control the virtual image.

[0085] Please refer to Figure 2 , the application scenario of the control information generation method provided in the embodiment of the present application may also include a live broadcast provider 300 and a live broadcast receiver 400, and the above-mentioned server 100 may be a live broadcast server. The live broadcast provider 300 and the live broadcast receiver 400 may be connected to the live broadcast server in communication. Among them, the live broadcast provider 300 may be a terminal device (such as a mobile phone, a tablet computer, a computer, etc.) used by the anchor during the live broadcast, and the live broadcast receiver 400 may be a terminal device (such as a mobile phone, a tablet computer, a computer, etc.) used by the audience when watching the live broadcast.

[0086] The live broadcast provider 300 can send the live video stream to the live broadcast server, and the audience can access the live broadcast server through the live broadcast receiving terminal 400 to watch the live video. The live broadcast server can also receive the control instructions for the virtual image sent by the live broadcast provider 300, and then control the virtual image accordingly based on the control information obtained above to generate a live video stream, and push it to the live broadcast receiving terminal 400.

[0087] Combination Figure 3 The embodiment of the present application also provides a control information generation method based on a personalized expression base that can be applied to an electronic device to generate control information for driving a virtual image. The electronic device can be the above-mentioned server 100. The method steps defined in the process related to the control information generation method can be implemented by the electronic device. Figure 3 The specific process shown is described in detail.

[0088] Step S110, obtaining a plurality of facial images of the reference object in different states.

[0089] Step S120, based on the preset multiple expression base templates and the multiple facial images, split the facial features contained in each of the facial images to generate multiple personalized expression bases.

[0090] Step S130, obtaining video frames of the reference object in different set expressions.

[0091] Step S140, generating control information including a personalized expression base corresponding to each of the set expressions according to the video frame and the personalized expression base, and saving the control information.

[0092] In this embodiment, a plurality of preset expression base templates are pre-stored in the electronic device, and each expression base template can be a basic template drawn by an animation production staff. Each expression base template contains a plurality of facial features, such as eyes, nose, mouth, eyebrows, etc. The corresponding facial features in different expression base templates may be in different states. For example, in some expression base templates, the eyes are in an open state, in some expression base templates, the eyes are in a closed state, in some expression base templates, the mouth is in an open state, in some expression base templates, the mouth is in a pouting state, and so on.

[0093] By setting each facial feature to a different state in the preset multiple expression base templates, different facial feature states can be reflected individually or in combination of multiple facial features.

[0094] Since the preset expression base template is often a unified template, not based on a certain character object, it is difficult to use it for subsequent operations such as comparison based on character information to determine control information. Therefore, in this embodiment, it is necessary to establish personalized expression bases for different specific characters based on the preset expression base template.

[0095] In this embodiment, the image acquisition device 200 can be used to acquire multiple facial images of the reference object in different states. The reference object can be any human object, for example, any female model, any male model, etc. Of course, when selecting the reference object, it can also be selectively selected based on the virtual image that needs to be driven later. For example, the virtual image can include a cartoon image, a lolita image, or a hero image. If it is necessary to drive the virtual image of the lolita image, a female model can be selected as the reference object, which can reduce the complexity of the subsequent adjustment operation to a certain extent.

[0096] In this embodiment, facial images of the reference object in different states can be collected, such as angry state, laughing state, sad state, etc. The facial images obtained in various states contain multiple facial features. Based on the setting method for each facial feature in the preset multiple expression base templates, the multiple facial features in the facial image can be split to generate multiple personalized expression bases.

[0097] For example, a facial image of a reference object in an angry state contains features of eyebrows, mouth, eyes, nose, etc. These features can be extracted separately, and each feature can be used to obtain a personalized expression base, that is, each personalized expression base can be used to reflect the information of a certain facial feature.

[0098] By means of the above method, the facial features of the reference object in different states can be separated separately, and each facial feature can be represented separately in a personalized expression base.

[0099] On this basis, video frames of the reference object in different set expressions can be collected, where the set expressions can be set according to needs, such as yawning, blinking, etc. Subsequently, the virtual image can be controlled to display the corresponding set expressions based on the information of the set expressions.

[0100] The obtained video frame contains the facial information of the reference object under the set expression, and the facial information should be jointly reflected by all the facial features contained on the face. The personalized expression base obtained above can represent each facial feature separately. Therefore, based on the comparative analysis of the video frame and the personalized expression base, the information in the video frame of the reference object can be reflected by the personalized expression base. In addition, the video frame of the reference object should be reflected by multiple personalized expression bases in a certain way, that is, the control information corresponding to the set expression containing the personalized expression base can be obtained.

[0101] By processing each set expression in the above manner, control information corresponding to each set expression can be obtained, and the control information can be saved. In subsequent live broadcasts, the saved control information can be used to control the virtual image so that the virtual image shows the corresponding expression to increase the fun of the live broadcast.

[0102] The control information generation method provided in this embodiment can generate a personalized expression base reflecting the characteristics of the reference object based on the expression base template, and based on the video frame and personalized expression base of the reference object under different set expressions, the set expression information represented by the personalized expression base can be obtained, which can be used for the subsequent control of the virtual image. In this way, only the image acquisition device 200 needs to be used in combination with the above processing method to obtain the control information, which reduces the complexity of the device and avoids excessive interference, and the obtained control information can accurately reflect each set expression.

[0103] In this embodiment, in order to make the virtual image vividly display different expressions, the constructed virtual image is generally a 3D image, and accordingly, each preset expression base template is a 3D model. Figure 4 In this embodiment, the generation of a personalized expression base can be achieved in the following ways:

[0104] Step S121, constructing three-dimensional models in different states according to the multiple facial images.

[0105] Step S122, based on a plurality of preset expression base templates and a plurality of three-dimensional models, the facial features in each of the three-dimensional models are split to generate a plurality of personalized expression bases.

[0106] In this implementation, first, a three-dimensional model of a reference object in different states can be constructed. For each state, multiple target facial images in the state can be obtained, and the multiple target facial images are images taken from different perspectives. Three-dimensional reconstruction is performed based on the multiple target facial images, and the model obtained by the three-dimensional reconstruction is topologically processed to obtain a three-dimensional model in the state.

[0107] In this embodiment, when capturing images of the reference object, multiple image capture devices 200 may be used to capture images from different angles, for example, a perspective facing the face of the object, a perspective from the left side, a perspective looking up, a perspective from the right side, etc. Figure 5 As shown in . In this way, in a certain state, such as an angry state, multiple facial images of the reference object in this state from different perspectives can be captured. Based on the facial images from different perspectives, the face of the reference object can be reconstructed in three dimensions, and the resulting model is as follows Figure 6 as shown in .

[0108] In addition, the model obtained by 3D reconstruction is composed of many points. The points in the obtained model may be unevenly distributed and the positions of each point may not be consistent with the standard reference position. Therefore, in order to facilitate subsequent processing and the standardization of the constructed model, the model obtained by 3D reconstruction can also be topologically processed. The topological processing method can include adjusting the points in the 3D model so that the points in the adjusted model are evenly distributed and the positions of each point are consistent with the standard reference position, such as Figure 7 shown.

[0109] Through the above method, three-dimensional models in different states can be constructed according to the multiple facial images, and the facial features in each three-dimensional model can be split based on the preset multiple expression base templates and the obtained multiple three-dimensional models, thereby generating multiple personalized expression base templates.

[0110] In this embodiment, the preset expression base template may include a reference template and multiple variable templates, such as Figure 8 As shown in , the reference template is a template when the face is expressionless (each facial feature is also expressionless accordingly), and each variable template is a template with a facial feature that changes with the facial feature corresponding to the reference template relative to the reference template.

[0111] For example, Figure 8 The first expression base template in the example can be a reference template, in which the face has no expression, and the subsequent expression base templates are variable templates, in which, compared with the first expression base template, only the mouth information of the second expression base template changes while other facial features remain unchanged. That is, each subsequent variable template can represent the information of a facial feature separately.

[0112] Correspondingly, in this embodiment, the facial image of the reference object captured also includes the facial image of the reference object when it is expressionless, and the multiple personalized expression bases established include a baseline expression base and multiple variable expression bases. Compared with the baseline expression base, each variable expression base has a facial feature that is different from the corresponding facial feature in the baseline expression base.

[0113] For example, Fig. 9 As shown in , the first personalized expression base is the base expression base, and the subsequent personalized expression bases are variable expression bases. The facial features in the first personalized expression base represent the features in the expressionless state, such as only the mouth features in the second personalized expression base are different from those in the first personalized expression base.

[0114] Through the above method, an expression base group including a reference expression base and multiple variable expression bases can be obtained.

[0115] On this basis, the image acquisition device 200 can be used to acquire video frames of the reference object. The image acquisition device 200 can be a depth camera, and the acquired video frames can be 3D images. Optionally, video frames of the reference object under different set expressions can be acquired.

[0116] For a certain set expression of the reference object, the set expression can be reflected by analyzing the video frame with the above expression base group. Fig.10 In this embodiment, control information that can reflect each set expression can be obtained in the following way.

[0117] Step S141, assigning base coefficients to the expression bases and then combining them to obtain a combined expression base.

[0118] Step S142, comparing the video frame with the combined expression base, and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0119] Step S143, generating control information based on the base coefficient combination and the personalized expression base.

[0120] From the above, it can be seen that each personalized expression base can represent a certain facial feature alone. Under a certain set expression, the facial information corresponding to the set expression should be comprehensively reflected by multiple personalized expression bases. Each personalized expression base comprehensively reflects the set expression with a certain coefficient ratio, that is, each expression base can be given a base coefficient to combine to obtain a combined expression base.

[0121] The above-mentioned combined expression base includes a reference expression base and a variable expression base, wherein the base coefficient of the reference expression base can be fixed to 1. What this embodiment needs to do is to optimize the base coefficients of each variable expression base. The ultimate goal of the optimization is to make the gap between the obtained combined expression base and the video frame as small as possible, that is, when the gap between the combined expression base and the video frame is lower than a preset threshold, it can be determined that the preset requirements are met, and the base coefficient obtained at this time is the base coefficient that finally corresponds to the set expression.

[0122] In this embodiment, the initial base coefficient of each variable expression base can be assigned a value of 0, and then the obtained combined expression base is compared with the video frame, and each base coefficient is adjusted multiple times according to the comparison result, and the combined expression base obtained with the adjusted base coefficient is compared with the video frame again. In this way, after multiple adjustments, when the preset requirements are met, a base coefficient combination that meets the requirements can be obtained. Fig.11 As shown, the base coefficient combination can be obtained by Fig.11 The Pexp in and the coefficient 1 of the first base expression base form the combined expression base. Fig.11 The rightmost expression in the.

[0123] A combination of multiple personalized expression bases and base coefficients can together constitute control information, and the virtual image can be subsequently controlled based on the control information.

[0124] In this embodiment, it can be seen from the above that the video frame captured by the depth camera is a 3D image, and the video frame contains point cloud information of three-dimensional feature points. When comparing the video frame with the combined expression basis to determine the basis coefficient, it can be based on the point cloud information in the video frame. For details, please refer to Fig.12 In this embodiment, the base coefficient can be determined in the following way.

[0125] Step S1421A, aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base.

[0126] Step S1422A, adjusting each of the base coefficients according to the result of the alignment operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0127] In this embodiment, the combined expression base obtained by the above method is also a 3D model, which contains multiple feature points, and the information of each feature point is three-dimensional information. The point cloud information obtained based on the captured video frame contains multiple three-dimensional points of the facial model of the reference object under the set expression.

[0128] Since the combined expression base is obtained by combining multiple personalized expression bases with corresponding basis coefficients, and the face model in the video frame is obtained by shooting, there is a large difference between the two when the basis coefficients are not optimized. Therefore, the process of continuously aligning the feature points in the combined expression base with the 3D points in the point cloud information is the process of continuously adjusting the basis coefficients. Optionally, please refer to Fig.13 , the process of alignment based on point cloud information can be achieved in the following ways:

[0129] Step S14211A, for each feature point in the combined expression base, determine the three-dimensional feature point corresponding to the feature point in the point cloud information of the three-dimensional feature point, and perform position alignment constraints on the corresponding feature point and the three-dimensional feature point in terms of coordinate values.

[0130] Step S14212A, determining the plane where the three-dimensional feature point corresponding to the feature point is located, and performing distance constraints between the feature point and the plane.

[0131] Step S14213A, obtaining the alignment operation result according to the results of the position alignment constraint and the distance constraint.

[0132] In this embodiment, the number of feature points and the positions of the feature points in the obtained combined expression base may be different from the three-dimensional feature points in the point cloud information. For each feature point in the combined expression base, the three-dimensional feature point with the smallest position difference with the feature point can be found, and the found three-dimensional feature point is used as the three-dimensional feature point corresponding to the feature point.

[0133] The feature points are aligned with the corresponding three-dimensional feature points in terms of point-to-point coordinate values, that is, the gap between the feature points and the three-dimensional feature points in terms of coordinate values ​​is reduced.

[0134] In addition, for a certain 3D feature point, the 3D feature point and several other adjacent 3D feature points can form a plane, and the difference between the corresponding feature point and the 3D feature point is also reflected in the distance between the feature point and the plane where the 3D feature point is located. Therefore, the distance between the feature point and the plane where the corresponding 3D feature point is located can be continuously reduced by adjusting the basis coefficient and then adjusting the combined expression basis.

[0135] Combination Fig.14By combining the above-mentioned position alignment constraints between points and distance constraints between points and surfaces, the basis coefficients in the combined expression basis are adjusted so that the results of the position alignment constraints and distance constraints can meet the required requirements.

[0136] In addition, in this embodiment, by mapping the video frame in the form of a 3D image to a two-dimensional coordinate system, the corresponding two-dimensional image information can be obtained. That is, the video frame also includes two-dimensional image information. When optimizing the basis coefficients in the combined expression basis, the optimization can also be performed based on the two-dimensional image information. For details, please refer to Fig.15 , which can be achieved by:

[0137] Step S1421B: performing a comparison operation on the two-dimensional image information and the two-dimensional information included in the combined expression base.

[0138] Step S1422B, adjusting each of the base coefficients according to the result of the comparison operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0139] In this embodiment, considering that the two-dimensional image information is generally represented by the position information of the point and the pixel value of the point, and the obtained combined expression base is a model without pixel values, therefore, in this embodiment, please refer to Fig.16 , the comparison operation between the two-dimensional image information and the combined expression base can be realized through the following processing method.

[0140] Step S14211B, rendering the combined expression base based on a preset texture map to obtain a comparison image.

[0141] Step S14212B, performing position comparison operation on the key points in the comparison image and corresponding key points in the two-dimensional image information in terms of coordinate values ​​and pixel comparison operation in terms of pixel values ​​respectively.

[0142] Step S14213B, obtaining a comparison operation result based on the results of the position comparison operation and the pixel comparison operation.

[0143] In this embodiment, a preset texture map can be obtained, which is a map with pixel information. By using the texture map to render the combined expression, a comparison image can be obtained, which is a two-dimensional image with pixel information.

[0144] Since there is a difference in pixel values ​​between the preset texture map and the two-dimensional image information obtained from the video frame of the reference object collected, the difference is mainly caused by the difference in position information of the key points corresponding to the two. Among them, the key points can be points at positions such as eyeballs, nose tips, mouth corners, eyebrow tips, eyebrow tails, etc.

[0145] Therefore, the corresponding key points in the comparison image and the two-dimensional image information can be obtained, and the corresponding key points are compared in coordinate values ​​and pixel values. The ultimate goal of the comparison is to make the corresponding key points between the two as consistent as possible in coordinate values ​​and pixel values.

[0146] Combination Fig.17 As shown, by means of the above-mentioned key point constraints and pixel value constraints, the base coefficients in the combined expressions can be continuously adjusted, and finally a base coefficient combination that meets the preset requirements is obtained.

[0147] It should be noted that, in the construction of optimizing the basis coefficients, the optimization may be performed based solely on the above-mentioned point cloud information, based solely on the two-dimensional image information, or in combination with the point cloud information and the two-dimensional image information.

[0148] If the basis coefficients are optimized by combining point cloud information and two-dimensional image information, the basis coefficients can be adjusted together according to the results of the alignment operation based on the point cloud information and the results of the comparison operation based on the two-dimensional image information until the preset requirements are met and the final basis coefficient combination corresponding to the set expression is obtained.

[0149] Through the above method, the base coefficient combination corresponding to each set expression can be obtained, and multiple personalized expression bases are fixed. By combining different base coefficient combinations with personalized expression bases, multiple different control information can be obtained. The obtained control information can be saved and can be directly called later.

[0150] The above process is an offline process of producing corresponding control information for different set expressions. In practical applications, for example, during live broadcast, the control information obtained in the above process can be used to control the constructed virtual image. Fig.18 In this embodiment, it can be implemented in the following ways:

[0151] Step S210, during the live broadcast process, obtain a control expression instruction for the constructed virtual image, and obtain a set expression corresponding to the control expression.

[0152] Step S220, extracting the saved control information corresponding to the set expression.

[0153] Step S230: controlling the virtual image to display the control expression according to the control information.

[0154] In this embodiment, a virtual image may be pre-constructed, and the constructed virtual image may be one or more, for example, may include a loli image, a cartoon image, a hero image, etc. During the live broadcast, the anchor may trigger the virtual image to show different expressions in order to enrich the diversity and fun of the live broadcast.

[0155] In this embodiment, the anchor can initiate a control expression instruction, and the control expression instruction can be used to instruct the virtual image to display the control expression. After receiving the control expression instruction, the electronic device can obtain the set expression corresponding to the control expression.

[0156] For example, different trigger buttons can be set in the live broadcast application to correspond to different control expressions, and the anchor can issue the corresponding control expression instruction by clicking the trigger button. In addition, an information input box can also be set in the live broadcast application, and the anchor can enter the corresponding control expression in the information input box to enter the corresponding trigger window to initiate the corresponding control expression instruction.

[0157] After determining the set expression corresponding to the control expression, the control information corresponding to the set expression can be extracted from the pre-generated control information. For example, if the control expression is a smiling expression for the virtual image, the control information corresponding to the smiling expression can be extracted from the pre-generated multiple control information. Then, based on the extracted control information, the virtual image is controlled to display the control expression, for example, the virtual image is controlled to display a smiling expression. Fig.19 shown.

[0158] In this embodiment, it is considered that an expression may not be limited to one image frame, but may need to be reflected by multiple continuous image frames, that is, the display of an expression is a dynamic process, which may last for multiple seconds. For example, for a yawning expression, if the expression is to be fully displayed, multiple continuous image frames may be required to reflect it.

[0159] Therefore, the video frames of the reference object collected in advance may include multiple continuous video frames under a certain set expression, and the control information corresponding to the set expression includes multiple control sub-information corresponding to each video frame. That is, each control sub-information may correspond to each video frame.

[0160] In this case, see Fig. 20 When the virtual image is controlled based on the control information to display the control expression, the above can be achieved in the following ways:

[0161] Step S231, for the continuous multiple live video frames to be controlled during the live broadcast process, obtain the control sub-information corresponding to each live video frame in the control information.

[0162] Step S232, using each of the control sub-information to control the virtual image in the corresponding live video frame, so that the virtual image in the continuous multiple frames of live video frames executes the display of the control expression.

[0163] In this embodiment, the continuous multi-frame live video frame to be controlled can be a multi-frame live video frame after the current time point, taking the current time point as the boundary point. For example, if the virtual image needs to be controlled to show a yawning expression, and the complete display of the expression requires 5 consecutive live video frames, then the continuous multi-frame live video frame to be controlled can be 5 consecutive live video frames after the current time point.

[0164] It should be noted that the above is only an example, and during implementation, the live video frame to be controlled can be determined according to actual control requirements.

[0165] In this embodiment, the obtained control information includes multiple control sub-information, and the control sub-information corresponding to each live video frame can be obtained. Then, each control sub-information is used to control the corresponding live video frame. In this way, each control sub-information can control the virtual image in the corresponding live video frame to be in a certain state, and the continuous multiple states can construct the required control expression, thereby making the virtual image in the continuous multiple video frames perform the display of the control expression.

[0166] Optionally, as can be seen from the above, the control information for a certain set expression includes multiple personalized expression bases and base coefficient combinations. A comprehensive expression base model can be obtained by weighting the multiple personalized expression bases in the control information according to their respective base coefficients.

[0167] The definition of feature points in the comprehensive expression base model is consistent with the definition of feature points on the avatar, and the corresponding feature points between the two can be identified with the same serial number. That is, for example, for eye features, the comprehensive expression base model can use N feature points to constitute the eye feature, and accordingly, the eye feature in the avatar can also be composed of N feature points to constitute its eye feature.

[0168] In this way, the coordinate information of the feature points in the comprehensive expression base model can be used to control the coordinate values ​​of the corresponding feature points in the virtual image, so that the facial features of the virtual image can be consistent with the state of the facial features in the comprehensive expression base model, and then the full facial expression of the virtual image can be consistent with the full facial expression of the comprehensive expression base.

[0169] The control information generation method based on the personalized expression base provided in this embodiment can generate a personalized expression base reflecting the characteristics of the reference object based on the expression base template, and obtain control information represented by the personalized expression base based on the video frames of different set expressions and the personalized expression base, so that the control information can be used to control the virtual image in the future, so that the virtual image can display the corresponding expression. This solution can realize the generation of control information without the need for auxiliary equipment, reduce the complexity of the equipment and avoid excessive interference, and the personalized expression base information generated based on the expression base template is more standardized, so that the obtained control information can be more standardized and can accurately reflect each set expression.

[0170] Furthermore, in this embodiment, during the live broadcast, the pre-generated control information can be directly extracted to control the virtual image, which is convenient and fast, and does not need to be bound to the host's status, thus avoiding interference with the host. In addition, the interactivity and fun of the live broadcast can be enhanced, and the audience's immersive experience of the live broadcast can be improved.

[0171] See also Fig.21 , is a schematic diagram of an exemplary component of an electronic device provided in an embodiment of the present application, and the electronic device may be Figure 1 The server 100 shown in . The electronic device may include a storage medium 110, a processor 120, a control information generating device 130 based on a personalized expression base, and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0172] The control information generating device 130 based on personalized expression base can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned control information generating method based on personalized expression base under the control of the electronic device.

[0173] like Fig. 22 As shown, the control information generating device 130 based on personalized expression base may include a first obtaining module 131, a splitting module 132, a second obtaining module 133 and a generating module 134. The functions of each functional module of the control information generating device 130 based on personalized expression base are described in detail below.

[0174] The first obtaining module 131 is used to obtain a plurality of facial images of a reference object in different states.

[0175] It can be understood that the first obtaining module 131 can be used to execute the above step S110. The detailed implementation of the first obtaining module 131 can refer to the above content related to step S110.

[0176] The splitting module 132 is used to split the facial features contained in each of the facial images based on the preset multiple expression base templates and the multiple facial images to generate multiple personalized expression bases.

[0177] It can be understood that the splitting module 132 can be used to execute the above step S120. The detailed implementation of the splitting module 132 can refer to the above content related to step S120.

[0178] The second obtaining module 133 is used to obtain video frames of the reference object in different set expressions.

[0179] It can be understood that the second obtaining module 133 can be used to execute the above step S130. The detailed implementation of the second obtaining module 133 can refer to the above content related to step S130.

[0180] The generating module 134 is used to generate control information including a personalized expression base corresponding to each of the set expressions according to the video frame and the personalized expression base, and save the control information.

[0181] It can be understood that the generation module 134 can be used to execute the above step S140. The detailed implementation of the generation module 134 can refer to the above content related to step S140.

[0182] In a possible implementation, the control information generating device further includes a control module, which can be used to:

[0183] During the live broadcast, a control expression instruction for the constructed virtual image is obtained, and a set expression corresponding to the control expression is obtained;

[0184] Extracting the saved control information corresponding to the set expression;

[0185] The virtual image is controlled according to the control information to display the control expression.

[0186] In a possible implementation, the generating module 134 may be specifically used for:

[0187] After assigning base coefficients to each of the personalized expression bases, the bases are combined to obtain a combined expression base;

[0188] Comparing the video frame with the combined expression base, and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met;

[0189] Control information is generated based on the base coefficient combination and the personalized expression base.

[0190] In a possible implementation, the video frame includes point cloud information of three-dimensional feature points, and the generating module 134 may generate a basis coefficient combination in the following manner:

[0191] Aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base;

[0192] The base coefficients are adjusted according to the result of the alignment operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0193] In a possible implementation, the generating module 134 may be specifically used for:

[0194] For each feature point in the combined expression base, determine the three-dimensional feature point corresponding to the feature point in the point cloud information of the three-dimensional feature point, and perform position alignment constraints on the corresponding feature point and the three-dimensional feature point in terms of coordinate values;

[0195] Determine the plane where the three-dimensional feature point corresponding to the feature point is located, and impose a distance constraint between the feature point and the plane;

[0196] The alignment operation result is obtained according to the results of the position alignment constraint and the distance constraint.

[0197] In a possible implementation, the video frame includes two-dimensional image information, and the generation module 134 may also be used to generate a basis coefficient combination in the following manner:

[0198] Comparing the two-dimensional image information with the two-dimensional information included in the combined expression base;

[0199] The base coefficients are adjusted according to the result of the comparison operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

[0200] In a possible implementation, the generating module 134 may be specifically used for:

[0201] Rendering the combined expression base based on a preset texture map to obtain a comparison image;

[0202] Performing a position comparison operation on the key points in the comparison image and the corresponding key points in the two-dimensional image information in terms of coordinate values ​​and a pixel comparison operation in terms of pixel values;

[0203] Based on the results of the position comparison operation and the pixel comparison operation, a comparison operation result is obtained.

[0204] In a possible implementation, the video frame under the set expression includes multiple continuous video frames, and the control information corresponding to the set expression includes multiple control sub-information corresponding to each of the video frames. The above control module can be specifically used for:

[0205] For a plurality of continuous live video frames to be controlled during the live broadcast process, obtaining control sub-information corresponding to each live video frame in the control information;

[0206] Each of the control sub-information is used to control the virtual image in the corresponding live video frame, so that the virtual image in the continuous multiple frames of live video frames executes the display of the control expression.

[0207] In a possible implementation, the splitting module 132 may be used to:

[0208] constructing three-dimensional models in different states according to the multiple facial images;

[0209] Based on a plurality of preset expression base templates and a plurality of three-dimensional models, facial features in each of the three-dimensional models are split to generate a plurality of personalized expression bases;

[0210] The multiple personalized expression bases include a baseline expression base and multiple variable expression bases, and each variable expression base has a facial feature that is different from the corresponding facial feature in the baseline expression base compared to the baseline expression base.

[0211] In a possible implementation, the splitting module 132 may be specifically used to:

[0212] For each state, a plurality of target facial images in the state are obtained, wherein the plurality of target facial images are images taken from different viewing angles;

[0213] Three-dimensional reconstruction is performed based on the multiple target facial images, and topological processing is performed on the model obtained by the three-dimensional reconstruction to obtain the three-dimensional model in the state.

[0214] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0215] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the control information generation method based on the personalized expression base provided in the above embodiment is implemented.

[0216] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned control information generation method based on the personalized expression base can be executed. Regarding the processes involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant descriptions in the above-mentioned method embodiments, which will not be described in detail here.

[0217] In summary, the embodiments of the present application provide a method, device, electronic device and readable storage medium for generating control information based on a personalized expression base, by obtaining multiple facial images of a reference object in different states, splitting the facial features contained in each facial image based on multiple preset expression base templates, and generating multiple personalized expression bases. Then, a video frame of the reference object in different set expressions is obtained, and control information containing a personalized expression base corresponding to each set expression is generated and saved based on the video frame and the personalized expression base. This scheme can generate a personalized expression base reflecting the characteristics of the reference object based on the expression base template, and can obtain control information represented by the personalized expression base based on the video frames of different set expressions and the personalized expression base, which can be used for the subsequent control of the virtual image. In this way, the complexity of the device can be reduced and excessive interference can be avoided, and the obtained control information can accurately reflect each set expression.

[0218] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A control information generation method based on personalized expression base, characterized in that: The method comprises: obtaining a plurality of facial images of a reference object in different states; Based on a plurality of preset expression base templates and the plurality of facial images, facial features contained in each of the facial images are split to generate a plurality of personalized expression bases; Obtaining video frames of the reference object in different set expressions; Generate control information corresponding to each of the set expressions and including the personalized expression base according to the video frame and the personalized expression base, and save the control information, wherein the control information also includes a basis coefficient of the personalized expression base; The steps of generating a plurality of the personalized expression bases include: Three-dimensional models in different states are constructed according to the multiple facial images, and based on multiple preset expression base templates and multiple three-dimensional models, facial features in each of the three-dimensional models are split to generate multiple personalized expression bases.

2. The method for generating control information based on personalized expression base according to claim 1, characterized in that: The method further comprises: During the live broadcast, a control expression instruction for the constructed virtual image is obtained, and a set expression corresponding to the control expression is obtained; Extracting the saved control information corresponding to the set expression; The virtual image is controlled according to the control information to display the control expression.

3. The method for generating control information based on personalized expression base according to claim 1, characterized in that: The step of generating control information including a personalized expression base corresponding to the set expression according to the video frame and the personalized expression base comprises: After assigning base coefficients to each of the personalized expression bases, the bases are combined to obtain a combined expression base; Comparing the video frame with the combined expression base, and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met; Control information is generated based on the base coefficient combination and the personalized expression base.

4. The method for generating control information based on personalized expression base according to claim 3, characterized in that: The video frame includes point cloud information of three-dimensional feature points; The step of comparing the video frame with the combined expression base and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met includes: Aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base; The base coefficients are adjusted according to the result of the alignment operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

5. The method for generating control information based on personalized expression base according to claim 4, characterized in that: The step of aligning the point cloud information of the three-dimensional feature points with the feature points in the combined expression base comprises: For each feature point in the combined expression base, determine the three-dimensional feature point corresponding to the feature point in the point cloud information of the three-dimensional feature point, and perform position alignment constraints on the corresponding feature point and the three-dimensional feature point in terms of coordinate values; Determine the plane where the three-dimensional feature point corresponding to the feature point is located, and impose a distance constraint between the feature point and the plane; The alignment operation result is obtained according to the results of the position alignment constraint and the distance constraint.

6. The method for generating control information based on personalized expression base according to claim 3 or 4, characterized in that: The video frame includes two-dimensional image information; The step of comparing the video frame with the combined expression base and adjusting each of the base coefficients based on the comparison result until a base coefficient combination corresponding to the set expression is finally obtained when the preset requirements are met includes: Comparing the two-dimensional image information with the two-dimensional information included in the combined expression base; The base coefficients are adjusted according to the result of the comparison operation until a base coefficient combination corresponding to the set expression is finally obtained when a preset requirement is met.

7. The method for generating control information based on personalized expression base according to claim 6, characterized in that: The step of comparing the two-dimensional image information with the two-dimensional information included in the combined expression base comprises: Rendering the combined expression base based on a preset texture map to obtain a comparison image; Performing a position comparison operation on the key points in the comparison image and the corresponding key points in the two-dimensional image information in terms of coordinate values ​​and a pixel comparison operation in terms of pixel values; Based on the results of the position comparison operation and the pixel comparison operation, a comparison operation result is obtained.

8. The method for generating control information based on personalized expression base according to claim 2, characterized in that: The video frame under the set expression includes a plurality of continuous video frames, and the control information corresponding to the set expression includes a plurality of control sub-information corresponding to each of the video frames; The step of controlling the virtual image to display the control expression according to the control information includes: For a plurality of continuous live video frames to be controlled during the live broadcast process, obtaining control sub-information corresponding to each live video frame in the control information; Each of the control sub-information is used to control the virtual image in the corresponding live video frame, so that the virtual image in the continuous multiple frames of live video frames executes the display of the control expression.

9. The method for generating control information based on personalized expression base according to claim 1, characterized in that: The multiple personalized expression bases include a reference expression base and multiple variable expression bases. Compared with the reference expression base, each variable expression base has a facial feature that is different from the corresponding facial feature in the reference expression base.

10. The method for generating control information based on personalized expression base according to claim 9, characterized in that: The step of constructing three-dimensional models in different states according to the multiple facial images comprises: For each state, a plurality of target facial images in the state are obtained, wherein the plurality of target facial images are images taken from different viewing angles; Three-dimensional reconstruction is performed based on the multiple target facial images, and topological processing is performed on the model obtained by the three-dimensional reconstruction to obtain the three-dimensional model in the state.

11. A control information generating device based on personalized expression base, characterized in that: The device is used to implement the method for generating control information based on a personalized expression base according to any one of claims 1 to 10, comprising: A first acquisition module is used to obtain a plurality of facial images of a reference object in different states; A splitting module, used for splitting the facial features contained in each of the facial images based on a plurality of preset expression base templates and the plurality of facial images, to generate a plurality of personalized expression bases; A second obtaining module is used to obtain video frames of the reference object in different set expressions; The generating module is used for generating control information including the personalized expression base corresponding to each of the set expressions according to the video frame and the personalized expression base, and storing the control information.

12. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of claims 1-10 are implemented.

Images