Method, device, equipment and live broadcast system for determining expression coefficient and model training
Through the virtual target object's image and sample expression coefficients training, the expression coefficient prediction model is solved, and the problems of high cost and high calculation volume in the existing technology are realized, and the expression coefficient determination of low-cost and low-complexity are implemented, which is suitable for high-dimensional expression coefficient driving Virtual image.
Patent Information
- Application Number
- CN202210157073.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-02-21
AI Technical Summary
In the prior art, determining the expression coefficient of the target image requires training based on a large number of real sample images, resulting in high training costs and high computational cost, especially when high-dimensional expression coefficients drive high-precision virtual images.
By acquiring the image of the virtual target object, the expression coefficient prediction model is trained using the pre-generated sample expression coefficient and the sample image of the virtual object, and the expression coefficient of the real target object is determined, reducing the sample collection cost and calculation complexity.
The expression coefficients of real target objects can be determined without high cost without collecting large amounts of real sample images, which reduces the cost and complexity of training and prediction and improves efficiency.
Smart Images

Figure CN114529637B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of computer vision, and in particular to a method, device, equipment and live broadcast system for determining expression coefficients and model training. Background Art
[0002] In the field of computer vision, it is often necessary to drive a virtual image based on the expression coefficient of the target object. In some scenarios, it is necessary to determine the expression coefficient of the target object based on the image of the target object, and then drive the virtual image through the expression coefficient. The process of obtaining the expression coefficient can be achieved through a prediction model. However, the expression coefficient prediction model needs to be trained based on a large number of sample images, and the training cost is very high. Summary of the invention
[0003] In order to overcome the problems existing in the related art, this specification provides a method, device, equipment and live broadcast system for determining expression coefficient and model training.
[0004] According to a first aspect of an embodiment of this specification, a method for determining an expression coefficient is provided, comprising:
[0005] Acquire an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0006] Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0007] The first expression coefficient is determined based on the second expression coefficient.
[0008] According to a second aspect of the embodiments of this specification, a system for live broadcasting of a virtual image is provided, including a host client, a viewer client, and a server:
[0009] The anchor client is used to: obtain an image of a virtual target object, where the image of the virtual target object is converted based on an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0010] Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0011] determining the first expression coefficient based on the second expression coefficient;
[0012] driving the expression of the avatar based on the first expression coefficient;
[0013] sending the virtual image to the server;
[0014] The server is used to: receive the virtual image sent by the anchor client, and send the virtual image to the viewer client;
[0015] The audience client is used to receive and display the virtual image sent by the server.
[0016] According to a third aspect of the embodiments of this specification, another system for live broadcasting of a virtual image is provided, including a host client, a viewer client and a server:
[0017] The anchor client is used to: obtain an image of a real target object, and send the image of the real target object to the server;
[0018] The server is used to: receive the image of the real target object sent by the anchor client, and obtain the image of the virtual target object, where the image of the virtual target object is obtained by converting the image of the real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object;
[0019] Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0020] determining the first expression coefficient based on the second expression coefficient;
[0021] driving the expression of the avatar based on the first expression coefficient;
[0022] Sending the virtual image to the host client and the viewer client;
[0023] The anchor client is also used to: receive and display the virtual image sent by the server;
[0024] The audience client is used to receive and display the virtual image sent by the server.
[0025] According to a fourth aspect of an embodiment of this specification, a model training method is provided, including:
[0026] Driving the expression of the virtual object based on the pre-generated sample expression coefficient to generate a sample image including the virtual object;
[0027] Training the expression coefficient prediction model based on the sample image and the sample expression coefficient;
[0028] The expression coefficient prediction model is used to obtain a second expression coefficient of the virtual target object based on an image of the virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object;
[0029] The second expression coefficient is used to determine the first expression coefficient; the first expression coefficient is used to drive the expression of the virtual image.
[0030] According to a fifth aspect of an embodiment of this specification, there is provided a device for determining an expression coefficient, comprising:
[0031] An acquisition module is used to acquire an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0032] Prediction module: used for inputting the image of the virtual target object into the expression coefficient prediction model, and obtaining the second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained by using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0033] Determination module: used to determine the first expression coefficient based on the second expression coefficient.
[0034] According to a sixth aspect of an embodiment of the present specification, there is provided a device for determining an expression coefficient, comprising a camera; a processor; and a memory for storing processor executable instructions, wherein the processor is configured to implement the method described in any of the above embodiments.
[0035] The embodiment of the present specification obtains an image of a virtual target object corresponding to the image of a real target object, and based on the corresponding properties of the expression coefficients of the real target object and the virtual target object, determines the expression coefficient of the virtual target object in the image of the virtual target object by using an expression coefficient prediction model, thereby determining the expression coefficient of the real target object; since the virtual object expression prediction model only needs to use pre-generated sample expression coefficients and drive the expression of the virtual object based on the sample expression coefficients to generate a sample image including the virtual object for training, it can be seen that the method of the present application does not require the high-cost collection of a large number of real sample images, nor does it require a high-computing-power algorithm to determine the expression coefficient of the real target object.
[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.
[0038] Figure 1 It is a flowchart of determining the expression coefficient of a target image using model X in the prior art provided by an exemplary embodiment.
[0039] Figure 2 It is a flowchart of training model X in the prior art provided by an exemplary embodiment.
[0040] Figure 3 It is a flow chart of a method for determining an expression coefficient provided by an exemplary embodiment.
[0041] Figure 4 is a schematic image of a virtual target object provided by an exemplary embodiment.
[0042] Figure 5 A flowchart of converting an image of a real object into an image of a virtual object by converting a model is provided by an exemplary embodiment.
[0043] Figure 6 It is a schematic diagram of the effect before and after style conversion of a conversion model provided by an exemplary embodiment.
[0044] Figure 7 is a flowchart of a training conversion model provided by an exemplary embodiment.
[0045] Figure 8 It is a flow chart of a method for determining 240-dimensional expression coefficients provided by an exemplary embodiment.
[0046] Fig. 9This is a system interaction diagram of a virtual image live broadcast provided by an exemplary embodiment.
[0047] Fig.10 This is another system interaction diagram for live broadcasting of a virtual image provided by an exemplary embodiment.
[0048] Fig.11 It is a flow chart of a method for training an expression coefficient prediction model provided by an exemplary embodiment.
[0049] Fig.12 It is a structural schematic diagram of a device for determining an expression coefficient provided by an exemplary embodiment.
[0050] Fig.13 It is a schematic structural diagram of a device for determining an expression coefficient provided by an exemplary embodiment. DETAILED DESCRIPTION
[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.
[0052] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0053] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0054] In the field of computer vision, expression coefficients (expression blendshape coefficients) can be used to characterize facial expressions. For example, a set of expression coefficients can be used to characterize a face's degree of smiling expression, angry expression, crying expression, surprised expression, etc., wherein the face can be a real human face, a real animal face, or a virtual model face, etc. The dimension of the expression coefficient determines the accuracy and richness of the expression. For example, a virtual face driven by a 52-dimensional expression coefficient has higher expression accuracy and richer expression than a virtual face driven by a 240-dimensional expression coefficient. Of course, it is not limited to the 52-dimensional expression coefficient or the 240-dimensional expression coefficient. The dimension value of the specific expression coefficient can be determined according to actual needs.
[0055] At present, ordinary consumer-grade avatar expression drivers generally use 52-dimensional expression coefficients, which can express limited expression accuracy. To drive high-precision avatars, hundreds of expression coefficients are generally used. For example, the commonly used 240-dimensional expression coefficients to drive high-precision avatars can capture micro-expressions on the face, making the avatar more vivid and lifelike.
[0056] In order to drive a virtual image, such as a virtual human face, a virtual animal face, etc., usually, a pre-trained model for determining facial expression coefficients in an image is used to determine the expression coefficients, and then the virtual image is driven according to the determined expression coefficients. The method for determining the expression coefficients is as follows: Figure 1 As shown, the target image including the target object is input into a pre-trained expression coefficient prediction model (hereinafter referred to as model X), and model X can output the expression coefficient of the target object in the target image.
[0057] The current training process of model X can be found in Figure 2 First, a certain number of real sample images are collected, and the expression coefficient of each sample image is calculated using the expression resolution algorithm, and then the sample images and the expression coefficients of the sample images are used as sample training model X. In this process, it is often necessary to collect a large number of sample images, for example, no less than 30,000 sample images, and the time and cost of collecting sample images are long; and the amount of calculation of the expression resolution algorithm is also very large.
[0058] Especially when using high-dimensional expression coefficients to drive high-precision virtual images, for example, when using 240-dimensional expression coefficients to drive high-precision virtual images, the acquisition cost and time of collecting a large number of high-precision sample images are higher, and the computational complexity of the expression resolution algorithm is also greater.
[0059] Therefore, in order to reduce the cost of determining the expression coefficient of the target image, the present application provides a method for determining the expression coefficient of the target image, using pre-generated sample expression coefficients to generate an image of a virtual object, and using the sample expression coefficients and the image of the virtual object as samples to train an expression coefficient prediction model, thereby reducing the sample collection cost during the training process. At the same time, it reduces the complexity of expression resolution and improves the efficiency of expression resolution, thereby improving the training and prediction efficiency.
[0060] Next, the embodiments of this specification are described in detail.
[0061] like Figure 3 As shown, Figure 3 This is a flowchart of a special effect rendering provided by an embodiment of this specification, including the following steps:
[0062] S302, acquiring an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0063] S304, inputting the image of the virtual target object into an expression coefficient prediction model, and obtaining a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0064] S306: Determine the first expression coefficient based on the second expression coefficient.
[0065] In the embodiments of this specification, the real object may be a real photographed object (such as a human face, an animal face), etc.; the real target object may be a real object for which the first expression coefficient needs to be determined. The virtual object may be a non-real photographed object (such as a virtual digital human face, a virtual animal face); and the virtual target object may be a virtual object converted from the real target object.
[0066] In S302, an image of a virtual target object obtained by converting an image of a real target object is obtained, wherein the image of the real target object may be an image containing the real target object captured in real time, or may be an image containing the real target object stored in a storage device; wherein the real target object may be a real face with expression, such as a real human face or a real animal face, and the expression may be represented by an expression coefficient; correspondingly, the virtual target object may be a virtual face with expression, such as a virtual human face or a virtual animal face, and the expression may also be represented by an expression coefficient. The image may be one or more images collected separately, or one or more video frames in a collected video.
[0067] Taking the live broadcast scene as an example, the image of the real target object may be an image including the host's face captured by the host's client-side camera, and the host's face is the real target object.
[0068] After acquiring the image of the real target object, the image of the virtual target object can be converted based on the image of the real target object, for example, converting the image of a real human face into the image of a virtual human face, or converting the image of a real human face into the image of a virtual cat face, etc.
[0069] The image of the real target object may be converted into the image of the virtual target object according to certain rules, so there is a certain correlation between the expression of the real target object and the expression of the virtual target object before and after the conversion, that is, the expression coefficient of the real target object (first expression coefficient) and the expression coefficient of the virtual target object (second expression coefficient) are corresponding. For example, the two are the same, or there is a certain conversion relationship between the two. The same situation is used as an example for explanation below.
[0070] In some embodiments, the style of the image of the real target object can be converted, for example, the image of the real target object can be converted into a virtual reality style, or into an animation style, etc. After the conversion, the real target object can be converted into a virtual target object, for example, a real face can be converted into a digital face. Figure 4 shown.
[0071] The expression of the virtual target object obtained by the style conversion technology may be consistent with the expression of the real target object before conversion. In this case, the first expression coefficient of the target object and the second expression coefficient of the virtual target object are the same.
[0072] In some embodiments, the conversion of the image of the real target object by the style conversion technology can be completed by multiple conversion models in collaboration, or by a single conversion model. In the embodiments of this specification, in order to reduce the model cost and the complexity of style conversion, the style conversion of the image of the real target object can be performed by a single conversion model. Figure 5 , input an image of a real object into the conversion model, and the conversion model can output an image of a virtual object corresponding to the image of the real object. The effect diagram of the conversion model can be seen in Figure 6 , Figure 6 (6-1) is a real face image (mosaiced). Figure 6 In (6-2), the image containing the digital face is obtained by converting the style of the real face image in (6-1) into an image containing the digital face. Before and after the style conversion, the expressions of the real face and the digital face are consistent.
[0073] In some embodiments, various types of conversion models can be selected according to actual needs. In order to achieve the effect of converting an image of a real object into an image of a virtual object, and to obtain samples for training the conversion model at a low cost, in the embodiments of this specification, the generator in the cycle generation network cycleGAN is used to implement the conversion function. The structure and training method of cycleGAN refer to Figure 7 .
[0074] CycleGAN includes a generator (i.e., a conversion model) for converting an image of a real object into an image of a virtual object; an inverse generator (i.e., an inverse conversion model) for converting an image of a virtual object generated by the generator into an image of a real object; and a discriminator for determining whether the image generated by the inverse generator is a generated image.
[0075] During training, the first sample image of the real sample object can be input into the conversion model, wherein the real sample object can be a sample face, etc.; the first sample image of the real sample object is style converted by the conversion model to obtain the sample image of the virtual sample object, wherein the virtual sample object can be a virtual sample face, etc. Since the sample image of the virtual sample object is converted from the style of the first sample image of the real sample object, the expressions of the virtual sample object and the real sample object are corresponding, so the expression coefficient of the real sample object and the virtual expression coefficient of the virtual sample object are corresponding. Then the sample image of the virtual sample object is input into the reverse conversion model with parameters corresponding to the conversion model parameters, and the reverse conversion model is used to convert the sample image of the virtual sample object into the second sample image of the real sample object. Then the second sample image is input into the discriminator, wherein the discriminator is used to determine whether the second sample image is a generated image or an image of the real sample object. Then the conversion model is trained based on the discrimination result of the discriminator. The L1 loss function can be used as the loss function for training, and the optimization function uses Adam. In some specific embodiments, the conversion model can also be trained for multiple generations so that the conversion model can achieve better conversion effect.
[0076] In S304, the image of the virtual target object is input into the expression coefficient prediction model, so that the expression coefficient prediction model predicts and outputs the second expression coefficient of the virtual target object. In the embodiment of the present specification, the expression coefficient prediction model can be trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients; (i.e., the sample image is generated based on the sample expression coefficients) Therefore, the expression coefficient prediction model can predict the model of the expression coefficient of the virtual object.
[0077] In some embodiments, a large number of sample expression coefficients may be randomly generated in advance, such as a large number of 52-dimensional expression coefficients or 240-dimensional expression coefficients, etc.; sample images may be generated based on the sample expression coefficients, such as an image of a virtual face. The sample expression coefficients and the sample images may be used as inputs of the expression coefficient prediction model to train the expression coefficient prediction model.
[0078] In some embodiments, sample expression coefficients can be generated in advance, and images of virtual animal faces can be generated using a portion of the sample expression coefficients, and images of virtual human faces can be generated using another portion of the sample expression coefficients. An expression coefficient prediction model can be trained by combining the images of virtual human faces, images of virtual animal faces, and sample expression coefficients, so that the expression coefficient prediction model can predict both the expression coefficients of virtual animal faces and the expression coefficients of virtual human faces.
[0079] In some embodiments, a rendering engine (e.g., Unreal Engine) may be used to obtain a sample image. Specifically, a sample expression coefficient may be input into the rendering engine, and the rendering engine may drive the expression of the virtual image based on the sample expression coefficient, and then generate a sample image containing the virtual image and corresponding to the sample expression coefficient. The virtual image may be a virtual image pre-stored in the rendering engine, such as a digital face pre-stored in the rendering engine.
[0080] For example, a set of 240-dimensional expression coefficients is input into the rendering engine, and the set of 240-dimensional expression coefficients corresponds to expression N. The rendering engine will drive expression N of the digital face pre-stored in the rendering engine based on the set of 240-dimensional expression coefficients, and then generate a sample image containing the digital face.
[0081] In some embodiments, sample images and sample expression coefficients can be used, the sample expression coefficients can be used as labels for sample images, and the labeled sample images can be input into a coefficient regression network (i.e., an expression coefficient prediction model) to train the coefficient regression network. In some embodiments, since the size of each image in the acquired sample images may be different, and the number of input terminals in the input layer of the coefficient regression network is fixed, the pixel size of the sample image can also be adjusted to 224*224 (pixels), and the adjusted image can be used as an image for input into the coefficient regression network to improve the efficiency of the coefficient regression network in extracting image features. In some embodiments, shufflenet (efficient and lightweight network) is used as the backbone network, L1 loss function is used as the loss function, and the Adam optimization function is used to optimize the network.
[0082] Of course, the selection of the above backbone network, loss function and optimization network is only exemplary and is not intended to be a limitation of the embodiments of this specification. Those skilled in the art may also select other backbone networks, loss functions and optimization networks to implement the training of the above coefficient regression network.
[0083] In S306, the first expression coefficient is determined based on the second expression coefficient. That is, the expression coefficient of the real target object is determined based on the expression coefficient of the virtual target object. Since the image of the virtual target object is obtained by converting the image of the real target object, the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object, and the first expression coefficient of the real target object can be determined based on the second expression coefficient of the virtual target object according to the corresponding relationship.
[0084] In some embodiments, when the first expression coefficient is the same as the second expression coefficient, the second expression coefficient can be directly determined as the first expression coefficient; when there is a conversion relationship between the second expression coefficient and the first expression coefficient, the second expression coefficient can be converted into the first expression coefficient based on the conversion relationship.
[0085] In some embodiments, after determining the first expression coefficient of the real target object, the expression of the virtual image can be driven based on the first expression coefficient. For example, the expression of a virtual human face or a virtual animal face can be driven based on the first expression coefficient to enrich the way of using the image of the real target object.
[0086] For example, in a live broadcast, the host client captures an image containing the host's face. After determining the first expression coefficient of the host's face, it drives the expression of a cat's face based on the first label coefficient, and enables the host client and the audience client to display the cat face to replace the host's face, thereby enabling the host to conduct a virtual live broadcast.
[0087] The above embodiments are all examples of the embodiments of this specification. The present application obtains an image of a virtual target object corresponding to an image of a real target object, and based on the corresponding properties of the expression coefficients of the real target object and the virtual target object, determines the expression coefficient of the virtual target object in the image of the virtual target object by using an expression coefficient prediction model, thereby determining the expression coefficient of the real target object; since the virtual object expression prediction model only needs to use pre-generated sample expression coefficients and images of virtual objects generated based on the sample expression coefficients as sample training, it can be seen that the method of the present application does not require the high-cost collection of a large number of real sample images, nor does it require a high-computing-power algorithm to determine the expression coefficient of the real target object.
[0088] Figure 8FIG. 1 is a flow chart of an overall method for determining expression coefficients provided in an embodiment of the present specification. In the embodiment shown in the figure, it is assumed that facial expressions are driven based on 240-dimensional expression coefficients.
[0089] First, the expression coefficient prediction model is pre-trained: a 240-dimensional sample expression coefficient is randomly generated, and the sample expression coefficient is input into a rendering engine for generating a rendered image based on the expression coefficient, so that the rendering engine generates a sample image according to the sample expression coefficient, wherein the generated sample image contains a virtual object. Then, the sample expression coefficient and the sample image are used as training samples and input into the expression coefficient prediction model to train the expression coefficient prediction model, so that the expression coefficient prediction model has the ability to determine the expression coefficient of the virtual object in the image of the virtual object.
[0090] Furthermore, the first expression coefficient of the real target object in the image of the real target object can be determined in the following manner: obtaining an image of the real target object, such as obtaining an image of a real human face; inputting the image of the real target object into a conversion model for the conversion model to perform style conversion, converting the image of the real target object into an image of a virtual target object, such as converting the image of a real human face into an image of a virtual face (digital face); after the conversion model outputs the image of the virtual target object, inputting the image of the virtual target object into an expression coefficient prediction model for the expression coefficient prediction model to predict a second expression coefficient of the virtual target object; and then determining the first expression coefficient of the real target object according to the second expression coefficient of the virtual target object.
[0091] Since the conversion model is to convert the style of the image of the real target object, the expression of the real target object is consistent with the expression of the virtual target object before and after the conversion, so the expression coefficient of the real target object is the same as the expression coefficient of the virtual target object. Therefore, in this embodiment, the second expression coefficient of the virtual target object can be used as the first expression coefficient of the real target object.
[0092] In this embodiment, the real object may be a human face, an animal face, etc. that is actually photographed; the real target object may be a real object for which the first expression coefficient needs to be determined. The virtual object may be a virtual human face, a virtual animal face, etc. that is not actually photographed; the virtual target object may be a virtual human face, a virtual animal face, etc. that is converted from the real target object, and this specification embodiment does not limit this.
[0093] Through the method of this embodiment, it is only necessary to randomly generate 240-dimensional sample expression coefficients, use a rendering engine to generate sample images based on the sample expression coefficients, and use the sample expression coefficients and sample images as samples to train the expression coefficient prediction model, so as to predict the second expression coefficient of the virtual target object in the image of the virtual target object obtained by style conversion of the image of the real target object by the conversion model through the expression coefficient prediction model, and then determine the first expression coefficient of the real target object. It can be seen that this solution does not require the high-cost collection of high-precision sample images with 240-dimensional expression coefficients to train the expression coefficient prediction model, nor does it require a high-computing algorithm to calculate the expression coefficient of the real target object in the image of the real target object, which reduces the cost of determining the expression coefficient of the real target object and saves computing resources.
[0094] See also Fig. 9 The embodiment of this specification also provides a live broadcast system of a virtual image live broadcast, which includes an anchor client, an audience client and a server:
[0095] The anchor client is used to: obtain an image of a virtual target object, where the image of the virtual target object is converted based on an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0096] Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0097] determining the first expression coefficient based on the second expression coefficient;
[0098] driving the expression of the avatar based on the first expression coefficient;
[0099] sending the virtual image to the server;
[0100] The server is used to: receive the virtual image sent by the anchor client, and send the virtual image to the viewer client;
[0101] The audience client is used to receive and display the virtual image sent by the server.
[0102] According to S901, the anchor client obtains an image of the real target object, which may be acquired by a camera. The real target object may be the real anchor's face, and the like.
[0103] According to S902, the anchor client can convert the image of the real target object based on the image of the real target object to obtain the image of the virtual target object; since the image of the virtual target object is converted based on the image of the real target object, the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object.
[0104] According to S903, the anchor client inputs the image of the virtual target object into the expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein, the training process of the expression coefficient prediction model includes: driving the expression of the virtual object based on a pre-generated sample expression coefficient, thereby generating a sample image containing the virtual object, and training the expression coefficient prediction model using the sample expression coefficient and the sample image; therefore, the expression coefficient prediction model after training can predict the expression coefficient of the virtual object.
[0105] According to S904, since the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object, the anchor client can determine the first expression coefficient based on the second expression coefficient.
[0106] According to S905, the anchor client can drive the expression of the virtual image based on the determined first expression coefficient, such as driving the expression of a cat, the expression of a virtual person, etc.
[0107] According to S906, the anchor client sends the virtual image to the server, so that the server executes S907 to send the virtual image to the viewer client; so that after receiving the virtual image, the viewer client executes S908 to display the virtual image.
[0108] See also Fig.10 The embodiment of this specification also provides another live broadcast system of virtual image live broadcast, which includes an anchor client, an audience client and a server:
[0109] The anchor client is used to: obtain an image of a real target object, and send the image of the real target object to the server;
[0110] The server is used to: receive the image of the real target object sent by the anchor client, and obtain the image of the virtual target object, where the image of the virtual target object is obtained by converting the image of the real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object;
[0111] Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0112] determining the first expression coefficient based on the second expression coefficient;
[0113] driving the expression of the avatar based on the first expression coefficient;
[0114] Sending the virtual image to the host client and the viewer client;
[0115] The anchor client is also used to: receive and display the virtual image sent by the server;
[0116] The audience client is used to receive and display the virtual image sent by the server.
[0117] According to S1001, the anchor client obtains an image of the real target object, which may be that the anchor client collects the image of the real target object through a camera, and the real target object may be the anchor's face.
[0118] According to S1002, after acquiring the image of the real target object, the anchor client sends the image of the real target object to the server.
[0119] According to S1003, after receiving the image of the real target object, the server converts the image of the frame target object into the image of the virtual target object. Since the image of the virtual target object is converted based on the image of the real target object, the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object.
[0120] According to S1004, the server inputs the image of the virtual target object into the expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein, the training process of the expression coefficient prediction model includes: driving the expression of the virtual object based on a pre-generated sample expression coefficient, thereby generating a sample image containing the virtual object, and training the expression coefficient prediction model using the sample expression coefficient and the sample image; therefore, the expression coefficient prediction model after training can predict the expression coefficient of the virtual object.
[0121] According to S1005, since the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object, the server can determine the first expression coefficient based on the second expression coefficient.
[0122] According to S1006, the server may drive the expression of the virtual image based on the determined first expression coefficient, for example, driving the expression of a cat, the expression of a virtual person, and so on.
[0123] According to S1007, the server sends the virtual image to the host client and the viewer client, so that after the host client receives the virtual image, S1008 is executed to display the virtual image; and after the viewer client receives the virtual image, S1009 is executed to display the virtual image.
[0124] See also Fig.11 The embodiment of this specification also provides a method for model training, comprising the following steps:
[0125] S1102, driving the expression of the virtual object based on the pre-generated sample expression coefficient to generate a sample image including the virtual object;
[0126] The expression of the virtual object may be driven by a rendering engine based on the randomly generated sample expression coefficients to generate a sample image including the virtual object.
[0127] S1104, training the expression coefficient prediction model based on the sample image and the sample expression coefficient;
[0128] The expression coefficient prediction model is used to obtain a second expression coefficient of the virtual target object based on an image of the virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object;
[0129] The second expression coefficient is used to determine the first expression coefficient; the first expression coefficient is used to drive the expression of the virtual image.
[0130] By training an expression coefficient prediction model including sample images of virtual objects and sample expression coefficients corresponding to the virtual objects, the expression coefficient prediction model can predict the expression coefficient of the virtual object; and since the image of the real target object can be converted into the image of the virtual target object, the expression coefficient prediction model can be used to determine the second expression coefficient of the virtual target object, and then determine the first expression coefficient of the real target object based on the second expression coefficient.
[0131] See also Fig.12 The embodiment of this specification also provides a device for determining an expression coefficient, including:
[0132] Acquisition module 1201: used to acquire an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object;
[0133] Prediction module 1202: used to input the image of the virtual target object into the expression coefficient prediction model, and obtain the second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained by using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients;
[0134] Determination module 1203: used to determine the first expression coefficient based on the second expression coefficient.
[0135] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0136] The embodiments of the device for determining the expression coefficient of this specification can be applied to a computer device. In some embodiments, the computer device also includes a camera; wherein the camera is used to obtain an image of a real target object. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Fig.13 As shown in FIG. 1 , a hardware structure diagram of a computer device where the device for determining the expression coefficient in the embodiment of this specification is located is shown. Fig.13 In addition to the processor 1310, memory 1330, network interface 1320, and non-volatile memory 1340 shown, the server or electronic device where the device 1331 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.
[0137] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0138] Those skilled in the art will readily appreciate other embodiments of the present specification after considering the specification and practicing the inventions claimed herein. The present specification is intended to cover any variations, uses or adaptations of the present specification that follow the general principles of the present specification and include common knowledge or customary technical means in the art that are not claimed in the present specification.
[0139] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
[0140] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.
Claims
1. A method for determining an expression coefficient, characterized in that: The method comprises: Acquire an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object; Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients; The first expression coefficient is determined based on the second expression coefficient.
2. The method according to claim 1, characterized in that: The step of acquiring the image of the virtual target object comprises: Style conversion is performed on the image of the real target object to obtain the image of the virtual target object.
3. The method according to claim 2, characterized in that: The performing style conversion on the real target object includes: The style of the image of the real target object is transferred by the conversion model.
4. The method according to claim 3, characterized in that: The conversion model is trained in the following way: Performing style conversion on a first sample image of a real sample object by using a conversion model to obtain a sample image of a virtual sample object; Inputting the sample image of the virtual sample object into the inverse conversion model to obtain a second sample image of the real sample object; The second sample image is input into a discriminator, and the discriminator is used to determine whether the second sample image is a generated image or an image of a real sample object, and the conversion model is trained based on the determination result of the discriminator.
5. The method according to claim 1, characterized in that: The first expression coefficient is used to drive the expression of the virtual image.
6. The method according to claim 1, characterized in that: The sample image is obtained based on the following method: The sample expression coefficient is input into a rendering engine, and in the rendering engine, the expression of the virtual object is driven based on the sample expression coefficient to generate a sample image including the virtual object.
7. A live broadcast system for a virtual image, characterized in that: Including anchor client, audience client and server: The anchor client is used to: obtain an image of a virtual target object, where the image of the virtual target object is converted based on an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object; Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients; determining the first expression coefficient based on the second expression coefficient; driving the expression of the avatar based on the first expression coefficient; sending the virtual image to the server; The server is used to: receive the virtual image sent by the anchor client, and send the virtual image to the viewer client; The audience client is used to receive and display the virtual image sent by the server.
8. A live broadcast system for a virtual image, characterized in that: Including anchor client, audience client and server: The anchor client is used to: obtain an image of a real target object, and send the image of the real target object to the server; The server is used to: receive the image of the real target object sent by the anchor client, and obtain the image of the virtual target object, where the image of the virtual target object is obtained by converting the image of the real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object; Inputting an image of a virtual target object into an expression coefficient prediction model to obtain a second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients; determining the first expression coefficient based on the second expression coefficient; driving the expression of the avatar based on the first expression coefficient; Sending the virtual image to the anchor client and the viewer client; The anchor client is also used to: receive and display the virtual image sent by the server; The audience client is used to receive and display the virtual image sent by the server.
9. A method for model training, characterized in that: The method comprises: Driving the expression of the virtual object based on the pre-generated sample expression coefficient to generate a sample image including the virtual object; Training an expression coefficient prediction model based on the sample image and the sample expression coefficient; The expression coefficient prediction model is used to obtain a second expression coefficient of the virtual target object based on an image of the virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and the first expression coefficient of the real target object corresponds to the second expression coefficient of the virtual target object; The second expression coefficient is used to determine the first expression coefficient; the first expression coefficient is used to drive the expression of the virtual image.
10. A device for determining an expression coefficient, characterized in that: The device comprises: An acquisition module is used to acquire an image of a virtual target object, wherein the image of the virtual target object is obtained by converting an image of a real target object, and a first expression coefficient of the real target object corresponds to a second expression coefficient of the virtual target object; Prediction module: used for inputting the image of the virtual target object into the expression coefficient prediction model, and obtaining the second expression coefficient output by the expression coefficient prediction model; wherein the expression coefficient prediction model is trained by using pre-generated sample expression coefficients and sample images including virtual objects, and the expression of the virtual object is driven based on the sample expression coefficients; Determination module: used to determine the first expression coefficient based on the second expression coefficient.
11. A device for determining an expression coefficient, comprising a camera, a memory processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Video display method and system, computer equipment, storage medium and terminal
CN109727303A
Facial expression migration method and device, electronic equipment and storage medium
CN112541445A
Expression redirection method and device, equipment and medium
CN113066156A
Virtual image driving method and device, equipment and medium
CN113537056A
Facial expression detection method, device and system, facial expression driving method, device and system, and storage medium
US20190384967A1
