Virtual character model control method, device and electronic equipment

Abstract

The virtual character model control method, device and electronic device provided in the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models to process them to obtain first key point three-dimensional posture data corresponding to different limb parts, and then combine the three-dimensional posture data of each group of first key points to control the same virtual character model to perform corresponding actions. In this way, even if there are fewer training samples of the prediction model, the decoupling between the three-dimensional posture data of the first key points corresponding to different limbs can be achieved, thereby avoiding erroneous limb linkage of the virtual character model when using the three-dimensional posture data of each group of first key points to control the same virtual character model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and electronic device for controlling a virtual character model. Background Art

[0002] In some image processing scenarios, key point recognition and prediction can be performed on a two-dimensional video image of a person, so as to obtain three-dimensional posture data (such as spatial position coordinates and posture angles) of the key points of the person's limbs for modeling or model control. For example, in some live broadcast scenarios, the key point positions of human limbs can be recognized for the two-dimensional live video image obtained from the anchor terminal, and then the three-dimensional posture data can be predicted based on the coordinate position data of the limb key points in the two-dimensional image to obtain the three-dimensional posture data of the limb key points, and finally the corresponding virtual character model is driven according to the three-dimensional posture data to imitate the anchor's actions. Among them, the action of obtaining three-dimensional posture data based on the prediction of the two-dimensional coordinate data of the key points is usually performed by a machine learning model, but due to the limitation of the number of training samples of the machine learning model or the diversity of the training sample data, the coupling of the three-dimensional posture data corresponding to the relatively independent limbs in the prediction results may be too high, so that the subsequent modeling or model control process will produce erroneous limb linkage, affecting the modeling or model control effect. Summary of the invention

[0003] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a virtual character model control method, the method comprising:

[0004] Obtain the two-dimensional coordinate data of the key points of the target person from the two-dimensional image;

[0005] For at least two limb parts of the human body, extracting corresponding at least two sets of first key point data sets from the key point two-dimensional coordinate data respectively;

[0006] Inputting the at least two sets of first key point data sets into at least two different first prediction models for processing, respectively, to obtain first key point three-dimensional pose data corresponding to the at least two limb parts respectively;

[0007] The same virtual character model is controlled to perform corresponding actions according to each group of obtained three-dimensional position and posture data of the first key points.

[0008] In a possible implementation, the step of extracting corresponding at least two sets of first key point data sets from the key point two-dimensional coordinate data for at least two limb parts of the human body, respectively, includes:

[0009] For each of the at least two limb parts, the key point two-dimensional coordinate data corresponding to the limb part and the key point two-dimensional coordinate data corresponding to the torso part are extracted from the key point two-dimensional coordinate data as the first key point data set of the limb part.

[0010] In a possible implementation, the three-dimensional posture data of the first key point includes spatial position data and posture angle data of preset joint points in the corresponding limb part.

[0011] In a possible implementation, the method further includes:

[0012] The two-dimensional coordinate data of the key points of the target person are used as a second key point data set;

[0013] Inputting the second key point data set into the second prediction model for processing to obtain overall three-dimensional pose data, wherein the overall three-dimensional pose data includes the third key point three-dimensional pose data corresponding to the at least two limb parts and the second key point three-dimensional pose data corresponding to the torso part;

[0014] The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises:

[0015] The first key point three-dimensional pose data is used to replace the third key point three-dimensional pose data in the overall three-dimensional pose data, and the replaced overall three-dimensional pose data is used to control the virtual character model to perform corresponding actions.

[0016] In a possible implementation, the at least two limb parts include a left arm and a right arm, and the at least two different first prediction models include a left arm prediction model and a right arm prediction model;

[0017] The left arm prediction model includes a first fully connected network of the left arm, a second fully connected network of the left arm, and a third fully connected network of the left arm, which are connected in sequence; the input of the first fully connected network of the left arm is the first key point data set of the left arm of 44 dimensions, and the output of the first fully connected network of the left arm is 512-dimensional data; the input of the second fully connected network of the left arm is the 512-dimensional data output by the first fully connected network of the left arm, and the output of the second fully connected network of the left arm is 512-dimensional data; the input of the third fully connected network of the left arm is the 512-dimensional data output by the second fully connected network of the left arm, and the output of the third fully connected network of the left arm is 12-dimensional three-dimensional pose data of the first key point of the left arm;

[0018] The right arm prediction model includes a right arm first fully connected network, a right arm second fully connected network and a right arm third fully connected network connected in sequence; the input of the right arm first fully connected network is the first key point data set of the right arm of 44 dimensions, and the output of the right arm first fully connected network is 512-dimensional data; the input of the right arm second fully connected network is the 512-dimensional data output by the right arm first fully connected network, and the output of the right arm second fully connected network is 512-dimensional data; the input of the right arm third fully connected network is the 512-dimensional data output by the right arm second fully connected network, and the output of the right arm third fully connected network is 12-dimensional three-dimensional pose data of the first key point of the right arm;

[0019] The second prediction model includes a first fully connected network of the trunk, a second fully connected network of the trunk, and a third fully connected network of the trunk, which are connected in sequence; the input of the first fully connected network of the trunk is 48-dimensional three-dimensional pose data of the second key point, and the output of the first fully connected network of the trunk is 512-dimensional data; the input of the second fully connected network of the trunk is the 512-dimensional data output of the first fully connected network of the trunk, and the output of the second fully connected network of the trunk is 512-dimensional data; the input of the third fully connected network of the trunk is the 512-dimensional data output of the second fully connected network of the trunk, and the output of the third fully connected network of the trunk is 144-dimensional overall three-dimensional pose data.

[0020] In a possible implementation, the method further includes:

[0021] Extracting a second key point data set including key points of the torso from the key point two-dimensional coordinate data;

[0022] Inputting the second key point data set into a second prediction model for processing to obtain second key point three-dimensional pose data;

[0023] The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises:

[0024] The first key point three-dimensional pose data and the second key point three-dimensional pose data are used to control the same virtual character model to perform corresponding actions.

[0025] In a possible implementation, the step of acquiring two-dimensional coordinate data of key points of the target person from the two-dimensional image includes:

[0026] Acquire the two-dimensional coordinate data of the key points of the anchor user from the first live video image of the anchor user; the first live video image is the two-dimensional image;

[0027] The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises:

[0028] According to each group of the obtained three-dimensional position and posture data of the first key points, the virtual character model corresponding to the anchor user is controlled to perform corresponding actions, so that the virtual character model performs actions similar to those of the anchor user.

[0029] Another object of the present application is to provide a virtual character model control device, the device comprising:

[0030] An acquisition module is used to acquire two-dimensional coordinate data of key points of a target person from a two-dimensional image;

[0031] An extraction module, for extracting corresponding at least two first key point data sets from the key point two-dimensional coordinate data for at least two limb parts of the human body;

[0032] A prediction module, used for inputting the at least two sets of first key point data sets into at least two different first prediction models for processing, to obtain first key point three-dimensional posture data corresponding to the at least two limb parts respectively;

[0033] The model control module is used to control the same virtual character model to perform corresponding actions according to the obtained groups of three-dimensional position and posture data of the first key points.

[0034] Another object of the present application is to provide an electronic device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the virtual character model control method provided in the present application is implemented.

[0035] Another object of the present application is to provide a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed by one or more processors, they implement the virtual character model control method provided by the present application.

[0036] Compared with the prior art, this application has the following beneficial effects:

[0037] The virtual character model control method, device and electronic device provided in the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models to process them to obtain first key point three-dimensional posture data corresponding to different limb parts, and then combine the three-dimensional posture data of each group of first key points to control the same virtual character model to perform corresponding actions. In this way, even if there are fewer training samples of the prediction model, the decoupling between the three-dimensional posture data of the first key points corresponding to different limbs can be achieved, thereby avoiding erroneous limb linkage of the virtual character model when using the three-dimensional posture data of each group of first key points to control the same virtual character model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the steps of the virtual character model control method provided in an embodiment of the present application.

[0040] Figure 2 A schematic diagram of a live broadcast system provided in an embodiment of the present application.

[0041] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application.

[0042] Figure 4 A schematic diagram of the functional modules of the virtual character model control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0046] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0047] In the description of the present application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0048] The inventors have found that the process of predicting the three-dimensional posture data of the key points of a person's limbs based on the two-dimensional video image of the person is usually performed by a trained machine learning model. In the training process of the machine learning model, a person wearing a three-dimensional posture sensor usually performs various actions first, and the three-dimensional posture data of the key points of the limbs are collected by the three-dimensional posture sensor, and the two-dimensional image is collected by a two-dimensional image acquisition device (such as a camera) to obtain the two-dimensional coordinates of the key points of the limbs on the two-dimensional image. Then, the two-dimensional coordinates and three-dimensional posture data of each key point of the limbs are used as training samples, and the machine learning model is trained to predict the three-dimensional posture data of the key points of the limbs based on the two-dimensional coordinates of the key points of the limbs.

[0049] In the above method, due to the number of training samples or the diversity of training sample data, there may be a large number of scenes in which different limbs move simultaneously in the training samples, but there are few scenes in which a certain limb moves alone. For example, there may be a large number of scenes in which the left hand and the right hand move together in the training samples, and there are few scenes in which only one hand moves. This will cause the three-dimensional pose data prediction results output by the trained machine learning model to always tend to move different limbs together, and there is a problem of transition coupling of three-dimensional pose data of different limbs, which will lead to incorrect limb linkage in subsequent model reconstruction or model control. For example, there is only left hand movement in the actual two-dimensional image, but the prediction result of the machine learning model is left hand movement and the right hand also has slight movement.

[0050] In view of the discovery and research on the above-mentioned problems, this embodiment provides a solution that can reduce the erroneous body linkage of the virtual character model. The solution provided by this embodiment is described in detail below.

[0051] See also Figure 1 , Figure 1 A flowchart of a virtual character model control method provided in this embodiment, and each step of the method will be described in detail below.

[0052] Step S110, obtaining the two-dimensional coordinate data of the key points of the target person from the two-dimensional image.

[0053] In this embodiment, the limb key points may correspond to various limb joints of the target person, such as shoulders, elbows, wrists, etc. In a possible implementation, a pre-trained key point recognition model may be used to perform image recognition on a two-dimensional image containing the target person (such as a two-dimensional video image captured by a camera), thereby determining the positions of various limb key points of the target person in the two-dimensional image, and then obtaining the two-dimensional coordinate data of the key points of the target person.

[0054] Step S120, for at least two limb parts of the human body, extract corresponding at least two groups of first key point data sets from the key point two-dimensional coordinate data respectively.

[0055] In this embodiment, according to the human body key point recognition result of the two-dimensional image, for at least two human body parts that can move independently, corresponding at least two groups of first key point data sets can be extracted from the key point two-dimensional coordinate data. For example, the left arm (including the left shoulder, the left upper arm, the left lower arm, and the left hand) and the right arm (including the right shoulder, the right upper arm, the right lower arm, and the right hand) are two limbs that can move relatively independently. In this embodiment, at least one group of first key point data sets corresponding to the left arm and one group of first key point data sets corresponding to the right arm can be extracted from the key point two-dimensional coordinate data. The first key point data set may include the key point two-dimensional coordinate data of the key joints of the corresponding limb parts, for example, the first key point data set of the left arm at least includes the key point two-dimensional coordinate data of the left elbow and the left wrist, and the first key point data set of the right arm at least includes the key point two-dimensional coordinate data of the right elbow and the right wrist.

[0056] Optionally, in this embodiment, each of the first key point data sets may correspond to different limb parts. For example, the first key point data set corresponding to the left arm may not include the coordinate data of the key points corresponding to the right arm, and the first key point data set corresponding to the right arm may not include the coordinate data of the key points corresponding to the left arm.

[0057] Step S130, inputting the at least two groups of first key point data sets into at least two different first prediction models for processing, respectively, to obtain first key point three-dimensional posture data corresponding to the at least two limb parts respectively.

[0058] In this embodiment, the at least two different first prediction models may be machine learning models that do not share network parameters. It is understandable that, in some cases, the at least two different first prediction models may have the same model network structure, but may have different model parameters depending on the training samples.

[0059] In this embodiment, the first key point data sets corresponding to different limb parts can be input into different first prediction models for relatively independent prediction, so that the predicted three-dimensional pose data of each group of first key points are decoupled. For example, the first key point data sets of the left arm and the right arm are respectively input into different first prediction models for prediction, so the two-dimensional coordinate data of the key points corresponding to the right arm will not affect the three-dimensional pose data of the first key points corresponding to the left arm, and the two-dimensional coordinate data of the key points corresponding to the left arm will not affect the three-dimensional pose data of the first key points corresponding to the right arm, thereby achieving the decoupling of the three-dimensional pose data of the first key points between the left arm and the right arm.

[0060] The first key point three-dimensional posture data may include spatial position data and posture angle data of preset joint points in the corresponding limb part. The spatial position data may be the three-dimensional spatial position coordinates corresponding to the joint points, and the posture angle data may be the rotation change angle of the joint points in three directions of the three-dimensional space compared to the initial posture.

[0061] Step S140: controlling the same virtual character model to perform corresponding actions according to the obtained groups of three-dimensional position and posture data of the first key points.

[0062] In this embodiment, the limbs corresponding to the virtual character model can be controlled respectively according to each set of the first key point three-dimensional posture data. Since each set of the first key point three-dimensional posture data is decoupled, when each set of the first key point three-dimensional posture data is used together to control the same virtual character model, incorrect limb linkage of the virtual character model is avoided.

[0063] In one possible implementation, in step S120, for each of the at least two limb parts, the key point two-dimensional coordinate data corresponding to the limb part and the key point two-dimensional coordinate data corresponding to the torso part can be extracted from the key point two-dimensional coordinate data as the first key point data set of the limb part.

[0064] For example, since the left arm is connected to the torso and has a strong linkage relationship, in this embodiment, when obtaining the first key point data set corresponding to the left arm, the key point two-dimensional coordinate data corresponding to each key joint of the left arm (such as the left elbow, left wrist, etc.) and the key point two-dimensional coordinate data corresponding to the torso can be extracted from the key point two-dimensional coordinate data as the first key point data set for the left arm. In this way, when the three-dimensional posture data of the left arm is predicted later, the prediction can be made together based on the key point two-dimensional coordinate data of the left arm and the torso, which can make the prediction result more accurate.

[0065] When controlling the virtual character model to move, in addition to the 3D position data of the limbs, the 3D position data of the torso may also be required. Therefore, in a possible implementation, the method may further include the following steps.

[0066] Step S210: taking the two-dimensional coordinate data of the key points of the target person as a whole as a second key point data set.

[0067] Step S220, input the second key point data set into the second prediction model for processing to obtain overall three-dimensional pose data, wherein the overall three-dimensional pose data includes third key point three-dimensional pose data corresponding to the at least two limb parts and second key point three-dimensional pose data corresponding to the torso part.

[0068] In this embodiment, since all limbs are connected to the torso, in this implementation, in order to accurately predict the three-dimensional posture data of the torso, the two-dimensional coordinate data of the key points of the target person can be input as a whole as the second key point data set into the second prediction model for processing. The data output by the second prediction model may include the three-dimensional posture data of the third key points corresponding to the at least two limb parts and the three-dimensional posture data of the second key points corresponding to the torso. The torso may include the upper body of the human body except the left and right arms, such as the body, neck, and head.

[0069] It is understandable that in this embodiment, due to the number of training samples or the diversity of training samples, the three-dimensional pose data corresponding to the third key point of each limb part may be over-coupled. Therefore, in step S140, the three-dimensional pose data of the first key point can be used to replace the three-dimensional pose data of the third key point in the overall three-dimensional pose data, and the overall three-dimensional pose data after replacement can be used to control the virtual character model to perform corresponding actions. In this way, the prediction result (the first key three-dimensional pose data) that may be over-coupled in the overall three-dimensional pose data is replaced by the decoupled prediction result (the third key three-dimensional pose data), and the overall three-dimensional pose data after replacement is used to control the virtual character model to perform corresponding actions, which can avoid the situation where the virtual character model has incorrect linkage of limb movements.

[0070] Specifically, in this embodiment, taking the three-dimensional posture data prediction processing of the upper body of the target person as an example, the at least two limb parts include a left arm and a right arm, and the at least two different first prediction models include a left arm prediction model and a right arm prediction model.

[0071] In this case, the left arm prediction model includes a first fully connected network for the left arm, a second fully connected network for the left arm, and a third fully connected network for the left arm, which are connected in sequence. The input of the first fully connected network for the left arm is a 44-dimensional data set of the first key point of the left arm, which may include two-dimensional coordinate data of 22 joints of the left arm and the trunk. The output of the first fully connected network for the left arm is 512-dimensional data, the input of the second fully connected network for the left arm is the 512-dimensional data output by the first fully connected network for the left arm, and the output of the second fully connected network for the left arm is 512-dimensional data. The input of the third fully connected network for the left arm is the 512-dimensional data output by the second fully connected network for the left arm, and the output of the third fully connected network for the left arm is 12-dimensional three-dimensional pose data of the first key point of the left arm, which may include 6D pose data of the two joints of the left elbow and the left wrist.

[0072] The right arm prediction model includes a first fully connected network for the right arm, a second fully connected network for the right arm, and a third fully connected network for the right arm, which are connected in sequence. The input of the first fully connected network for the right arm is a 44-dimensional data set of the first key point of the right arm, which may include the two-dimensional coordinate data of the 22 joints of the right arm and the trunk. The output of the first fully connected network for the right arm is 512-dimensional data, the input of the second fully connected network for the right arm is the 512-dimensional data output by the first fully connected network for the right arm, and the output of the second fully connected network for the right arm is 512-dimensional data. The input of the third fully connected network for the right arm is the 512-dimensional data output by the second fully connected network for the right arm, and the output of the third fully connected network for the right arm is 12-dimensional three-dimensional pose data of the first key point of the right arm, which may include 6D pose data of the two joints of the right elbow and the right wrist.

[0073] The second prediction model includes a first fully connected network of the trunk, a second fully connected network of the trunk, and a third fully connected network of the trunk, which are connected in sequence. The input of the first fully connected network of the trunk is 48-dimensional three-dimensional pose data of the second key point, which may include two-dimensional coordinate data of 24 joints of the left arm, the right arm, and the trunk. The output of the first fully connected network of the trunk is 512-dimensional data, the input of the second fully connected network of the trunk is 512-dimensional data output by the second fully connected network of the trunk, the output of the second fully connected network of the trunk is 512-dimensional data, the input of the third fully connected network of the trunk is 512-dimensional data output by the second fully connected network of the trunk, and the output of the third fully connected network of the trunk is 144-dimensional overall three-dimensional pose data, which may include 6D pose data of 24 joints of the left arm, the right arm, and the trunk.

[0074] As another possible implementation manner, the method further includes the following steps.

[0075] Step S310: extracting a second key point data set including key points of the torso from the key point two-dimensional coordinate data.

[0076] Step S320: input the second key point data set into a second prediction model for processing to obtain second key point three-dimensional pose data.

[0077] And in step S140, the first key point three-dimensional pose data and the second key point three-dimensional pose data can be used to control the same virtual character model to perform corresponding actions.

[0078] The torso may include the upper body of the human body except the left and right arms, such as the body, neck, and head. In this implementation, by extracting the two-dimensional coordinate data of the key points corresponding to the torso separately and processing them using a separate prediction model, the predicted three-dimensional pose data of the key points of the torso and other limbs are also decoupled, further avoiding the generation of erroneous limb or body linkage in subsequent modeling or control.

[0079] In this embodiment, the above scheme can be applied to the virtual image control in the live broadcast system, wherein the two-dimensional image containing the target task can be a live video image obtained from the live broadcast terminal, and the virtual character model can be a virtual character image corresponding to the anchor.

[0080] Specifically, in a possible implementation, please refer to Figure 2 , Figure 2 Schematic diagram of a live broadcast system, which may include an anchor terminal 201, a server 202 and an audience terminal 203.

[0081] The anchor user can shoot a live video image through the anchor terminal 201, and the live video image can include a half-body or full-body image of the anchor user.

[0082] The server 202 can be an independent device or a cluster of multiple devices working together. The server 202 can obtain the two-dimensional coordinate data of the key points of the anchor user from the live video image of the anchor user, and predict the first three-dimensional posture data of the key points corresponding to each limb part based on the two-dimensional coordinate data of the key points. Then, based on each group of the obtained three-dimensional posture data of the first key points, the virtual character model corresponding to the anchor user is controlled to perform corresponding actions, so that the virtual character model performs actions similar to those of the anchor user. Then the server 202 can send the second live video image containing the virtual image and the second live video image to the audience terminal 203 or the anchor terminal 201 for display.

[0083] Based on the same inventive concept, this embodiment also provides an electronic device, which may have a certain image processing capability. For example, the electronic device may be a personal computer or Figure 2 Server 202 is shown.

[0084] Please refer to Figure 3 , Figure 3 1 is a block diagram of the electronic device 100. The electronic device 100 includes a virtual character model control device 110, a machine-readable storage medium 120, and a processor 130.

[0085] The machine-readable storage medium 120, the processor 130 and the communication unit 140 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The virtual character model control device 110 includes at least one software function module that can be stored in the machine-readable storage medium 120 in the form of software or firmware or solidified in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the virtual character model control device 110.

[0086] The machine-readable storage medium 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store a program, and the processor 130 executes the program after receiving an execution instruction.

[0087] The processor 130 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0088] Please refer to Figure 4This embodiment further provides a virtual character model control device 110, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in the form of software. From a functional perspective, the virtual character model control device 110 may include an acquisition module 111, an extraction module 112, a prediction module 113, and a model control module 114.

[0089] The acquisition module 111 is used to acquire the two-dimensional coordinate data of key points of the target person from the two-dimensional image.

[0090] In this embodiment, the acquisition module 111 can be used to execute Figure 1 As shown in step S110, for a detailed description of the acquisition module 111, please refer to the description of step S110.

[0091] The extraction module 112 is used to extract corresponding at least two sets of first key point data sets from the key point two-dimensional coordinate data for at least two limb parts of the human body.

[0092] In this embodiment, the extraction module 112 can be used to perform Figure 1 As shown in step S120, for a detailed description of the extraction module 112, please refer to the description of step S120.

[0093] The prediction module 113 is used to input the at least two groups of first key point data sets into at least two different first prediction models for processing, so as to obtain first key point three-dimensional posture data corresponding to the at least two limb parts respectively.

[0094] In this embodiment, the prediction module 113 can be used to perform Figure 1 As shown in step S130, for a detailed description of the prediction module 113, please refer to the description of step S130.

[0095] The model control module 114 is used to control the same virtual character model to perform corresponding actions according to the obtained groups of the first key point three-dimensional posture data.

[0096] In this embodiment, the model control module 114 can be used to execute Figure 1 As shown in step S140 , for a detailed description of the model control module 114 , please refer to the description of step S140 .

[0097] In summary, the virtual character model control method, device and electronic device provided in the embodiments of the present application extract different first key point data sets for different limb parts, and use different first prediction models to process them to obtain first key point three-dimensional posture data corresponding to different limb parts, and then combine the three-dimensional posture data of each group of first key points to control the same virtual character model to perform corresponding actions. In this way, even if there are fewer training samples for the prediction model, the decoupling between the first key point three-dimensional posture data corresponding to different limbs can be achieved, thereby avoiding erroneous limb linkage of the virtual character model when using each group of first key point three-dimensional posture data to control the same virtual character model.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0100] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0102] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling a virtual character model, characterized in that: The method comprises: Obtain the two-dimensional coordinate data of the key points of the target person from the two-dimensional image; For at least two limb parts of the human body, extracting corresponding at least two sets of first key point data sets from the key point two-dimensional coordinate data respectively; Inputting the at least two sets of first key point data sets into at least two different first prediction models for processing, respectively, to obtain first key point three-dimensional pose data corresponding to the at least two limb parts respectively; Controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional position and posture data of each group of the first key points; The step of extracting corresponding at least two sets of first key point data sets from the key point two-dimensional coordinate data for at least two limb parts of the human body, respectively, comprises: For each of the at least two limb parts, the key point two-dimensional coordinate data corresponding to the limb part and the key point two-dimensional coordinate data corresponding to the torso part are extracted from the key point two-dimensional coordinate data as the first key point data set of the limb part.

2. The method according to claim 1, characterized in that The three-dimensional posture data of the first key point includes spatial position data and posture angle data of preset joint points in the corresponding limb part.

3. The method according to claim 1, characterized in that The method further comprises: The two-dimensional coordinate data of the key points of the target person are used as a second key point data set; Inputting the second key point data set into the second prediction model for processing to obtain overall three-dimensional pose data; wherein the overall three-dimensional pose data includes the third key point three-dimensional pose data corresponding to the at least two limb parts and the second key point three-dimensional pose data corresponding to the torso part; The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises: The first key point three-dimensional pose data is used to replace the third key point three-dimensional pose data in the overall three-dimensional pose data, and the replaced overall three-dimensional pose data is used to control the virtual character model to perform corresponding actions.

4. The method according to claim 3, characterized in that The at least two limb parts include a left arm and a right arm, and the at least two different first prediction models include a left arm prediction model and a right arm prediction model; The left arm prediction model includes a first fully connected network of the left arm, a second fully connected network of the left arm, and a third fully connected network of the left arm, which are connected in sequence; the input of the first fully connected network of the left arm is the first key point data set of the left arm of 44 dimensions, and the output of the first fully connected network of the left arm is 512-dimensional data; the input of the second fully connected network of the left arm is the 512-dimensional data output by the first fully connected network of the left arm, and the output of the second fully connected network of the left arm is 512-dimensional data; the input of the third fully connected network of the left arm is the 512-dimensional data output by the second fully connected network of the left arm, and the output of the third fully connected network of the left arm is 12-dimensional three-dimensional pose data of the first key point of the left arm; The right arm prediction model includes a right arm first fully connected network, a right arm second fully connected network and a right arm third fully connected network connected in sequence; the input of the right arm first fully connected network is the first key point data set of the right arm of 44 dimensions, and the output of the right arm first fully connected network is 512-dimensional data; the input of the right arm second fully connected network is the 512-dimensional data output by the right arm first fully connected network, and the output of the right arm second fully connected network is 512-dimensional data; the input of the right arm third fully connected network is the 512-dimensional data output by the right arm second fully connected network, and the output of the right arm third fully connected network is 12-dimensional three-dimensional pose data of the first key point of the right arm; The second prediction model includes a first fully connected network of the trunk, a second fully connected network of the trunk, and a third fully connected network of the trunk, which are connected in sequence; the input of the first fully connected network of the trunk is 48-dimensional three-dimensional pose data of the second key point, and the output of the first fully connected network of the trunk is 512-dimensional data; the input of the second fully connected network of the trunk is the 512-dimensional data output of the first fully connected network of the trunk, and the output of the second fully connected network of the trunk is 512-dimensional data; the input of the third fully connected network of the trunk is the 512-dimensional data output of the second fully connected network of the trunk, and the output of the third fully connected network of the trunk is 144-dimensional overall three-dimensional pose data.

5. The method according to claim 1, characterized in that The method further comprises: Extracting a second key point data set including key points of the torso from the key point two-dimensional coordinate data; Inputting the second key point data set into a second prediction model for processing to obtain second key point three-dimensional pose data; The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises: The first key point three-dimensional pose data and the second key point three-dimensional pose data are used to control the same virtual character model to perform corresponding actions.

6. The method according to claim 1, characterized in that The step of obtaining the two-dimensional coordinate data of the key points of the target person from the two-dimensional image includes: Acquire the two-dimensional coordinate data of the key points of the anchor user from the first live video image of the anchor user; the first live video image is the two-dimensional image; The step of controlling the same virtual character model to perform corresponding actions according to each group of the obtained three-dimensional position and posture data of the first key points comprises: According to each group of the obtained three-dimensional position and posture data of the first key points, the virtual character model corresponding to the anchor user is controlled to perform corresponding actions, so that the virtual character model performs actions similar to those of the anchor user.

7. A virtual character model control device, characterized in that: The device comprises: An acquisition module is used to acquire two-dimensional coordinate data of key points of a target person from a two-dimensional image; An extraction module, for extracting corresponding at least two first key point data sets from the key point two-dimensional coordinate data for at least two limb parts of the human body; A prediction module, used for inputting the at least two sets of first key point data sets into at least two different first prediction models for processing, to obtain first key point three-dimensional posture data corresponding to the at least two limb parts respectively; A model control module, used for controlling the same virtual character model to perform corresponding actions according to the obtained three-dimensional position and posture data of each group of the first key points; The extraction module is also used to extract, for each of the at least two limb parts, the key point two-dimensional coordinate data corresponding to the limb part and the key point two-dimensional coordinate data corresponding to the torso part from the key point two-dimensional coordinate data as the first key point data set of the limb part.

8. An electronic device, characterized in that: The method comprises a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by one or more processors, the method according to any one of claims 1 to 6 is implemented.

Images