A method and device for predicting leg movements of a virtual live broadcast character

Abstract

The present invention discloses a method and device for predicting the leg movements of a virtual live broadcast character, which belongs to the field of virtual live broadcast technology and is used to solve the technical problems of the existing leg movement prediction methods, such as the large amount of training data required, the slow model training speed, and the low accuracy of the model prediction results. The method comprises: constructing a leg movement prediction network model; collecting the leg movement data and foot pressure data of the experimental subject, and compiling them into a model training set; training the leg movement prediction network model through the model training set; for the live broadcast subject that lacks actual leg movement in the virtual live broadcast, generating initial leg movement data corresponding to the first picture frame, inputting the data into the leg movement prediction network model, obtaining the leg movement prediction data and foot pressure prediction data corresponding to the second picture frame, generating the leg prediction movement of the second picture frame, and displaying it on the virtual image of the live broadcast subject. The present invention reduces the amount of training data and increases the authenticity of the predicted leg movement.

Description

Technical Field

[0001] The present application relates to the field of virtual live broadcast technology, and in particular to a method and device for predicting leg movements of a virtual live broadcast character. Background Art

[0002] Virtual live streaming is a new live streaming method achieved through the integration of "virtual reality" and "reality." Compared to traditional live streaming, it is faster to set up and more flexible, allowing for quick switching between different spaces and avatars. A large number of virtual live streamers have emerged, and they are very popular among young people. Gesture recognition technology is a key supporting technology in the implementation of virtual live streaming. Currently, a variety of technologies for generating half-body and full-body virtual characters have been developed. By recognizing the posture and movements of the host, the virtual character can perform the same movements.

[0003] However, during a live broadcast of a full-body virtual character, the host's legs may appear outside the camera's field of view, such as when the host is too close to the camera. In this case, since the leg movements cannot be obtained, the virtual character's legs will move erratically, greatly affecting the live broadcast effect. The existing solution is to predict the movement posture of invisible key points by the position of visible key points, but the training data used in these methods is key point position data, the amount of data required for training is huge, and the model training speed is slow. In addition, when predicting multiple invisible key points from a few visible key points, the number of key points used as reference data is small, and other factors that can reflect leg movements are not combined, so the accuracy of the prediction results cannot be guaranteed. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for predicting the leg movements of a virtual live broadcast character, which are used to solve the following technical problems: the existing leg movement prediction methods require a large amount of training data, the model training speed is slow, and the model prediction results are less accurate.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for predicting leg movements of a virtual live broadcast character, the method comprising: constructing a leg movement prediction network model; wherein the leg movement prediction network model is composed of an LSTM prediction network and a foot pressure classifier; collecting leg movement data and foot pressure data of an experimental subject, and compiling them into a model training set; wherein the leg movement data at least includes the body displacement of the experimental subject and the relative direction of each key point; training the leg movement prediction network model through the model training set; for a live broadcast subject that lacks actual leg movement during a virtual live broadcast, generating initial leg movement data corresponding to a first frame; and inputting the initial leg movement data into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to a second frame; generating predicted leg movements of the second frame based on the leg movement prediction data and the foot pressure prediction data, and displaying them on the virtual image of the live broadcast subject.

[0007] In a feasible embodiment, a leg motion prediction network model is constructed, specifically including: building an LSTM prediction network and setting the network parameters of the LSTM prediction network; wherein the network parameters include at least time step, BatchSize, input data length, output data length, hidden layer Size and number of hidden layer layers; constructing a classifier structure and setting the classifier parameters of the classifier structure to obtain the foot pressure classifier; wherein the classifier parameters include at least time step and output data length; connecting one output end of the LSTM prediction network to the input end of the foot pressure classifier, and using the other output end of the LSTM prediction network as the first output end of the leg motion prediction network model, and using the output end of the foot pressure classifier as the second output end of the leg motion prediction network model.

[0008] In a feasible embodiment, the leg motion data and foot pressure data of the experimental subject are collected and compiled into a model training set, which specifically includes: determining a number of key points on the body of the experimental subject and creating a parent-child node relationship between the key points; within a preset experimental time, continuously collecting the relative direction of each key point during the movement of the experimental subject through a motion capture device; wherein the relative direction is the quaternion of the rotation angle of the current key point relative to the parent node; in the parent-child node relationship, obtaining the current position and the position of the key point corresponding to the root node at the previous moment, and subtracting the current position from the position at the previous moment to obtain the body displacement of the experimental subject at each moment; collecting the foot pressure data of the experimental subject's feet through pressure sensors installed on the soles of the experimental subject's feet; and compiling the relative direction of each key point, the experimental subject's body displacement and foot pressure data collected at each moment within the preset experimental time into the model training set.

[0009] In a feasible implementation, several key points on the body of the experimental subject are determined, and parent-child node relationships are created between the several key points, specifically including: determining preset parts on the body of the experimental subject as the key points; wherein the preset parts include the spine, pelvis, right hip, right knee, right ankle, left hip, left knee and left ankle; determining the key point corresponding to the pelvis as the root node, and determining the spine key point, left hip key point and right hip key point as child nodes of the root node; and creating parent-child node relationships for the remaining key points based on the upper and lower positional relationships of the leg joints.

[0010] In a feasible embodiment, the leg motion prediction network model is trained through the model training set, specifically including: taking the leg motion data of each frame in the model training set as the input of the leg motion prediction network model, taking the corresponding next frame leg motion data as the output of the LSTM prediction network, and taking the corresponding next frame foot pressure data as the output of the foot pressure classifier to train the leg motion prediction network model; testing the trained leg motion prediction network model through a test data set; calculating the mean square error between the leg motion prediction data obtained after the test and the actual leg motion data through a first loss function; calculating the binary cross entropy between the foot pressure prediction data obtained after the test and the actual foot pressure data through a second loss function; updating the weight parameters of the leg motion prediction network model according to the sum of the mean square error and the binary cross entropy, and retraining the leg motion prediction network model until the model converges.

[0011] In a feasible implementation, for a live broadcast object that lacks actual leg movements during a virtual live broadcast, initial leg movement data corresponding to a first frame is generated; and the initial leg movement data is input into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to a second frame, specifically including: during a virtual live broadcast, when it is detected that the leg key points of the live broadcast object are blocked or not within the lens frame, the frame at the current moment is determined as the first frame, and the previous frame of the first frame is determined as the zeroth frame; wherein all key points in the zeroth frame are visible; according to the position coordinates of the visible key points in the first frame , generate initial position coordinates for invisible key points; wherein, the visible key points include at least spine key points and pelvic key points; determine the initial leg motion data based on the position coordinates of all key points in the first picture frame and the known position coordinates of all key points in the zeroth picture frame; wherein, the initial leg motion data includes: the relative direction of each key point in the current frame and the body displacement of the live broadcast object; input the initial leg motion data into the leg motion prediction network model, so that the LSTM prediction network in the leg motion prediction network model outputs the leg motion prediction data corresponding to the second picture frame, and the foot pressure classifier outputs the foot pressure prediction data.

[0012] In a feasible implementation, based on the leg movement prediction data and the foot pressure prediction data, the predicted leg movement of the second picture frame is generated and displayed on the virtual image of the live broadcast object, specifically including: arranging the key points in the leg movement prediction data according to their respective relative directions, and driving the virtual image of the live broadcast object to make the same predicted leg movement and body displacement based on the arrangement results and the body displacement in the leg movement prediction data; determining the actual position of the foot of the live broadcast object in the second picture frame based on the foot pressure prediction data, and adjusting the predicted leg movement of the virtual image based on the actual position of the foot.

[0013] In a feasible implementation manner, based on the foot pressure prediction data, the actual position of the foot of the live broadcast object in the second picture frame is determined, specifically including: if at least one pressure value in the foot pressure prediction data is 1, it is determined that the foot corresponding to the foot pressure prediction data is in contact with the ground; if all pressure values ​​in the foot pressure prediction data are 0, it is determined that the foot corresponding to the foot pressure prediction data is not in contact with the ground.

[0014] In a feasible embodiment, after generating the predicted leg movement of the second frame based on the leg movement prediction data and the foot pressure prediction data, and displaying it on the virtual image of the live broadcast object, the method also includes: performing displacement calculation on the pelvic key point position of the second frame and the pelvic key point position of the first frame to obtain the real body displacement of the live broadcast object in the second frame; and replacing the predicted body displacement in the leg movement prediction data of the second frame with the real body displacement; determining the real relative direction between the visible key points according to the position of the visible key points in the second frame; and replacing the corresponding predicted relative direction in the leg movement prediction data of the second frame with the real relative direction; inputting the replaced leg movement prediction data into the leg movement prediction network model to obtain the leg movement prediction data of the third frame, and generating the corresponding leg prediction movement; looping this step to obtain the leg movement prediction data of each frame until the complete actual leg movement picture is re-displayed in the lens or the virtual live broadcast ends.

[0015] On the other hand, an embodiment of the present application also provides a device for predicting leg movements of a virtual live broadcast character, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a method for predicting leg movements of a virtual live broadcast character according to any of the above embodiments.

[0016] Compared with the prior art, the method and device for predicting leg movements of a virtual live broadcast character provided by the embodiments of the present application have at least the following beneficial effects:

[0017] First, the present application adds a foot pressure classifier to the traditional LSTM network model, so that the leg movement prediction network model can output foot pressure prediction information while outputting leg key point prediction information, making the predicted leg movement more accurate, real and of higher reference value. In addition, the training data used in the present application is the local rotation of the relative direction between the key points of the legs and the body displacement, rather than the position of each key point. Local rotation can represent postures with the same action but different directions. In other words, two identical leg movements with different directions can be represented by only one data, local rotation. This greatly reduces the amount of training data for the leg movement prediction network model. It can achieve the same training effect as the prior art by using less training data than the training set of the prior art, reducing the difficulty of collecting training data and saving the time of collecting training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 A flowchart of a method for predicting leg movements of a virtual live broadcast character provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the structure of a leg movement prediction network model provided in an embodiment of the present application;

[0021] Figure 3 A schematic structural diagram of a device for predicting leg movements of a virtual live broadcast character provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] The embodiment of the present application provides a method for predicting leg movements of a virtual live broadcast character, such as Figure 1 As shown, the leg movement prediction method of the virtual live character specifically includes steps S101-S105:

[0024] S101. Construct a leg movement prediction network model.

[0025] Specifically, the leg motion prediction network model proposed in this application consists of an LSTM prediction network and a foot pressure classifier. First, based on the basic structure of the LSTM network, the LSTM prediction network required for this application is built and the network parameters of the LSTM prediction network are set. The network parameters include at least the time step, batch size, input data length, output data length, hidden layer size, and the number of hidden layers.

[0026] Furthermore, a classifier structure is constructed and classifier parameters of the classifier structure are set to obtain a foot pressure classifier. The classifier parameters include at least a time step and an output data length.

[0027] Finally, one output end of the LSTM prediction network is connected to the input end of the foot pressure classifier, and the other output end of the LSTM prediction network is used as the first output end of the leg movement prediction network model, and the output end of the foot pressure classifier is used as the second output end of the leg movement prediction network model.

[0028] As a feasible implementation method, Figure 2 A schematic diagram of the structure of a leg movement prediction network model provided in this application is shown as follows: Figure 2 As shown, data is input from the input end and enters the LSTM prediction network. After the LSTM prediction network performs prediction processing on the input data, the obtained leg movement prediction data is input into the foot pressure classifier and output end 1 respectively. The foot pressure classifier then makes predictions based on the received leg movement prediction data and outputs the obtained foot pressure prediction data from output end 2. Therefore, it can be seen that the output data of the leg movement prediction network model provided in this application consists of two parts: leg movement prediction data and foot pressure prediction data.

[0029] In one embodiment, the input data of the leg motion prediction network model of the present application is the relative local rotation of the key points and the body displacement. Since 8 key points are selected in this application, the local rotation of each key point is in quaternion format, and the body displacement is represented by three-dimensional coordinates. Therefore, the input data length and output data length of the leg motion prediction network model of the present application are both 4*8+3=35, the model time step is set to 200 frames, and BatchSize is set to 1. In addition, the output of the foot pressure classifier includes 6 pressure data, including 3 pressure data of the left foot and 3 pressure data of the right foot. Therefore, the output data length of the foot pressure classifier is 6, and the time step of the foot pressure classifier is also set to 200 frames.

[0030] S102: Collect the subject's leg movement data and foot pressure data, and compile them into a model training set.

[0031] Specifically, before model training, the subject wears a motion capture device for motion capture. Several key points on the subject's body are assigned, and parent-child node relationships are created between these key points. Specifically, preset body parts are identified as key points; these include the spine, pelvis, right hip, right knee, right ankle, left hip, left knee, and left ankle. The key point corresponding to the pelvis is then identified as the root node, and the spine key points, left hip key points, and right hip key points are identified as child nodes of the root node. Finally, parent-child node relationships are created for the remaining key points based on the vertical positional relationships of the leg joints.

[0032] In one embodiment, the right knee key point is a child node of the right hip key point, and the right ankle key point is a child node of the right knee key point; the left knee key point is a child node of the left hip key point, and the left ankle key point is a child node of the left knee key point.

[0033] Furthermore, after research, it was found that there are many cases in which leg movements have the same action but different directions. In general training sets, these cases are often collected as different data. For example, the same action is collected 4 times due to different directions. This will greatly increase the amount of redundant data and will not produce a substantial gain in the model training effect. Therefore, after research, this application uses the relative direction localrotation between key points for model training. Localrotation can represent multiple data with the same action but different directions with one data, which greatly reduces the amount of redundant data and increases the amount of useful training data. Therefore, within the preset experimental time, this application continuously collects the relative direction of each key point during the movement of the experimental subject through a non-wearable motion capture device. The relative direction localrotation is the quaternion of the rotation angle of the current key point relative to the parent node. This is an existing mathematical concept and will not be explained in detail here.

[0034] Furthermore, in the parent-child node relationship, the current and previous positions of the key points corresponding to the root node are obtained, and the current position is subtracted from the previous position to obtain the subject's body displacement at each moment. The subject's body displacement and the relative orientation of each key point constitute the leg movement data.

[0035] In one embodiment, the current and previous positions of the pelvic key points are obtained and subtracted to obtain the subject's body displacement, which is represented by three-dimensional coordinates. For example, if the current position of the pelvic key point is (x1, y1, z1) and the previous position is (x2, y2, z2), then the subject's body displacement is (x1-x2, y1-y2, z1-z2).

[0036] Furthermore, pressure sensors installed on the soles of the subjects' feet collected foot pressure data. Finally, the relative directions of key points, the subjects' body displacements, and foot pressure data collected at each moment during the pre-set experimental time were compiled into a model training set.

[0037] In one embodiment, pressure sensors are placed at three different points on the bottom of each foot of the subject. Therefore, three pressure data points are collected for each foot, for a total of six pressure data points for both feet. Each pressure data point contains only the values ​​of 0 and 1: if no pressure is detected, the pressure value is 0, and if pressure is detected, the pressure value is 1. If any one of the three pressure data points corresponding to each foot is 1, it indicates that the foot is touching the ground. If all three pressure data points are 0, it indicates that the foot is not touching the ground. For example, if the pressure sensor shows that the pressure data for the left foot is (1, 0, 0) and the pressure data for the right foot is (0, 0, 0), it indicates that the subject's left foot is touching the ground and the right foot is not.

[0038] S103. Train the leg movement prediction network model using the model training set.

[0039] Specifically, the leg motion data of each frame in the model training set is used as the input of the leg motion prediction network model, the corresponding next frame of leg motion data is used as the output of the LSTM prediction network, and the corresponding next frame of foot pressure data is used as the output of the foot pressure classifier to train the leg motion prediction network model.

[0040] After training is complete, the trained leg motion prediction network model is tested using a test dataset. A first loss function is then used to calculate the mean squared error (MSE) between the predicted leg motion data and the actual leg motion data. A second loss function is then used to calculate the binary cross entropy between the predicted foot pressure data and the actual foot pressure data. Finally, based on the sum of the mean squared error and the binary cross entropy, the weight parameters of the leg motion prediction network model are updated, and the model is retrained until convergence.

[0041] S104. For the live broadcast object that lacks actual leg movements during the virtual live broadcast process, generate initial leg movement data corresponding to the first picture frame; and input the initial leg movement data into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to the second picture frame.

[0042] Specifically, during a virtual live broadcast, if it is detected that key points of the live subject's legs are obscured or out of the camera frame, the current frame is determined as the first frame, and the frame immediately preceding the first frame is determined as the zeroth frame. All key points in the zeroth frame are visible. Initial position coordinates are then generated for the invisible key points based on the position coordinates of the visible key points in the first frame. Visible key points include at least spine key points and pelvic key points.

[0043] As a feasible implementation method, when it is detected that the key points of the legs of the live broadcast subject are blocked or not in the camera frame, the position coordinates of the visible key points (such as the pelvic key points and the spine key points) in the first frame are first obtained. This application sets the static neutral state of the legs as the initial state, and calculates the position coordinates of other invisible key points that should be in the static neutral state of the legs based on the position coordinates of the visible key points and the height of the live broadcast subject, and determines them as the initial position coordinates of these invisible key points in the first frame, thereby obtaining the position coordinates of all 8 key points in the first frame.

[0044] Furthermore, the initial leg motion data is determined based on the position coordinates of all key points in the first frame and the known position coordinates of all key points in the zeroth frame; wherein the initial leg motion data includes: the relative directions of the key points in the current frame and the body displacement of the live broadcast object.

[0045] In one embodiment, before the first frame in which no leg movement is detected, actual leg movement can still be detected in the previous frame. Therefore, the position coordinates of all key points in the previous frame are known. The position coordinates of the pelvic key point in the current first frame are subtracted from the position coordinates of the pelvic key point in the previous frame to obtain the body displacement of the live broadcast subject at the current moment relative to the previous moment. The position coordinates of each key point in the first frame are then used to calculate the relative local rotation of each key point.

[0046] Furthermore, the initial leg motion data is input into the leg motion prediction network model, so that the LSTM prediction network in the leg motion prediction network model outputs the leg motion prediction data corresponding to the second picture frame, and the foot pressure classifier outputs the foot pressure prediction data.

[0047] S105. Generate predicted leg movements corresponding to the second frame based on the visible key points of the live broadcast object, the leg movement prediction data, and the foot pressure prediction data, and display them on the virtual image of the live broadcast object.

[0048] Specifically, the key points in the leg movement prediction data are arranged according to their respective relative directions, and based on the arrangement results and the body displacement in the leg movement prediction data, the virtual image of the live broadcast object is driven to make the same leg movement and body displacement.

[0049] Furthermore, based on the foot pressure prediction data, the actual position of the foot of the live object in the second frame is determined, and the leg movement of the virtual image is adjusted according to the actual position of the foot.

[0050] As a feasible implementation method, the relative position relationship between the eight key points can be obtained based on the relative direction localrotation between each key point, thereby constructing a leg movement. By inputting these relative direction data and body displacement into the virtual image driving device, the virtual image can be driven to make the same leg movement and make the body produce the same displacement. In the present application, a parameter that can improve the confidence of the leg movement is also added, namely the foot pressure data. If at least one pressure value in the foot pressure prediction data is 1, it is determined that the foot corresponding to the foot pressure prediction data is in contact with the ground, and the virtual image driving device adjusts the initial leg prediction movement to the foot touching the ground. If all pressure values ​​in the foot pressure prediction data are 0, it is determined that the foot corresponding to the foot pressure prediction data is not in contact with the ground, and the virtual image driving device adjusts the initial leg prediction movement to the foot being lifted.

[0051] Furthermore, the pelvic key point positions of the second frame and the pelvic key point positions of the first frame are displaced to obtain the real body displacement of the live object in the second frame, and the real body displacement is used to replace the predicted body displacement in the leg movement prediction data of the second frame.

[0052] Then, according to the positions of the visible key points in the second picture frame, the true relative directions between the visible key points are determined, and the corresponding predicted relative directions in the leg movement prediction data of the second picture frame are replaced with the true relative directions.

[0053] The replaced leg movement prediction data is then fed into the leg movement prediction network model to obtain the leg movement prediction data for the third frame, and the corresponding predicted leg movement is generated. This process is repeated, obtaining leg movement prediction data for each frame until the complete leg movement image is displayed again or the virtual live broadcast ends.

[0054] As a feasible implementation method, before inputting the predicted data of the second frame into the model, since the second frame has already been displayed, the data in the predicted data that can be calculated through visible key points is first replaced with the actually calculated data, and then the replaced predicted data is input into the model to obtain a more realistic prediction result. The above steps are repeated to continuously predict the leg movement prediction data of the next frame until the live broadcast ends or the legs of the live broadcast subject reappear on the screen. Although the predicted identity displacement output by the model does not play a role in the subsequent processing process, using body displacement as training data to train the model can increase the constraints of the model and make the leg movement prediction network model training more accurate.

[0055] It should be noted that the leg movements completed in this application are not to restore the real leg movements of the live broadcast object, but to generate a smooth leg movement that conforms to the laws of leg movement, thereby avoiding the situation where the virtual image's legs move irregularly when the real leg movements cannot be detected, and can give the audience a more realistic effect in terms of perception.

[0056] In addition, the embodiment of the present application also provides a device for predicting leg movements of a virtual live character, such as Figure 3 As shown, the leg movement prediction device of the virtual live broadcast character specifically includes:

[0057] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores instructions executable by at least one processor, so as to enable the at least one processor to perform:

[0059] Constructing a leg motion prediction network model; wherein the leg motion prediction network model is composed of an LSTM prediction network and a foot pressure classifier;

[0060] Collecting the subject's leg motion data and foot pressure data and compiling them into a model training set; wherein the leg motion data at least includes the subject's body displacement and the relative directions of key points;

[0061] Training the leg motion prediction network model using the model training set;

[0062] For live broadcast subjects that lack actual leg movements during the virtual live broadcast process, initial leg movement data corresponding to the first frame is generated; and the initial leg movement data is input into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to the second frame;

[0063] Based on the leg movement prediction data and the foot pressure prediction data, the leg movement prediction of the second picture frame is generated and displayed on the virtual image of the live broadcast object.

[0064] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0065] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0066] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for predicting leg movements of a virtual live broadcast character, characterized in that: The method comprises: Construct a leg motion prediction network model; wherein the leg motion prediction network model is composed of an LSTM prediction network and a foot pressure classifier; specifically including: Build an LSTM prediction network and set the network parameters of the LSTM prediction network; wherein the network parameters include at least the time step, batch size, input data length, output data length, hidden layer size, and number of hidden layers; Constructing a classifier structure and setting classifier parameters of the classifier structure to obtain the foot pressure classifier; wherein the classifier parameters include at least a time step and an output data length; Connecting one output end of the LSTM prediction network to the input end of the foot pressure classifier, and using another output end of the LSTM prediction network as the first output end of the leg movement prediction network model, and using the output end of the foot pressure classifier as the second output end of the leg movement prediction network model; Collecting the subject's leg motion data and foot pressure data and compiling them into a model training set; wherein the leg motion data at least includes the subject's body displacement and the relative directions of key points; Training the leg motion prediction network model using the model training set; For live broadcast subjects that lack actual leg movements during the virtual live broadcast process, initial leg movement data corresponding to the first frame is generated; and the initial leg movement data is input into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to the second frame; Based on the leg movement prediction data and the foot pressure prediction data, the leg movement prediction of the second picture frame is generated and displayed on the virtual image of the live broadcast object.

2. The method for predicting leg movements of a virtual live broadcast character according to claim 1, wherein: Collect the subject's leg motion data and foot pressure data and compile them into a model training set, including: Determining a plurality of key points on the body of the experimental subject, and creating a parent-child node relationship between the plurality of key points; During a preset experimental time, the relative directions of each key point during the movement of the experimental subject are continuously collected by a motion capture device; wherein the relative direction is the quaternion of the rotation angle of the current key point relative to the parent node; In the parent-child node relationship, obtaining the current position and the previous moment position of the key point corresponding to the root node, and subtracting the current position from the previous moment position to obtain the body displacement of the experimental subject at each moment; collecting foot pressure data of the experimental subject's feet by means of pressure sensors installed on the soles of the experimental subject's feet; The relative directions of the key points at each moment, the body displacement of the experimental subject, and the foot pressure data collected within the preset experimental time are collected as the model training set.

3. The method for predicting leg movements of a virtual live broadcast character according to claim 2, wherein: Determine a number of key points on the subject's body and create parent-child node relationships between the key points, specifically including: Determining predetermined locations on the subject's body as the key points; wherein the predetermined locations include the spine, pelvis, right hip, right knee, right ankle, left hip, left knee, and left ankle; Determine the key point corresponding to the pelvis as a root node, and determine the spine key point, the left hip key point, and the right hip key point as child nodes of the root node; Create parent-child node relationships for the remaining key points based on the up-and-down position relationship of the leg joints.

4. The method for predicting leg movements of a virtual live broadcast character according to claim 1, wherein: Training the leg movement prediction network model using the model training set specifically includes: Using the leg motion data of each frame in the model training set as the input of the leg motion prediction network model, using the corresponding next frame of leg motion data as the output of the LSTM prediction network, and using the corresponding next frame of foot pressure data as the output of the foot pressure classifier, to train the leg motion prediction network model; Testing the trained leg movement prediction network model using a test data set; Calculate the mean square error between the leg movement prediction data and the actual leg movement data obtained after the test through the first loss function; The binary cross entropy between the foot pressure prediction data and the actual foot pressure data obtained after the test is calculated through the second loss function; According to the sum of the mean square error and the binary cross entropy, the weight parameters of the leg movement prediction network model are updated, and the leg movement prediction network model is retrained until the model converges.

5. The method for predicting leg movements of a virtual live broadcast character according to claim 1, wherein: For live broadcast subjects that lack actual leg movements during virtual live broadcast, initial leg movement data corresponding to the first frame is generated; and the initial leg movement data is input into the leg movement prediction network model to obtain leg movement prediction data and foot pressure prediction data corresponding to the second frame, specifically including: During the virtual live broadcast, when it is detected that the key points of the legs of the live broadcast subject are blocked or not within the camera frame, the current frame is determined as the first frame, and the frame before the first frame is determined as the zeroth frame; wherein all the key points in the zeroth frame are visible; Generating initial position coordinates for invisible key points based on the position coordinates of visible key points in the first image frame; wherein the visible key points include at least spine key points and pelvic key points; Determining the initial leg motion data based on the position coordinates of all key points in the first frame and the known position coordinates of all key points in the zeroth frame; wherein the initial leg motion data includes: the relative directions of the key points in the current frame and the body displacement of the live broadcast subject; The initial leg motion data is input into the leg motion prediction network model so that the LSTM prediction network in the leg motion prediction network model outputs the leg motion prediction data corresponding to the second picture frame, and the foot pressure classifier outputs the foot pressure prediction data.

6. The method for predicting leg movements of a virtual live broadcast character according to claim 1, wherein: Generating a predicted leg movement of a second frame according to the leg movement prediction data and the foot pressure prediction data, and displaying the predicted leg movement on the virtual image of the live broadcast subject, specifically comprising: Arranging the key points in the leg movement prediction data according to their respective relative directions, and driving the virtual image of the live broadcast subject to perform the same predicted leg movement and body displacement based on the arrangement result and the body displacement in the leg movement prediction data; Based on the foot pressure prediction data, the actual position of the foot of the live object in the second picture frame is determined, and the predicted leg movement of the virtual image is adjusted according to the actual position of the foot.

7. The method for predicting leg movements of a virtual live broadcast character according to claim 6, wherein: Determining the actual foot position of the live broadcast subject in the second picture frame according to the foot pressure prediction data specifically includes: If at least one pressure value in the foot pressure prediction data is 1, it is determined that the foot corresponding to the foot pressure prediction data is in contact with the ground; If all pressure values ​​in the foot pressure prediction data are 0, it is determined that the foot corresponding to the foot pressure prediction data is not in contact with the ground.

8. The method for predicting leg movements of a virtual live broadcast character according to claim 1, wherein: After generating the predicted leg motion of the second frame according to the leg motion prediction data and the foot pressure prediction data and displaying the predicted leg motion on the virtual image of the live broadcast subject, the method further includes: Calculating the displacement of the pelvic key point positions in the second frame and the pelvic key point positions in the first frame to obtain the real body displacement of the live broadcast object in the second frame; and replacing the predicted body displacement in the leg movement prediction data of the second frame with the real body displacement; determining the true relative directions between the visible key points according to the positions of the visible key points in the second picture frame; and replacing the corresponding predicted relative directions in the leg movement prediction data of the second picture frame with the true relative directions; Inputting the replaced leg movement prediction data into the leg movement prediction network model to obtain the leg movement prediction data of the third frame and generate the corresponding leg movement prediction; This step is repeated repeatedly to obtain the leg movement prediction data for each frame until the complete actual leg movement image is displayed again in the camera or the virtual live broadcast ends.

9. A device for predicting leg movements of a virtual live broadcast character, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the leg movement prediction method of a virtual live broadcast character according to any one of claims 1-8.

Images