Human action pose data recovery method, device, medium and system

Abstract

The application discloses a human action posture data recovery method, device, medium and system, belongs to the action posture data processing field, and comprises the following steps: image the time sequence of action posture data, generates an action change image, realizes the association of action posture data events and space; noise is injected to the action change image until the original image becomes a noise image; then the generated noise image is denoised by using a neural network algorithm model and restored into the original image; finally, the action posture data is recovered by an image reverse restoration method. In the noise image restoration process, the noise data carried in the original action posture data can be removed, and the wrong data can be automatically corrected. The method can also automatically generate complete and reasonable actions in the case that the action nodes are few or the time sequence is missing.

Description

Technical Field

[0001] This invention relates to the field of posture data processing, and more specifically, to a method, device, medium, and system for recovering human motion posture data. Background Technology

[0002] Currently, human posture data is generally collected using motion capture equipment. However, motion capture equipment has some problems in practical use. Optical motion capture equipment offers the highest accuracy, but it requires a large space and is expensive, making it unaffordable for individual users or small companies. Inertial motion capture equipment is relatively inexpensive, making it the best choice for individuals and small companies, but its accuracy is relatively low, and it is very sensitive to the distribution of magnetic fields in the environment, making it susceptible to environmental interference. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method, device, medium, and system for restoring human motion posture data. In the process of restoring noisy images, this invention can remove noise data from the original motion posture data and automatically correct erroneous data. Furthermore, the method of this invention can automatically generate complete and reasonable movements even when there are few motion nodes or missing time series data.

[0004] The objective of this invention is achieved through the following solution:

[0005] A method for recovering human motion posture data includes the following steps:

[0006] The time series of motion posture data is visualized to generate motion change images, and the correlation between motion posture data events and space is realized; noise is injected into the motion change images until the original images become noisy images.

[0007] Then, the generated noisy image is denoised using a neural network algorithm model to restore the original image;

[0008] Finally, the motion posture data was recovered using the method of image inversion reconstruction.

[0009] Furthermore, each point on the motion change image corresponds to the posture data of a part at a certain moment, and the image can reflect the temporal and spatial relationships between the points.

[0010] Furthermore, after injecting noise into the motion change image until the original image becomes a noisy image, the method further includes the step of using the noisy image and the intensity of the noise as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the noise intensity.

[0011] Furthermore, the step of visualizing the time series of motion posture node data to generate motion change images includes the following sub-steps:

[0012] Human motion and posture data are described using the human skeleton. Each bone in the skeleton records a human posture data, which includes the three-dimensional coordinate offset (Xposition, Yposition, Zposition) and the three-dimensional coordinate rotation (Xrotation, Yrotation, Zrotation) relative to the parent bone, for a total of 6 data points. All the bone posture data at a certain moment is called a frame of motion posture data, and the 6 data points of a bone are called a pixel (Xposition, Yposition, Zposition, Xrotation, Yrotation, Zrotation), which has 6 channels.

[0013] First, arrange all human motion and posture data in a frame according to the set rules to form a row of the image, with m pixels in each row, where m is an integer;

[0014] Then, the motion posture data of the 2nd frame, the 3rd frame, and so on up to the nth frame are stitched together into n rows of data in the same way;

[0015] Finally, these n rows of data are stitched together according to the set rules to form a motion posture data image. The dimensions of this image are (n, m, 6), where n is an integer.

[0016] Furthermore, the method of recovering motion posture data through image inverse reconstruction includes the following sub-steps: removing random noise from the original data and supplementing the missing data in the original data.

[0017] Furthermore, the artificial intelligence algorithm employs a trained neural network model capable of predicting currently added noise based on the image and noise intensity.

[0018] A human motion posture data recovery device, comprising:

[0019] The motion posture data visualization unit is used to visualize the time series of motion posture data, generate motion change images, and realize the correlation between motion posture data events and space.

[0020] The noise injection unit is used to inject noise into the motion change image until the original image becomes a noisy image.

[0021] The motion data recovery unit is used to denoise the generated noisy image using a neural network model algorithm to restore the original image; then, the motion posture data is recovered by the image inverse reconstruction method.

[0022] Furthermore, it also includes a neural network model training unit, which uses the noise image and the intensity of the noise as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the noise intensity.

[0023] A readable storage medium storing a program that, when loaded by a processor, executes the human motion posture data recovery method as described above.

[0024] A human motion posture data recovery system includes the human motion posture data recovery device described above.

[0025] The beneficial effects of this invention include:

[0026] This invention provides an automatic method for restoring human motion posture data. In the process of restoring noisy images, it can remove noise data from the original motion posture data and automatically correct erroneous data. This method can also automatically generate complete and reasonable movements when there are few motion nodes or missing time series data. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the steps of the method in an embodiment of the present invention;

[0029] Figure 2 This is a motion posture data model for an embodiment of the present invention;

[0030] Figure 3 This is a method for stitching together a frame of motion posture data according to an embodiment of the present invention;

[0031] Figure 4 This is an embodiment of the present invention for stitching together n frames of motion posture data;

[0032] Figure 5 This is the step-by-step noise addition process for motion posture data images in this embodiment of the invention;

[0033] Figure 6 This describes the step-by-step restoration process of the original motion posture data image in an embodiment of the present invention. Detailed Implementation

[0034] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0035] The concept of this invention is to generate a motion change image by visualizing a time series of correct motion posture node data, thus establishing a spatial correlation between motion posture data events. Multiple rounds of noise injection are applied to a series of generated motion change images until the original image becomes a purely noisy image. Then, artificial intelligence algorithms are used to continuously denoise the generated noisy image until the original image is restored. Finally, the motion data is recovered through image inverse reconstruction. This invention's method can remove noise data from the original motion posture data during the noisy image restoration process and can automatically correct erroneous data. This invention can also automatically generate complete and reasonable motions even when there are few motion nodes or missing time series data.

[0036] In a further inventive concept, the present invention designs an automatic method for recovering human motion posture data, including a motion posture data visualization step, a neural network model training step, and a motion posture data recovery step.

[0037] In the motion posture data visualization step, a series of motion posture data time series are constructed into a long multi-dimensional image. Each point on the image corresponds to the posture data of a part at a certain moment. At the same time, the image can also reflect the temporal and spatial relationships between points.

[0038] In the neural network model training step, a series of images generated by the motion pose visualization step are made into a dataset. Noise with progressively increasing intensity is added to the images. Then, the noisy images and the intensity of the noise are used as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the noise intensity.

[0039] In the motion data recovery step, a network model trained by a neural network is used to obtain the noise level by taking the current image to be repaired and the noise intensity as inputs. The current image and the predicted noise are then used to calculate the corrected motion posture data.

[0040] In a further embodiment, the present invention provides a method for automatically recovering human motion posture data. For example... Figure 1As shown, firstly, the raw motion posture data, including noisy or missing data segments, collected by the motion capture device over a period of time is converted into a two-dimensional motion posture image. Then, this motion posture image is input into the motion data recovery process. After recovery processing, a recovered motion posture image is returned. Finally, the recovered motion posture data is extracted from the image using a reverse transformation method. This process requires three steps: motion posture data visualization, neural network model training, and motion posture data recovery.

[0041] The motion posture data visualization step converts raw motion posture data into image data. The specific implementation steps are as follows: Human motion posture data is described by a human skeleton, such as... Figure 2 As shown, in Figure 2 In motion posture data models, a skeleton model contains a certain number of bones, such as... Figure 2 The skeleton includes bones such as Hips, Chest, and Neck. There is a parent-child relationship between bones; the tail (tip) of the parent root bone connects to the head (width) of the child bone. Bones without a parent bone are called root bones. Each bone in the skeleton records pose data, including the 3D coordinate offset (Xposition, Yposition, Zposition) and the 3D coordinate rotation (Xrotation, Yrotation, Zrotation) relative to the parent bone, for a total of six data points. If the bone is a root bone, these six data points are relative to world coordinates. The pose data of all bones at a given moment is called a frame of motion pose data. The six data points of a bone are called a pixel (Xposition, Yposition, Zposition, Xrotation, Yrotation, Zrotation), and this pixel has six channels. Figure 3 As shown, this invention employs a method for stitching motion posture data into a single frame. All motion posture data within a frame are arranged according to a specific rule (not limited in this invention), forming a single row of the image, with m pixels per row. Then, the motion posture data from the 2nd, 3rd, and up to the nth frame (n is not limited here and can be determined based on the motion duration) are stitched together into n rows of data. Finally, these n rows of data are arranged according to… Figure 4 As shown, the images are stitched together to form a motion pose data image, which has dimensions (n, m, 6). To obtain n frames of data from the image, simply extract the data by reversing the steps to get the original data.

[0042] The neural network model training step involves training the required neural network model. The specific steps are as follows: First, Gaussian noise is added to the motion pose data image, such as... Figure 5 The diagram illustrates the step-by-step noise addition process for motion posture data images according to the present invention. Here, the original motion posture data image is defined as x0. A Gaussian-distributed noise z is added to x0 to generate an intermediate image x1. Then, the value of the intermediate image x1 is reduced, and the noise intensity is continuously increased until the original image is completely submerged in noise, resulting in the image x after the Tth noise addition. T The specific formula is as follows:

[0043] Define α t =1-β t , where β t Let be the t-th number that is uniformly distributed from 0.0001 to 0.002.

[0044] 0≤t≤T

[0045] β1 = 0.0001.

[0046] β T =0.002

[0047] In the formula, t represents the number of times noise is added, T represents the total number of times noise is added, β1 represents the coefficient introduced during the first addition of noise, and β T The coefficient introduced when adding noise for the Tth time.

[0048] Then after adding noise For x t Simplify to get x t Formula for x0:

[0049]

[0050]

[0051] Where, x t For the image obtained after adding noise for the tth time, x t-1 For the image after adding noise for the (t-1)th time, β t z is the coefficient introduced when noise is added for the tth time. t The noise is the noise level when noise is added for the t-th time.

[0052] Design a deep learning neural network model. The input to the neural network model is a motion pose data image x with noise added. t Given t, the output is y, whose dimension is the same as x. t The same applies. The type of neural network in this invention is not limited; it can be a Convolutional Neural Network (CNN), a Fully Connected Neural Network (FCN), or a Unet network, etc. The model is defined as follows: y = net(x,t). The training process is as follows:

[0053] 1. Prepare the training dataset

[0054] Take a large amount of motion data with correct posture and no noise interference, convert all motion data into motion posture data images in the same way, and the dimensions (n, m, 6) of all images must be consistent.

[0055] 2. Training methods

[0056] Step 0: Define the number of iterations n and initialize the parameters;

[0057] Step 1: Randomly select a number from (1, T) as t, and the number of iterations n = n-1;

[0058] Step 2: Generate random noise z, with the same dimensions as the image (n, m, 6);

[0059] Step 3: Calculate x t ;

[0060] Step 4: Calculate the loss function: loss = z-net(x) t ,t);

[0061] Step 5: Optimize the loss function;

[0062] Step 6: If n is less than 0, the training ends and the trained neural network model y = net(x,t) is obtained; otherwise, return to step 1.

[0063] As can be seen from the training process, the value predicted by the model net(x,t) is the added noise z.

[0064] The motion data recovery step restores the motion posture data, removes random noise from the original data, and fills in missing data. The specific steps are as follows: The motion data recovery step, unlike neural network model training, is a step-by-step noise reduction process. Based on the current image X... T T and the trained net(x,t) can predict X step by step. T-1 Then according to X T-1 Predict X T-2 Until X0, X0 is a motion pose data image without noise or with padding, the process is as follows: Figure 6 The diagram illustrates the step-by-step restoration process of the original motion posture data image according to the present invention. The specific steps are as follows:

[0065] Step 1: Convert the original motion pose data that needs noise removal or data completion into an image X with the same dimensions as the model input. T Where T is less than or equal to the number of times noise is added during the training of the neural network model, and t is initialized to T;

[0066] Step 2: If t>1, generate random noise z, with the same dimensions as the image (n,m,6); otherwise, z is 0.

[0067] Step 3, Calculation

[0068] Step 4, t = t-1;

[0069] Step 5: If t = 0, proceed to step 6; otherwise, return to step 2.

[0070] Step 6: Convert the obtained x0 image into motion pose data, and finally obtain motion pose data with noise removed or data completed.

[0071] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0072] Example 1

[0073] A method for recovering human motion posture data includes the following steps:

[0074] The time series of motion posture data is visualized to generate motion change images, and the correlation between motion posture data events and space is realized; noise is injected into the motion change images until the original images become noisy images.

[0075] Then, the generated noisy image is denoised using a neural network algorithm model to restore the original image;

[0076] Finally, the motion posture data was recovered using the method of image inversion reconstruction.

[0077] Example 2

[0078] Based on Example 1, each point on the motion change image corresponds to the posture data of a part at a certain moment, and the image can reflect the temporal and spatial relationships between the points.

[0079] Example 3

[0080] Based on Example 1, after injecting noise into the motion change image until the original image becomes a noisy image, the method further includes the step of using the noisy image and the intensity of the noise as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the intensity of the noise.

[0081] Example 4

[0082] Based on Example 1, the step of visualizing the time series of motion posture node data to generate motion change images includes the following sub-steps:

[0083] Human motion and posture data are described using the human skeleton. Each bone in the skeleton records a human posture data, which includes the three-dimensional coordinate offset (Xposition, Yposition, Zposition) and the three-dimensional coordinate rotation (Xrotation, Yrotation, Zrotation) relative to the parent bone, for a total of 6 data points. All the bone posture data at a certain moment is called a frame of motion posture data, and the 6 data points of a bone are called a pixel (Xposition, Yposition, Zposition, Xrotation, Yrotation, Zrotation), which has 6 channels.

[0084] First, arrange all human motion and posture data in a frame according to the set rules to form a row of the image, with m pixels in each row, where m is an integer;

[0085] Then, the motion posture data of the 2nd frame, the 3rd frame, and so on up to the nth frame are stitched together into n rows of data in the same way;

[0086] Finally, these n rows of data are stitched together according to the set rules to form a motion posture data image. The dimensions of this image are (n, m, 6), where n is an integer.

[0087] Example 5

[0088] Based on Example 1, the method of recovering motion posture data by image inverse reconstruction includes the following sub-steps: removing random noise from the original data and supplementing the missing data in the original data.

[0089] Example 6

[0090] Based on Example 3, the artificial intelligence algorithm uses a trained neural network model that can predict the currently added noise based on the image and noise intensity.

[0091] Example 7

[0092] A human motion posture data recovery device, comprising:

[0093] The motion posture data visualization unit is used to visualize the time series of motion posture data, generate motion change images, and realize the correlation between motion posture data events and space.

[0094] The noise injection unit is used to inject noise into the motion change image until the original image becomes a noisy image.

[0095] The motion data recovery unit is used to denoise the generated noisy image using a neural network model algorithm to restore the original image; then, the motion posture data is recovered by the image inverse reconstruction method.

[0096] Example 8

[0097] Based on Embodiment 7, a neural network model training unit is also included, which is used to train a neural network model that can predict the currently added noise based on the image and noise intensity, by using the noise image and noise intensity as input parameters during training.

[0098] Example 9

[0099] A readable storage medium storing a program that, when loaded by a processor, executes the human motion posture data recovery method as described in any one of Examples 1 to 6.

[0100] Example 10

[0101] A human motion posture data recovery system includes the human motion posture data recovery device as described in any one of Embodiments 7 to 8.

[0102] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0103] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0104] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0105] All parts not covered in this invention are the same as or can be implemented using existing technologies.

[0106] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the above specific embodiments of the present invention. Therefore, the methods described above are only preferred and are not restrictive.

[0107] In addition to the examples above, other embodiments may be obtained by those skilled in the art based on the above disclosure or by making modifications using knowledge or technology in related fields. The features of each embodiment may be interchanged or replaced. Modifications and changes made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for recovering human motion posture data, characterized in that, Includes the following steps: The time series of motion posture data is visualized to generate motion change images, and the correlation between motion posture data events and space is realized. Noise is injected into the motion change image until the original image becomes a noisy image. The noise image and the intensity of the noise are used as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the noise intensity. Then, the generated noisy image is denoised using a neural network algorithm model to restore the original image; Finally, the motion posture data was recovered using the image inverse reconstruction method; The step of visualizing the time series of motion posture data to generate motion change images includes the following sub-steps: Human motion and posture data are described using the human skeleton. Each bone in the skeleton records a human posture data, which includes the three-dimensional coordinate offset (Xposition, Yposition, Zposition) and the three-dimensional coordinate rotation (Xrotation, Yrotation, Zrotation) relative to the parent bone, for a total of 6 data points. All the bone posture data at a certain moment is called a frame of motion posture data, and the 6 data points of a bone are called a pixel (Xposition, Yposition, Zposition, Xrotation, Yrotation, Zrotation), which has 6 channels. First, arrange all human motion and posture data in a frame according to the set rules to form a row of the image, with m pixels in each row, where m is an integer; Then, the motion posture data of the 2nd frame, the 3rd frame, and so on up to the nth frame are stitched together into n rows of data in the same way; Finally, these n rows of data are stitched together according to the set rules to form a motion posture data image. The dimensions of the image are (n, m, 6), where n is an integer. The method of recovering motion posture data through image inversion includes the following sub-steps: removing random noise from the original data and supplementing the missing data in the original data.

2. The method for recovering human motion posture data according to claim 1, characterized in that, Each point on the motion change image corresponds to the posture data of a part at a certain moment, and the image can reflect the temporal and spatial relationships between the points.

3. A human motion posture data recovery device, characterized in that, include: The motion posture data visualization unit is used to visualize the time series of motion posture data, generate motion change images, and realize the correlation between motion posture data events and space. The noise injection unit is used to inject noise into the motion change image until the original image becomes a noisy image. The motion data recovery unit is used to denoise the generated noisy image using a neural network model algorithm to restore the original image; then, the motion posture data is recovered by the image inverse reconstruction method. Furthermore, the device is used to perform the human motion posture data recovery method according to any one of claims 1 to 2.

4. The human motion posture data recovery device according to claim 3, characterized in that, It also includes a neural network model training unit, which uses the noise image and the intensity of the noise as input parameters during training to train a neural network model that can predict the currently added noise based on the image and the noise intensity.

5. A readable storage medium, characterized in that, A program is stored in a readable storage medium, which executes the human motion posture data recovery method as described in any one of claims 1 to 2 when the program is loaded by a processor.

6. A human motion posture data recovery system, characterized in that, Including the human motion posture data recovery device as described in any one of claims 3 to 4.

Images