Human motion data recovery method and device based on tensor representation

Abstract

The present application relates to the technical field of motion data recovery, and provides a human motion data recovery method and device based on tensor representation, which can better maintain the spatial structure and time structure of the human motion data to be recovered by determining the third-order tensor of the human motion data to be recovered. Considering the time sequence of the human motion data to be recovered, a frame difference regularization term is introduced into the objective function, so that the non-smoothness in the recovery process of the human motion data to be recovered can be avoided. The objective function combines the tensor core norm term of the recovery variable and the frame difference regularization term of the recovery variable, simultaneously considers tensor tube rank minimization and frame difference smoothing, can significantly reduce the recovery error, and obtain a more superior recovery result.

Description

Technical Field

[0001] This invention relates to the field of motion data recovery technology, and in particular to a method and apparatus for recovering human motion data based on tensor representation. Background Technology

[0002] Data-driven methods and low-rank matrix approximation (LRMA) methods are two popular approaches for 3D human motion data reconstruction. Data-driven methods often require a large amount of training data, covering as much of the data space as possible. Because the lack of 3D human motion data is random, and the amount of data will continue to increase over time, constructing a motion dataset that can continuously accommodate all motion states is extremely difficult.

[0003] LRMA-based methods typically process 3D human motion data by first vectorizing each frame of 3D human motion data into a one-dimensional vector. Multiple frames of 3D human motion data form a matrix, and then traditional two-dimensional matrix denoising methods are applied to recover missing or noisy outlier points in the matrix. This method does not require a large dataset as a reference to automatically establish relationships between all data frames, utilizing only the inherent characteristics of 3D human motion data to quickly recover outlier points. However, 3D human motion data possesses a three-dimensional spatial structure in the real environment, and vectorization of 3D human motion data can destroy this spatial characteristic, leading to unstable recovery results. Summary of the Invention

[0004] This invention provides a method and apparatus for recovering human motion data based on tensor representation, in order to overcome the deficiencies existing in the prior art.

[0005] This invention provides a method for recovering human motion data based on tensor representation, comprising:

[0006] Acquire human motion data to be recovered;

[0007] Based on the three-dimensional position of the joints in each frame of the human motion data to be recovered, the third-order tensor of the human motion data to be recovered is determined.

[0008] The third-order tensor is mapped to a restored variable represented by a low-rank tensor, and an objective function is constructed based on the tensor kernel norm of the restored variable and the inter-frame difference regularization term of the restored variable.

[0009] The objective function is solved to obtain the recovery result.

[0010] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein the inter-frame difference regularization term is the L1 norm of the recovered variable after performing difference operations on adjacent elements based on a weighted three-dimensional difference operator.

[0011] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein solving the objective function to obtain the recovery result includes:

[0012] The difference operation result in the inter-frame difference regularization term is represented as the first auxiliary variable, and the recovery variable in the inter-frame difference regularization term is represented as the second auxiliary variable, thus obtaining the variable transformation term;

[0013] Based on the tensor kernel norm terms and the variable transformation terms, the augmented Lagrangian function of the objective function is established;

[0014] The augmented Lagrangian function is solved to obtain the recovery result.

[0015] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein solving the augmented Lagrangian function to obtain the recovery result includes:

[0016] Based on the alternating direction multiplier method, the problem of solving the augmented Lagrange function is decomposed into subproblems concerning the restored variable, the first auxiliary variable, and the second auxiliary variable;

[0017] Within the stochastic gradient descent framework, each of the sub-problems is solved synchronously and iteratively to obtain the recovery result.

[0018] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein the recovery result is obtained by synchronously iteratively solving each sub-problem within a stochastic gradient descent framework, including:

[0019] For the first subproblem concerning the recovered variable, the closed-form solution obtained by the recovered variable in each iteration is calculated according to the tensor singular value threshold contraction operator;

[0020] For the second subproblem concerning the first auxiliary variable, the closed-form solution obtained by the first auxiliary variable in each iteration is calculated according to the soft thresholding operation;

[0021] For the third subproblem concerning the second auxiliary variable, the third subproblem is represented as a system of linear equations, and the system of linear equations is solved based on the three-dimensional Fourier transform matrix to obtain the closed-form solution of the second auxiliary variable in each iteration.

[0022] Based on the multiplier update formula of the augmented Lagrange function, the Lagrange multipliers are updated by applying the closed-form solutions obtained by the restored variable, the first auxiliary variable, and the second auxiliary variable in each iteration until the difference between the restored variable in two adjacent iterations meets the preset condition.

[0023] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein the representation of the third-order tensor in tensor space includes... or m is the number of joints in each frame of data, and n is the number of data frames of the human motion data to be recovered.

[0024] According to the present invention, a method for recovering human motion data based on tensor representation is provided, wherein the objective function is expressed based on the following formula:

[0025] ;

[0026] in, For the restored variable, For the tensor nuclear norm, For hyperparameters, The inter-frame difference regularization term is the L1 norm. Indicates constraints. For mapping functions, For the third-order tensor, Let be the coordinate space of the elements of the third-order tensor.

[0027] The present invention also provides a human motion data recovery device based on tensor representation, comprising:

[0028] The data acquisition module is used to acquire the human motion data to be recovered.

[0029] The tensor representation module is used to determine the third-order tensor of the human motion data to be recovered based on the three-dimensional position of the joints in each frame of the human motion data to be recovered.

[0030] The function construction module is used to map the third-order tensor to the recovery variable represented by the low-rank tensor, and to construct the objective function based on the tensor kernel norm of the recovery variable and the inter-frame difference regularization term of the recovery variable.

[0031] The function solving module is used to solve the objective function and obtain the recovery result.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the tensor-based human motion data recovery method as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human motion data recovery method based on tensor representation as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the human motion data recovery method based on tensor representation as described above.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention provides a method and apparatus for recovering human motion data based on tensor representation. By determining the third-order tensor of the human motion data to be recovered, the spatial and temporal structure of the data can be better preserved. Considering the temporal sequence of the human motion data to be recovered, an inter-frame difference regularization term is introduced into the objective function to avoid unevenness in the recovery process. By combining the tensor kernel norm term and the inter-frame difference regularization term of the recovered variables, and simultaneously considering tensor tube rank minimization and inter-frame difference smoothing, the objective function can significantly reduce the recovery error and obtain a superior recovery result. Compared with the existing two-dimensional matrix denoising methods that require vectorization of each frame of data, this method does not require unfolding the human motion data to be recovered, and can maintain the spatiotemporal characteristics of the human motion data to be recovered. Even with the loss of a large number of data points, a stable recovery result can still be obtained. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0038] Figure 1 This is a flowchart illustrating the human motion data recovery method based on tensor representation provided by the present invention.

[0039] Figure 2 This is a comparative diagram of the human motion data recovery method based on tensor representation provided by the present invention and the existing two-dimensional matrix denoising method;

[0040] Figure 3 This is a schematic diagram of the human motion data recovery device based on tensor representation provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] Figure 1 This is a flowchart illustrating a method for recovering human motion data based on tensor representation provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0044] S1, acquire the human motion data to be recovered;

[0045] S2, Based on the three-dimensional position of the joints in each frame of the human motion data to be recovered, determine the third-order tensor of the human motion data to be recovered;

[0046] S3, map the third-order tensor to a restored variable represented by a low-rank tensor, and construct an objective function based on the tensor kernel norm of the restored variable and the inter-frame difference regularization term of the restored variable;

[0047] S4, Solve the objective function to obtain the recovery result.

[0048] Specifically, the human motion data recovery method based on tensor representation provided in this embodiment of the invention is executed by a human motion data recovery device based on tensor representation. This device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0049] First, perform step S1 to obtain the human motion data to be restored. The human motion data to be restored can be three-dimensional (3D) human motion data, including multiple frames of data, and each frame of data includes the three-dimensional positions of the joints of the human body. In addition, the human motion data to be restored also includes: the positions of key points of the human body, the motion trajectory, the posture information, the gait characteristics, the muscle activity information, the plantar pressure of the human body during the motion, the heart rate, the respiratory rate, and the joint range of motion, etc. Among them, the three-dimensional positions of the joints of the human body and the positions of key points of the human body can be obtained by identifying the human body images collected by a camera or a depth camera, or can be determined by an optical motion capture system. The motion trajectory can be determined by the real-time positioning data of the human body, and the real-time positioning data can be determined by a global positioning system. The posture information and the gait characteristics can be collected by wearable devices such as a gyroscope or an inertial measurement unit. The muscle activity information can be measured by a surface electromyography sensor. The plantar pressure can be collected by a plantar pressure plate or a force platform. The heart rate and the respiratory rate can be collected by wearable devices such as a heart rate and respiratory monitoring device. The joint range of motion can be obtained by manual evaluation.

[0050] The human motion data to be restored can be represented by the following matrix:

[0051] ;

[0052] where, is the motion data matrix, 1 < t < n, n is the number of data frames of the human motion data to be restored, m is the number of joints in each frame of data, is the t-th frame of data, and .

[0053] Then, perform step S2. Consider each frame of the human motion data to be restored as a matrix of m×3, and consider the human motion data to be restored as a third-order tensor. Each frame of data is a side slice or a front slice of this third-order tensor.

[0054] Since the structure represented by the tensor has a flexible feature, its three directions can represent the two-dimensional coordinate features and the time feature of the human motion data to be restored, and it breaks through the defect that the two-dimensional matrix representation form needs to vectorize the human motion data to be restored. On the basis of flexibility, 3D human data can be represented as two types in the tensor space, or . The representation of the third-order tensor in the tensor space includes or , m is the number of joints in each frame of data, and n is the number of data frames of the human motion data to be restored.

[0055] Then, step S3 is executed, which maps the third-order tensor to the recovery variable represented by the low-rank tensor based on the low-rank prior of the human motion data to be recovered, so as to describe the global correlation between the frames of the human motion data to be recovered through the low-rank tensor approximation.

[0056] Given the prior knowledge that the human motion data to be recovered has a global low-rank property, its recovery problem can be formulated as a general constraint tensor rank minimization problem. Considering the need to maintain the temporal stability of the human motion data to be recovered in the tensor space, an objective function is constructed using the tensor kernel norm term and the inter-frame difference regularization term of the recovered variables. The objective function is expressed by the following formula:

[0057] ;

[0058] in, For the restored variable, It is the tensor nuclear norm (TNN). This is a hyperparameter. To recover the tensor nuclear norm of the variable, used for constraint The global low-rank characteristic of the tensor tube rank, as a convex relaxation of the tensor tube rank, enables the tensor tube rank minimization problem, which is an NP-hard problem, to obtain a good solution. The inter-frame difference regularization term is the L1 norm. Indicates constraints. For the third-order tensor, Let be the coordinate space of the elements of the third-order tensor. This is a mapping function used to map incomplete third-order tensors. Recovery variables mapped to low-rank tensors If the representation of a third-order tensor in tensor space is... Then there are If the representation of a third-order tensor in tensor space is Then there are .

[0059] Finally, step S4 is executed to solve the objective function and obtain the restored result. This restored result is the value of the restored variable that minimizes the objective function.

[0060] The tensor-based human motion data recovery method provided in this embodiment of the invention first acquires the human motion data to be recovered; then, based on the three-dimensional positions of the joints in each frame of the human motion data to be recovered, the third-order tensor of the human motion data to be recovered is determined; subsequently, the third-order tensor is mapped to a recovery variable represented by a low-rank tensor, and an objective function is constructed using the tensor kernel norm term and the inter-frame difference regularization term of the recovery variable; finally, the objective function is solved to obtain the recovery result. This method, by determining the third-order tensor of the human motion data to be recovered, can better preserve the spatial and temporal structure of the human motion data to be recovered. Considering the temporal sequence of the human motion data to be recovered, the introduction of the inter-frame difference regularization term in the objective function can avoid non-smoothness in the recovery process. By combining the tensor kernel norm term and the inter-frame difference regularization term of the recovery variable, and simultaneously considering tensor rank minimization and inter-frame difference smoothing, the objective function can significantly reduce recovery errors and obtain superior recovery results. Compared with existing two-dimensional matrix denoising methods that require vectorization of each frame of data, this method does not require unfolding the human motion data to be recovered. It can maintain the spatiotemporal characteristics of the human motion data to be recovered and can still obtain stable recovery results even when a large number of data points are lost.

[0061] Based on the above embodiments, the inter-frame difference regularization term is the L1 norm of the recovered variable after performing difference operations on adjacent elements based on a weighted three-dimensional difference operator.

[0062] Specifically, the form of the inter-frame difference regularization term can be flexible, mainly inspired by the total variation regularization method, and can be used to minimize the distance between two frames. In this embodiment of the invention, the inter-frame difference regularization term can be expressed as:

[0063] ;

[0064] in, It is an L1 norm. This represents the weighted three-dimensional difference operator, while , , It is the effect on Three first-order difference operators in different directions.

[0065] The expression for the inter-frame difference regularization term can be expanded as follows:

[0066] ;

[0067] in, express The Each element, and the weight coefficient for The The weights of each mode are used to adjust the regularization strength. Here, the c-th mode of the tensor represents the matrix obtained by expanding the tensor through the c-th dimension.

[0068] exist The purpose of setting weights is to enable the same solution framework to be applicable to different forms of third-order tensors, when At that time, by setting weights , It can constrain the distance between two frames of data. When Weights can be set at this time. This constrains the distance between two frames of data without modifying the solution framework.

[0069] Based on the above embodiments, solving the objective function to obtain the recovery result includes:

[0070] The difference operation result in the inter-frame difference regularization term is represented as the first auxiliary variable, and the recovery variable in the inter-frame difference regularization term is represented as the second auxiliary variable, thus obtaining the variable transformation term;

[0071] Based on the tensor kernel norm terms and the variable transformation terms, the augmented Lagrangian function of the objective function is established;

[0072] The augmented Lagrangian function is solved to obtain the recovery result.

[0073] Specifically, when solving the objective function, the tensor kernel norm term and the inter-frame difference regularization term in the objective function are separated so that each term can obtain a closed-form solution.

[0074] Here, we introduce the first auxiliary variable. Second auxiliary variable The result of the difference operation in the inter-frame difference regularization term can be represented as the first auxiliary variable, i.e. The recovery variable in the inter-frame difference regularization term is represented as a second auxiliary variable, i.e. , to obtain variable transformation terms At this point, the objective function transforms into:

[0075] ;

[0076] To transform the problem with equality constraints into an unconstrained optimization problem, we establish the augmented Lagrangian function of the objective function:

[0077] ;

[0078] in, and All are Lagrange multipliers. Represents the tensor inner product. and All are penalty parameters. It is a Frobenious norm.

[0079] The restored result can be obtained by solving the augmented Lagrangian function.

[0080] Based on the above embodiments, solving the augmented Lagrangian function to obtain the recovery result includes:

[0081] Based on the alternating direction multiplier method, the problem of solving the augmented Lagrange function is decomposed into subproblems concerning the restored variable, the first auxiliary variable, and the second auxiliary variable;

[0082] Within the stochastic gradient descent framework, each of the sub-problems is solved synchronously and iteratively to obtain the recovery result.

[0083] Specifically, when solving for the augmented Lagrange function, the Alternating Direction Method of Multipliers (ADMM) can be used to decompose the problem of solving the augmented Lagrange function into subproblems concerning the restored variable, the first auxiliary variable, and the second auxiliary variable.

[0084] The first subproblem concerning the recovery variable can be expressed as:

[0085] ;

[0086] in, This represents the recovery variable for the (k+1)th iteration. and For the k-th iteration and .

[0087] The second subproblem concerning the first auxiliary variable can be expressed as:

[0088] ;

[0089] in, This represents the (k+1)th iteration. , For the k-th iteration , For the k-th iteration .

[0090] The third subproblem concerning the second auxiliary variable can be expressed as:

[0091] .

[0092] Within the framework of Stochastic Gradient Descent (SGD), the recovery result can be obtained by simultaneously iterating through each subproblem.

[0093] Based on the above embodiments, the step of simultaneously iteratively solving each of the sub-problems within the stochastic gradient descent framework to obtain the recovery result includes:

[0094] For the first subproblem concerning the recovered variable, the closed-form solution obtained by the recovered variable in each iteration is calculated according to the tensor singular value threshold contraction operator;

[0095] For the second subproblem concerning the first auxiliary variable, the closed-form solution obtained by the first auxiliary variable in each iteration is calculated according to the soft thresholding operation;

[0096] For the third subproblem concerning the second auxiliary variable, the third subproblem is represented as a system of linear equations, and the system of linear equations is solved based on the three-dimensional Fourier transform matrix to obtain the closed-form solution of the second auxiliary variable in each iteration.

[0097] Based on the multiplier update formula of the augmented Lagrange function, the Lagrange multipliers are updated by applying the closed-form solutions obtained by the restored variable, the first auxiliary variable, and the second auxiliary variable in each iteration until the difference between the restored variable in two adjacent iterations meets the preset condition.

[0098] Specifically, when solving the first subproblem, the closed-form solution obtained by the recovery variable in each iteration can be calculated using the tensor singular value threshold contraction operator (t-SVT):

[0099] ;

[0100] in, This indicates that the recovered variable obtained in the (k+1)th iteration is in The values ​​in, and middle The value of equal, This indicates that the recovered variable obtained in the (k+1)th iteration is in The values ​​in for The tensor space outside of that. Indicates to Perform tensor singular value decomposition. and All are orthogonal tensors obtained from singular value decomposition of tensors. express The first front slice is a tensor of a diagonal matrix, and the remaining front slices are 0. , This is the inverse fast Fourier transform, where ma is the thresholding function. This indicates that a subarray is extracted from the j-th row and j-th column along the third dimension. To The third dimension is subjected to a fast Fourier transform.

[0101] Indicates when Mapped to At that time, if If no elements are missing, then the value is directly assigned to it. . Indicates when Mapped to At that time, if If elements are missing, calculate the closed-form solution using the tensor singular value threshold operator.

[0102] When solving the second subproblem, the closed-form solution of the first auxiliary variable is calculated in each iteration based on the soft thresholding operation:

[0103] ;

[0104] in, It is a soft threshold operator, and we have:

[0105] .

[0106] When solving the third subproblem, it is represented as a system of linear equations:

[0107] ;

[0108] in, express The adjoint operator.

[0109] Using the three-dimensional Fourier transform matrix, the linear equation system is solved to obtain the closed-form solution of the second auxiliary variable in each iteration:

[0110] because The corresponding matrix has a block cyclic structure and can be diagonalized using a 3D Fast Fourier Transform matrix. Therefore, we can obtain:

[0111] .

[0112] in, and All are intermediate variables.

[0113] Using the multiplier update formula of the augmented Lagrange function, the Lagrange multipliers are updated by applying the closed-form solutions obtained in each iteration using the restored variable, the first auxiliary variable, and the second auxiliary variable. The multiplier update formula of the augmented Lagrange function can be expressed as:

[0114] .

[0115] The condition for the end of the Lagrange multiplier update is a preset condition, which is:

[0116] .

[0117] in, The difference between two adjacent iterations is the solution process. It is a very small constant and can take values ​​of .

[0118] like Figure 2 As shown, the human motion data recovery method based on tensor representation provided in this embodiment of the invention is different from the existing two-dimensional matrix denoising method: the existing two-dimensional matrix denoising method directly represents the human motion data to be recovered through a two-dimensional matrix, and then obtains the recovery result through matrix technology; in this invention, it is necessary to determine the third-order tensor of the human motion data to be recovered, and then obtain the recovery result by considering tensor tube rank minimization and inter-frame difference smoothing through tensor technology.

[0119] The tensor-based human motion data recovery method provided in this embodiment of the invention can significantly reduce recovery errors and obtain superior recovery results. Compared with the existing two-dimensional matrix denoising method that requires vectorization processing of each frame of data, this method does not require unfolding the human motion data to be recovered, and can maintain the spatiotemporal characteristics of the human motion data to be recovered. Even when a large number of data points are lost, stable recovery results can still be obtained.

[0120] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a human motion data recovery device based on tensor representation, comprising:

[0121] Data acquisition module 31 is used to acquire human motion data to be recovered;

[0122] Tensor representation module 32 is used to determine the third-order tensor of the human motion data to be recovered based on the three-dimensional position of the joints in each frame of the human motion data to be recovered;

[0123] The function construction module 33 is used to map the third-order tensor to a recovery variable represented by a low-rank tensor, and to construct a target function based on the tensor kernel norm of the recovery variable and the inter-frame difference regularization term of the recovery variable.

[0124] The function solving module 34 is used to solve the objective function and obtain the recovery result.

[0125] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention has an inter-frame difference regularization term that is the L1 norm of adjacent elements in the recovery variable after difference operation based on a weighted three-dimensional difference operator.

[0126] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0127] The difference operation result in the inter-frame difference regularization term is represented as the first auxiliary variable, and the recovery variable in the inter-frame difference regularization term is represented as the second auxiliary variable, thus obtaining the variable transformation term;

[0128] Based on the tensor kernel norm terms and the variable transformation terms, the augmented Lagrangian function of the objective function is established;

[0129] The augmented Lagrangian function is solved to obtain the recovery result.

[0130] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0131] Based on the alternating direction multiplier method, the problem of solving the augmented Lagrange function is decomposed into subproblems concerning the restored variable, the first auxiliary variable, and the second auxiliary variable;

[0132] Within the stochastic gradient descent framework, each of the sub-problems is solved synchronously and iteratively to obtain the recovery result.

[0133] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0134] For the first subproblem concerning the recovered variable, the closed-form solution obtained by the recovered variable in each iteration is calculated according to the tensor singular value threshold contraction operator;

[0135] For the second subproblem concerning the first auxiliary variable, the closed-form solution obtained by the first auxiliary variable in each iteration is calculated according to the soft thresholding operation;

[0136] For the third subproblem concerning the second auxiliary variable, the third subproblem is represented as a system of linear equations, and the system of linear equations is solved based on the three-dimensional Fourier transform matrix to obtain the closed-form solution of the second auxiliary variable in each iteration.

[0137] Based on the multiplier update formula of the augmented Lagrange function, the Lagrange multipliers are updated by applying the closed-form solutions obtained by the restored variable, the first auxiliary variable, and the second auxiliary variable in each iteration until the difference between the restored variable in two adjacent iterations meets the preset condition.

[0138] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention includes the representation of the third-order tensor in tensor space. or m is the number of joints in each frame of data, and n is the number of data frames of the human motion data to be recovered.

[0139] Based on the above embodiments, the human motion data recovery device based on tensor representation provided in this embodiment of the invention has an objective function expressed based on the following formula:

[0140] ;

[0141] in, For the restored variable, For the tensor nuclear norm, For hyperparameters, The inter-frame difference regularization term is the L1 norm. Indicates constraints. For mapping functions, For the third-order tensor, Let be the coordinate space of the elements of the third-order tensor.

[0142] Specifically, the functions of each module in the tensor-based human motion data recovery device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-mentioned method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0143] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the tensor-based human motion data recovery method provided in the above embodiments.

[0144] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the human motion data recovery method based on tensor representation provided in the above embodiments.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the tensor-based human motion data recovery method provided in the above embodiments.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recovering human motion data based on tensor representation, characterized in that, include: Acquire human motion data to be recovered; the human motion data to be recovered includes multiple frames of data, each frame of data includes the three-dimensional position of each joint of the human body, the three-dimensional position of the joint is obtained by identifying human images captured by a camera or depth camera; Based on the three-dimensional position of the joints in each frame of the human motion data to be recovered, the third-order tensor of the human motion data to be recovered is determined. The third-order tensor is mapped to a restored variable represented by a low-rank tensor, and an objective function is constructed based on the tensor kernel norm of the restored variable and the inter-frame difference regularization term of the restored variable. The objective function is solved to obtain the recovery result; The inter-frame difference regularization term is the L1 norm of the recovered variable after performing difference operations on adjacent elements based on a weighted three-dimensional difference operator.

2. The method for recovering human motion data based on tensor representation according to claim 1, characterized in that, Solving the objective function to obtain the recovery result includes: The difference operation result in the inter-frame difference regularization term is represented as the first auxiliary variable, and the recovery variable in the inter-frame difference regularization term is represented as the second auxiliary variable, thus obtaining the variable transformation term; Based on the tensor kernel norm terms and the variable transformation terms, the augmented Lagrangian function of the objective function is established; The augmented Lagrangian function is solved to obtain the recovery result.

3. The method for recovering human motion data based on tensor representation according to claim 2, characterized in that, Solving the augmented Lagrangian function to obtain the restored result includes: Based on the alternating direction multiplier method, the problem of solving the augmented Lagrange function is decomposed into subproblems concerning the restored variable, the first auxiliary variable, and the second auxiliary variable; Within the stochastic gradient descent framework, each of the sub-problems is solved synchronously and iteratively to obtain the recovery result.

4. The method for recovering human motion data based on tensor representation according to claim 3, characterized in that, Within the stochastic gradient descent framework, the sub-problems are solved synchronously and iteratively to obtain the recovery result, including: For the first subproblem concerning the recovered variable, the closed-form solution obtained by the recovered variable in each iteration is calculated according to the tensor singular value threshold contraction operator; For the second subproblem concerning the first auxiliary variable, the closed-form solution obtained by the first auxiliary variable in each iteration is calculated according to the soft thresholding operation; For the third subproblem concerning the second auxiliary variable, the third subproblem is represented as a system of linear equations, and the system of linear equations is solved based on the three-dimensional Fourier transform matrix to obtain the closed-form solution of the second auxiliary variable in each iteration. Based on the multiplier update formula of the augmented Lagrange function, the Lagrange multipliers are updated by applying the closed-form solutions obtained by the restored variable, the first auxiliary variable, and the second auxiliary variable in each iteration until the difference between the restored variable in two adjacent iterations meets the preset condition.

5. The method for recovering human motion data based on tensor representation according to any one of claims 1-4, characterized in that, The representation of the third-order tensor in tensor space includes or m is the number of joints in each frame of data, and n is the number of data frames of the human motion data to be recovered.

6. The method for recovering human motion data based on tensor representation according to any one of claims 1-4, characterized in that, The objective function is expressed based on the following formula: ； in, For the restored variable, For the tensor nuclear norm, For hyperparameters, The inter-frame difference regularization term is the L1 norm. Indicates constraints. For mapping functions, For the third-order tensor, Let be the coordinate space of the elements of the third-order tensor.

7. A human motion data recovery device based on tensor representation, characterized in that, include: The data acquisition module is used to acquire human motion data to be recovered; the human motion data to be recovered includes multiple frames of data, each frame of data includes the three-dimensional position of each joint of the human body, and the three-dimensional position of the joint is obtained by identifying human images captured by a camera or depth camera; The tensor representation module is used to determine the third-order tensor of the human motion data to be recovered based on the three-dimensional position of the joints in each frame of the human motion data to be recovered. The function construction module is used to map the third-order tensor to the recovery variable represented by the low-rank tensor, and to construct the objective function based on the tensor kernel norm of the recovery variable and the inter-frame difference regularization term of the recovery variable. The function solving module is used to solve the objective function and obtain the recovery result; The inter-frame difference regularization term is the L1 norm of the recovered variable after performing difference operations on adjacent elements based on a weighted three-dimensional difference operator.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the human motion data recovery method based on tensor representation as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human motion data recovery method based on tensor representation as described in any one of claims 1-6.

Images