Model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a model training method and device, equipment and a storage medium, and relates to the technical field of motion capture. The method comprises the following steps: training a motion capture model according to a preset training set, wherein the motion capture model comprises a time sequence prediction unit; sequentially performing quantization operation and inverse quantization operation on the model data passing through a quantization node through the quantization node in the time sequence prediction unit; weight parameters of the time sequence prediction unit are updated and adjusted according to the gradient of the time sequence prediction unit until the motion capture model converges, and the weight parameters comprise weight scaling parameters and weight directions. In the whole training process, quantization errors are continuously introduced, and weight parameters are decomposed into weight scaling parameters and weight directions, so that the model can be trained to converge towards a quantization friendly direction, and the model trained to converge can eliminate positioning errors brought by an inertial measurement unit, and the prediction precision of the model is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of motion capture technology, and in particular to a model training method, apparatus, device and storage medium. Background Art

[0002] Motion capture technology (abbreviated as motion capture technology) refers to the real-time capture of posture-related information of target objects in natural scenes through certain sensors, such as inertial measurement units (IMUs) and cameras. Based on the captured posture-related information, it is possible to drive a virtual image in a virtual scene, or perform behavior analysis in a natural scene.

[0003] However, there is a positioning error in the posture-related information of the inertial measurement unit, and as time accumulates, the positioning error continues to accumulate, resulting in a decrease in the accuracy of motion capture based on posture-related information. Summary of the invention

[0004] The embodiments of the present application provide a model training method, apparatus, device and storage medium, which continuously introduce quantization errors throughout the training process and decompose weight parameters into weight scaling parameters and weight directions, so that the model can be trained in a quantization-friendly direction until convergence, and the model after training to convergence can eliminate the positioning error caused by the inertial measurement unit and improve the prediction accuracy of the model.

[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0006] A motion capture model is trained according to a preset training set, wherein the motion capture model includes a timing prediction unit; through a quantization node in the timing prediction unit, quantization operations and dequantization operations are performed in sequence on the model data passing through the quantization node; and a weight parameter of the timing prediction unit is adjusted according to a gradient update of the timing prediction unit until the motion capture model converges, wherein the weight parameter includes a weight scaling parameter and a weight direction.

[0007] Optionally, the model data passing through the quantization node is subjected to quantization and dequantization operations in sequence, including: performing a quantization operation on the model data based on a preset maximum value, a minimum value and a scaling factor; and performing the dequantization operation on the model data after the quantization operation based on the scaling factor.

[0008] Optionally, the gradient update of the time series prediction unit satisfies a preset normal form constraint, and the preset normal form constraint includes a weight parameter and a diagonal matrix corresponding to the weight parameter.

[0009] Optionally, the weight parameters include one or more of parameters of a hidden layer forget gate, parameters of a hidden layer input gate, parameters of a hidden layer output gate, and parameters of a hidden layer activation gate.

[0010] Optionally, the preset paradigm constraint includes:

[0011]

[0012] in, represents the partial derivative, ξm represents the parameters of the network, D represents the diagonal matrix, W hf , g f Represent the parameters of the hidden layer forget gate and the corresponding gradient, W hi , g i Represent the parameters of the hidden layer input gate and the corresponding gradient, W ho , go respectively represent the parameters of the hidden layer output gate and the corresponding gradient, W ha , g a They represent the parameters of the hidden layer activation gate and the corresponding gradient respectively.

[0013] Optionally, the model data includes at least one of the weight of the input data of the time series prediction unit, the weight of the recent memory data, the bias data and the output data of the time series prediction unit.

[0014] Optionally, the method further comprises: performing a weight normalization operation on the motion capture model so that the weights obey a normal distribution within a preset range.

[0015] Optionally, the activation function of the time series prediction unit includes a Relu activation function and a LeakyRelu activation function.

[0016] Optionally, the weight direction is determined according to the weight of the weight parameter and the modulus of the weight.

[0017] Optionally, the method further includes: when the motion capture model converges, canceling the quantization node in the timing prediction unit.

[0018] In a second aspect, an embodiment of the present application provides a model training device, comprising:

[0019] A training module, used for training a motion capture model according to a preset training set, wherein the motion capture model includes a timing prediction unit, wherein the timing prediction unit includes a quantization node, and wherein the quantization node is used for sequentially performing quantization operations and dequantization operations on model data passing through the quantization node;

[0020] An adjustment module is used to adjust the weight parameters of the timing prediction unit according to the gradient update of the timing prediction unit until the motion capture model converges, and the weight parameters include weight scaling parameters and weight directions.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method in the first aspect or its various implementations.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed by a processor, the processor executes the method of the first aspect.

[0023] In summary, in an embodiment of the present application, when training a motion capture model according to a preset training set, a quantization node is inserted into a timing prediction unit (such as a long short-term memory network (LSTM) unit) of the motion capture model, and the model data passing through the quantization node will be quantized and dequantized in turn, thereby introducing a quantization error in the model data, and the introduced quantization error can simulate the positioning error of the inertial measurement unit, thereby reducing the impact of the quantization error during the training process. As the gradient of the model is updated, the model parameters are adjusted so that the gradient is continuously reduced, and in the model parameters of the present application, due to the introduction of the quantization error, the weight parameters of the timing prediction unit are decomposed into weight scaling parameters and weight directions. Compared with only adjusting the weight of the weight parameter, due to the accumulation of the quantization error, the model may not be able to be trained in the expected quantization-friendly direction. By adjusting the weight scaling parameters and the weight direction, the direction of the gradient update can be controlled to ensure that the model can be trained in the quantization-friendly direction until convergence as the gradient is updated. The converged motion capture model can not only achieve quantization friendliness, but also solve the problem of decreased accuracy caused by the accumulation of the positioning error of the inertial measurement unit over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of an application scenario of a model training method provided in an embodiment of the present application;

[0026] Figure 2A flowchart of a model training method provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the principle of a model training method provided in an embodiment of the present application;

[0029] Figures 5 to 7 A distribution diagram of the weight of a time series prediction unit before and after training provided in an embodiment of the present application;

[0030] Figure 8 A flowchart of a model training method provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of a timing prediction unit provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0033] Fig.11 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0036] Before introducing the technical solution of this application, the following is an introduction to the relevant knowledge of this application:

[0037] Long short-term neural network is a basic network structure in contemporary neural networks, and is classified as a recurrent neural network (RNN).

[0038] Usually we use LSTM to extract timing information. Analogous to images and videos, a single image has no timing information, but a video composed of continuous images contains timing information. In combination with the "motion capture" described in this article, there is no timing information for a single frame of the target object (such as a human body), but the continuous multi-frame postures of the target object contain timing information. This application uses LSTM to extract the timing information of the target object during movement for the final motion capture.

[0039] The function of LSTM for data processing is as follows:

[0040]

[0041] c t =σ(i t )⊙tanh(a t )+σ(f t )⊙c t-1 (2)

[0042] h t =σ(o t )⊙tanh(c t ) (3)

[0043] x t =[c t-1 , x input ] (4)

[0044] Among them, four b * Both represent the weight matrix of the LSTM structure itself, that is, the learnable parameter matrix; X t represents the input data at time t (such as the current frame), x input It is the current frame posture data collected by the posture sensor. t 、f t 、a t , o t Describe the intermediate calculation results of LSTM at time t respectively.

[0045] The output information of LSTM consists of two parts: h t and c t ,h t is the recent memory data, c t That is the long-term memory data; h t-1represents the recent memory data extracted by LSTM at time (t-1), that is, the recent memory data of the previous frame; c t-1 Represents the long-term memory data extracted by LSTM at time (t-1).

[0046] σ is an activation function, such as the Sigmoid activation function, tanh is the hyperbolic tangent function, and ⊙ is an element-by-element multiplication operation. For example, in matrices A and B of the same size, after performing this operation, a matrix C of the same size is obtained, and the elements in corresponding positions in C are equal to the product of the elements in corresponding positions in A and B.

[0047] It should be understood that the technical solution of the present application can be applied to the following scenarios, but is not limited to:

[0048] For example, Figure 1 A schematic diagram of an application scenario of a model training method provided in an embodiment of the present application, such as Figure 1 As shown, the application scenario involves a terminal device 110 and a server 120 , and the terminal device 110 can communicate with the server 120 .

[0049] Optionally, Figure 1 The application scenarios shown may also include: base stations, core network side equipment, etc. In addition, Figure 1 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and the embodiments of the present application do not limit this.

[0050] Optionally, Figure 1 The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This embodiment of the application does not limit this.

[0051] Alternatively, if Figure 1 The terminal device 110 shown may be installed with an application client, and when the application client runs in the terminal device 110, data can be exchanged with the server 120. For example, the client here may specifically include: a car client, a smart home client, a game client, a multimedia client (such as a video client), a social client, and an information client (such as a news client).

[0052] Optionally, in the embodiment of the present application, the terminal device 110 may be a device with rich human-computer interaction modes, the ability to access the Internet, usually equipped with various operating systems, and having strong processing capabilities. The terminal device 110 may be a smart phone, a smart TV, a tablet computer, a car terminal, etc., but is not limited thereto.

[0053] Optionally, the server 120 and the terminal device 110 may execute the model training method provided in the embodiment of the present application in an interactive manner, or the model training method provided in the embodiment of the present application may be executed by the terminal device 110.

[0054] During the motion capture model training stage, the server 120 or the terminal device 110 uses the model training method provided in the embodiment of the present application to train a motion capture model.

[0055] During the posture detection stage of the target object, the user can run the motion capture model through the application client, browser client, etc. installed in the terminal device 110, obtain the posture data of the IMU installed on the target object through the motion model, and input it into the motion capture model to realize the posture detection of the target object.

[0056] Among them, if the target object is the human body as an example, the inertial measurement unit can be worn on different parts of the human body (such as the left hand, the right hand, the left wrist, the right wrist, the left knee, the right knee, the head and the waist), and the posture information of the human body can be described by the rotation information of multiple bones of the human body. As the skeleton is a rigid body, the rotation information of any point on the skeleton is consistent. Therefore, the posture information of the human body can be represented by the posture data collected by the inertial measurement units of different parts of the human body.

[0057] Optionally, the posture data may include 6DoF posture data of the human head: position (Position), angle (Orientation, represented by R6D), and 6DoF posture data of human hands: position (Position), angle (Orientation, represented by R6D).

[0058] After inputting the posture data into the motion capture model, the motion capture model can output the posture of the target object, such as the 6DoF position information of the target object's whole body key points: (x, y, z, R_x, R_y, R_z).

[0059] Optionally, the motion capture model provided in the embodiment of the present application can also run on the server 120, obtain posture data through the terminal device 110, and then upload it to the server 120, and then the server 120 runs the motion capture model to process the posture data, so that the detected posture of the target object is sent to the terminal device 110. The terminal device 110 can realize motion capture of the target object according to the posture of the target object, thereby driving the virtual image in the virtual scene, or performing behavior analysis in the natural scene, etc.

[0060] It should be noted that the training process of the motion capture model and the posture detection process of the target object can be completed in the server 120 or in the terminal device 110. Optionally, the trained model file (model file) in the server 120 can be transplanted to the terminal device 110.

[0061] In the related technologies, full-body motion capture technology refers to the technology of recording, capturing and processing human body movements, which has a very wide range of applications. It is applicable to sports, games, virtual reality, 3D character animation production, visual effects and film production. Traditional motion capture methods have many limitations, such as low recognition accuracy and low processing efficiency.

[0062] When performing motion capture, an inertial measurement unit can generally be used to collect the target object's posture data. Based on IMU positioning, the positioning solution is based on motion integration. The positioning error in the previous positioning result will be accumulated to the current positioning result. At the same time, due to the zero bias of the IMU itself, the final performance is the accumulation of positioning error over time.

[0063] In order to solve this technical problem, the present application inserts a quantization node in the timing prediction unit of the motion capture model. The model data passing through the quantization node will be quantized and dequantized in sequence, thereby introducing a quantization error in the model data. The introduced quantization error can simulate the positioning error of the inertial measurement unit, thereby reducing the impact of the quantization error during the training process. As the gradient of the model is updated, the model parameters are adjusted so that the gradient is continuously reduced. In the model parameters of the present application, due to the introduction of the quantization error, the weight parameters of the timing prediction unit are decomposed into weight scaling parameters and weight directions. Compared with only adjusting the weight of the weight parameters, due to the accumulation of quantization errors, the model may not be able to be trained in the expected quantization-friendly direction. By adjusting the weight scaling parameters and weight directions, the direction of the gradient update can be controlled to ensure that the model can be trained in the quantization-friendly direction until convergence as the gradient is updated. The converged motion capture model can not only achieve quantization friendliness, but also solve the problem of decreased accuracy caused by the accumulation of the positioning error of the inertial measurement unit over time.

[0064] The technical solution of this application will be described in detail below:

[0065] Figure 2 A flowchart of a model training method provided in an embodiment of the present application, the method can be, for example, Figure 1 The terminal device 110 shown executes, but is not limited to, Figure 2 As shown, the method may include the following steps:

[0066] S101 . Training a motion capture model according to a preset training set, where the motion capture model includes a timing prediction unit.

[0067] Specifically, in an embodiment of the present application, a training set may be generated by posture data collected by an inertial measurement unit and an actual posture corresponding to the posture data.

[0068] In the related technologies, the current motion capture technology that obtains posture information end-to-end based on deep learning models usually uses deep learning models (hereinafter referred to as models) containing temporal structures, such as: Temporal Convolutional Network (TCN), Multilayer Perceptron (MLP), Transformer, LSTM, Gated Recurrent Unit (GRU), etc.

[0069] The TCN network and MLP network use a fixed time window to help predict the full-body motion capture model. However, when the time window is set too small, the neural network cannot use long-term historical data. When the time window is set too large, it will bring huge memory and computing overhead, which will make it impossible to implement on embedded devices. The Transformer network based on the self-attention mechanism is also an option, and using its structure can indeed achieve significant performance improvements in deep neural networks. However, the Transformer network introduces a lot of computational complexity, which cannot meet the needs of real-time reasoning for embedded devices such as VR headsets and mobile terminals. In addition, the Transformer network has many hardware-unfriendly operators, which makes it difficult to deploy on heterogeneous hardware.

[0070] For the LSTM structure, after the model completes the inference of the current frame, it only needs to cache the hidden state of the current frame and input it as input data when inferring the next frame. This greatly reduces the computational cost of model inference.

[0071] The motion capture model of the present application can adopt an LSTM structure, and the motion capture model includes one or more timing prediction units (i.e., LSTM units). In this way, since the LSTM unit can reduce the calculation complexity of the model, it is conducive to realizing the lightweight model, so that the model can get rid of the calculation bottleneck and be deployed to more resource-constrained devices. The LSTM structure can also improve the model detection capability, making the model structure lightweight and efficient.

[0072] S102, performing quantization operations and dequantization operations on the model data passing through the quantization nodes in the time series prediction unit in sequence;

[0073] Specifically, during the training process of the motion capture model, the timing prediction unit can process the input data. In the timing prediction unit, the input data includes the current frame sample data in the training set and the recent memory data extracted by the timing prediction unit in the previous frame. Both the current frame sample data and the recent memory data have corresponding weights.

[0074] In order to simulate the positioning error of the IMU, the timing prediction unit can set a quantization node corresponding to the current frame sample data and a quantization node corresponding to the recent memory data. The quantization node corresponding to the current frame sample data can perform quantization and dequantization operations on the weight of the current frame sample data, and the quantization node corresponding to the recent memory data can perform quantization and dequantization operations on the weight of the recent memory data, thereby introducing quantization errors in both the current frame sample data and the recent memory data to simulate the positioning error in the current frame sample data and the positioning error of the recent memory data extracted from the previous frame.

[0075] See also Figure 3 Optionally, step S102: sequentially performing quantization operations and dequantization operations on the model data that has passed the quantization node, including:

[0076] Step S1021: performing a quantization operation on the model data based on a preset maximum value, minimum value and scaling factor;

[0077] Quantization operation refers to the process of approximating floating-point activation values ​​or weights (usually represented by 32-bit floating-point numbers) to low-bit fixed-point numbers (16 bits or 8 bits), and then completing the calculation under the low-bit representation.

[0078] For example, to quantize the weight of recent memory data, we can implement it as follows:

[0079] Among them, x int That is, the quantized data, x is the model data to be quantized (such as recent memory data), and s is the scaling factor, which is 0 and 2 b-1 are the minimum and maximum values ​​of the quantized model data respectively. In this way, the quantization operation of the model data can be completed.

[0080] Optionally, the model data may include at least one of a weight of input data of the time series prediction unit, a weight of recent memory data, bias data, and output data of the time series prediction unit.

[0081] Specifically, Figure 4 In the above, Quantize represents a quantizer, which is used to quantize the corresponding model data, and De-Quantize represents a dequantizer, which is used to dequantize the corresponding model data. Weight represents weight, and bias represents bias.

[0082] By introducing quantization error to at least one of the weight of the input data of the time series prediction unit (such as input's Weight), the weight of the input recent memory data (such as hx's Weight), the bias data (such as Bias) and the output data of the time series prediction unit (such as the output recent memory data hx and the long-term memory data cx), quantization error can be introduced into the model. It can be understood that the more types of quantization errors introduced, the more conducive it is to improving the adaptability of the converged model to different quantization errors, so that the model can better eliminate various quantization errors.

[0083] Therefore, corresponding quantization errors can be introduced into the weight of the input data of the time series prediction unit, the weight of the recent memory data, the bias data, and the output data of the time series prediction unit to maximize the model's adaptability to quantization errors and improve the model's ability to eliminate quantization errors, thereby improving the model's detection accuracy.

[0084] Step S1022: performing a dequantization operation on the quantized model data based on the scaling coefficient.

[0085] The dequantization operation is the opposite of the quantization operation and can be implemented as follows:

[0086] For asymmetric quantization, the function is x=s*(x int -z)

[0087] For symmetric quantization, the function is x = s*x int .

[0088] Wherein, s is the same scaling factor as the scaling factor of the quantization operation, and z is an integer corresponding to 0 in the real number after quantization.

[0089] Generally speaking, the quantization operation can compress the model data, so that the model data changes from floating point numbers to fixed point numbers, thereby reducing the model storage overhead and computational complexity. The quantized model data may change greatly, resulting in model data distortion. This application is only for the purpose of introducing quantization errors. Therefore, it is necessary to perform a dequantization operation on the quantized model data. The dequantized model data is still a fixed point number, and the difference with the data before quantization is small. The difference between the two is the introduced quantization error. In this way, through quantization and dequantization, not only the introduction of quantization errors can be achieved, but also when fixed-point numbers are used to represent model data, the loss of precision is also small, which reduces the hardware requirements required for the model.

[0090] Step S103: adjusting the weight parameters of the time series prediction unit according to the gradient update of the time series prediction unit until the motion capture model converges, the weight parameters including the weight scaling parameters and the weight direction.

[0091] Specifically, during the training process, as the gradient of the time series prediction unit is updated, the model will continuously adjust the weight parameters of the time series prediction unit, so that the gradient of the model continues to decrease until the model converges.

[0092] Since the present application inserts a quantization node in the timing prediction unit, a quantization error is introduced. As the model is trained, the error may continue to accumulate. If only the weight of the weight parameter is adjusted, the gradient of the model may not be reduced, and the model cannot be trained to converge in a quantization-friendly direction. Therefore, the present application decomposes the weight parameter into a scale (i.e., a weight scaling parameter) and a direction (i.e., a weight direction). By adjusting the weight scaling parameter and the weight direction, the direction of the gradient update can be controlled to ensure that the model can be trained to converge in a quantization-friendly direction as the gradient is updated. The converged motion capture model can not only achieve quantization friendliness, but also solve the problem of decreased accuracy caused by the accumulation of the positioning error of the inertial measurement unit over time.

[0093] This application can collect enough data sets (i.e., training sets) so that the model can learn the correlation information between data in a statistical sense. Secondly, by associating historical data information through the time series prediction unit, the forward kinematics (FK) and inverse kinematics (IK) mechanisms can be used to improve detection accuracy when determining the spatial position of the target object. Furthermore, since the quantization error is introduced to train the model, the model reasoning results can eliminate and correct the IMU accumulated drift error. In addition, during the training of the model, the entire update process introduces quantization errors, so that the training process is always resistant to quantization noise. The weights and activation values ​​are normally distributed throughout the training process, but it can be trained from a quantization-unfriendly normal distribution to a quantization-friendly normal distribution. Taking the distribution of weights as an example, Figure 5 The left and right figures are the weight distributions of the first layer of time series prediction units before and after training, respectively. Figure 6 The left and right figures are the weight distributions of the middle layer time series prediction unit before and after training, respectively. Figure 7 The left and right figures are the weight distribution of the last layer of time series prediction unit before and after training. It can be found that the quantization-friendly Recurrent structure is more dispersed than the original initialization weights. As the network is trained, it can achieve a balance between model performance and quantization-friendliness.

[0094] Optionally, the weight direction may be obtained based on the weight of the weight parameter and the modulus of the weight.

[0095] Specifically, in LSTM, each row W j , the function is expressed as:

[0096] Among them, g j represents a trainable scaling factor, i.e., a weight scaling parameter; W j Refers to the weight of the jth row in the time series prediction unit; This is the weight direction.

[0097] The weight of the weight parameter is a vector. The quotient of the weight and the modulus of the weight can be used to obtain a unit vector with the same direction as the weight. The direction of the unit vector is the weight direction. Then the weight parameter can be the product of the unit vector and the weight scaling parameter. In this way, by adjusting the weight scaling parameter and the weight direction, the scale and direction of the weight can be adjusted.

[0098] In some embodiments, the gradient update of the time series prediction unit satisfies a preset normal form constraint, and the preset normal form constraint includes a weight parameter and a diagonal matrix corresponding to the weight parameter.

[0099] Specifically, in order to ensure that the gradient of the time series prediction unit is updated in a quantization-friendly and convergent direction, when performing the gradient update, the gradient update can be constrained, such as the gradient update needs to meet the preset paradigm constraints, the preset paradigm constraints include weight parameters and the diagonal matrix corresponding to the weight parameters, and the diagonal matrix is ​​a matrix whose elements outside the main diagonal are all 0. Since the diagonal matrix corresponding to the weight parameter is obtained according to the weight of the diagonal of the matrix corresponding to the weight parameter, the weight parameter and the diagonal matrix corresponding to the weight parameter can be scaled synchronously, that is, the scaling ratio of the two is the same. By introducing the weight parameter and the diagonal matrix corresponding to the weight parameter in the preset paradigm, the back propagation of the parameters of the time series prediction unit can be unaffected by the scaling of the matrix corresponding to the weight parameter caused by quantization.

[0100] Optionally, the weight parameters include one or more of parameters of a hidden layer forget gate, parameters of a hidden layer input gate, parameters of a hidden layer output gate, and parameters of a hidden layer activation gate.

[0101] Optionally, the preset paradigm constraints include:

[0102]

[0103] in, represents the partial derivative, ξm represents the parameters of the network, and D represents the diagonal matrix (such as D f That is W hf The corresponding diagonal matrix, D i That is W hi The corresponding diagonal matrix, D a That is W ha The corresponding diagonal matrix, D o That is W ho The corresponding diagonal matrix), W hf , g f Represent the parameters of the hidden layer forget gate and the corresponding gradient, W hi , g i Represent the parameters of the hidden layer input gate and the corresponding gradient, W ho ,g o Represent the parameters of the hidden layer output gate and the corresponding gradient, W ha , g a They represent the parameters of the hidden layer activation gate and the corresponding gradient respectively.

[0104] It can be seen that the matrices corresponding to the weight parameters are multiplied by the transposed matrix of the diagonal matrix corresponding to the weight parameters, such as D f -1 *W hf , the weight parameters and the diagonal matrices corresponding to the weight parameters are synchronously scaled, so that the back propagation of the parameters of the timing prediction unit is not affected by the scaling of the matrices corresponding to the weight parameters caused by quantization.

[0105] When the gradient update satisfies the above-mentioned preset paradigm constraints, the gradient explosion problem in training can also be alleviated. In addition, the update gradient is restricted to prevent the weight of the time series prediction unit from deviating from the quantization-friendly expectation and ensure the stability of the training.

[0106] As the gradient is updated, the weight parameters (such as W hf , W hiWhen adjusting the IMU, the model will continuously adapt to the quantization error introduced, thereby ensuring that the model trained to convergence can eliminate the error caused by the IMU and improve the detection accuracy of the model. After the timing prediction unit with quantization-friendly design is trained and converged, the quantization error can be eliminated. During the hardware deployment process, even if the hardware will quantize and quantize the floating point number to a fixed point number, the accuracy loss caused by quantization is relatively small. This makes the model have lower requirements on hardware and can be deployed more widely and efficiently.

[0107] In some embodiments, when the motion capture model is trained to convergence, the quantization nodes in the timing prediction unit are deactivated.

[0108] Specifically, the quantization node is used to introduce quantization errors during the training process. If the model still has quantization nodes when the model is trained to convergence, the IMU data will not only introduce errors during the detection process, but the model itself will also introduce quantization errors again, which may cause the error to be too large and difficult for the model to eliminate accurately. Therefore, when the motion capture model is trained to convergence, the quantization nodes in the timing prediction unit can be revoked, so that the model only needs to eliminate the IMU error, ensuring the detection accuracy of the model.

[0109] Figure 3 A flowchart of a model training method provided in an embodiment of the present application, the method can be, for example, Figure 1 The terminal device 110 or the server 120 shown in the figure performs, but is not limited to, such as Figure 8 As shown, the method may include the following steps:

[0110] S104: performing a weight normalization operation on the motion capture model so that the weights obey a normal distribution within a preset range.

[0111] Specifically, the error of the timing prediction unit has a great impact on the overall performance. This error will continue to accumulate through the Hidden State. This accumulation process is related to the weight of the timing prediction unit, the value of the timing prediction unit input, and the distribution. Therefore, the distribution of the error is required to be symmetric about 0 (such as normal distribution), so that the quantization errors can offset each other without generating cumulative errors. Weight Normalization can scale the weights of the model to a preset range, and make the normalized weights obey the normal distribution, thereby eliminating the cumulative error to a certain extent, improving the generalization and stability of the model, and making the model easier to control.

[0112] However, only using weight normalization can only guarantee the stability of the initial process. As the network is trained, the network weights and activation values ​​will gradually lose their quantization-friendly properties. Therefore, this application introduces quantization errors during the training process, and makes the weight parameters constantly adapt to the quantization errors during the training process, thereby ensuring that the distribution of weights can adaptively eliminate quantization errors during the entire network update process, so that the model can always maintain its quantization-friendly characteristics.

[0113] In some embodiments, the activation function of the time series prediction unit includes a Relu activation function and a LeakyRelu activation function.

[0114] Specifically, in the full-body motion capture model, after the posture data is input, the timing prediction unit will first perform position estimation, then complete the full-body joints, and finally perform an inverse solution of the human body posture to output the posture of the target object.

[0115] During the model detection process, the model is mainly used to memorize historical trajectory information and affect the current joint point position and angle detection. Therefore, the Sigmoid activation function and Tanh activation function used in conventional LSTM are nonlinear activation functions and are not friendly to quantization. Therefore, it can be changed to a quantization-friendly (linear or piecewise linear) activation function, such as replacing the Sigmoid activation function with the ReLU activation function and replacing the Tanh activation function with the LeakyReLU activation function, such as Fig. 9 In the modified time series prediction unit shown, the activation functions only include the Relu activation function and the LeakyRelu activation function, which is conducive to achieving quantization friendliness while reducing the running time of the model.

[0116] Among them, the experimental results of LSTM using different activation functions are shown in the following table:

[0117]

[0118]

[0119] It can be seen that, firstly, in terms of network runtime latency and resource usage, a balance between accuracy and speed is achieved in the whole-body motion capture task through the combination of several linear activation functions. Secondly, according to the training scheme for introducing quantization error for LSTM units proposed in the present invention, the model does not have the problem of gradient vanishing and gradient exploding during the training process, and can be trained stably. Finally, the timing prediction unit with modified linear activation function can also alleviate the problem of model quantization dropout.

[0120] Comparing Model 1 and Model 5, after replacing the activation function, the accuracy of the original full-body motion capture model can be restored through reasonable piecewise linear combinations. The hardware indicators are improved from 82.9% to 88.7%, the quantization loss is reduced from 10% to 0.3%, and the running time is reduced from 20ms to 11ms.

[0121] Fig.10 A schematic diagram of the structure of a model training device 10 provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the model training device 10 may include: a training module 11 and an adjustment module 13. The training module 11 is used to train the motion capture model according to a preset training set, the motion capture model includes a timing prediction unit, the timing prediction unit includes a quantization node, and the quantization node is used to sequentially perform quantization operations and dequantization operations on the model data passing through the quantization node; the adjustment module is used to adjust the weight parameters of the timing prediction unit according to the gradient update of the timing prediction unit until the motion capture model converges, and the weight parameters include weight scaling parameters and weight direction.

[0122] The quantization node is specifically used to perform a quantization operation on the model data based on a preset maximum value, minimum value and scaling factor; and to perform a dequantization operation on the quantized model data based on the scaling factor.

[0123] The model training device 10 may further include a normalization module 14, which is used to perform a weight normalization operation on the motion capture model so that the weights obey a normal distribution within a preset range.

[0124] The model training device 10 may further include a revocation module 15, which is used to revoke the quantization nodes in the timing prediction unit when the motion capture model converges.

[0125] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.10 The model training device 10 shown can execute the model training method of any of the above-mentioned embodiments, which will not be described in detail here for the sake of brevity.

[0126] The model training device of the embodiment of the present application is described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to execute, or a combination of hardware and software modules in the coding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0127] Fig.11 1 is a schematic block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be a terminal device or a server in the above method embodiment.

[0128] like Fig.11 As shown, the electronic device 100 may include:

[0129] The memory 30 and the processor 40, the memory 30 is used to store the computer program and transmit the computer program to the processor 40. In other words, the processor 40 can call and run the computer program from the memory 30 to implement the model training method in any embodiment of the present application.

[0130] Optionally, the processor 40 may include but is not limited to:

[0131] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0132] Optionally, the memory 30 includes but is not limited to:

[0133] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0134] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 30 and executed by the processor 40 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 100.

[0135] like Fig.11 As shown, the electronic device may also include:

[0136] The transceiver 50 may be connected to the processor 40 or the memory 30 .

[0137] The processor 40 may control the transceiver 50 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 50 may include a transmitter and a receiver. The transceiver 50 may further include an antenna, and the number of antennas may be one or more.

[0138] It should be understood that the various components in the electronic device 100 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0139] The present application also provides a non-volatile computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the processor is enabled to execute the model training method of any of the above-mentioned embodiments. For the sake of brevity, it will not be repeated here.

[0140] An embodiment of the present application also provides a computer program product comprising computer program instructions, wherein the computer program instructions include instructions for executing the model training method of any of the above-mentioned embodiments, which will not be described in detail here for the sake of brevity.

[0141] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0142] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0144] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0145] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Training a motion capture model according to a preset training set, wherein the motion capture model includes a timing prediction unit; Through the quantization nodes in the time series prediction unit, sequentially performing quantization operations and dequantization operations on the model data passing through the quantization nodes; The weight parameters of the timing prediction unit are adjusted according to the gradient update of the timing prediction unit until the motion capture model converges, the weight parameters including weight scaling parameters and weight directions.

2. The model training method according to claim 1, characterized in that: The step of sequentially performing quantization operations and dequantization operations on the model data passing through the quantization node includes: Based on a preset maximum value, minimum value and scaling factor, quantizing the model data; and The dequantization operation is performed on the model data after the quantization operation based on the scaling factor.

3. The model training method according to claim 1, characterized in that: The gradient update of the time series prediction unit satisfies a preset normal form constraint, and the preset normal form constraint includes a weight parameter and a diagonal matrix corresponding to the weight parameter.

4. The model training method according to claim 3, characterized in that: The weight parameters include one or more of parameters of a hidden layer forget gate, parameters of a hidden layer input gate, parameters of a hidden layer output gate, and parameters of a hidden layer activation gate.

5. The model training method according to claim 4, characterized in that: The preset paradigm constraints include: in, represents the partial derivative, ξm represents the parameters of the network, D represents the diagonal matrix, W hf , g f Represent the parameters of the hidden layer forget gate and the corresponding gradient, W hi , g i Represent the parameters of the hidden layer input gate and the corresponding gradient, W ho ,g o Represent the parameters of the hidden layer output gate and the corresponding gradient, W ha , g a They represent the parameters of the hidden layer activation gate and the corresponding gradient respectively.

6. The model training method according to claim 1, characterized in that: The model data includes at least one of a weight of input data of the time series prediction unit, a weight of recent memory data, bias data, and output data of the time series prediction unit.

7. The model training method according to claim 1, characterized in that: Also includes: A weight normalization operation is performed on the motion capture model so that the weights obey a normal distribution within a preset range.

8. The model training method according to claim 1, characterized in that: The activation function of the time series prediction unit includes a Relu activation function and a LeakyRelu activation function.

9. The model training method according to claim 1, characterized in that: The weight direction is determined according to the weight of the weight parameter and the modulus of the weight.

10. The model training method according to claim 1, characterized in that: Also includes: When the motion capture model converges, the quantization node in the timing prediction unit is cancelled.

11. A model training device, characterized in that: include: A training module, used for training a motion capture model according to a preset training set, wherein the motion capture model includes a timing prediction unit, wherein the timing prediction unit includes a quantization node, and wherein the quantization node is used for sequentially performing quantization operations and dequantization operations on model data passing through the quantization node; An adjustment module is used to adjust the weight parameters of the timing prediction unit according to the gradient update of the timing prediction unit until the motion capture model converges, and the weight parameters include weight scaling parameters and weight directions.

12. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 10.

13. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

Images