Posture prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an attitude prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting time sequence information of input data through a long and short-term memory network, the time sequence information comprising recent memory data, and the input data comprising current frame attitude data of a target object collected by an attitude sensor; performing full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data; overlapping the recent memory data and the first intermediate data of the nonlinear learning unit through an overlapping unit to generate second intermediate data, and forming a basic learning module by the long and short-term memory network, the nonlinear learning unit and the overlapping unit; and predicting the current attitude of the target object according to the second intermediate data without adjusting the model according to the specific heterogeneous hardware platform adaptability, so that the deployment cost is low, and universal deployment can be realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of motion capture technology, and in particular to a posture prediction method, a posture prediction device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Motion capture technology (abbreviated as motion capture technology) refers to the real-time capture of posture-related information of target objects in natural scenes through certain sensors, such as inertial measurement units (IMUs) and cameras. Based on the captured posture-related information, it is possible to drive a virtual image in a virtual scene, or perform behavior analysis in a natural scene.

[0003] However, the current deep learning models for motion capture usually use deep models that contain timing structures. When combined with specific heterogeneous hardware, the model structure is often complex, which requires the model to be adaptively adjusted according to the specific heterogeneous hardware platform. The deployment cost is high and it is difficult to achieve universal deployment. Summary of the invention

[0004] The embodiments of the present application provide a posture prediction method, a posture prediction device, an electronic device, and a computer-readable storage medium. There is no need to adaptively adjust the model according to a specific heterogeneous hardware platform, the deployment cost is low, and universal deployment can be achieved.

[0005] In a first aspect, an embodiment of the present application provides a posture prediction method, comprising: extracting time series information of input data through a long short-term memory network, the time series information including recent memory data, the input data including current frame posture data of a target object collected by a posture sensor; performing full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data; superimposing the recent memory data and the first intermediate data of the nonlinear learning unit through a superposition unit to generate second intermediate data, the long short-term memory network, the nonlinear learning unit and the superposition unit forming a basic learning module; and predicting the current posture of the target object based on the second intermediate data.

[0006] In a second aspect, an embodiment of the present application provides a posture prediction device, comprising: an extraction module, a learning module, a superposition module and a prediction module. The extraction module is configured to extract the time series information of the input data through a long short-term memory network, the time series information includes recent memory data, and the input data includes the current frame posture data of the target object collected by the posture sensor; the learning module is configured to perform full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data; the superposition module is configured to superimpose the recent memory data and the first intermediate data of the nonlinear learning unit through a superposition unit to generate second intermediate data, the long short-term memory network, the nonlinear learning unit and the superposition unit constitute a basic learning module; and the prediction module is configured to predict the current posture of the target object according to the second intermediate data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed by a processor, the method in the first aspect is implemented.

[0009] In summary, in the embodiment of the present application, a long short-term memory network (Long Short-Term Memory, LSTM) is used to extract the time series information of the input data, wherein the input data not only includes the current frame posture data collected by the posture sensor, but also includes the recent memory data and / or long-term memory data extracted from the previous frame. After that, the recent memory data in the current frame and the first intermediate data learned by the nonlinear learning unit in the input data are superimposed to obtain the second intermediate data for predicting the current posture. Since the time series information is considered when predicting the current posture of the target object, it is beneficial to improve the prediction accuracy of the current posture. And since the nonlinear learning unit processes the input data through full connection and activation mapping, and both full connection and activation mapping only need to perform matrix multiplication and / or addition, the nonlinear learning unit can adapt to any heterogeneous hardware platform, and the current posture prediction is realized by combining the widely used LSTM and nonlinear learning unit, which not only ensures the prediction accuracy of the current posture, but also makes the model more adaptable to different heterogeneous hardware platforms, and there is no need to adjust the model according to the adaptability of the specific heterogeneous hardware platform, the deployment cost is low, and universal deployment can be achieved.

[0010] In addition, compared with the model combining multiple LSTMs, where multiple LSTMs are difficult to execute in parallel, the nonlinear learning unit of the present application can realize parallel computing with LSTM, thereby making full use of heterogeneous hardware platforms with a high degree of parallelization to realize data processing and improve the prediction efficiency of the model.

[0011] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of an application scenario of a posture prediction method provided in an embodiment of the present application;

[0014] Figure 2 A flowchart of a posture prediction method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the structure of a posture prediction model provided in an embodiment of the present application;

[0016] Figure 4 A flowchart of a posture prediction method provided in an embodiment of the present application;

[0017] Figure 5 A flowchart of a posture prediction method provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the structure of a posture prediction model provided in an embodiment of the present application;

[0019] Figure 7 A flowchart of a posture prediction method provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of a posture prediction model provided in an embodiment of the present application;

[0021] Fig. 9 A flowchart of a posture prediction method provided in an embodiment of the present application;

[0022] Fig.10 A schematic diagram of the structure of a posture prediction model provided in an embodiment of the present application;

[0023] Fig.11 A schematic diagram of a module of a posture prediction device provided in an embodiment of the present application; and

[0024] Fig.12 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] Before introducing the technical solution of this application, the following is an introduction to the relevant knowledge of this application:

[0028] 1. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The technical solutions provided in the embodiments of this application mainly involve natural language processing technology and machine learning / deep learning in artificial intelligence.

[0029] 2. Long short-term neural network, which is a basic network structure in contemporary neural networks and is classified as a recurrent neural network (RNN).

[0030] Usually we use LSTM to extract timing information. Analogous to images and videos, a single image has no timing information, but a video composed of continuous images contains timing information. In combination with the "motion capture" described in this article, there is no timing information for a single frame of the target object (such as a human body), but the continuous multi-frame postures of the target object contain timing information. This application uses LSTM to extract the timing information of the target object during movement for the final motion capture.

[0031] In the related technologies, the current motion capture technology that obtains posture information end-to-end based on deep learning models usually uses deep learning models (hereinafter referred to as models) containing temporal structures, such as: Temporal Convolutional Network (TCN), Multilayer Perceptron (MLP), Transformer, LSTM, Gated Recurrent Unit (GRU), etc.

[0032] For structures such as TCN, MLP, and Transformer, it is usually necessary to cache certain historical frame data to predict the results of the current frame, that is, a fixed time window is needed to cache the input data of the historical frame. If the time window is set too small, it often cannot meet the high-precision model requirements. If the time window is set too large, it will bring huge computing and power consumption costs, which are often unacceptable to mobile devices. In addition, when the above model structure is combined with specific heterogeneous hardware, there are often problems such as complex model structure, heterogeneous hardware platforms do not support operators, and it is difficult to carry out general deployment.

[0033] For LSTM and GRU structures, after the model completes the inference of the current frame, it only needs to cache the hidden state of the current frame and input it as input data when inferring the next frame, which greatly reduces the computational cost of model inference. However, when LSTM and GRU structures are combined with specific heterogeneous hardware, such as GPUs with a high degree of parallel computing, LSTM or GRU is difficult to parallelize and it is difficult to efficiently use computing resources, resulting in a significant increase in model inference time.

[0034] In order to solve the above technical problems, an embodiment of the present application provides a posture prediction method.

[0035] The following first introduces the application scenarios of the technical solution of this application. Figure 1 As shown, it is a schematic diagram of an application scenario of a posture prediction method provided in an embodiment of the present application. The application scenario involves a terminal device 110 and a server 120, and the terminal device 110 can communicate with the server 120.

[0036] In some possible implementations, Figure 1 The application scenarios shown may also include: base stations, core network side equipment, etc. In addition, Figure 1 One terminal device 110 and one server 120 are shown by way of example, but other numbers of terminal devices and servers may actually be included, and this embodiment of the present application does not impose any limitation on this.

[0037] In some possible implementations, Figure 1 The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This embodiment of the application does not limit this.

[0038] In some possible implementations, such as Figure 1The terminal device 110 shown may be installed with an application client, and when the application client runs in the terminal device 110, data can be exchanged with the server 120. For example, the client here may specifically include: a car client, a smart home client, a game client, a multimedia client (such as a video client), a social client, and an information client (such as a news client).

[0039] Optionally, in the embodiment of the present application, the terminal device 110 may be a device that has a rich human-computer interaction mode, has the ability to access the Internet, is usually equipped with various operating systems, and has strong processing capabilities. The terminal device 110 may be a smart phone, smart glasses, a handheld terminal, a smart TV, a tablet computer, a car terminal, etc., but is not limited thereto.

[0040] In one achievable manner, the server 120 and the terminal device 110 may execute the posture prediction method provided in the embodiment of the present application in an interactive manner, or the posture prediction method provided in the embodiment of the present application may be executed by the terminal device 110.

[0041] The technical solution of this application will be described in detail below:

[0042] Figure 2 A flowchart of a posture prediction method provided in an embodiment of the present application, the method can be performed as follows Figure 1 The terminal device 110 shown executes, but is not limited to, Figure 2 As shown, the method may include the following steps:

[0043] Step 011: extracting the timing information of the input data through the long short-term memory network 21, the timing information includes recent memory data, and the input data includes the current frame posture data of the target object collected by the posture sensor.

[0044] Specifically, see Figure 3 After obtaining the input data, the input data can be processed by LSTM to extract the time series information in the input data. The time series data includes recent memory data and long-term memory data.

[0045] Among them, recent memory data is also called hidden state in LSTM, and long-term memory data is also called cell state in LSTM. The cell state contains all long-term and short-term information, while the hidden state is the information most relevant to the current input extracted from the cell state.

[0046] The recent memory data and / or the long-term memory data can be used as part of the input data of the next frame, that is, the input data of the current frame generally includes the recent memory data and / or the long-term memory data extracted from the previous frame in addition to the current frame posture data of the target object collected by the posture sensor. Of course, for the first frame, since there is no previous frame, the input data may only include the posture data of the current frame.

[0047] Among them, the posture sensor can be an inertial measurement unit (IMU), a camera, etc. For example, if the posture sensor is an IMU and the target object is a human body, the IMU can be worn on different parts of the human body (such as the left wrist, the right wrist, the left knee, the right knee, the head and the waist). The posture information of the human body can be described by the rotation information of multiple bones of the human body. As the skeleton is a rigid body, the rotation information of any point on the skeleton is consistent. Therefore, the posture information of the human body can be represented by the posture data collected by the inertial measurement unit at different parts of the human body. For example, if the posture sensor is a camera and the target object is a human body, the image of the human body is collected by the camera as the collected posture data.

[0048] By using LSTM to extract timing information, the memory data of the previous frame can be used as part of the input data of the next frame. This not only reduces the computational cost of model reasoning, but also uses timing information to realize posture prediction of subsequent target objects, which is also conducive to improving the accuracy of posture prediction.

[0049] The function of LSTM for data processing is as follows:

[0050]

[0051] c t =σ(i t )⊙tanh(a t )+σ(f t )⊙c t-1 (2)

[0052] h t =σ(o t )⊙tanh(c t ) (3)

[0053] x t =[c t-1 , x input ] (4)

[0054] Among them, the four W_x, w_h, and b_i represent the weight matrix of the LSTM structure itself, that is, the learnable parameter matrix; X trepresents the input data at time t (such as the current frame), x input It is the current frame posture data collected by the posture sensor. t 、f t 、a t , o t Describe the intermediate calculation results of LSTM at time t respectively.

[0055] The output information of LSTM consists of two parts: h t and c t ,h t is the recent memory data, c t That is the long-term memory data; h t-1 represents the recent memory data extracted by LSTM at time (t-1), that is, the recent memory data of the previous frame; c t-1 Represents the long-term memory data extracted by LSTM at time (t-1).

[0056] σ is an activation function, such as the Sigmoid activation function, tanh is the hyperbolic tangent function, and ⊙ is an element-by-element multiplication operation. For example, in matrices A and B of the same size, after performing this operation, a matrix C of the same size is obtained, and the elements in corresponding positions in C are equal to the product of the elements in corresponding positions in A and B.

[0057] Step 012: Perform full connection and activation mapping on the input data through the nonlinear learning unit 22 to generate first intermediate data;

[0058] Specifically, while extracting the time series information of the input information through LSTM, it is also necessary to perform feature extraction on the input data to extract features for posture prediction. The nonlinear learning unit 22 can process the input data in parallel with the LSTM, perform full connection and activation mapping on the input data, and thus generate the first intermediate data.

[0059] The nonlinear learning unit 22 can also normalize the input data. Normalization can be layer normalization, which can be achieved through the LayerNorm operator. Layer normalization can ensure the smoothness of the model training process. The fully connected layer contains most of the learnable parameters, which can be used for matrix operations and linear mapping of input data, which can be achieved through the FullyConnect operator. Activation mapping introduces nonlinear factors into the model through activation functions, such as Elu or Relu functions. Elu is usually supported on most heterogeneous hardware platforms (i.e. electronic devices), and Relu is supported on all heterogeneous hardware platforms circulating in the market.

[0060] The calculation process of LayerNorm, FullyConnect, Elu / Relu is expressed as a function:

[0061]

[0062] Since normalization, full connection and activation mapping are just matrix multiplication and / or addition, any heterogeneous hardware platform can support matrix multiplication and addition, so that the nonlinear learning unit 22 can adapt to any heterogeneous hardware platform. By combining the widely used LSTM and nonlinear learning unit 22 to realize the prediction of the current posture, the model is better adapted to different heterogeneous hardware platforms. There is no need to adjust the model according to the adaptability of the specific heterogeneous hardware platform. The deployment cost is low and universal deployment can be achieved.

[0063] In addition, since the nonlinear learning unit 22 can introduce nonlinear factors, processing the input data through the nonlinear learning unit 22 is conducive to improving the fitting ability during the model training process. Therefore, the more nonlinear learning units 22 in parallel with LSTM, the stronger the model fitting ability.

[0064] Optionally, the nonlinear unit may perform normalization, full connection, and activation mapping on the input data in sequence to generate first intermediate data. By performing normalization first, a better gradient expression can be provided for the model, which is beneficial to the stability of the training of the model itself.

[0065] Optionally, the number of nonlinear learning units 22 that process input data in parallel with the LSTM may be one or more.

[0066] Specifically, for current heterogeneous hardware platforms (such as terminal device 110), the parallel capability is generally greater than 4. Therefore, the number of nonlinear learning units 22 can be set to 3, so as to maximize the use of the parallel capability of the hardware platform to improve the model performance while ensuring that the model can be adapted to any current heterogeneous hardware platform as much as possible.

[0067] And LSTM or GRU will inevitably cause precision loss errors. Among them, the precision loss error comes from the conversion of the model from the FLOAT32 data calculation method to FLOAT16, or INT8, INT16. The timing accumulation error mainly comes from the need to maintain the hidden state in the model when performing continuous frame reasoning, that is, the Hidden-State of the previous frame will become the model input of the next frame. As time accumulates, the calculation results of heterogeneous hardware will inevitably have precision errors, which will lead to an increase in the timing accumulation error.

[0068] In the embodiment of the present application, since the LSTM and the nonlinear learning unit 22 are in parallel, the number of LSTMs can be reduced. Compared with setting multiple LSTMs, due to the complex structure of the LSTM itself, the resource usage is large, and the precision loss error and timing accumulation error of the LSTM are large. When the LSTM and the nonlinear learning unit 22 are used to process the input data in parallel, since the data processing and the structure of the nonlinear learning unit 22 are relatively simple, the model of the present application takes up less resources for data processing and has smaller errors, which is beneficial to improving the accuracy of subsequent posture prediction.

[0069] Step 013: superimposing the recent memory data and the first intermediate data of the nonlinear learning unit 22 by the superposition unit 23 to generate second intermediate data, and the long short-term memory network 21, the nonlinear learning unit 22 and the superposition unit 23 constitute a basic learning module 25;

[0070] Specifically, after LSTM extracts the recent memory data of the input data and the nonlinear learning unit 22 generates the first intermediate data, the recent memory data and the first intermediate data can be superimposed to generate the second intermediate data.

[0071] Optionally, the superposition can be an axial stacking (Concat operation) or an add operation.

[0072] Among them, the Concat operation does not perform addition, but only merges the features of different structure outputs, that is, multiple matrices, into a larger matrix. The number of matrix channels increases, but the information under each feature is not superimposed. When heterogeneous hardware platforms need to perform calculations in the FLOAT16 or INT8 data format, compared with the ADD operation, the Concat operation has basically no quantization precision error and will not produce numerical overflow errors.

[0073] The Add operation accumulates multiple matrices element by element to form a matrix of the same size as the input matrix. The amount of information under each feature increases, but the number of channels does not increase. The Add operation is beneficial to improve the accuracy of classification.

[0074] Optionally, after superimposing the recent memory data and the first intermediate data, the superimposed data may be fully connected (e.g. Figure 3 The FullyConnect shown in FIG. 1 is used to perform dimensional transformation to generate second intermediate data. The calculation process for dimensional transformation is expressed as the following function:

[0075] d t =D(x t ); (6)

[0076] It can be understood that x in function (6) tRepresents the superimposed data, not the input data mentioned above.

[0077] Generate the second intermediate data p according to the input data t The calculation process of is expressed as the following function:

[0078]

[0079] That is to say, LSTM extracts h t , one or more nonlinear learning units 22 are respectively obtained Then h t , The second intermediate data p is finally generated by superimposing and transforming the dimension t .

[0080] Step 014: Predict the current posture of the target object based on the second intermediate data.

[0081] Specifically, the model includes one or more neural network structures (such as FullyConnect) that output the final desired result, and each neural network structure can implement different prediction tasks. After obtaining the second intermediate data, the second intermediate data can be input into the neural network structure to output the final result. If the neural network structure can predict the type of the target object, the neural network structure can output the type of the target object according to the second intermediate data; or, if the neural network structure can predict the current posture of the target object, the neural network structure can output the current posture of the target object according to the second intermediate data.

[0082] In the embodiment of the present application, at least a neural network structure capable of predicting the current posture of the target object is included, so as to predict the current posture of the target object based on the second intermediate data.

[0083] The posture prediction method provided in this embodiment uses the long short-term memory network 21 to extract the time series information of the input data, wherein the input data not only includes the posture data of the current frame collected by the posture sensor, but also includes the long-term memory data extracted from the previous frame. Afterwards, the recent memory data in the current frame and the first intermediate data learned by the nonlinear learning unit 22 in the input data are superimposed to obtain the second intermediate data for predicting the current posture. Since the time series information is considered when predicting the current posture of the target object, it is beneficial to improve the prediction accuracy of the current posture. And since the nonlinear learning unit 22 processes the input data through full connection and activation mapping, and both full connection and activation mapping only need to perform matrix multiplication and / or addition, the nonlinear learning unit 22 can adapt to any heterogeneous hardware platform, and then the model constructed by combining the nonlinear learning unit 22 and the widely used LSTM is used to realize the prediction of the current posture, which not only ensures the prediction accuracy of the current posture, but also makes the model more adaptable to different heterogeneous hardware platforms, and there is no need to adjust the model according to the adaptability of the specific heterogeneous hardware platform, the deployment cost is low, and universal deployment can be achieved.

[0084] In addition, building a model based on LSTM and nonlinear learning unit 22 can reduce computing costs (such as the amount of resources occupied by data processing), facilitate deployment on different heterogeneous hardware platforms, efficiently utilize the parallel computing capabilities of heterogeneous hardware platforms, and alleviate the precision loss error and timing accumulation error caused by the LSTM structure itself.

[0085] See also Figure 4 In some embodiments, step 014: predicting the current posture of the target object according to the second intermediate data includes:

[0086] Step 0141: Predict the current posture of the target object based on the second intermediate data output by multiple basic learning modules 25.

[0087] Specifically, the long short-term memory network 21, the nonlinear learning unit 22 and the superposition unit 23 can form a basic learning module 25. Multiple basic learning modules 25 can be set. Each basic learning module 25 can generate corresponding second intermediate data after steps 011 to 013. When predicting the current posture of the target object, the second intermediate data output by multiple basic learning modules 25 can be used to predict the current posture of the target object. In this way, the accuracy of posture prediction can be further improved.

[0088] See also Figure 5 In some embodiments, step 0141: predicting the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules 25, comprises:

[0089] Step 01411: superimposing the second intermediate data output by the plurality of basic learning modules 25 to generate third intermediate data;

[0090] Step 01412: Predict the current posture of the target object based on the third intermediate data.

[0091] Specifically, please combine Figure 6 In order to better utilize the parallel capabilities of the heterogeneous hardware platform (such as the heterogeneous hardware platform includes multiple processors, and each processor has parallel capabilities), multiple basic learning modules 25 can be connected in parallel to form a horizontal expansion structure. For example, each basic learning module 25 is deployed to a processor, and then the LSTM and nonlinear learning unit 22 are executed in parallel within the processor, so that each basic learning module 25 can output the corresponding second intermediate data.

[0092] Afterwards, when using the second intermediate data output by multiple basic learning modules 25 to predict the current posture of the target object, the second intermediate data output by multiple basic learning modules 25 can be superimposed to generate third intermediate data, wherein the superposition unit 23 can perform superposition in an axial stacking manner, thereby using the third intermediate data containing more channels to predict the current posture of the target object to improve the accuracy of posture prediction.

[0093] See also Figure 7 In some embodiments, step 0141: predicting the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules 25, comprises:

[0094] Step 01413: Predict the current posture of the target object based on the second intermediate data output by the last basic learning module 25.

[0095] Specifically, please combine Figure 8 In some low-bit computing hardware platforms, hardware platforms that use INT8 or INT12 data formats for computing, due to low-bit computing, the model will have serious quantization errors during the calculation process. The quantization error will be transmitted not only in the model but also in the timing structure.

[0096] Therefore, multiple basic learning modules 25 can be connected in series (that is, multiple basic learning modules 25 are connected in sequence) to form a longitudinal extension structure, so that the output of the previous basic learning module 25 serves as the input of the next basic learning module 25.

[0097] Optionally, when the depth of the longitudinally extending structure is not deep, the input of the basic learning module 25 does not superimpose the input data (i.e., the aforementioned input data including the current frame posture data of the target object collected by the posture sensor and the recent memory data and / or long-term memory data extracted by the long short-term memory network 21 in the previous frame), and the quantization error caused is also small. However, when the depth is relatively deep, it is necessary to make the input of each basic learning module 25 superimpose (such as axial stacking) the input data and the output of the previous basic learning module 25, so as to reduce the quantization error.

[0098] Finally, the current posture of the target object is predicted based on the second intermediate data output by the last basic learning module 25. This ensures that even after the model depth increases, the gradient during model training can be well transmitted. After multiple basic learning modules 25 are combined in a serial stacking manner, the generalization performance of the model can be further improved, and the anti-noise ability of the model can be enhanced, which can significantly reduce the quantization error caused by low-bit calculations. In addition, the LSTM and nonlinear learning units 22 in each basic module can still be executed in parallel, which can efficiently utilize the parallel capabilities of heterogeneous hardware platforms.

[0099] See also Fig. 9 In some embodiments, step 0141: predicting the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules 25, comprises:

[0100] Step 01414: superimposing the second intermediate data output by the plurality of basic learning modules 25 in the parallel learning module 26 to generate fourth intermediate data;

[0101] Step 01415: Predict the current posture of the target object based on the fourth intermediate data output by the last parallel learning module 26.

[0102] Specifically, please combine Fig.10 As mentioned above, the horizontal expansion structure makes better use of the parallel capabilities of heterogeneous hardware platforms, while the vertical extension structure can reduce the model quantization error of heterogeneous hardware platforms with low bit computing. Therefore, the horizontal expansion structure and the vertical extension structure can be combined to better utilize the parallel capabilities of heterogeneous hardware platforms while reducing the model quantization error of heterogeneous hardware platforms with low bit computing.

[0103] Among them, multiple basic learning modules 25 form a parallel learning module 26 to realize a horizontal expansion structure, and the fourth intermediate data is generated by superimposing the second intermediate data output by multiple basic learning modules 25 in the parallel learning module 26.

[0104] The parallel learning module 26 also includes multiple parallel learning modules 26, and then the multiple parallel learning modules 26 are connected in sequence to form a longitudinal extension structure, and the output of the previous parallel learning module 26 serves as the input of the next parallel learning module 26.

[0105] Optionally, when the depth of the longitudinally extending structure is not deep, the input of the parallel learning module 26 does not superimpose the input data (i.e., the aforementioned input data including the current frame posture data of the target object collected by the posture sensor and the recent memory data and / or long-term memory data extracted by the long short-term memory network 21 in the previous frame), and the quantization error is small. However, when the depth is relatively deep, it is necessary to make the input of each parallel learning module 26 superimpose (such as axial stacking) the input data and the output of the previous parallel learning module 26, so as to reduce the quantization error.

[0106] Therefore, the fourth intermediate data output by the previous parallel learning module 26 will serve as the input of the next parallel learning module 26. In this way, after multiple parallel learning modules 26 are combined in a serial stacking manner, the generalization performance of the model can be further improved, and the noise resistance of the model can be enhanced, which can significantly reduce the quantization error caused by low-bit calculation.

[0107] See also Fig.11 , the posture prediction method of the present application can be implemented by a posture prediction device 10, and the posture prediction device 10 includes an extraction module 11, a learning module 12, a superposition module 13 and a prediction module 14. The extraction module 11 is configured to extract the time series information of the input data through the long short-term memory network 21, the time series information includes recent memory data, and the input data includes the current frame posture data of the target object collected by the inertial measurement unit; the learning module 12 is configured to perform full connection and activation mapping on the input data through the nonlinear learning unit 22 to generate the first intermediate data; the superposition module 13 is configured to superimpose the recent memory data and the first intermediate data of the nonlinear learning unit 22 through the superposition unit 23 to generate the second intermediate data, and the long short-term memory network 21, the nonlinear learning unit 22 and the superposition unit 23 constitute a basic learning module 25; and the prediction module 14 is configured to predict the current posture of the target object according to the second intermediate data.

[0108] The prediction module 14 is further configured to predict the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules 25 .

[0109] The prediction module 14 is further configured to superimpose the second intermediate data output by the plurality of basic learning modules 25 to generate third intermediate data; and predict the current posture of the target object according to the third intermediate data.

[0110] The prediction module 14 is further configured to predict the current posture of the target object according to the second intermediate data output by the last basic learning module 25 .

[0111] The prediction module 14 is also configured to superimpose the second intermediate data output by multiple basic learning modules 25 in the parallel learning module 26 to generate fourth intermediate data; and predict the current posture of the target object based on the fourth intermediate data output by the last parallel learning module 26.

[0112] The learning module 12 is also configured to perform normalization, full connection and activation mapping on the input data in sequence through the nonlinear learning unit 22 to generate first intermediate data.

[0113] In the above text, the posture prediction device 10 is described from the perspective of the functional module in conjunction with the accompanying drawings. The functional module can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to execute, or can be executed by a combination of hardware and software modules in the coding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0114] Please refer again Figure 3 , is a schematic block diagram of a posture prediction model provided in an embodiment of the present application. The posture prediction model 20 can be deployed to an electronic device, so that the electronic device can implement the posture prediction method of any of the above embodiments.

[0115] The posture prediction model 20 includes a long short-term memory network 21, a nonlinear learning unit 22, a superposition unit 23 and a prediction unit 24. The long short-term memory network 21 is configured to extract the timing information of the input data, the timing information includes recent memory data, and the input data includes the current frame posture data of the target object collected by the inertial measurement unit; the nonlinear learning unit 22 is configured to perform full connection and activation mapping on the input data to generate first intermediate data, the recent memory data and the first intermediate data of the nonlinear learning unit 22 are superimposed to generate second intermediate data, the long short-term memory network 21, the nonlinear learning unit 22 and the superposition unit 23 constitute a basic learning module 25; and the prediction unit 24 is configured to predict the current posture of the target object according to the second intermediate data.

[0116] The long short-term memory network 21 and the nonlinear learning unit 22 are connected in parallel to realize parallel processing. The recent memory data extracted by the nonlinear learning unit 22 and the first intermediate data learned by the nonlinear learning unit 22 are superimposed by the superposition unit 23 to generate the second intermediate data, which are then input into the prediction unit 24. The prediction unit 24 can predict the current posture of the target object according to the second intermediate data and output it. Please refer to the description of step 011 to step 014 for details, which will not be repeated here.

[0117] Optionally, refer again to Figure 6 and Figure 8 The posture prediction model 20 includes multiple basic learning modules 25; the prediction unit 24 is also configured to predict the current posture of the target object based on the second intermediate data output by the multiple basic learning modules 25.

[0118] Among them, multiple basic learning modules 25 are connected in parallel to form the aforementioned horizontal expansion structure (please refer to step 01411 and step 01412 for details); or, multiple basic learning modules 25 are connected in series to form the aforementioned longitudinal extension structure (please refer to step 01413 for details).

[0119] Optionally, refer again to Fig.10 The posture prediction model 20 includes a plurality of parallel learning modules 26, each of which includes a plurality of parallel basic learning modules 25, and the plurality of parallel basic learning modules 25 in each parallel learning module 26 form the aforementioned horizontal extension structure (please refer to step 01414 for details); the plurality of parallel learning modules 26 are connected in series to form the aforementioned vertical extension structure (please refer to step 01415 for details).

[0120] Optionally, the number of basic learning modules 25 is 2 or 4.

[0121] Specifically, the more basic learning modules 25 are stacked, the higher the overall computational load will be; the higher the parallel computing capability required for heterogeneous hardware platforms will be. The parallel computing capability of heterogeneous hardware platforms determines the upper limit of parallelism. For example, if an electronic device includes 4 processors, and one processor can support one basic learning module 25, even if 12 basic learning modules 25 are stacked, there are still 4 basic learning models as a group, divided into three groups, and the same group is executed in parallel, while different groups are executed serially. Moreover, the parallel capability of current electronic devices is generally an even number. For example, if an electronic device includes 2 processors, 4 processors, 8 processors, etc., the number of basic learning modules 25 is 2 or 4, which can ensure that the parallel capability of the electronic device is fully utilized, and the overall required computational load is also small.

[0122] Optionally, the nonlinear learning unit 22 includes a normalization operator, a full connection operator and an activation operator, the normalization operator is configured to normalize the input data, the full connection operator is configured to perform a full connection operation on the normalized data, and the activation operator is configured to perform activation mapping processing on the fully connected data through a preset activation function to generate first intermediate data.

[0123] The posture prediction model 20 includes operators such as LSTM, normalization operator, full connection operator, activation operator and superposition operator. It has a simple structure and wide application, and can be widely adapted to various hardware platforms, thereby ensuring that the posture prediction model 20 is widely deployed on heterogeneous hardware platforms and can be deployed on multiple terminals, significantly reducing the deployment cost. And by implementing the parallel execution of the nonlinear learning unit 22 and LSTM, multiple basic learning units realize a horizontal expansion structure and / or a vertical extension structure, so that the posture prediction model 20 fully utilizes the parallel capabilities of the heterogeneous hardware platform while reducing the errors that may be caused by LSTM when reasoning on heterogeneous hardware.

[0124] Fig.12 1 is a schematic block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be the terminal device 110 or the server 120 in the above method embodiment.

[0125] like Fig.12 As shown, the electronic device 100 may include:

[0126] The memory 30 and the processor 40, the memory 30 is used to store the computer program and transmit the program code to the processor 40. In other words, the processor 40 can call and run the computer program from the memory 30 to implement the method in any embodiment of the present application.

[0127] In some embodiments of the present application, the processor 40 may include but is not limited to:

[0128] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0129] In some embodiments of the present application, the memory 30 includes but is not limited to:

[0130] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0131] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 30 and executed by the processor 40 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 100.

[0132] like Fig.12 As shown, the electronic device 100 may further include:

[0133] The transceiver 50 may be connected to the processor 40 or the memory 30 .

[0134] The processor 40 may control the transceiver 50 to communicate with other devices.

[0135] Specifically, information or data may be sent to other devices, or information or data sent by other devices may be received, such as receiving posture data collected by a posture sensor set on the target object through the transceiver 50. The transceiver 50 may include a transmitter and a receiver. The transceiver 50 may further include an antenna, and the number of antennas may be one or more.

[0136] It should be understood that the various components in the electronic device 100 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0137] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0138] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0139] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0141] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0142] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A posture prediction method, characterized in that: include: Extracting time series information of input data through a long short-term memory network, wherein the time series information includes recent memory data, and the input data includes current frame posture data of the target object collected by a posture sensor; Performing full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data; The recent memory data and the first intermediate data of the nonlinear learning unit are superimposed by a superposition unit to generate second intermediate data, and the long short-term memory network, the nonlinear learning unit and the superposition unit constitute a basic learning module; and The current posture of the target object is predicted according to the second intermediate data.

2. The posture prediction method according to claim 1, characterized in that: The basic learning modules are multiple, and predicting the current posture of the target object according to the second intermediate data includes: Predict the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules.

3. The posture prediction method according to claim 2, characterized in that: The step of predicting the current posture of the target object according to the second intermediate data output by the plurality of basic learning modules comprises: superimposing the second intermediate data output by a plurality of the basic learning modules to generate third intermediate data; The current posture of the target object is predicted according to the third intermediate data.

4. The posture prediction method according to claim 2, characterized in that: The multiple basic learning modules are connected in sequence, and the output of the previous basic learning module is used as the input of the next basic learning module. The current posture of the target object is predicted based on the second intermediate data output by the multiple basic learning modules, including: Predict the current posture of the target object according to the second intermediate data output by the last basic learning module.

5. The posture prediction method according to claim 2, characterized in that: The multiple basic learning modules form a parallel learning module, the parallel learning module includes multiple parallel learning modules, the multiple parallel learning modules are connected in sequence, the output of the previous parallel learning module is used as the input of the next parallel learning module, and the current posture of the target object is predicted according to the second intermediate data output by the multiple basic learning modules, including: superimposing the second intermediate data output by a plurality of the basic learning modules in the parallel learning module to generate fourth intermediate data; Predict the current posture of the target object according to the fourth intermediate data output by the last parallel learning module.

6. The posture prediction method according to claim 1, characterized in that: The superposition unit superimposes the recent memory data and the first intermediate data by axial stacking.

7. The posture prediction method according to claim 1, characterized in that: The step of performing full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data includes: The input data is sequentially normalized, fully connected, and activated mapped by the nonlinear learning unit to generate the first intermediate data.

8. A posture prediction device, characterized in that: include: An extraction module is configured to extract time sequence information of input data through a long short-term memory network, wherein the time sequence information includes recent memory data, and the input data includes current frame posture data of the target object collected by the posture sensor; A learning module, configured to perform full connection and activation mapping on the input data through a nonlinear learning unit to generate first intermediate data; A superposition module, configured to superimpose the recent memory data and the first intermediate data of the nonlinear learning unit through a superposition unit to generate second intermediate data, wherein the long short-term memory network, the nonlinear learning unit and the superposition unit constitute a basic learning module; and A prediction module is configured to predict a current posture of the target object based on the second intermediate data.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the posture prediction method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium comprising a computer program, wherein when the computer program is executed by a processor, the processor executes the posture prediction method according to any one of claims 1 to 7.

Images