Pose prediction method, apparatus and electronic device

Abstract

The application discloses a pose prediction method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring M first pose data, the i-th first pose data representing a pose of a target object at an i-th first time; extracting a first feature of the i-th first pose data; performing feature extraction on the first feature of the i-th first pose data according to the first features of other pose data, to obtain a second feature of the i-th first pose data; and determining N second pose data according to the second features of the M first pose data, the j-th second pose data representing a pose of the target object at a j-th second time, the j-th second time being after the i-th first time. The application can reduce the influence of individual first pose data on the prediction result, and improve the accuracy of the second pose data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a posture prediction method, apparatus, and electronic device. Background Technology

[0002] Exoskeleton robots are wearable robots frequently used in medical, outdoor sports, and other applications. In these scenarios, users wear exoskeleton robots, which provide assistance to help them get up or walk.

[0003] In general, to enable exoskeleton robots to provide timely assistance, it is necessary to predict the user's posture in advance. Therefore, how to predict the user's posture has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides an attitude prediction method, apparatus, and electronic device, which can be used to solve problems in related technologies. The technical solution includes the following contents.

[0005] On the one hand, a pose prediction method is provided, the method comprising:

[0006] Get M first pose data, where the i-th first pose data represents the pose of the target object at the i-th first time, where M is a positive integer greater than 1, and i takes any positive integer from 1 to M;

[0007] Extract the first feature of the i-th first pose data;

[0008] Based on the first features of other pose data, feature extraction is performed on the first features of the i-th first pose data to obtain the second features of the i-th first pose data. The other pose data includes each first pose data other than the i-th first pose data.

[0009] Based on the second features of the M first posture data, N second posture data are determined. The j-th second posture data represents the posture of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

[0010] On the other hand, an attitude prediction device is provided, the device comprising:

[0011] The acquisition module is used to acquire M first posture data, where the i-th first posture data represents the posture of the target object at the i-th first time, where M is a positive integer greater than 1, and i takes the value of any positive integer from 1 to M.

[0012] The extraction module is used to extract the first feature of the i-th first pose data;

[0013] The extraction module is further configured to perform feature extraction on the first feature of the i-th first pose data based on the first feature of other pose data, to obtain the second feature of the i-th first pose data, wherein the other pose data includes each first pose data other than the i-th first pose data;

[0014] The determining module is used to determine N second posture data based on the second features of the M first posture data. The j-th second posture data represents the posture of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

[0015] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the attitude prediction methods described above.

[0016] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement any of the attitude prediction methods described above.

[0017] On the other hand, a computer program is also provided, wherein the computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement any of the above-described attitude prediction methods.

[0018] On the other hand, a computer program product is also provided, which stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described attitude prediction methods.

[0019] The technical solution provided in this application brings at least the following beneficial effects:

[0020] The technical solution provided in this application extracts the first features of each first pose data, and then uses the first features of other pose data to perform feature extraction on the first feature of the i-th first pose data to obtain the second feature of the i-th first pose data. This achieves compensation or correction of the features of each first pose data based on global information, improving the accuracy of the features. Since the second features of the first pose data have high accuracy, accurate second pose data can be determined based on the second features of each first pose data, reducing the impact of individual first pose data on the prediction results and improving the accuracy of the second pose data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a computer system provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart of an attitude prediction method provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram illustrating the prediction of a pose sequence provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of an LSTM encoder provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a Transformer decoder provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of an attitude prediction device provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] like Figure 1 As shown, Figure 1 This is a schematic diagram of a computer system for an attitude prediction method provided in an embodiment of this application. The computer system includes a terminal device 101 and a server 102. The terminal device 101 has a client installed and running, and the server 102 provides background services to the client. The attitude prediction method provided in this embodiment can be executed by the terminal device 101, by the server 102, or by both the terminal device 101 and the server 102; this embodiment does not limit the execution of this method.

[0033] In one possible implementation, server 102 undertakes the primary computational work, while terminal device 101 undertakes the secondary computational work. Alternatively, server 102 undertakes the secondary computational work, while terminal device 101 undertakes the primary computational work. Or, terminal device 101 and server 102 collaborate on computation using a distributed computing architecture.

[0034] Optionally, terminal device 101 can be any electronic device product capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, remote control, voice interaction, or handwriting device. For example, terminal device 101 can be a mobile phone or a computer, wherein mobile phones include, but are not limited to, smartphones, foldable phones, slider phones, or flip phones, and computers include, but are not limited to, tablets, laptops, desktop computers, PPCs (Pocket PCs), or PCs (Personal Computers). In practical applications, terminal device 101 may also include smart speakers, smartwatches, PDAs (Personal Digital Assistants), wearable devices, smart car systems, smart TVs, etc.

[0035] Terminal device 101 can refer to one of a plurality of terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will know that the number of terminal devices 101 can be more or less. For example, there may be only one terminal device 101, or there may be dozens or hundreds of terminal devices 101, or more. This application embodiment does not limit the number or type of terminal devices 101.

[0036] Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: a cloud computing platform or a virtualization center. This embodiment of the application does not limit this. Server 102 communicates directly or indirectly with terminal device 101 via a wired or wireless network. Server 102 has data receiving, data processing, and data sending functions. Of course, server 102 may also have other functions, which are not limited in this embodiment of the application.

[0037] Those skilled in the art should understand that the terminal device 101 and server 102 described above are merely illustrative examples. Other existing or future terminal devices or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0038] This application provides a posture prediction method, which can be applied to the aforementioned computer system and executed by at least one of terminal device 101 or server 102. For ease of description, terminal device 101 and server 102 are collectively referred to as electronic devices; that is, the method of this application embodiment is executed by an electronic device. Figure 2 As shown, the method includes the following steps.

[0039] Step 201: Obtain M first pose data. The i-th first pose data represents the pose of the target object at the i-th first time. M is a positive integer greater than 1, and i takes any positive integer from 1 to M.

[0040] In this embodiment, the electronic device may include a server or a terminal device. The terminal device may be mounted on the exoskeleton robot or may be a mobile phone, computer, or other device. The target object wears the exoskeleton robot, and the exoskeleton robot collects first posture data, which represents the posture of the target object.

[0041] The content of the first posture data is not limited here. Exemplarily, the first posture data includes coordinate data of at least one skeleton of the target object, such as the coordinate data of the thigh skeleton, lower leg skeleton, etc. The coordinate data of any skeleton can be three-dimensional data, used to characterize the position of the skeleton in three-dimensional space. Alternatively, the first posture data includes motion data of at least one skeleton, where the motion data of any skeleton characterizes the motion state of the skeleton. This application embodiment does not limit the content of the skeleton's motion data; for example, the skeleton's motion data includes the skeleton's velocity, acceleration, angle, and angular velocity. Optionally, at least one inertial measurement unit (IMU) is installed on the exoskeleton robot, and an IMU is used to measure the motion data of one skeleton.

[0042] In practical applications, a time interval can be preset, and first attitude data can be collected once at each time interval. Assuming the time for collecting the first attitude data is the first time point, the difference between two consecutive first time points is the time interval. The size of the time interval is not limited here; for example, the time interval can be 0.05s (Second). Based on this, the electronic device can acquire M first attitude data points. M is a positive integer greater than 1; for example, M is 7. Assuming the time interval is 0.05s, then 7 first attitude data points are equivalent to the poses of the target object at 7 first time points, and these poses can form a 0.3s motion segment.

[0043] Step 202: Extract the first feature of the i-th first pose data.

[0044] In this example, the electronic device can acquire an attitude prediction model. An attitude prediction model is a network model used to predict attitude data; its structure and prediction method are not limited here. The attitude prediction model has a feature extraction function. The electronic device inputs M first attitude data points into the attitude prediction model, and the model performs feature extraction on each first attitude data point to obtain its first feature. The first feature of the i-th first attitude data point is a vector representing the attitude of the target object at the i-th time step; the size of the vector is not limited here.

[0045] This application does not limit the method of extracting the first feature. For example, the first feature of the i-th first pose data can be obtained by performing convolution processing on the i-th first pose data through a pose prediction model. Alternatively, step 202 includes steps 2021 to 2022 (not shown in the figure).

[0046] Step 2021: Obtain the action feature of the i-th first time point. The action feature of the i-th first time point represents the action of the target object up to the i-th first time point.

[0047] It is understandable that posture is the physical expression of a target object at a certain point in time, which is static information. Action, on the other hand, is the process of change in the physical expression of a target object over a period of time, which is dynamic information. Therefore, action includes posture at various points in time within a corresponding time period.

[0048] Based on this, for the first first time interval, the electronic device can determine the motion feature of the first first time interval based on the first posture data of the first first time interval, so as to characterize the action of the target object up to the first first time interval. For the i-th first time interval among the second to the M-th first time intervals, the electronic device can determine the motion feature of the i-th first time interval based on the first posture data of the first to the i-th first time intervals, so as to characterize the action of the target object up to the i-th first time interval. Here, the motion feature is a vector, and the magnitude of the vector is not limited here.

[0049] This application does not limit the method by which the electronic device obtains the action features of the i-th first time period. For example, the electronic device can perform convolution processing on the first posture data of the first first time period using a posture prediction model to obtain the action features of the first first time period. For the i-th first time period among the second to the M-th first time periods, the electronic device can concatenate the first posture data of the first to the i-th first time periods to obtain concatenated data, and then perform convolution processing on the concatenated data using a posture prediction model to obtain the action features of the i-th first time period.

[0050] Alternatively, the pose prediction model includes an LSTM (Long Short-Term Memory) encoder, where i takes any positive integer from 2 to M. Step 2021 includes: obtaining the action features at the (i-1)th time step, which represent the actions of the target object up to the (i-1)th time step; filtering the action features at the (i-1)th time step to obtain filtered features; and determining the action features at the ith time step based on the filtered features and the ith first pose data.

[0051] In this example, the electronic device can input M first posture data into the LSTM encoder, and the LSTM encoder can determine the action features at each first time point.

[0052] Optionally, the electronic device encodes each first time point to obtain encoded data for that first time point. For example, a sine / cosine position encoding scheme can be used to encode each first time point. The encoded data for each first time point is then concatenated with the corresponding first attitude data to obtain the input data for that first time point. For example, the input data Xtime(t) corresponding to the first time point t is determined according to Xtime(t) = X(t) + Etime(t), where X(t) represents the first attitude data corresponding to the first time point t, and Etime(t) represents the encoded data for the first time point t.

[0053] Subsequently, the encoded data from each first time step can be directly input into the LSTM encoder to determine the action features at each first time step. Alternatively, a Multilayer Perceptron (MLP) can be used to extract features from each input data at each first time step to better fuse the encoded data and the first pose data, obtaining the input features for that first time step. These input features fully represent the pose of the target object at that first time step. Then, the input features from each first time step are input into the LSTM encoder to determine the action features at each first time step. By inputting the encoded data from each first time step into the LSTM encoder, the network is explicitly provided with the first time step, improving the orderliness and accuracy of the network's processing of temporal data.

[0054] Optionally, the input X of the LSTM encoder satisfies: X∈R B×T×F Where R represents a real number, B represents the batch size, T represents the sequence length, and F represents the dimension of each data point in the sequence. In this example, M first pose data points constitute a first pose sequence. T represents the length of the first pose sequence; assuming M = 7 and the time interval is 0.05s, then T represents 7 or 0.3s. Each first pose data point is a data point within the first pose sequence. Assuming the first pose data includes the coordinate data and motion data of each skeleton, then the dimension of the first pose data is 2, i.e., F represents 2. B represents the number of first pose sequences input to the encoder; any two first pose sequences correspond to the same or different target objects.

[0055] like Figure 3As shown, the first time and corresponding first pose data are concatenated to obtain the first pose sequence. The dimensions of the first pose sequence are [B, T, F]. Electronic devices can input the first pose sequence into an LSTM encoder. LSTM encoders have a natural advantage in modeling temporal data, capable of sequentially capturing pose features that evolve over time. Furthermore, the structure of LSTM encoders is relatively simple, with low computational cost during training and inference, making them suitable for deployment in electronic devices. The LSTM encoder can then perform feature extraction on the first pose sequence to obtain the action features at each first time point.

[0056] Specifically, such as Figure 4 As shown, when i takes any positive integer from 2 to M, the LSTM encoder obtains the action features for each first time step as follows: Step 1: Obtain the action feature c of the (i-1)th first time step. t-1 The method for obtaining the action feature of the (i-1)th time step is similar to that for the action feature of the ith time step, and will not be repeated here. Step 2: For the action feature c of the (i-1)th time step... t-1 The filtering process is performed to obtain the filtered features. Specifically, based on the filtering information f t For the (i-1)th action feature c at the first moment t-1 The filtered features f are obtained. t ⊙c t-1 Filter information f t Satisfy: f t =σ(W xf x t +W hf h t-1 +b f Step 3, based on the filtered features f t ⊙c t-1 and the i-th first pose data x t Determine the action feature c of the i-th time step. t Specifically, on the one hand, for the i-th first pose data x t The first feature h of the (i-1)th first pose data t-1 The concatenated data is then subjected to Sigmoid activation processing to obtain the first activation result, i. t Satisfy: i t =σ(W xi X t +W hi h t-1 +b i On the other hand, for the i-th first pose data x t The first feature h of the (i-1)th first pose data t-1The concatenated data is then subjected to tanh activation to obtain a second activation result, which satisfies: tanh(W xc x t +W hc h t-1 +b c Next, the first activation result and the second activation result are multiplied together to obtain the multiplied result. This multiplied result is then added to the filtered features to obtain the i-th action feature c at the first time step. t And c t Satisfy: c t =f t ⊙c t-1 +i t ⊙tanh(W xc x t +W hc h t-1 +b c Where σ represents the Sigmoid activation process, and tanh represents the tanh activation process. t W represents the i-th first pose data. xf W xi W xc Both represent the weights of the i-th first pose data, and any two weights can be the same or different. t-1 The first feature representing the (i-1)th first pose data, W hf W hi W hc Both weights represent the weights of the first feature of the (i-1)th first pose data, and any two weights can be the same or different. f b i b c Both represent biases, and any two biases can be the same or different.

[0057] It should be noted that when i is 1, the electronic device can also determine the first action feature c1 at the first moment according to steps 1 to 3 above. In this process, c0 represents the initial action feature and is a pre-set vector. h0 represents the initial attitude feature and is also a pre-set vector.

[0058] By filtering the action features of the (i-1)th time step, key information in the action is captured, invalid information is removed, and the accuracy of the action features is improved. The action features of the ith time step are determined by combining the filtered features with the ith first pose data. This allows for the overlay of historical actions and current poses to obtain the action at the current time, which in turn facilitates the extraction of current pose features based on the current action.

[0059] Step 2022: Determine the first feature of the first pose data based on the action feature of the first time step and the first pose data of the first time step.

[0060] Since the action feature of the i-th first time period represents the action of the target object up to the i-th first time period, it can reflect the posture of the target object at the i-th first time period. Furthermore, the i-th first posture data represents the posture of the target object at the i-th first time period. Therefore, the electronic device can determine the first feature of the i-th first posture data based on the action feature of the i-th first time period and the i-th first posture data, so as to more fully and accurately represent the posture of the target object at the i-th first time period through the first feature.

[0061] By acquiring the action feature at the i-th time step, modeling is achieved for the first to i-th first pose data, i.e., modeling the local temporal sequence. This modeling captures short-term dependencies in poses, allowing the acquisition of the target object's actions within a short period. Generally, first pose data contains noise. By using the action feature at the i-th time step and the i-th first pose data, the first feature of the i-th first pose data is determined. This enables the determination of the current pose feature based on the target object's actions within a short period and its current pose, reducing the impact of noise on pose features. Even if the first pose data contains missing or abnormal data, these missing or abnormal data can be compensated for by the target object's actions within a short period, thus smoothing the data and improving robustness. In summary, extracting the first feature of the i-th first pose data improves the accuracy of the first feature.

[0062] The method for determining the first feature is not limited here. For example, the electronic device can filter features from the action features of the i-th first time interval. The filtered features can characterize the posture of the target object at the i-th first time interval. Furthermore, the electronic device performs feature extraction on the i-th first posture data, and the extracted features can also characterize the posture of the target object at the i-th first time interval. The filtered features and the extracted features are then fused to obtain the first feature of the i-th first posture data.

[0063] Alternatively, i can take any positive integer from 2 to M. Step 2022 includes: determining filtering information based on the first features of the i-th first pose data and the (i-1)-th first pose data; and filtering out the first features of the i-th first pose data from the motion features of the i-th first time step based on the filtering information.

[0064] In this example, such as Figure 4As shown, when i takes any positive integer from 2 to M, the LSTM encoder obtains the first feature of each first pose data in the following manner. Step 1, based on the i-th first pose data x t The first feature h of the (i-1)th first pose data t-1 Determine the filter information. t Among them, filtering information o t It is a vector whose size and the first-time action feature c are given by... t They are the same size. Filter information o t Each element in the table represents whether c is filtered. t The corresponding element in [the context]. Generally, filtering information [is done]. t The elements in the array are numerical values ​​between 0 and 1. When an element is 0, it indicates that c is not selected. t The corresponding element in the table; when the element is 1, it represents the selection of c. t The corresponding element in the table; when the element is greater than 0 and less than 1, it represents the proportional selection of c. t The corresponding element in. Specifically, filtering information o t Satisfy: o t =σ(W xo x t +W ho h t-1 +b o Step 2, based on the filtering information. t From the i-th first-time action feature c t The first feature h of the i-th first pose data is selected from the data. t Specifically, for the i-th action feature c in the first moment... t Perform tanh activation processing to obtain the activation result tanh(c) t The activation result tanh(c) will be used. t ) and filtering information t Multiply to obtain the first feature h of the i-th first pose data. t And the first feature h of the i-th first pose data t Satisfy: h t =o t ⊙tanh(c t Where σ represents the Sigmoid activation process, and tanh represents the tanh activation process. t W represents the i-th first pose data. xo The weight representing the i-th first pose data. t-1 The first feature representing the (i-1)th first pose data, W ho The weight representing the first feature of the (i-1)th first pose data. o Characterization bias.

[0065] It should be noted that when i is 1, the electronic device can also determine the first feature h1 of the first attitude data according to steps 1 to 2 above. In this process, h0 represents the initial attitude feature and is a pre-set vector.

[0066] Optionally, the first features of the M first pose data satisfy: H∈R B×T×C Where H represents the first feature of M first pose data, R represents a real number, B represents the batch size, T represents the sequence length, and C represents the dimension of each first feature in the sequence.

[0067] By determining the filtering information and selecting the first feature of the first pose data from the first action feature of the first time step based on the filtering information, it is possible to filter the current pose feature from continuous action features. This helps to reduce the impact of noise in the first pose data on the pose feature and improve the accuracy of the pose feature.

[0068] Step 203: Based on the first features of other pose data, perform feature extraction on the first features of the i-th first pose data to obtain the second features of the i-th first pose data. Other pose data include each first pose data other than the i-th first pose data.

[0069] Assume M = 7. When i = 1, the other pose data includes the 2nd to 7th first pose data; when i = 2, the other pose data includes the 1st, 3rd to 7th first pose data; and so on. After extracting the first features of M first pose data, feature extraction can be performed again on the first feature of the i-th first pose data based on the first features of the other pose data to obtain the second feature of the i-th first pose data. This allows for updating the features of the current pose based on context information, improving the accuracy of the second feature. This application does not limit the method of feature extraction.

[0070] In one possible implementation, step 203 includes steps 2031 to 2032 (not shown in the figure).

[0071] Step 2031: Based on the first features of other pose data, perform attention processing on the first features of the i-th first pose data to obtain the attention features of the i-th first pose data.

[0072] In this example, the pose prediction model may include an attention layer. The electronic device can input the first features of M first pose data points into the attention layer, which then performs attention processing to obtain the attention features of the M first pose data points. The type of attention layer is not limited here. For example, it could be a cross-attention layer or a multi-head attention layer. Different types of attention layers result in different attention processing methods, which will not be elaborated further here.

[0073] Generally, the attention layer performs the following processing: Step 1, performs a linear mapping on the first feature of the i-th first pose data to obtain a query vector. Step 2, performs two linear mappings on the first feature of any other pose data to obtain a key vector and a value vector, respectively. Step 3, based on the query vector, key vector, and value vector, determines the attention sub-feature, which is the attention processing result between the first feature of another pose data and the first feature of the i-th first pose data. Optionally, according to... Determine the attention sub-feature Attn(Q,K,V). Here, Q represents the query vector, K represents the key vector, V represents the value vector, softmax represents the normalization function, T is the transpose sign, and d... k The dimension of the key vector K is represented. Step 4: Combine the attention sub-features to obtain the attention feature of the i-th first pose data.

[0074] Optionally, the pose prediction model includes an LSTM encoder and a Transformer decoder, such as Figure 3 As shown, the Transformer decoder is cascaded after the LSTM encoder. First, the LSTM encoder determines the first features of M first pose data. Then, these first features can be directly input into the Transformer decoder, or the encoded data from the first time step and the corresponding first features of the first pose data can be concatenated to obtain the input data for the first time step. Each of these first-time-step input data is then input into the Transformer decoder. For example, the input data Z(t) corresponding to the first time step t is determined according to Z(t) = H(t) + Etime(t), where H(t) represents the first feature of the first pose data at the first time step t, and Etime(t) represents the encoded data at the first time step t. The Transformer decoder includes an attention layer, through which attention processing is performed.

[0075] In an exemplary embodiment, the first feature of the i-th first pose data includes features of L modal data, where L is a positive integer greater than 1. Step 2031 includes: performing attention processing on the features of the k-th modal data based on the first features of other pose data to obtain a first processing result, where k is any positive integer from 1 to L; performing attention processing on the features of the k-th modal data based on the features of each modal data other than the features of the k-th modal data to obtain a second processing result; and determining the attention feature of the i-th first pose data based on the first processing result and the second processing result.

[0076] As mentioned above, the first pose data can include at least one of the coordinate data or motion data of each skeleton. Coordinate data is one modality of data, while motion data is another. In practical applications, the first pose data can also include data from other modalities; for example, it may include a pose image of the target object, etc. Therefore, each set of first pose data includes at least one modal of data.

[0077] When there are at least two modal data points, the first feature of each first pose data point includes the features of all modal data points. Assuming there are L modal data points, then the first feature of each first pose data point includes the features of all L modal data points.

[0078] For the feature of the k-th modal data in the first feature of the i-th first pose data, a linear mapping can be performed on the feature of the k-th modal data to obtain a query vector. On one hand, two linear mappings are performed on the first feature of any other pose data to obtain a first key vector and a first value vector, respectively. Based on the query vector, the first key vector, and the first value vector, the first attention sub-feature is determined. In this way, the first attention sub-features corresponding to each other pose data can be determined. Then, the various first attention sub-features are merged to obtain the first processing result. On the other hand, two linear mappings can be performed on the feature of any other modal data to obtain a second key vector and a second value vector, respectively. Based on the query vector, the second key vector, and the second value vector, the second attention sub-feature is determined. In this way, the attention sub-features corresponding to each other modal data can be determined. Then, the various second attention sub-features are merged to obtain the second processing result.

[0079] In this context, "other modal data" refers to modal data other than the features of the k-th modal data. For example, if L is 3, then: when k=1, other modal data refers to the 2nd and 3rd modal data; when k=2, other modal data refers to the 1st and 3rd modal data; and when k=3, other modal data refers to the 1st and 2nd modal data.

[0080] Then, the first processing result and the second processing result are merged to obtain the attention feature of the i-th first pose data. For example, the first processing result and the second processing result are weighted and calculated to obtain the attention feature of the i-th first pose data.

[0081] Since the first pose data includes data from multiple modalities, attention processing is performed on the features of the k-th modal data using the first features of other pose data. This achieves information compensation between different pose data and improves the accuracy of the attention features. Furthermore, attention processing is performed on the features of the k-th modal data using features from all modalities other than the k-th modal data. This achieves the fusion of multimodal data, which not only improves model compatibility but also captures pose features more accurately, thereby improving the accuracy of the second pose data.

[0082] Step 2032: Determine the second feature of the i-th first pose data based on the attention features of the i-th first pose data.

[0083] The attention features of the i-th first pose data can be used as the second feature of the i-th first pose data. Alternatively, the attention features of the i-th first pose data can be processed by mapping, convolution, activation, normalization, etc., to obtain the second feature of the i-th first pose data. For example, the attention features of the i-th first pose data can be added to the first feature of the i-th first pose data, the added feature can be normalized, and the second feature of the i-th first pose data can be determined based on the normalization result.

[0084] Suppose that the first feature of the i-th first pose data is Z, and the attention feature of the i-th first pose data is Attn(Z), then the normalization result Z mid It can be represented as: Z mid =LayerNorm(Z+Attn(Z)), where LayerNorm represents the normalization process of the representation layer, Z is also the input of the attention layer, and Attn(Z) is also the output of the attention layer.

[0085] By applying attention processing to the first feature of the i-th first pose data using the first features of other pose data, information compensation for each pose feature is achieved using contextual information. This improves the accuracy of pose features, reduces the impact of individual data on the prediction result (i.e., the second pose data), and enhances the accuracy of the prediction result. When some skeleton coordinate or motion data is missing, the attention mechanism can mitigate the impact of missing information on the prediction result. When some first pose data contains noise, the attention mechanism can reduce the impact of noise, making the prediction result more stable.

[0086] In one possible implementation (implementation A), the attention feature of the i-th first pose data includes features from the first channel and features from the second channel. Step 2032 includes: merging the features from the first channel and the first feature of the i-th first pose data to obtain merged features; and determining the second feature of the i-th first pose data based on the merged features and the features from the second channel.

[0087] In this example, such as Figure 5 As shown, the first feature Z(t) of the i-th first pose data is input into the attention layer. Attention processing is performed by the attention layer to obtain the attention feature of the i-th first pose data. The attention feature of the i-th first pose data includes features from multiple channels, which are divided into first channels and second channels. The number of first channels and second channels is not limited here. For example, the number of first channels and second channels is equal; that is, in the channels corresponding to the attention feature of the i-th first pose data, 50% of the channels are first channels, and the remaining 50% are second channels. In practical applications, the number of first channels and second channels can be unequal. For example, in the channels corresponding to the attention feature of the i-th first pose data, 30% of the channels are first channels, and the remaining 70% are second channels.

[0088] The electronic device can merge the features of the first channel and the first feature of the i-th first pose data. For example, it can concatenate or add the features of the first channel and the first feature of the i-th first pose data to obtain a merged feature. Then, it concatenates the merged feature and the features of the second channel to obtain a concatenated feature. Optionally, the concatenated feature can be used as the second feature of the i-th first pose data, or normalization can be performed on the concatenated feature to obtain the second feature of the i-th first pose data. In this example, this feature processing method is called Half-Residual Stacking.

[0089] By using a semi-residual superposition method, a portion of the attention layer's output retains its original value. This portion of the output, based on features determined by global information, boasts high accuracy. The remaining portion of the attention layer's output is added to or concatenated with the input. This method introduces temporal information into the features, improving the effectiveness of subsequent feature processing. In other words, determining the second feature of the first pose data through semi-residual superposition ensures both the accuracy of the second feature and enhances its temporal information, thereby improving the accuracy of the prediction result.

[0090] In another possible implementation (implementation B), step 2032 includes: performing at least one of linear mapping or nonlinear mapping on the attention features of the i-th first pose data to obtain a mapping result; and determining the second feature of the i-th first pose data based on the mapping result.

[0091] like Figure 3 As shown, the Transformer decoder includes an attention layer and a feedforward layer, with the feedforward layer concatenated after the attention layer. Attention processing can be performed on the first features of the M first pose data points through the attention layer to obtain attention features for the M first pose data points. Then, at least one of linear or nonlinear mapping is performed on the attention features of the M first pose data points through the feedforward layer to obtain the mapping result.

[0092] Optionally, the structure of the Transformer decoder is as follows: Figure 5 As shown. In this example, the first feature of the i-th first pose data and the attention feature of the i-th first pose data can be added or concatenated to obtain the first processed feature. Alternatively, following the implementation principle of implementation method A, the features of the first channel and the first feature of the i-th first pose data are first merged, and then the merged features and the features of the second channel are concatenated to obtain the first processed feature. Then, the first processed feature is normalized through a normalization layer to obtain the second processed feature. Next, at least one of linear mapping or nonlinear mapping is performed on the second processed feature through a feedforward layer to obtain the mapping result. Optionally, according to Z... out =W2σ(W1Z) mid +b1)+b2, determine the mapping result Z out Here, W2 and W1 represent two weights, which can be the same or different. b1 and b2 represent two biases, which can also be the same or different. σ represents the sigmoid activation process, and Z... mid Characterize the second processing feature.

[0093] Subsequently, the mapping result can be used as the second feature of the i-th first pose data. Alternatively, the mapping result and the second processed feature can be added or concatenated to obtain a third processed result, which can then be used as the second feature of the i-th first pose data. Alternatively, the third processed result can be normalized using a normalization layer to obtain the second feature of the i-th first pose data.

[0094] Optionally, such as Figure 5As shown, the mapping result includes features from multiple channels, which are divided into third and fourth channels. The number of third and fourth channels is not limited here; for example, the number of third and fourth channels is equal. That is, in the mapping result of the i-th first pose data, 50% of the channels are third channels, and the remaining 50% are fourth channels. In practical applications, the number of third and fourth channels can be unequal; for example, in the mapping result of the i-th first pose data, 30% of the channels are third channels, and the remaining 70% are fourth channels.

[0095] The electronic device can merge the features of the third channel with the attention features (or second processing features) of the i-th first pose data. For example, it can concatenate or add the features of the third channel and the attention features (or second processing features) of the i-th first pose data to obtain a merged feature. Then, it concatenates the merged feature with the features of the fourth channel to obtain a concatenated feature. Optionally, the concatenated feature can be used as the second feature of the i-th first pose data, or normalization processing can be performed on the concatenated feature to obtain the second feature of the i-th first pose data. Optionally, as... Figure 5 As shown, according to Z l+1 =LayerNorm(Z) mid +Z out Determine the second feature Z of the i-th first pose data. l+1 Among them, LayerNorm characterization layer normalization processing, Z mid Z represents the input to the feedforward network. out Characterizes the output of the feedforward network.

[0096] In practical applications, there can be at least one Transformer decoder. When there are two or more Transformer decoders, they are cascaded. The electronic device can input the first features of M first pose data into the first Transformer decoder, and the Transformer decoder performs feature extraction on the first features of each first pose data according to steps 2031 to 2032. The features of each first pose data output by the first Transformer decoder are input into the second Transformer decoder, and the Transformer decoder performs feature extraction on the features of each first pose data again according to steps 2031 to 2032. The features of each first pose data output by the second Transformer decoder are input into the third Transformer decoder, and the Transformer decoder performs feature extraction on the features of each first pose data again according to steps 2031 to 2032. This process of feature extraction is repeated until the last Transformer decoder outputs the features of each first pose data. The second feature of the i-th first pose data is the feature of the i-th first pose data output by the last Transformer decoder.

[0097] Optionally, the second features of the M first pose data satisfy: Z∈R B×T×C Where Z represents the second feature of M first pose data, R represents a real number, B represents the batch size, T represents the sequence length, and C represents the dimension of each second feature in the sequence. The dimension of the second feature is the same as the dimension of the first feature.

[0098] Step 204: Based on the second features of the M first posture data, determine N second posture data. The j-th second posture data represents the posture of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

[0099] like Figure 3 As shown, the second features of M first pose data can be determined using a Transformer decoder. The pose prediction model also includes an output layer, which is concatenated after the Transformer decoder. The output layer can map the second features of the M first pose data into a second pose sequence. The structure of the output layer is not limited here; for example, it could be a multilayer perceptron. Based on this, the second pose sequence... satisfy: Among them, MLP outCharacterizing the multilayer perceptron, Z final The second feature characterizes the M first pose data.

[0100] The second pose sequence includes N second pose data points, where the value of N is not limited here; for example, N equals M. The second pose data may include at least one of the coordinate data and motion data of each skeleton. Optionally, the second pose sequence satisfies: Where R represents a real number, B represents the batch size, T represents the sequence length, and F represents the dimension of each data point in the sequence. In this example, N second pose data points constitute a second pose sequence. T represents the length of the second pose sequence; assuming N = 7 and the time interval is 0.05s, then T represents 7 or 0.3s. Each second pose data point is a data point in the second pose sequence. Assuming the second pose data includes the coordinate data and motion data of each skeleton, then the dimension of the second pose data is 2, i.e., F represents 2. B represents the number of output second pose sequences; any two second pose sequences correspond to the same or different target objects.

[0101] In this example, an LSTM encoder is first used to model the local temporal sequence, capturing the short-term dependencies of pose, which effectively smooths local noise and improves the accuracy of the first feature. Next, a Transformer decoder is used to model the global temporal sequence, capturing the global dependencies of pose, which further improves the accuracy of the second feature. In summary, this application can ensure prediction accuracy while taking into account the dependencies between short-term and long-term temporal sequences, improving the model's robustness and the scalability of multimodal data. It can determine the second pose data in real time, meeting real-time requirements.

[0102] Optionally, the second attitude data and the first attitude data have the same format. This setting facilitates comparison between the second attitude data and the true attitude data, allowing for the calculation of the error or evaluation index between the two, thereby assessing the accuracy of the second attitude data.

[0103] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the attitude data involved in this application was obtained with full authorization.

[0104] In the above method, by extracting the first features of each first pose data, and then using the first features of other pose data, feature extraction is performed on the first features of the i-th first pose data to obtain the second features of the i-th first pose data. This achieves feature compensation or correction for each first pose data based on global information, improving the accuracy of the features. Since the second features of the first pose data have high accuracy, accurate second pose data can be determined based on the second features of each first pose data, reducing the impact of individual first pose data on the prediction results and improving the accuracy of the second pose data.

[0105] Figure 6 The diagram shown is a structural schematic of an attitude prediction device provided in an embodiment of this application. Figure 6 As shown, the device includes:

[0106] The acquisition module 601 is used to acquire M first posture data, where the i-th first posture data represents the posture of the target object at the i-th first time, M is a positive integer greater than 1, and i takes any positive integer from 1 to M.

[0107] Extraction module 602 is used to extract the first feature of the i-th first pose data;

[0108] The extraction module 602 is also used to perform feature extraction on the first feature of the i-th first pose data based on the first feature of other pose data to obtain the second feature of the i-th first pose data. The other pose data includes each first pose data other than the i-th first pose data.

[0109] The determination module 603 is used to determine N second attitude data based on the second features of M first attitude data. The j-th second attitude data represents the attitude of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

[0110] In one possible implementation, the extraction module 602 is used to obtain the action features of the i-th first time period, which characterize the action of the target object up to the i-th first time period; and to determine the first feature of the i-th first posture data based on the action features of the i-th first time period and the i-th first posture data.

[0111] In one possible implementation, i takes the value of any positive integer from 2 to M;

[0112] The extraction module 602 is used to obtain the action features of the (i-1)th first time period, which represent the action of the target object up to the (i-1)th first time period; to filter the action features of the (i-1)th first time period to obtain the filtered features; and to determine the action features of the ith first time period based on the filtered features and the ith first posture data.

[0113] In one possible implementation, i takes the value of any positive integer from 2 to M;

[0114] The extraction module 602 is used to determine filtering information based on the first feature of the i-th first posture data and the (i-1)-th first posture data; and to filter out the first feature of the i-th first posture data from the action features of the i-th first time step based on the filtering information.

[0115] In one possible implementation, the extraction module 602 is used to perform attention processing on the first feature of the i-th first pose data based on the first feature of other pose data to obtain the attention feature of the i-th first pose data; and to determine the second feature of the i-th first pose data based on the attention feature of the i-th first pose data.

[0116] In one possible implementation, the first feature of the i-th first pose data includes features of L modal data, where L is a positive integer greater than 1;

[0117] The extraction module 602 is used to perform attention processing on the features of the kth modal data based on the first features of other pose data to obtain a first processing result, where k is any positive integer from 1 to L; perform attention processing on the features of the kth modal data based on the features of each modal data other than the features of the kth modal data to obtain a second processing result; and determine the attention features of the i-th first pose data based on the first processing result and the second processing result.

[0118] In one possible implementation, the attention features of the i-th first pose data include features from the first channel and features from the second channel;

[0119] Extraction module 602 is used to merge the features of the first channel and the first feature of the i-th first pose data to obtain the merged feature; and to determine the second feature of the i-th first pose data based on the merged feature and the features of the second channel.

[0120] In one possible implementation, the extraction module 602 is used to perform at least one of linear mapping or nonlinear mapping on the attention features of the i-th first pose data to obtain a mapping result; and to determine the second feature of the i-th first pose data based on the mapping result.

[0121] In the aforementioned device, by extracting the first features of each first pose data, and then using the first features of other pose data, feature extraction is performed on the first feature of the i-th first pose data to obtain the second feature of the i-th first pose data. This achieves feature compensation or correction for each first pose data based on global information, improving the accuracy of the features. Since the second features of the first pose data have high accuracy, accurate second pose data can be determined based on the second features of each first pose data, reducing the impact of individual first pose data on the prediction result and improving the accuracy of the second pose data.

[0122] It should be understood that the above Figure 6 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0123] Figure 7 A structural block diagram of a terminal device 700 provided in an exemplary embodiment of this application is shown. The terminal device 700 includes a processor 701 and a memory 702.

[0124] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0125] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one computer program, which is executed by the processor 701 to implement the attitude prediction method provided in the method embodiments of this application.

[0126] In some embodiments, the terminal device 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, and a power supply 708.

[0127] Peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0128] The radio frequency (RF) circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0129] Display screen 705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 705 may be a single screen, disposed on the front panel of terminal device 700; in other embodiments, display screen 705 may be at least two screens, disposed on different surfaces of terminal device 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of terminal device 700. Furthermore, display screen 705 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0130] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0131] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0132] Power supply 708 is used to supply power to the various components in terminal device 700. Power supply 708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 708 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0133] In some embodiments, the terminal device 700 further includes one or more sensors 709. The one or more sensors 709 include, but are not limited to, an accelerometer 711, a gyroscope 712, a pressure sensor 713, an optical sensor 714, and a proximity sensor 715.

[0134] Accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 700. For example, accelerometer 711 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 701 can control display screen 705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 711. Accelerometer 711 can also be used for games or for acquiring user motion data.

[0135] The gyroscope sensor 712 can detect the orientation and rotation angle of the terminal device 700. The gyroscope sensor 712, in conjunction with the accelerometer sensor 711, can collect 3D motion data from the user on the terminal device 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0136] The pressure sensor 713 can be disposed on the side bezel of the terminal device 700 and / or on the lower layer of the display screen 705. When the pressure sensor 713 is disposed on the side bezel of the terminal device 700, it can detect the user's grip signal on the terminal device 700, and the processor 701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is disposed on the lower layer of the display screen 705, the processor 701 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0137] An optical sensor 714 is used to collect ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 based on the ambient light intensity collected by the optical sensor 714. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity collected by the optical sensor 714.

[0138] The proximity sensor 715, also known as a distance sensor, is typically located on the front panel of the terminal device 700. The proximity sensor 715 is used to detect the distance between the user and the front of the terminal device 700. In one embodiment, when the proximity sensor 715 detects that the distance between the user and the front of the terminal device 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from a screen-on state to a screen-off state; when the proximity sensor 715 detects that the distance between the user and the front of the terminal device 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from a screen-off state to a screen-on state.

[0139] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the terminal device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0140] Figure 8This is a schematic diagram of the server structure provided in the embodiments of this application. The server 800 can vary considerably due to different configurations or performance. It may include one or more processors 801 and one or more memories 802. The one or more memories 802 store at least one computer program, which is loaded and executed by the one or more processors 801 to implement the attitude prediction method provided in the above-described method embodiments. For example, the processor 801 is a CPU. Of course, the server 800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 800 may also include other components for implementing device functions, which will not be elaborated here.

[0141] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described attitude prediction methods.

[0142] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0143] In an exemplary embodiment, a computer program is also provided, which is at least one such computer program, loaded and executed by a processor to enable the electronic device to implement any of the above-described attitude prediction methods.

[0144] In an exemplary embodiment, a computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described attitude prediction methods.

[0145] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0146] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A posture prediction method, characterized in that, The method includes: Get M first pose data, where the i-th first pose data represents the pose of the target object at the i-th first time, where M is a positive integer greater than 1, and i takes any positive integer from 1 to M; Extract the first feature of the i-th first pose data; Based on the first features of other pose data, feature extraction is performed on the first features of the i-th first pose data to obtain the second features of the i-th first pose data. The other pose data includes each first pose data other than the i-th first pose data. Based on the second features of the M first posture data, N second posture data are determined. The j-th second posture data represents the posture of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

2. The method according to claim 1, characterized in that, The extraction of the first feature of the i-th first pose data includes: Obtain the action feature of the i-th first time period, wherein the action feature of the i-th first time period represents the action presented by the target object up to the i-th first time period; Based on the action features at the i-th first time moment and the i-th first posture data, determine the first feature of the i-th first posture data.

3. The method according to claim 2, characterized in that, The value of i is any positive integer from 2 to M; obtaining the action feature of the i-th time moment includes: Obtain the action features of the (i-1)th first time interval, wherein the action features of the (i-1)th first time interval characterize the action presented by the target object up to the (i-1)th first time interval; The action features of the (i-1)th time step are filtered to obtain the filtered features; Based on the filtered features and the i-th first posture data, the action feature of the i-th first time moment is determined.

4. The method according to claim 2, characterized in that, The value of i is any positive integer from 2 to M; the step of determining the first feature of the i-th first posture data based on the action feature of the i-th first time moment and the i-th first posture data includes: Based on the first features of the i-th first pose data and the (i-1)-th first pose data, the filtering information is determined; Based on the filtering information, the first feature of the first pose data of the i-th time is filtered out from the action features of the i-th first time.

5. The method according to claim 1, characterized in that, The step of performing feature extraction on the first feature of the i-th first pose data based on the first feature of other pose data to obtain the second feature of the i-th first pose data includes: Based on the first features of other pose data, attention processing is performed on the first features of the i-th first pose data to obtain the attention features of the i-th first pose data; Based on the attention features of the i-th first pose data, determine the second feature of the i-th first pose data.

6. The method according to claim 5, characterized in that, The first feature of the i-th first pose data includes features of L modal data, where L is a positive integer greater than 1; the step of performing attention processing on the first feature of the i-th first pose data based on the first features of other pose data to obtain the attention feature of the i-th first pose data includes: Based on the first feature of other pose data, attention processing is performed on the features of the kth modality data to obtain a first processing result, where k is any positive integer from 1 to L; Based on the features of each modality data other than the features of the k-th modality data, attention processing is performed on the features of the k-th modality data to obtain a second processing result; Based on the first processing result and the second processing result, the attention features of the i-th first pose data are determined.

7. The method according to claim 5, characterized in that, The attention features of the i-th first pose data include features from the first channel and features from the second channel; determining the second feature of the i-th first pose data based on the attention features of the i-th first pose data includes: The features of the first channel and the first features of the i-th first pose data are combined to obtain the combined features; Based on the merged features and the features of the second channel, the second feature of the i-th first pose data is determined.

8. The method according to claim 5, characterized in that, The step of determining the second feature of the i-th first pose data based on the attention features of the i-th first pose data includes: Perform at least one of linear or nonlinear mapping on the attention features of the i-th first pose data to obtain the mapping result; Based on the mapping result, the second feature of the i-th first pose data is determined.

9. An attitude prediction device, characterized in that, The device includes: The acquisition module is used to acquire M first posture data, where the i-th first posture data represents the posture of the target object at the i-th first time, where M is a positive integer greater than 1, and i takes the value of any positive integer from 1 to M. The extraction module is used to extract the first feature of the i-th first pose data; The extraction module is further configured to perform feature extraction on the first feature of the i-th first pose data based on the first feature of other pose data, to obtain the second feature of the i-th first pose data, wherein the other pose data includes each first pose data other than the i-th first pose data; The determining module is used to determine N second posture data based on the second features of the M first posture data. The j-th second posture data represents the posture of the target object at the j-th second time. The j-th second time is located after the i-th first time. N is a positive integer, and j takes any positive integer from 1 to N.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the attitude prediction method as described in any one of claims 1 to 8.

Images