Posture recognition method based on neural network

CN118968632AInactive Publication Date: 2024-11-15UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202411440220.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing posture recognition technology cannot be used indoors or without a camera, and it has poor adaptability to complex scenes and lack of texture information, high computing resource requirements, and overfitting problems.

Method used

Using a neural network-based attitude recognition method, data is collected through acceleration sensors and gyroscopes, attitude database is established, and attitude recognition is performed using the time recursive neural network modified by the long and short-term memory network (LSTM) and the Dense network to calculate the rolling and pitch of the human body. and yaw angle.

Benefits of technology

It realizes high-precision human posture recognition in indoor environments, enhances privacy protection, reduces the rate of error recognition, improves recognition performance and efficiency, and adapts to rapid dynamic changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968632A_ABST
    Figure CN118968632A_ABST
Patent Text Reader

Abstract

The invention provides a posture recognition method based on a neural network. According to the method, a data-based posture recognition technology is adopted, data of an acceleration sensor and a gyroscope sensor during user operation are collected, a posture database is established, a human body posture rule is learned from the posture database by utilizing a neural network, and therefore the human body posture is estimated according to the data of the current user sensor. The acceleration sensor measures the acceleration of an object on three axes, the gyroscope sensor measures the angular velocity of the object on the three axes, and the attitude angle of the object is calculated through the measured acceleration and angular velocity, so that the attitude of the object is judged. The adopted neural network is a new time recurrent neural network modified according to a long short-term memory (LSTM) network and a Dense network. Through the scheme of the invention, the occurrence of overfitting is reduced, and the performance of the classifier is effectively improved, so that the attitude recognition efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a posture recognition method based on a neural network. Background Art

[0002] Most of the existing posture recognition algorithms belong to posture capture system algorithms, which use multiple devices such as multiple cameras or sensors to capture and recognize human postures, and combine 3D modeling technology to reconstruct the human body model and perform posture analysis. Its advantage is that it can perform more accurate posture reconstruction and analysis, but it requires more equipment and space, and the cost is high. In addition, this type of algorithm is suitable for outdoor scenes with large range of human activities, and is mostly used in intelligent monitoring, film and television production, and athlete training. For indoor situations or when the device does not carry a camera, there will be a problem that this method cannot be used or that using this method will cause a waste of resources.

[0003] The existing gesture recognition technologies are mainly the following:

[0004] (1) Posture estimation based on motion geometry

[0005] This method usually relies on image changes and motion information between consecutive frames to estimate posture. It uses optical flow, feature matching, or stereo vision to infer the joint position and motion trajectory of the target object (such as the human body) from a 2D or 3D image sequence. For example, in a multi-view system, the relative motion of the same object captured by cameras at different angles is analyzed by the principle of triangulation to calculate the posture. However, the problem is that it has poor adaptability to complex scenes such as occlusion, rapid motion, and lighting changes, and it is difficult to extract stable feature points in the absence of texture information.

[0006] (2) Posture Estimation Based on Deep Learning

[0007] Using deep neural networks for posture recognition has become a mainstream method. These networks can directly predict the coordinates of key points from image pixels, such as OpenPose, CPN (Cascaded Pyramid Network) and other models. After training on large-scale annotated datasets, they can effectively extract and learn human features and map these features to the locations of joints. In recent years, with the development of deep learning technology, many studies have also explored 3D posture estimation, processing monocular images in an end-to-end manner to obtain 3D posture information. The advantage is that it can handle complex visual environments and has high accuracy; but the disadvantage is that this method requires a large amount of annotated training data, has high computing resource requirements, and may have overfitting problems.

[0008] (3) Model-based approach

[0009] In the field of computer vision, "model-based methods" usually refer to pre-establishing a physical or mathematical model to simulate the structure and motion of the target object. In posture estimation, it may refer to using a known human skeleton model and combining it with an optimization algorithm (such as iterative closest point method ICP, particle filter, etc.) to fit the model to the observed data. This method requires searching and optimizing the model parameters to minimize the difference between the model and the actual observation. Its advantage is a deep understanding of the dynamic characteristics of the object, which is suitable for precise control tasks; but the difficulty lies in the accuracy of model establishment and the uncertainty of model errors in the actual environment.

[0010] (4) Data-based methods.

[0011] Data-based methods mainly refer to those techniques that rely heavily on large amounts of labeled data for machine learning. This covers a variety of statistical learning and machine learning algorithms, but with the rise of deep learning, most modern data-based gesture recognition methods have turned to deep neural networks. These methods first collect a dataset containing various gestures, and then train the model to learn the relationship between gesture changes and image features. In some cases, the term "data-based" may emphasize big data-driven and data-driven learning processes rather than just a specific type of learning algorithm. The advantage of this method is that it is data-driven and has a certain generalization ability; but there are also problems such as limited performance when there is insufficient training data and weak robustness to noise in complex dynamic environments. Summary of the invention

[0012] In view of the problem that existing gesture recognition methods cannot be used indoors or on devices without cameras, the present invention proposes a gesture recognition method based on a neural network, which can recognize the user's current gesture through sensor data.

[0013] The invention proposes a posture recognition method based on a neural network, which adopts a posture recognition technology based on data, collects data of an acceleration sensor and a gyroscope sensor when a user operates in different postures, and numbers the postures, establishes a posture database according to the sensor data and the posture numbers corresponding thereto, and uses a neural network to learn the law of human postures therefrom, obtains a trained neural network model, and thus estimates the human posture according to the data of the current user sensor; the acceleration sensor measures the acceleration of an object on three axes of x, y and z, and the gyroscope sensor measures the angular velocity of an object on three axes of x, y and z, and the posture angle of the human body is calculated by the measured acceleration and angular velocity, so as to judge the posture of the human body, and the posture angle includes a roll angle, a pitch angle and a yaw angle, and the adopted neural network is a new time recursive neural network modified according to a long short-term memory network LSTM and a Dense network, and in the new time recursive neural network, the output of the LSTM network is used as the input of the Dense network.

[0014] Furthermore, the LSTM network includes an input gate, a forget gate and an output gate; the input gate includes a memory unit, and the input gate determines which new input information will be stored in the memory unit at the current time step; the forget gate determines which information is discarded from the memory unit at the previous time step, and the candidate memory unit creates a new candidate memory content, which will be updated according to the information determined by the input gate; the output gate controls how much information of the memory unit will be passed to the hidden state of the next time step, and ultimately used to calculate the output of the model; the memory unit information update combines the operations of the forget gate and the input gate to achieve selective retention and discarding of information; the hidden state is the output of the LSTM at the current time step, which is the mapping of the memory unit after the output gate, is used for the calculation of the next time step, and is also one of the final outputs of the entire network; the Dense network includes multiple dense blocks, transition layers, and one or more fully connected layers.

[0015] Furthermore, the working principle of the new time recurrent neural network modified based on the long short-term memory network LSTM and the Dense network is:

[0016] Set the input data X t represents the input vector of the current time step; the hidden state h of the previous time step t-1 Represents the historical information carried to the current time step; the memory unit information C of the previous time step t-1 ,The memory unit stores the information from the beginning of the sequence to the current time step, which is saved after being processed by the gate;

[0017] Step S1. Determine the information that needs to be discarded in the memory unit, and process it through a sigmoid unit called a forget gate to determine which information in the memory unit of the previous time step needs to be discarded by checking X t and h t-1 Information output is a vector with element values ​​between 0 and 1. The element values ​​in the vector represent the memory unit information C t-1 Which information in the data should be retained or how much should be discarded, where 0 means no information should be retained and 1 means all information should be retained.

[0018] Step S2. Determine the new information to be added to the memory unit, decide which new information should be stored in the memory unit at the current time step, and use X t and h t-1 Through an operation called the input gate, we decide which information to update, and then use X t and h t-1 Get new candidate memory unit information through a layer with tanh activation function , using the new candidate memory unit information Update old memory unit information Ct-1 , becomes the new memory unit information C t The update rule is to discard part of the old memory unit information through the forget gate and add candidate memory unit information through the input gate. part of the memory cell information C t ;

[0019] Step S3. After updating the memory unit, according to the input X t and h t-1 Determine the state characteristics of the output memory unit based on the output of the forget gate and the input gate. Pass the output gate result through a sigmoid layer called the output gate as the judgment condition, and pass the memory unit information through the tanh layer to obtain a vector with element values ​​between -1 and 1. Multiply the vector with the judgment condition obtained by the output gate to get the output of the LSTM network. The forget gate determines which old information to discard, and the input gate determines which new information to add.

[0020] Step S4. Use the output of the LSTM network as the input of the Dense network;

[0021] Step S5. The input data first enters a dense block. In each dense block, there are multiple convolutional layers. The output of each convolutional layer is not only passed to the next layer, but also to all subsequent layers. Therefore, each layer receives the output of all previous layers as input;

[0022] Step S6. In the dense network, each convolution operation is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer is used to standardize the input, and the ReLU activation function is used to increase the nonlinearity of the network.

[0023] Step S7. There is a transition layer between dense blocks, the transition layer includes a batch normalization layer, a ReLU activation function, a 1x1 convolution layer and a 2x2 pooling layer, the 1x1 convolution layer is used to reduce the number of feature maps, and the 2x2 pooling layer is used for downsampling;

[0024] Step S8. The end of the Dense network is one or more fully connected layers, which are used to generate the final output of the network.

[0025] Furthermore, the processing method of the sigmoid unit in step S1 is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function. The value in the brackets is calculated and inserted into it to obtain a value between 0 and 1.

[0026] Furthermore, using Xt and h t-1 Get new candidate memory unit information through a layer with tanh activation function The calculation method is: ,in, is the weight of the current memory unit, is the bias of the current memory cell;

[0027] Furthermore, the input information i of the input gate t The calculation method is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function; new memory unit information C t for: .

[0028] Furthermore, the output of the LSTM network is: ,in , W o is the weight of the output gate, b o is the bias of the output gate.

[0029] The beneficial technical effects of the present invention are:

[0030] (1) Enhanced privacy protection: Since the image or video data is no longer collected by visual sensors (such as cameras), the problem of possible user privacy leakage is avoided. Especially in situations where personal privacy needs to be strictly protected, the present invention provides a more secure and reliable human posture recognition solution;

[0031] (2) Improving recognition accuracy: Through algorithmic processing of sensor data, the present invention can achieve high-precision posture recognition and accurately capture and track subtle changes in human joints even in fast and dynamic movement changes, thereby reducing the misrecognition rate.

[0032] (3) Improving network performance: The Dense layer is located after the LSTM and is used to further integrate the high-level features extracted by the LSTM and make the final decision or classification. The Dense layer allows each neuron in the network to be connected to all neurons in the previous layer, thereby performing pattern recognition in the entire feature space, improving recognition performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a model diagram of the LSTM network provided by an embodiment of the present invention;

[0035] Figure 2 It is a model diagram of a Dense network provided by an embodiment of the present invention;

[0036] Figure 3 is a structural diagram of a DenseBlock provided by an embodiment of the present invention;

[0037] Figure 4 It is a structural diagram of a new time recursive neural network modified according to the LSTM+Dense network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The present invention mainly uses a data-based method. Traditional data-based methods usually use images or video data captured by visual methods (such as cameras) for processing. The present invention improves on this basis and changes the data method used. It no longer uses image or video data, but uses acceleration sensors and gyroscopes for gesture recognition. It does not rely on camera image information, and can better protect user privacy in sensitive occasions or applications with high privacy requirements. In addition, the present invention can respond more quickly to fast and dynamic action changes, especially when high-frequency updates of gesture information are required in a short time. In addition, the present invention further introduces an improved time recursive neural network architecture when processing time series data collected by sensors. Different from traditional methods, we designed a deep learning model that integrates LSTM and Dense layers. The model is good at capturing long-term dependencies and instantaneous dynamic changes in sensor data. Through the memory mechanism of the LSTM network, the system can retain historical information in complex and rapidly changing gesture recognition scenarios and make accurate predictions at subsequent moments, thereby improving the system's real-time response capability and recognition accuracy of various dynamic actions.

[0040] The present invention proposes a posture recognition method based on a neural network. When a user uses an application on a device that requires an acceleration sensor and a gyroscope sensor (such as a sports app on a mobile phone, a wearable device, or other electronic device), the generated acceleration sensor and gyroscope sensor data are collected, a posture database is established, and the neural network is used to learn the rules of human posture, thereby estimating the human posture. Specifically, the method includes:

[0041] The data-based posture recognition technology is adopted to collect the data of the acceleration sensor and the gyroscope sensor when the user operates in different postures, and the postures are numbered. A posture database is established according to the sensor data and the posture numbers corresponding to them. A neural network is used to learn the rules of human postures from it to obtain a trained neural network model, so as to estimate the human posture according to the data of the current user sensor; the acceleration sensor measures the acceleration of the object on the three axes of x, y and z, and the gyroscope sensor measures the angular velocity of the object on the three axes of x, y and z. The posture angle of the human body is calculated by the measured acceleration and angular velocity, so as to judge the posture of the human body. The posture angle includes roll angle, pitch angle and yaw angle. The neural network adopted is a new time recursive neural network modified according to the long short-term memory network LSTM and the Dense network. In the new time recursive neural network, the output of the LSTM network is used as the input of the Dense network.

[0042] The existing neural network models for processing time series data mainly include the Transformer model and the long short-term memory network LSTM. The present invention selects the LSTM network to process the time series data obtained by the sensor, and adopts a new time recursive neural network modified according to the LSTM+Dense network for posture recognition.

[0043] The LSTM network consists of multiple LSTM units, each of which is responsible for capturing and transmitting long-term dependency information in time series data processing. LSTM is a special recurrent neural network (RNN) structure designed specifically to solve long-term dependency problems. Figure 1As shown in the figure, in LSTM, there are mainly the following core layers or components: input gate, including memory cells, determines which new input information will be stored in the memory cells at the current time step; forget gate, determines which information is discarded from the memory cells at the previous time step; candidate memory cells, create a new candidate memory content, which will be updated according to the information determined by the input gate; output gate, controls how much information of the memory cell will be passed to the hidden state of the next time step, and is ultimately used to calculate the output of the model; memory cell information update, combines the operations of the forget gate and the input gate to achieve selective retention and discarding of information; hidden state, the hidden state is the output of LSTM at the current time step, which is the mapping of the memory cell after the output gate, is used for the calculation of the next time step, and is also one of the final outputs of the entire network. The working process of LSTM is as follows: (1) Calculate the forget gate and select the information to be forgotten; (2) Calculate the memory gate and select the information to be remembered; (3) The memory unit updates its own state through the input data and the memory unit state at the previous moment; (4) The input data, the output data of the hidden state at the previous moment, and the memory unit state at the current moment are jointly applied to the output gate to output the hidden layer information at the current moment.

[0044] Dense refers to the fully connected layer in deep learning, such as Figure 2 As shown in the figure, the basic structural units of the Dense network (densely connected convolutional network) include: (1) Initialization layer: At the beginning, the network usually contains a large-scale convolution layer (such as 7x7 convolution with a step size of 2) to reduce the dimension of the input image, followed by a maximum pooling layer or an average pooling layer for further downsampling the spatial size; (2) Dense Block: This is the core part of the densely connected convolutional network, such as Figure 3As shown in the figure, in a dense block, each layer receives the feature maps of all previous layers as input and merges them with its own output (usually through a concatenation operation). This means that the lth layer receives not only the output of the l-1th layer, but also the outputs of all layers from the input layer to the l-2th, l-3th, ..., and up to the first layer. This design realizes the propagation and reuse of features. This connection method is called a dense connection. Inside a dense block, each layer usually adopts a bottleneck layer structure, which consists of a series of operations, including batch normalization. Normalization (BN) layer, activation function (such as ReLU) and 1x1 convolution and a 3x3 convolution structure. 1x1 convolution is used to reduce the number of channels and computational complexity. At the end of the dense block, the feature maps of all layers are concatenated together to form the input of the next dense block. (3) Transition layer: Connecting two adjacent dense blocks, the transition layer includes a batch normalization layer (BN layer), a ReLU activation function, a 1x1 convolution layer and a 2x2 pooling layer. The main task of the transition layer is to reduce the spatial dimension of the feature map (through 1x1 convolution and 2x2 pooling layer) and reduce the number of channels (through 1x1 convolution and compression factor θ). This is done to control the complexity and scale of the model while maintaining the effective flow of information. (4) The last layer: The end of the network usually contains a global pooling layer and a fully connected layer to generate the final classification result. For large-scale image classification tasks such as ImageNet, the last layer often passes the globally pooled feature vector into a fully connected layer to generate a category probability distribution.

[0045] A densely connected convolutional network is a neural network architecture in which all nodes are fully connected between adjacent layers. It is widely used to deal with various machine learning problems such as classification and regression, and is particularly powerful when processing non-image and non-sequential structured data.

[0046] The sensor data collected by the present invention is time series data, and each segment of data is relatively long. Using the LSTM model may cause overfitting problems. The present invention adopts the method of adding a Dense network to the LSTM model. The Dense layer can directly perform corresponding regularization to reduce the occurrence of overfitting. At the same time, the Dense layer can increase the nonlinear expression ability of the model by introducing a nonlinear activation function (such as ReLU), so that the model can better adapt to complex time series data. Adding a Dense network can also help the LSTM model to better handle multi-classification problems. In the LSTM model, Softmax is usually used as the activation function of the last layer for classification, but this method is only applicable to the case where the number of output categories is small. The present invention sends the output of the LSTM model to a Dense layer, and then uses activation functions such as Softmax for classification, which can effectively improve the performance of the classifier.

[0047] The structure of the new time recursive neural network modified according to the LSTM+Dense network proposed in the present invention is as follows: Figure 4 As shown, its working principle is:

[0048] Set the input data X t represents the input vector of the current time step; the hidden state h of the previous time step t-1 Represents the historical information carried to the current time step; the memory unit information C of the previous time step t-1 ,The memory unit stores the information from the beginning of the sequence to the current time step, which is saved after being processed by the gate;

[0049] Step S1. Determine the information that needs to be discarded in the memory unit, and process it through a sigmoid unit called a forget gate to determine which information in the memory unit of the previous time step needs to be discarded by checking X t and h t-1 Information output is a vector with element values ​​between 0 and 1. The element values ​​in the vector represent the memory unit information C t-1 Which information in the data should be retained or how much should be discarded, where 0 means no information should be retained and 1 means all information should be retained.

[0050] The processing method of the sigmoid unit in the forget gate is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function, and the value in the brackets is calculated and inserted into it to obtain a value between 0 and 1;

[0051] Step S2. Determine the new information to be added to the memory unit, decide which new information should be stored in the memory unit at the current time step, and use X t and h t-1 Through an operation called the input gate, we decide which information to update, and then use X t and h t-1 Get new candidate memory unit information through a layer with tanh activation function , using the new candidate memory unit information Update old memory unit information C t-1 , becomes the new memory unit information C t The update rule is to discard part of the old memory unit information through the forget gate and add candidate memory unit information through the input gate. part of the memory cell information C t ;

[0052] Input information i of the input gate tThe calculation method is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function;

[0053] Utilizing X t and h t-1 Get new candidate memory unit information through a layer with tanh activation function The calculation method is: ,in, is the weight of the current memory unit, is the bias of the current memory cell;

[0054] New memory unit information C t for: ;

[0055] Step S3. After updating the memory unit, according to the input X t and h t-1 Determine the state characteristics of the output memory unit based on the output of the forget gate and the input gate. Pass the output gate result through a sigmoid layer called the output gate as the judgment condition, and pass the memory unit information through the tanh layer to obtain a vector with element values ​​between -1 and 1. Multiply the vector with the judgment condition obtained by the output gate to get the output of the LSTM network. The forget gate determines which old information to discard, and the input gate determines which new information to add.

[0056] Specifically, the output of the LSTM network is: ,in , W o is the weight of the output gate, b o is the bias of the output gate;

[0057] Step S4. Use the output of the LSTM network as the input of the Dense network;

[0058] Step S5. The input data first enters a dense block. In each dense block, there are multiple convolutional layers. The output of each convolutional layer is not only passed to the next layer, but also to all subsequent layers. Therefore, each layer receives the output of all previous layers as input;

[0059] Step S6. In the dense network, each convolution operation is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer is used to standardize the input, and the ReLU activation function is used to increase the nonlinearity of the network.

[0060] Batch normalization is a regularization technique used in deep learning network training, which helps improve the convergence of the training process. Its main purpose is to make the model training faster and more stable by normalizing the distribution of eigenvalues ​​output by each layer of the neural network, and to reduce overfitting. Specifically, batch normalization performs the following two core steps on the small batch data of each training:

[0061] Normalization: Calculate the mean and variance of a small batch of samples on the same channel, and then use these statistics to adjust the input features of the channel to zero mean and unit variance, so that each batch of data has similar distribution characteristics;

[0062] Learnable scaling and translation parameters: In order to preserve the learning ability of the network, batch normalization adds a learnable bias term (β) and translation term (γ) to the normalized result. These two parameters are automatically adjusted during training to optimize model performance.

[0063] Step S7. There is a transition layer between dense blocks, which includes a batch normalization layer, a ReLU activation function, a 1x1 convolution layer and a 2x2 pooling layer. The 1x1 convolution layer is used to reduce the number of feature maps, and the 2x2 pooling layer is used for downsampling.

[0064] Step S8. The network ends with one or more fully connected layers to generate the final output of the network.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A gesture recognition method based on a neural network, characterized in that: The method adopts data-based posture recognition technology to collect data from acceleration sensors and gyroscope sensors when users operate in different postures, and numbers the postures. A posture database is established based on the sensor data and the posture numbers corresponding to them, and a neural network is used to learn the rules of human postures from the database to obtain a trained neural network model, thereby estimating the human posture based on the data from the current user's sensors. The acceleration sensor measures the acceleration of the object on the x, y, and z axes, and the gyroscope sensor measures the angular velocity of the object on the x, y, and z axes. The posture angle of the human body is calculated by measuring the acceleration and angular velocity, thereby judging the posture of the human body. The posture angle includes the roll angle, the pitch angle, and the yaw angle. The neural network used is a new time recursive neural network modified according to the long short-term memory network LSTM and the Dense network. In the new time recursive neural network, the output of the LSTM network is used as the input of the Dense network.

2. The method according to claim 1, characterized in that The LSTM network includes an input gate, a forget gate and an output gate; the input gate includes a memory unit, and the input gate determines which new input information will be stored in the memory unit at the current time step; the forget gate determines which information is discarded from the memory unit at the previous time step, and the candidate memory unit creates a new candidate memory content, which will be updated according to the information determined by the input gate; the output gate controls how much information of the memory unit will be passed to the hidden state of the next time step, and is ultimately used to calculate the output of the model; the memory unit information update combines the operations of the forget gate and the input gate to achieve selective retention and discarding of information; the hidden state is the output of the LSTM at the current time step, which is the mapping of the memory unit after the output gate, is used for the calculation of the next time step, and is also one of the final outputs of the entire network; the Dense network includes multiple dense blocks, a transition layer, and one or more fully connected layers.

3. The method according to claim 2, characterized in that The working principle of the new time recurrent neural network modified based on the long short-term memory network LSTM and Dense network is: Set the input data X t represents the input vector of the current time step; the hidden state h of the previous time step t-1 Represents the historical information carried to the current time step; the memory unit information C of the previous time step t-1 ,The memory unit stores the information from the beginning of the sequence to the current time step, which is saved after being processed by the gate; Step S1. Determine the information that needs to be discarded in the memory unit, and process it through a sigmoid unit called a forget gate to determine which information in the memory unit of the previous time step needs to be discarded by checking X t and h t-1 Information output is a vector with element values ​​between 0 and 1. The element values ​​in the vector represent the memory unit information C t-1 Which information in the data should be retained or how much should be discarded, where 0 means no information should be retained and 1 means all information should be retained. Step S2. Determine the new information to be added to the memory unit, decide which new information should be stored in the memory unit at the current time step, and use X t and h t-1 Through an operation called the input gate, we decide which information to update, and then use X t and h t-1 Get new candidate memory unit information through a layer with tanh activation function , using the new candidate memory unit information Update old memory unit information C t-1 , becomes the new memory unit information C t The update rule is to discard part of the old memory unit information through the forget gate and add candidate memory unit information through the input gate. part of the memory cell information C t ; Step S3. After updating the memory unit, according to the input X t and h t-1 Determine the state characteristics of the output memory unit based on the output of the forget gate and the input gate. Pass the output gate result through a sigmoid layer called the output gate as the judgment condition, and pass the memory unit information through the tanh layer to obtain a vector with element values ​​between -1 and 1. Multiply the vector with the judgment condition obtained by the output gate to get the output of the LSTM network. The forget gate determines which old information to discard, and the input gate determines which new information to add. Step S4. Use the output of the LSTM network as the input of the Dense network; Step S5. The input data first enters a dense block. In each dense block, there are multiple convolutional layers. The output of each convolutional layer is not only passed to the next layer, but also to all subsequent layers. Therefore, each layer receives the output of all previous layers as input; Step S6. In the dense network, each convolution operation is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer is used to standardize the input, and the ReLU activation function is used to increase the nonlinearity of the network. Step S7. There is a transition layer between dense blocks, which includes a batch normalization layer, a ReLU activation function, a 1x1 convolution layer and a 2x2 pooling layer. The 1x1 convolution layer is used to reduce the number of feature maps, and the 2x2 pooling layer is used for downsampling. Step S8. The network ends with one or more fully connected layers to generate the final output of the network.

4. The method according to claim 3, characterized in that The processing method of the sigmoid unit in step S1 is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function. The value in the brackets is calculated and inserted into it to obtain a value between 0 and 1.

5. The method according to claim 4, characterized in that Utilizing X t and h t-1 Get new candidate memory unit information through a layer with tanh activation function The calculation method is: ,in, is the weight of the current memory unit, is the bias of the current memory cell.

6. The method according to claim 5, characterized in that Input information i of the input gate t The calculation method is: ,in, is the weight of the current input, is the current input bias, Represents the sigmoid activation function; new memory unit information C t for: .

7. The method according to claim 6, characterized in that The output of the LSTM network is: ,in , W o is the weight of the output gate, b o is the bias of the output gate.

Citation Information

Patent Citations

  • Children motion posture automatic identification technology based on 3D convolution long-term and short-term memory network

    CN111079656A

  • Shield attitude dynamic prediction method, system and device

    CN112879024A

  • Human body posture recognition model building method and human body posture recognition method

    CN114897025A

  • Attitude estimation method, device and system

    CN116045971A

  • Future pose predictor for a controller

    US20240169677A1

Cited By

  • Humanoid robot welding pose adjusting method and system

    CN120326636A

  • Humanoid robot welding pose adjustment method and system

    CN120326636B