Posture recognition method and device, computer equipment and storage medium

Abstract

The invention relates to a posture recognition method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring an image set, and inputting the image set into a posture model; wherein the to-be-processed images in the image set are sorted according to a time sequence, and the attitude model comprises a long short-term memory (LSTM) network; determining a first feature of the difference region; wherein the difference region comprises changed joints of a pedestrian in two adjacent frames of images to be processed, and the first feature corresponds to time; inputting the first feature into the LSTM to obtain a second feature; and determining the attitude information of the pedestrian based on the first feature and the second feature. By adopting the method, the accuracy of the recognized pedestrian posture can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a gesture recognition method, device, computer equipment and storage medium. Background Art

[0002] In the fields of video surveillance analysis and intelligent driving, compared with manual image recognition, the method of using deep learning models to identify human actions and output corresponding action labels (for example, waving, bending, etc.) has been promoted due to its accuracy and efficiency. Taking intelligent driving as an example, the human action recognition method based on deep learning models helps intelligent driving vehicles to timely identify the action intentions of pedestrians, avoid them in time, and ensure the safety of people and vehicles.

[0003] Currently, it is common to process RGB images (Red Green Blue images) through ConvNets (Convolutional Neural Networks) to identify the actions of pedestrians in the image. However, ConvNets are easily affected by the background environment (e.g., lighting, noise) in the image, which leads to the problem of insufficient accuracy in the current method of identifying pedestrian actions. Summary of the invention

[0004] Based on this, it is necessary to provide a posture recognition method, device, computer equipment and storage medium to address the above technical problems, so as to improve the accuracy of the recognized pedestrian posture.

[0005] In a first aspect, an embodiment of the present application provides a gesture recognition method, comprising:

[0006] Acquire an image set, and input the image set into a posture model; wherein the images to be processed in the image set are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM;

[0007] Determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the image to be processed, and the first feature corresponds to time;

[0008] Inputting the first feature into the LSTM to obtain a second feature;

[0009] Based on the first feature and the second feature, posture information of the pedestrian is determined.

[0010] In one embodiment, determining the posture information of the pedestrian based on the first feature and the second feature includes:

[0011] Combining the first feature and the second feature to obtain a third feature;

[0012] Processing the third feature based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time;

[0013] The first feature is enhanced by using the attention coefficient to obtain the posture information of the pedestrian.

[0014] In one embodiment, the processing the third feature based on a preset attention mechanism to obtain an attention coefficient includes:

[0015] The third feature is input into the first fully connected layer and the first activation function to obtain the attention coefficient.

[0016] In one embodiment, the step of enhancing the first feature by using the attention coefficient to obtain the posture information of the pedestrian includes:

[0017] Obtaining a fourth feature by multiplying the attention coefficient by the first feature;

[0018] The fourth feature is input into the second fully connected layer and the second activation function to obtain the posture information.

[0019] In one embodiment, obtaining the fourth feature by multiplying the attention coefficient by the first feature comprises:

[0020] Performing a product operation on the attention coefficient at a first time and the first feature at a second time to obtain an intermediate feature, wherein the first time is after the second time;

[0021] A sum operation is performed on the intermediate feature and the first feature at the first time to obtain the fourth feature.

[0022] In one of the embodiments, the posture model includes a graph convolutional neural network GCN;

[0023] Then the first feature of determining the difference area includes:

[0024] Determine a joint graph of a pedestrian in the image to be processed;

[0025] Inputting the joint map into the GCN to obtain the fifth feature of the pedestrian; wherein the fifth feature indicates the joint information of the pedestrian in the image to be processed;

[0026] The fifth feature of the pedestrian in two adjacent frames of the image to be processed is subjected to difference processing to obtain the first feature.

[0027] In one embodiment, determining a joint graph of a pedestrian in the image to be processed includes:

[0028] Determine node information of joints of the pedestrian in the image to be processed;

[0029] The joint graph corresponding to the pedestrian in the image to be processed is determined according to the connection relationship between the joints and the node information.

[0030] In a second aspect, an embodiment of the present application provides a gesture recognition device, including:

[0031] A collection module, used for acquiring an image collection and inputting the image collection into a posture model; wherein the images to be processed in the image collection are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM;

[0032] A difference module, used to determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the to-be-processed image, and the first feature corresponds to time;

[0033] A feature model, used for inputting the first feature into the LSTM to obtain a second feature;

[0034] A posture module is used to determine the posture information of the pedestrian based on the first feature and the second feature.

[0035] In one embodiment, the posture module is specifically used to fuse the first feature and the second feature to obtain a third feature; process the third feature based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time; and enhance the first feature by using the attention coefficient to obtain the posture information of the pedestrian.

[0036] In one of the embodiments, the posture module is specifically used to input the third feature into a first fully connected layer and a first activation function to obtain the attention coefficient.

[0037] In one of the embodiments, the posture module is specifically used to obtain a fourth feature by multiplying the attention coefficient and the first feature; and input the fourth feature into a second fully connected layer and a second activation function to obtain the posture information.

[0038] In one embodiment, the posture module is also used to perform a product operation on the attention coefficient at a first time and the first feature at a second time to obtain an intermediate feature; the first time is located after the second time; and the intermediate feature and the first feature at the first time are summed to obtain the fourth feature.

[0039] In one embodiment, the posture model includes a graph convolutional neural network (GCN); the difference module is specifically used to determine a joint graph of the pedestrian in the image to be processed; the joint graph is input into the GCN to obtain the fifth feature of the pedestrian; wherein the fifth feature indicates the joint information of the pedestrian in the image to be processed; and difference processing is performed on the fifth features of the pedestrian in two adjacent frames of the image to be processed to obtain the first feature.

[0040] In one of the embodiments, the difference module is further used to determine the node information of the joints of the pedestrian in the image to be processed; and determine the joint graph corresponding to the pedestrian in the image to be processed based on the connection relationship between the joints and the node information.

[0041] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect and any possible implementation manner when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect and any possible implementation manner are implemented.

[0043] In the above-mentioned posture recognition method, device, computer equipment and storage medium, by inputting the first feature of the difference area into LSTM, the posture change information of the pedestrian at different times is extracted to obtain the time series feature (i.e., the second feature), and then the posture information of the pedestrian is determined by the time series feature and the first feature. In this posture recognition method, by determining the first feature of the difference area, the feature information of the joints of the pedestrian that have changed is processed, the dependence on the external environment and background during posture recognition is reduced, and the calculation focusing on the change feature (i.e., the first feature) is realized, so that the accuracy of the posture information obtained by recognition is improved. On this basis, due to the reduction of feature information, the complexity of feature calculation is also reduced, so the influence of noise in the feature information on posture recognition can be avoided, so that the accuracy of posture information can be further improved, and the efficiency of posture recognition is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A diagram of an application environment of a gesture recognition method in an embodiment;

[0046] Figure 2 is a flow chart of a gesture recognition method in one embodiment;

[0047] Figure 3 A schematic diagram of a method for determining an attention coefficient step in one embodiment;

[0048] Figure 4 A schematic flow chart of a method for determining a first feature in one embodiment;

[0049] Figure 5 A schematic diagram of a method for determining a difference area and a fifth feature in one embodiment;

[0050] Figure 6 is a structural block diagram of a gesture recognition device in one embodiment;

[0051] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0053] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0054] The gesture recognition method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via the network.

[0055] In one embodiment, the terminal 102 can be used to collect multiple frames (sheets) of images to be processed containing pedestrians, and transmit the collected multiple images to be processed in the form of an image set to the server 104 through the network. After receiving the image set to be processed, the server 104 inputs the images to be processed into a posture model to identify the posture information of the pedestrians in the images to be processed. The posture information can be, for example, bending, raising hands, waving, raising legs or bending knees.

[0056] In one embodiment, the terminal 102 may be used to collect multiple frames (sheets) of images to be processed containing pedestrians to obtain an image set, and process the image set using a posture model preset on the terminal 102 to obtain posture information. At the same time, the posture model on the terminal 102 is maintained and updated via the network by the server 104. The posture information may be, for example, bending over, raising hands, waving hands, raising legs, or bending knees.

[0057] The terminal 102 may be, but is not limited to, various video cameras, cameras, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0058] In one embodiment, Figure 2 As shown, a gesture recognition method is provided, the method comprising the following steps:

[0059] Step 201, obtain an image set, and input the image set into a posture model.

[0060] The images to be processed in the image set are sorted according to time sequence, and the posture model includes LSTM (Long Short-Term Memory).

[0061] Specifically, the above-mentioned image set may be one or more videos.

[0062] Alternatively, the image set may also be composed of a plurality of images to be processed with equal time intervals.

[0063] Alternatively, the image set may also be composed of multiple image subsets, the time intervals of the images to be processed in each image subset are equal, and the time intervals of the images to be processed in different image subsets may be the same or different.

[0064] Step 202: Determine the first feature of the difference area.

[0065] The difference area includes a portion of the pedestrian where changes occur in two adjacent frames of images to be processed, and the first feature corresponds to time.

[0066] Specifically, two adjacent frames of images to be processed are the first image and the second image in sequence, and when one or more parts of the pedestrian in the first image and the second image change, the area where the joints that change are located is the first area. Therefore, the difference area is the local area where the pedestrian in the first image and the second image are located.

[0067] The first feature includes feature information of all difference regions between the first image and the second image.

[0068] Since the image to be processed corresponds to time, the difference area and the first feature both correspond to time.

[0069] In a possible implementation, the position information of the joints of the pedestrian in each frame of the image to be processed can be determined respectively, and then the position information of the corresponding joints of the pedestrian in two adjacent frames of the image to be processed can be compared. When the difference in the coordinates corresponding to the position information is greater than the position threshold, it is determined that the corresponding joint has changed. Then, a detection frame of a preset size is used to determine the difference area containing the joint, and the first feature of the difference area is determined.

[0070] In a possible implementation, difference processing can be performed on the features of the area where the pedestrians are located in two adjacent frames of the image to be processed to obtain a difference feature. When the feature value in the difference feature is greater than a first threshold, it can be determined that the area where the feature value greater than the first threshold is located is the difference area, and it can be determined that the feature value greater than the first threshold constitutes the first feature.

[0071] Step 203: input the first feature into LSTM to obtain the second feature.

[0072] Specifically, since the first feature corresponds to time, the first feature can be input into the LSTM in time order, and the time series feature of the first feature is obtained as the second feature.

[0073] Through LSTM, the first feature of the past time can be integrated into the hidden layer feature of the current time, so as to capture the dynamic changes of the first feature of the difference area, and the output hidden layer feature h i That is the time series feature of the first feature mentioned above, then the second feature h is obtained i , i=1,2,…,t,

[0074] Step 204: Determine the posture information of the pedestrian based on the first feature and the second feature.

[0075] Specifically, based on the feature information of the difference area corresponding to the first feature and the second feature (ie, the time series feature of the first feature), the posture information of the pedestrian can be determined.

[0076] In the above-mentioned posture recognition method, in the posture model, the first feature of the difference area is input into LSTM to extract the temporal features of the difference area where the change occurs, so that the posture model can focus on processing the feature information of the difference area, effectively improving the accuracy of posture recognition. And when determining the posture information, the processed features are the first feature and the temporal features of the first feature: the second feature, so the feature information processed by the posture model is also greatly reduced, and thus its recognition efficiency is also effectively improved.

[0077] Further, the following provides examples 1-2 to illustrate the determination of the posture information of the pedestrian: Example 1

[0078] First, the second feature is processed using a preset attention mechanism to obtain an attention coefficient, which corresponds to time.

[0079] Then, the first feature is enhanced by using the attention coefficient to obtain the fourth feature.

[0080] Finally, the posture information of the pedestrian is determined based on the fourth feature. Specifically, the fourth feature can be processed by the second fully connected layer and the second activation function to obtain the probability of each possible label, and the possible label corresponding to the label with the highest probability is determined as the posture information of the pedestrian.

[0081] Example 2

[0082] First, the first feature and the second feature are combined to obtain the third feature. Specifically, the elements in the first feature and the elements in the second feature can be added together to obtain the third feature. Alternatively, the first feature and the second feature can be concatenated to obtain the third feature.

[0083] Then, the third feature is processed based on a preset attention mechanism to obtain an attention coefficient, which corresponds to time.

[0084] Finally, the first feature is enhanced by the attention coefficient to obtain the posture information of the pedestrian. Since the attention coefficient and the first feature each correspond to time, the corresponding relationship between the attention coefficient and the first feature can be determined according to time, and the feature enhancement is performed accordingly. The following is a detailed description:

[0085] The attention coefficient and the first feature can be multiplied to obtain the fourth feature.

[0086] Then the fourth feature is input into the second fully connected layer and the second activation function to obtain the posture information. That is, the fourth feature is screened by the second fully connected layer, and the output of the second fully connected layer is used as the input of the second activation function. The action classification is performed in the second activation function to obtain the probability of each possible label, and the possible label corresponding to the label with the highest probability is determined as the posture information of the pedestrian. The possible labels include but are not limited to bending, raising hands, waving, raising legs, and bending knees.

[0087] In a possible implementation, in order to obtain the fourth feature, the above-mentioned attention coefficient corresponding to the same time may be multiplied with the first feature to obtain an enhanced feature: the fourth feature.

[0088] Preferably, in order to avoid the problem of partial information loss due to feature enhancement, in a possible implementation, the attention coefficient at the first time and the first feature at the second time may be multiplied to obtain an intermediate feature.

[0089] Then, the intermediate feature and the first feature at the first time are summed to obtain the fourth feature. The first time is after the second time. The first time and the second time may be adjacent in the aforementioned time sequence, and the fourth feature can be calculated by the following formula:

[0090] X' ui =X ui +a ui ·X u(i-1) ; where i represents time, a represents the attention coefficient, u represents the difference area, X' is the fourth feature, and X is the first feature. Then X' ui It represents the fourth feature of the pedestrian’s difference area u at time i.

[0091] Thus, i may be the aforementioned first time, and (i-1) may be the aforementioned second time.

[0092] In a possible implementation, the preset attention mechanism is implemented by the first fully connected layer and the first activation function. Specifically, the third feature can be input into the first fully connected layer and the first activation function to obtain an attention coefficient.

[0093] The following combination Figure 3 , taking the concatenation of the first feature and the second feature and the fusion of the first feature and the second feature as an example, the determination of the attention coefficient is explained as follows:

[0094] First, input the first feature corresponding to time into LSTM to obtain the second feature h i , and the first feature and the second feature h iThe first feature and the second feature are concatenated to achieve the fusion of the first feature and the second feature to obtain the third feature. The second feature corresponds to time (i.e., the time step in LSTM).

[0095] The third feature is processed by the first fully connected layer and the first activation function sigmoid to obtain the attention coefficient a corresponding to time ui . Then the attention coefficient can be calculated by the following formula:

[0096] a ui =sigmoid(W(h i / / X ui )+b);

[0097] Wherein, a represents the attention coefficient, i represents the time, u represents the difference area, X represents the first feature, h represents the second feature, represents the splicing operation, W represents the preset weight matrix, and b represents the preset bias matrix. The preset weight matrix and the preset bias matrix here can be obtained through pre-training.

[0098] Then a ui is the attention coefficient of the different area of ​​the pedestrian at time i, h i is the second feature of the difference area of ​​the pedestrian at time i, that is, the second feature of the LSTM output at time i, X ui is the first feature of the difference area of ​​the pedestrian when i. The value of the attention coefficient can be between (0, 1).

[0099] In order to further improve the accuracy of posture recognition, in one possible implementation, the posture model includes GCN (Graph Convolutional Networks); in the posture model, the pedestrian's joint point information is extracted from the image to be processed, so as to accurately describe the pedestrian's posture through the joint point information, and the key information such as the joint point association information in the joint point information is processed by GCN, thereby effectively improving the accuracy of the difference area and the first feature. For detailed description below, please refer to Figure 4 :

[0100] Step 401: Determine a joint graph of a pedestrian in an image to be processed.

[0101] Specifically, the joint map corresponds to the time corresponding to the image to be processed.

[0102] Moreover, the joint graph corresponds to the pedestrian one by one. In each joint graph, the joints of the pedestrian are nodes, and the information of the joints of the pedestrian is the node information.

[0103] The node information includes the position information of each joint of the pedestrian at a corresponding time. The node information may also include the coordinates, confidence, speed, etc. of the node.

[0104] When determining the joint graph, the node information of the joints of the pedestrian in the image to be processed can be determined first. The node information includes the position information of the node. Specifically, the image to be processed can be processed based on a preset human key point algorithm to identify the joints of the pedestrian in the image to be processed as key points.

[0105] Then, according to the connection relationship between the joints and the node information, a joint graph corresponding to the pedestrian in the image to be processed is determined. The joint graph corresponds to the time corresponding to the image to be processed where the pedestrian is located.

[0106] The connection relationship between the joints can be expressed by the n×n adjacency matrix A i Represents, where n is the number of joints of the pedestrian. Adjacency matrix A i The value of the element in represents the connection relationship between the corresponding two nodes.

[0107] For example, for nodes s and f, when the corresponding joints of the two nodes are connected, the element e sf The value of is 1; if there is no connection between the two corresponding joints, then the element e sf The value of is 0.

[0108] For any two nodes in the joint graph, the actual distance between the two nodes can be determined based on their joint information, and the preset rule of connecting the two nodes with the closest actual distance can be used to determine whether there is a connection relationship between the joints.

[0109] Then the above joint graph can be expressed as G i =(A i , X i ), where i=1,2,…,t, and i is time.

[0110] G i is the pedestrian joint graph at time i, A i is the adjacency matrix representing the connection relationship between the joints of the pedestrian at time i, X i is the node information of the pedestrian’s joint at time i.

[0111] Step 402: Input the joint graph into GCN to obtain the fifth feature.

[0112] The fifth feature indicates the joint information of the pedestrian in the image to be processed. Therefore, the feature values ​​in the fifth feature correspond to the joints in the joint map.

[0113] Specifically, consistent with the time corresponding to the joint diagram, the fifth feature corresponds to the time corresponding to the image to be processed.

[0114] GCN can process the node information in the joint graph by the following formula to extract the joint feature as the fifth feature. The fifth feature includes the spatial features of the joint points of the pedestrian:

[0115]

[0116] Among them, H i is the spatial aggregation feature output by GCN, i.e., the fifth feature of the pedestrian at time i, σ is the activation function Relu, A i is the adjacency matrix representing the connection relationship between the joints of the pedestrian at time i, X i is the node information of the pedestrian’s joint at time i, W i is the preset weight matrix, Represents the adjacency matrix A i A symmetric normalization operation is performed. The preset weight matrix can be obtained through pre-training.

[0117] Step 403: performing difference processing on the fifth feature of the pedestrian in the to-be-processed images of two adjacent frames to obtain the aforementioned first feature.

[0118] The first feature indicates characteristic information of changes in joint information of the same pedestrian at different times (corresponding to the image to be processed).

[0119] Specifically, the feature values ​​in the difference feature obtained by subtracting the fifth feature correspond to the joints of the pedestrian. Therefore, when the first feature value in the difference feature is greater than the first threshold, the node corresponding to the feature value is determined to be a change node, and then the connection relationship between the nodes in the change node is determined according to the connection relationship between the aforementioned nodes, and then the difference area can be determined, and the feature of the change node is determined as the first feature of the difference area.

[0120] Furthermore, in order to accurately identify the difference area, in a possible implementation manner, the feature values ​​in the difference features obtained by performing difference processing on the fifth feature of the pedestrian in two adjacent frames of images to be processed may be normalized.

[0121] For the fifth feature obtained by the normalization processing, when the second eigenvalue is greater than the second threshold, the node corresponding to the second eigenvalue is determined to be a changed node. Then, based on the connection relationship between the aforementioned nodes, the connection relationship of the nodes in the changed node is determined, and then the difference area can be determined, and the feature of the changed node is determined to be the first feature of the difference area.

[0122] Thus, the above difference area can be expressed by the following formula:

[0123] U i =(A ui , X ui ), where i is the time, U i is the difference area when i, A ui is the connection relationship of the changed nodes in the difference area at time i, X ui is the node feature of the difference region at time i.

[0124] Then the node feature X of all changed nodes in the difference area ui Constitute the above-mentioned first feature. Or, the node feature X of all changed nodes in the difference area ui and change node A ui The connection relationship constitutes the above-mentioned first feature.

[0125] Please refer to Figure 5 , the joint map of pedestrians in the image to be processed at time T = i-1, T = i, T = i + 1 is processed by GCN, and the fifth feature of the output is subtracted, and the difference area obtained is as follows Figure 5 As shown. In this way, by determining the first feature of the difference area, the spatial local adaptability and local feature adaptability of the posture model are improved, so that the accuracy of the posture information obtained by recognition is further improved.

[0126] It should be understood that although Figure 2-4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0127] In one embodiment, Figure 6 As shown, a gesture recognition device is provided, including: a collection module 601, a difference module 602, a feature module 603 and a gesture module 604, wherein:

[0128] The collection module 601 is used to obtain an image collection and input the image collection into a posture model; wherein the images to be processed in the image collection are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM.

[0129] A difference module 602 is used to determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the to-be-processed image, and the first feature corresponds to time;

[0130] The feature module 603 is used to determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the to-be-processed images, and the first feature corresponds to time.

[0131] The posture module 604 is used to determine the posture information of the pedestrian based on the first feature and the second feature.

[0132] In one embodiment, the gesture module 604 is specifically used for:

[0133] The first feature and the second feature are integrated to obtain a third feature; the third feature is processed based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time; the first feature is feature enhanced by the attention coefficient to obtain the posture information of the pedestrian.

[0134] In one embodiment, the state module 604 is specifically used to: input the third feature into the first fully connected layer and the first activation function to obtain the attention coefficient.

[0135] In one embodiment, the gesture module 604 is specifically used for:

[0136] A fourth feature is obtained by multiplying the attention coefficient by the first feature; and the fourth feature is input into a second fully connected layer and a second activation function to obtain the posture information.

[0137] In one embodiment, the gesture module 604 is further configured to:

[0138] The attention coefficient at the first time is multiplied by the first feature at the second time to obtain an intermediate feature; the first time is located after the second time; the intermediate feature and the first feature at the first time are summed to obtain the fourth feature.

[0139] In one embodiment, the posture model includes a graph convolutional neural network GCN; then the difference module 602 is specifically used to:

[0140] Determine a joint map of a pedestrian in the image to be processed; input the joint map into the GCN to obtain a fifth feature of the pedestrian; wherein the fifth feature indicates joint information of the pedestrian in the image to be processed; perform difference processing on the fifth features of the pedestrian in two adjacent frames of the image to be processed to obtain the first feature.

[0141] In one embodiment, the difference module 602 is further used to determine the node information of the joints of the pedestrian in the image to be processed; and determine the joint graph corresponding to the pedestrian in the image to be processed based on the connection relationship between the joints and the node information.

[0142] For the specific definition of the gesture recognition device, please refer to the definition of the gesture recognition method above, which will not be repeated here. Each module in the above-mentioned gesture recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0143] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a gesture recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0144] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0146] Acquire an image set, and input the image set into a posture model; wherein the images to be processed in the image set are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM;

[0147] Determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the image to be processed, and the first feature corresponds to time;

[0148] Inputting the first feature into the LSTM to obtain a second feature;

[0149] Based on the first feature and the second feature, posture information of the pedestrian is determined.

[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0151] The first feature and the second feature are integrated to obtain a third feature; the third feature is processed based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time; the first feature is feature enhanced by the attention coefficient to obtain the posture information of the pedestrian.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] The third feature is input into the first fully connected layer and the first activation function to obtain the attention coefficient.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] The attention coefficient at the first time is multiplied by the first feature at the second time to obtain an intermediate feature; the first time is located after the second time; the intermediate feature and the first feature at the first time are summed to obtain the fourth feature.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] The posture model includes a graph convolutional neural network (GCN); determining a joint graph of a pedestrian in the image to be processed; inputting the joint graph into the GCN to obtain a fifth feature of the pedestrian; wherein the fifth feature indicates joint information of the pedestrian in the image to be processed; performing difference processing on the fifth features of the pedestrian in two adjacent frames of the image to be processed to obtain the first feature.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] Determine the node information of the joints of the pedestrian in the image to be processed; and determine the joint graph corresponding to the pedestrian in the image to be processed according to the connection relationship between the joints and the node information.

[0160] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0161] Acquire an image set, and input the image set into a posture model; wherein the images to be processed in the image set are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM;

[0162] Determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the image to be processed, and the first feature corresponds to time; input the first feature into the LSTM to obtain a second feature; and determine the posture information of the pedestrian based on the first feature and the second feature.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0164] The first feature and the second feature are integrated to obtain a third feature; the third feature is processed based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time; the first feature is feature enhanced by the attention coefficient to obtain the posture information of the pedestrian.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0166] The third feature is input into the first fully connected layer and the first activation function to obtain the attention coefficient.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0168] A fourth feature is obtained by multiplying the attention coefficient by the first feature; and the fourth feature is input into a second fully connected layer and a second activation function to obtain the posture information.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0170] The attention coefficient at the first time is multiplied by the first feature at the second time to obtain an intermediate feature; the first time is located after the second time; the intermediate feature and the first feature at the first time are summed to obtain the fourth feature.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] The posture model includes a graph convolutional neural network GCN; the first feature of the difference area is determined, including: determining a joint map of the pedestrian in the image to be processed; inputting the joint map into the GCN to obtain a fifth feature of the pedestrian; wherein the fifth feature indicates the joint information of the pedestrian in the image to be processed; performing difference processing on the fifth features of the pedestrian in two adjacent frames of the image to be processed to obtain the first feature.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] Determine the node information of the joints of the pedestrian in the image to be processed; and determine the joint graph corresponding to the pedestrian in the image to be processed according to the connection relationship between the joints and the node information.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A gesture recognition method, characterized in that: include: Acquire an image set, and input the image set into a posture model; wherein the images to be processed in the image set are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM; Determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the image to be processed, and the first feature corresponds to time; Inputting the first feature into the LSTM to obtain a second feature; Based on the first feature and the second feature, posture information of the pedestrian is determined.

2. The method according to claim 1, characterized in that The determining the posture information of the pedestrian based on the first feature and the second feature includes: Combining the first feature and the second feature to obtain a third feature; Processing the third feature based on a preset attention mechanism to obtain an attention coefficient; wherein the attention coefficient corresponds to the time; The first feature is enhanced by using the attention coefficient to obtain the posture information of the pedestrian.

3. The method according to claim 2, characterized in that The processing of the third feature based on a preset attention mechanism to obtain an attention coefficient includes: The third feature is input into the first fully connected layer and the first activation function to obtain the attention coefficient.

4. The method according to claim 2 or 3, characterized in that The step of enhancing the first feature by using the attention coefficient to obtain the posture information of the pedestrian includes: Obtaining a fourth feature by multiplying the attention coefficient by the first feature; The fourth feature is input into the second fully connected layer and the second activation function to obtain the posture information.

5. The method according to claim 4, characterized in that The obtaining of the fourth feature by multiplying the attention coefficient by the first feature comprises: Performing a product operation on the attention coefficient at a first time and the first feature at a second time to obtain an intermediate feature, wherein the first time is after the second time; A sum operation is performed on the intermediate feature and the first feature at the first time to obtain the fourth feature.

6. The method according to any one of claims 1 to 3 and 5, characterized in that: The posture model includes a graph convolutional neural network GCN; Then the first feature of determining the difference area includes: Determine a joint graph of a pedestrian in the image to be processed; Inputting the joint map into the GCN to obtain the fifth feature of the pedestrian; wherein the fifth feature indicates the joint information of the pedestrian in the image to be processed; The fifth feature of the pedestrian in two adjacent frames of the image to be processed is subjected to difference processing to obtain the first feature.

7. The method according to claim 6, characterized in that The determining of the joint graph of the pedestrian in the image to be processed comprises: Determine node information of joints of the pedestrian in the image to be processed; The joint graph corresponding to the pedestrian in the image to be processed is determined according to the connection relationship between the joints and the node information.

8. A gesture recognition device, characterized in that: include: A collection module, used for acquiring an image collection and inputting the image collection into a posture model; wherein the images to be processed in the image collection are sorted according to time sequence, and the posture model includes a long short-term memory network LSTM; A difference module, used to determine a first feature of a difference region; wherein the difference region includes a joint of a pedestrian that changes in two adjacent frames of the to-be-processed image, and the first feature corresponds to time; A feature model, used for inputting the first feature into the LSTM to obtain a second feature; A posture module is used to determine the posture information of the pedestrian based on the first feature and the second feature.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images