Continuous gesture recognition method and device and computer equipment

Abstract

The invention relates to a continuous gesture recognition method and device and computer equipment. The method comprises the following steps: extracting a hand key point in each image frame from continuous image frames, and obtaining first hand key point information with time sequence characteristics; performing down-sampling on the first hand key point information to obtain second hand key point information; and performing feature fusion on the first hand key point information and the second hand key point information by using a pre-trained neural network model, and identifying gesture semantic categories contained in continuous image frames. According to the continuous gesture semantic recognition method based on the hand key points, the time consumed by model reasoning is greatly shortened, the good recognition effect on continuous gestures at different speeds is achieved, and the real-time performance and robustness of continuous gesture recognition are improved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a method, device and computer equipment for continuous gesture recognition. Background Art

[0002] As a body language, gestures play an important role in human communication. A simple gesture contains rich information. Therefore, people can convey a lot of information through gestures. At present, gesture recognition has become a technology in the field of computer vision and artificial intelligence, which is used to control devices or interact by identifying and analyzing people's hand movements. This technology usually uses sensors, cameras and image processing algorithms to capture and interpret gestures.

[0003] Most of the gesture recognition currently known is for static gesture recognition. For example, to recognize the game of rock-paper-scissors, you only need to recognize which gesture is being made at the current moment. However, continuous gesture recognition requires combining historical gesture information with future gesture information to derive the meaning of the complete gesture. For example, waving to let the other person come over requires continuous gesture movement recognition to be understood.

[0004] For continuous gesture recognition tasks, most existing solutions use RGB images, depth maps, etc. as input, and the artificial neural network design is relatively complex, resulting in high network latency and unable to meet the real-time performance of the model output. Summary of the invention

[0005] Based on this, it is necessary to provide a continuous gesture recognition method, device and computer equipment with a lightweight design to address the above technical problems and improve the speed of continuous gesture recognition.

[0006] In a first aspect, the present application provides a method for continuous gesture recognition. The method comprises:

[0007] Extracting the hand key points in each image frame from the continuous image frames to obtain the first hand key point information with time sequence features;

[0008] Down-sampling the first hand key point information to obtain the second hand key point information;

[0009] The pre-trained neural network model is used to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic categories contained in continuous image frames.

[0010] In one embodiment, the first hand key point information includes the number of the image frame, the number of the hand key point, and the three-dimensional coordinates of the hand key point.

[0011] In one embodiment, extracting the hand key points in each image frame from consecutive image frames includes:

[0012] For each image frame, locate the hand position and output the hand image;

[0013] Recognize hand key points based on hand images.

[0014] In one embodiment, obtaining first-hand key point information with time sequence features includes:

[0015] Extracting the hand key points in the current image frame and the hand key points in the historical fixed-length image frames; the current image frame and the historical fixed-length image frames are continuous image frames;

[0016] Arrange all hand key points according to the time sequence of the image frames, and obtain the first hand key point information corresponding to the current image frame with time sequence features.

[0017] In one embodiment, the pre-trained neural network model includes a flattening layer, a convolutional layer, a concatenation layer, and a fully connected layer;

[0018] The pre-trained neural network model is used to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic categories contained in the continuous image frames, including:

[0019] Inputting the first hand key point information and the second hand key point information into the flattening layer for flattening processing respectively, and obtaining the first one-dimensional data and the second one-dimensional data respectively;

[0020] Inputting the first one-dimensional data and the second one-dimensional data into the convolution layer for convolution respectively, and obtaining the first local feature and the second local feature respectively;

[0021] Input the first local feature and the second local feature into the concatenation layer to obtain a fusion feature;

[0022] The fused features are input into the fully connected layer to obtain the semantic category of the gesture.

[0023] In one embodiment, a sliding window-based smoothing algorithm is used to take the gesture with the largest number of occurrences in the sliding window as the output gesture of the continuous image frames in the current sliding window, and the gesture semantic category is identified based on the output gesture.

[0024] In a second aspect, the present application also provides a continuous gesture recognition device. The device comprises:

[0025] A first extraction module, used to extract the hand key points in each of the image frames from the continuous image frames, and obtain first hand key point information with time sequence features;

[0026] A second extraction module, used for down-sampling the first hand key point information to obtain second hand key point information;

[0027] A recognition module is used to use a pre-trained neural network model to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic category contained in the continuous image frames.

[0028] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0029] Extracting the hand key points in each image frame from the continuous image frames to obtain the first hand key point information with time sequence features;

[0030] Down-sampling the first hand key point information to obtain the second hand key point information;

[0031] The pre-trained neural network model is used to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic categories contained in continuous image frames.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0033] Extracting the hand key points in each image frame from the continuous image frames to obtain the first hand key point information with time sequence features;

[0034] Down-sampling the first hand key point information to obtain the second hand key point information;

[0035] The pre-trained neural network model is used to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic categories contained in continuous image frames.

[0036] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] Extracting the hand key points in each image frame from the continuous image frames to obtain the first hand key point information with time sequence features;

[0038] Down-sampling the first hand key point information to obtain the second hand key point information;

[0039] The pre-trained neural network model is used to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic categories contained in continuous image frames.

[0040] The above-mentioned continuous gesture recognition method, device and computer equipment first extract the first hand key point information with time series characteristics from the continuous image frame, then downsample the first hand key point information to obtain the second hand key point information, and finally use the pre-trained neural network model to identify the gesture semantic category. The present application performs gesture semantic recognition based on hand key points, which effectively reduces the data complexity compared to the traditional gesture semantic recognition based on RGB images, thereby well solving the problem of high time consumption of continuous gesture recognition. At the same time, the present application performs gesture semantic category recognition based on the first hand key point information and the second hand key point information. The first hand key point information contains the hand key point features at the original hand speed, and the second hand key point information contains the hand key point features at a relatively faster hand speed. By fusing these two features, the robustness of the continuous gesture recognition framework structure is improved, and the second hand key point information is obtained based on the first hand key point information, which can effectively control the data complexity and improve the performance of the continuous gesture recognition framework structure under the premise of low time consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic flow chart of a continuous gesture recognition method in one embodiment;

[0042] Figure 2 is a structural block diagram of a hand key point detection model in one embodiment;

[0043] Figure 3 is a structural block diagram of a pre-trained neural network model in one embodiment;

[0044] Figure 4 is a flow chart of a continuous gesture recognition method according to another embodiment;

[0045] Figure 5 FIG. 4 is a structural block diagram of a continuous gesture recognition device in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The continuous gesture recognition method provided by the embodiment of the present application is as follows: Figure 1 As shown, the following steps are included:

[0048] Step 102: extract the hand key points in each image frame from the continuous image frames to obtain the first hand key point information with time sequence features.

[0049] The method for extracting hand key points from the image frame can be a regression model for single-stage detection. For example, Mediapipe can be used to detect and locate the joint positions of the thumbs and the remaining four fingers of both hands. The commonly used hand key point model is a model with 21 key points for one hand and 42 key points for both hands.

[0050] After determining the positions of all the hand key points on an image frame, the position-related information of all the hand key points on the image frame is summarized to obtain a set of hand key point information corresponding to the image frame, and the hand gesture in the image frame can be restored based on the set of hand key point information of the image frame. According to the time sequence of continuous image frames, the set of hand key point information corresponding to each image frame is summarized to finally obtain the first hand key point information with time sequence features.

[0051] Step 104: downsample the first hand key point information to obtain second hand key point information.

[0052] Downsampling is the process of selecting a portion of the data from the original sampled data in order to reduce the amount of data while retaining the important information in the original data as much as possible.

[0053] Since the first hand key point information reflects the gesture on each image frame in the continuous image frames, that is, the first hand key point information represents the original action to a certain extent. This embodiment obtains the second hand key point information by downsampling based on the first hand key point information, which is equivalent to using the downsampling technology on the original action to obtain a fast action. For example, the original action is the first hand key point information with a length of 2n (n is a positive integer) frames. After downsampling, the second hand key point information of n frames is obtained. Since the time is shortened and the gesture is complete, it becomes a fast action relative to the original action.

[0054] Step 106: Use the pre-trained neural network model to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic category contained in the continuous image frames.

[0055] The first hand key point and the second hand key point are fused to identify the semantic category of the gesture. That is, the pre-trained neural network model considers the features of the hand key points at both fast and slow speeds when performing the classification task, making the pre-trained neural network model more robust to movements at different speeds.

[0056] This embodiment first extracts the first hand key point information with time series features from the continuous image frames, then downsamples the first hand key point information to obtain the second hand key point information, and finally uses the pre-trained neural network model to identify the gesture semantic category. This application performs gesture semantic recognition based on hand key points, which effectively reduces the data complexity compared to the traditional gesture semantic recognition based on RGB images, thereby well solving the problem of high time consumption of continuous gesture recognition. At the same time, this application performs gesture semantic category recognition based on the first hand key point information and the second hand key point information. The first hand key point information contains the hand key point features at the original hand speed, and the second hand key point information contains the hand key point features at a relatively faster hand speed. By fusing these two features, the robustness of the continuous gesture recognition framework structure is improved, and the second hand key point information is obtained based on the first hand key point information, which can effectively control the data complexity and improve the performance of the continuous gesture recognition framework structure under the premise of low time consumption.

[0057] In one embodiment, the first hand key point information includes the number of the image frame, the number of the hand key point, and the three-dimensional coordinates of the hand key point.

[0058] The first hand key point information includes the number of the image frame, which is related to the time sequence of the image frame. The hand key point information set corresponding to the image frame is distinguished by the image frame number, reflecting the time sequence characteristics contained in the first hand key point information.

[0059] Taking the model of 21 hand key points as an example, a number is defined for each hand key point as the index of each hand key point, so as to more accurately locate the hand key point and recognize the gesture later.

[0060] If there are 21 hand key points, there are 21 corresponding 3D coordinates of the hand key points. Each coordinate consists of x, y and z coordinates, where the z coordinate represents the depth, with the depth at the wrist as the origin. The smaller the value, the closer it is to the camera. The 3D coordinates of the 21 hand key points are also presented in world coordinates. Each coordinate consists of x, y and z, representing the real world 3D coordinate in meters, with the origin at the geometric center of the hand.

[0061] Finally, each hand key point can be expressed as (a, b, c), where a represents the number of consecutive image frames, a takes an integer from 1 to n, and n represents the total number of consecutive image frames; b represents the key point number of the two hands, b takes an integer from 1 to m, and m represents the total number of hand key points of the two hands; c represents the three-dimensional coordinates of the hand key point.

[0062] This embodiment converts the gestures in the image into digital first-hand key point information, effectively reducing the data complexity of the input pre-trained neural network model, reducing the calculation time, and improving the real-time performance of continuous procedure recognition.

[0063] In one embodiment, extracting the hand key points in each image frame from continuous image frames includes: locating the hand position for each image frame and outputting the hand image; and identifying the hand key points based on the hand image.

[0064] like Figure 2 As shown in the figure, a framework model for extracting hand key points is shown. The framework model includes a palm detection model and a hand key point detection model. The palm detection model locates the hand position in the input original image; after detecting the hand, the image of the hand area is cropped and then sent to the key point detection model; the hand key point detection model identifies specific hand landmarks on the hand image output by the palm detection model.

[0065] In one embodiment, obtaining the first hand key point information with timing features in step 102 includes: extracting the hand key points in the current image frame and the hand key points in the historical fixed-length image frames; the current image frame and the historical fixed-length image frames are continuous image frames; arranging all hand key points in time sequence of the image frames, and obtaining the first hand key point information corresponding to the current image frame with timing features.

[0066] For example, 32 consecutive image frames are used for gesture recognition, so the length of the historical fixed-length image frame is 31 frames. Since this solution needs to predict continuous gesture information, the first hand key point information also needs to have time series features. The continuous image frames are input into the framework model for hand key point extraction to obtain the first hand key point information. Since the first hand key point information contains a set of hand key point information corresponding to each image frame, and the image frames are arranged in chronological order, the first hand key point information also contains time series features.

[0067] The input to the pre-trained neural network model is the first hand key point information corresponding to 32 consecutive image frames. Based on the first hand key point information input of these 32 frames, the pre-trained neural network model will predict a gesture semantic category.

[0068] This embodiment may also use continuous image frames of other lengths, and the frame length is not specifically limited here.

[0069] In one embodiment, the pre-trained neural network model includes a flattening layer, a convolutional layer, a splicing layer and a fully connected layer; step 106 includes: inputting the first hand key point information and the second hand key point information into the flattening layer for flattening processing, respectively, to obtain the first one-dimensional data and the second one-dimensional data, respectively; inputting the first one-dimensional data and the second one-dimensional data into the convolutional layer for convolution, respectively, to obtain the first local feature and the second local feature, respectively; inputting the first local feature and the second local feature together into the splicing layer to obtain the fused feature; inputting the fused feature into the fully connected layer to obtain the gesture semantic category.

[0070] like Figure 3 As shown in the figure, after the hand key points are input into the pre-trained neural network model, the pre-trained neural network model will be divided into two branches. The first branch processes the original action and the second branch processes the fast action (the fast action is obtained by downsampling the original action). After the hand key points are flattened, they are processed by one-dimensional (1D) convolution respectively, and then the original action and the fast action are concatenated (concat) in the feature dimension, and then sent to the fully connected layer, and finally the predicted continuous gesture semantic category is output.

[0071] In this embodiment, the pre-trained neural network model is trained based on about 30,000 real-world images and multiple synthetic hand models rendered under different backgrounds. The training data is a self-collected, self-annotated data set, with a total of 100 videos in the training set and 20 videos in the test set. Each video is 60 seconds long, containing 1,800 pictures and about 30 to 40 continuous gestures. Because the video is continuous, each frame needs to be labeled during the annotation process. Combined with the experience of interpersonal gesture interaction in reality, continuous gestures all have a forward shaking action. After the forward shaking action, people can understand the meaning of the gesture. In the annotation process of this data set, the same concept is also used to make the performance of the trained model better.

[0072] The training set is a randomly selected n (n is 500 in this embodiment, 500*100=5w segments in total) fixed-length frame segments from 0 to 1800 frames of each video. Each independent segment may contain one or more complete gestures, or may contain incomplete gestures. The corresponding training true value label is the label of the last frame in the fixed-length frame.

[0073] In addition to the complexity of the task, data quality is also a decisive factor affecting the performance of the neural network. At present, there is a small amount of annotated continuous gesture data available. In order to solve this problem, this solution uses the data collected and annotated by itself as the training data of the model. On this dataset, this solution has achieved good results.

[0074] Due to the lightweight design and small training data volume, the training parameters used for the pre-trained neural network model in this embodiment are: batch size = 1024, epoch = 200, the loss function is Crossentropy, the optimization function uses ADAM (Adaptive Moment Estimation), the learning rate is 0.01, and finally the trained model can be obtained after 2 hours of training on a 4090 GPU.

[0075] In one embodiment, after extracting the three-dimensional coordinates of the gesture key points, they need to be pre-processed, that is, the hand key points in the fixed-length frame segment are standardized. Specifically, for the coordinates of all the hand key points in the fixed-length frame segment, the mean and the maximum absolute value corresponding to each coordinate axis are calculated according to the three coordinate axes x, y, and z, and the coordinate value of each hand key point is subtracted from the mean corresponding to x, y, and z, and then divided by the corresponding maximum absolute value to obtain the normalized key point coordinate value.

[0076] In one embodiment, a sliding window-based smoothing algorithm is used to take the gesture with the largest number of occurrences in the sliding window as the output gesture of the continuous image frames in the current sliding window, and the gesture semantic category is identified based on the output gesture.

[0077] In order to ensure the temporal continuity of the output of the pre-trained neural network model, the model output needs to be smoothed. This embodiment adopts a smoothing algorithm based on a sliding window, and takes the gesture that appears most times in the sliding window as the output gesture of the current frame.

[0078] The length of the sliding window in this embodiment is set to 32 frames, the step size is 1, and each frame contains the x, y, and z coordinates of 42 key points of the hand. The prediction is performed through the continuous pre-trained neural network model, and the obtained results are smoothed through the sliding window to obtain the final result. Specifically, Figure 4 As shown in the figure, every 32 image frames are a sliding window, and the gesture key points corresponding to the image frames in the sliding window are input into the pre-trained neural network model (NN) to predict the gesture semantic category (Prediction). The sliding window is changed with a step size of 1, and the gesture semantic categories corresponding to other sliding windows are obtained in turn until the video stream ends. The prediction output corresponding to each sliding window is post-processed (Postprocess), that is, smoothed (Smooth), and the gesture with the most occurrences in each sliding window is taken as the predicted gesture of the current frame, and the gesture semantic category is output.

[0079] This application proposes a lightweight continuous gesture recognition solution based on hand key points. The operation time on the mobile terminal Jetson is about 1 to 2 ms, which greatly reduces the model inference time on the mobile terminal and solves the problem of high time consumption of continuous gesture recognition. At the same time, in the self-built data set, the classification accuracy can reach 73%. After processing and filtering, the accuracy of continuous gesture recognition is about 0.72 and the recall rate is 0.80. If mediapipe is used for hand key point extraction, thanks to its longer gesture key point recognition distance, the effective distance of the gesture semantic recognition method can reach about 7m.

[0080] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0081] Based on the same inventive concept, the embodiment of the present application also provides a continuous gesture recognition device for implementing the continuous gesture recognition method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more continuous gesture recognition device embodiments provided below can refer to the limitations of the continuous gesture recognition method above, and will not be repeated here.

[0082] In one embodiment, Figure 5 As shown, a continuous gesture recognition device is provided, comprising: a first extraction module 502, a second extraction module 504 and a recognition module 506, wherein:

[0083] The first extraction module 502 is used to extract the hand key points in each of the continuous image frames to obtain the first hand key point information with time sequence characteristics.

[0084] A second extraction module 504 is used to downsample the first hand key point information to obtain second hand key point information;

[0085] The recognition module 506 is used to use a pre-trained neural network model to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic category contained in the continuous image frames.

[0086] In one embodiment, the first hand key point information includes the number of the image frame, the number of the hand key point, and the three-dimensional coordinates of the hand key point.

[0087] In one embodiment, the first extraction module 502 is further used to locate the hand position for each image frame, output the hand image, and identify the hand key points based on the hand image.

[0088] In one embodiment, the first extraction module 502 is also used to extract hand key points in the current image frame and hand key points in historical fixed-length image frames; the current image frame and the historical fixed-length image frames are continuous image frames; all hand key points are arranged in chronological order of the image frames, and the first hand key point information corresponding to the current image frame with timing features is obtained.

[0089] In one embodiment, the pre-trained neural network model includes a flattening layer, a convolution layer, a splicing layer and a fully connected layer. The recognition module 506 is also used to input the first hand key point information and the second hand key point information into the flattening layer for flattening processing, respectively, to obtain the first one-dimensional data and the second one-dimensional data; input the first one-dimensional data and the second one-dimensional data into the convolution layer for convolution, respectively, to obtain the first local feature and the second local feature; input the first local feature and the second local feature together into the splicing layer to obtain the fused feature; input the fused feature into the fully connected layer to obtain the gesture semantic category.

[0090] In one embodiment, the recognition module 506 uses a sliding window-based smoothing algorithm to take the gesture with the largest number of occurrences in the sliding window as the output gesture of the continuous image frames in the current sliding window, and recognizes the gesture semantic category according to the output gesture.

[0091] Each module in the above-mentioned continuous gesture recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0092] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in all the above method embodiments when executing the computer program.

[0093] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in all the above method embodiments are implemented.

[0094] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in all the above method embodiments when executed by a processor.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0096] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0097] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A continuous gesture recognition method, characterized in that: The method comprises: Extracting the hand key points in each of the image frames from the continuous image frames to obtain first hand key point information with time sequence features; Down-sampling the first hand key point information to obtain second hand key point information; The first hand key point information and the second hand key point information are feature fused using a pre-trained neural network model to identify the gesture semantic category contained in the continuous image frames.

2. The method according to claim 1, characterized in that The first hand key point information includes the number of the image frame, the number of the hand key point, and the three-dimensional coordinates of the hand key point.

3. The method according to claim 1, characterized in that The step of extracting the hand key points in each of the image frames from the continuous image frames comprises: For each of the image frames, locate the hand position and output the hand image; The hand key points are identified based on the hand image.

4. The method according to claim 1, characterized in that The obtaining of first-hand key point information with time series features includes: Extracting the hand key points in the current image frame and the hand key points in the image frames of a fixed length in history; the current image frame and the image frames of a fixed length in history are continuous image frames; Arrange all the hand key points in time sequence of the image frames, and obtain the first hand key point information corresponding to the current image frame with time sequence features.

5. The method according to claim 1, characterized in that The pre-trained neural network model includes a flattening layer, a convolutional layer, a concatenation layer and a fully connected layer; The using of the pre-trained neural network model to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic category contained in the continuous image frames includes: Inputting the first hand key point information and the second hand key point information into the flattening layer for flattening processing respectively, and obtaining first one-dimensional data and second one-dimensional data respectively; Inputting the first one-dimensional data and the second one-dimensional data into the convolution layer for convolution respectively, and obtaining a first local feature and a second local feature respectively; Inputting the first local feature and the second local feature into the concatenated layer to obtain a fused feature; The fused features are input into a fully connected layer to obtain the semantic category of the gesture.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: A smoothing algorithm based on a sliding window is adopted, the gesture with the largest number of occurrences in the sliding window is taken as the output gesture of the continuous image frames in the current sliding window, and the gesture semantic category is identified according to the output gesture.

7. A continuous gesture recognition device, characterized in that: The device comprises: A first extraction module, used to extract the hand key points in each of the image frames from the continuous image frames, and obtain first hand key point information with time sequence features; A second extraction module, used for down-sampling the first hand key point information to obtain second hand key point information; A recognition module is used to use a pre-trained neural network model to perform feature fusion on the first hand key point information and the second hand key point information to identify the gesture semantic category contained in the continuous image frames.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images