Gesture recognition interaction method and device, equipment, storage medium and product

Abstract

The invention discloses a gesture recognition interaction method and device, equipment, a storage medium and a product, and relates to the technical field of man-machine interaction, and the method comprises the steps: obtaining single-frame image data including a gesture action in real time, inputting the single-frame image data into a hand shape feature extraction model, obtaining single-frame hand shape features, and obtaining a hand shape feature extraction model; and determining multi-frame sequence features based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data. And inputting the multi-frame sequence features into a gesture action judgment model to obtain a target gesture action, and based on the target gesture action, triggering an interaction response corresponding to the target gesture action. According to the method, the target gesture action is obtained through the gesture action judgment model, direct image-level feature extraction can be achieved, effective information is utilized to the maximum extent, the calculation power of gesture interaction in an AR scene is reduced, and the model can run on a low-calculation-power platform in real time.

Description

Technical Field

[0001] The present application relates to the technical field, and in particular to gesture recognition interaction methods, devices, equipment, storage media and products. Background Art

[0002] With the rapid development of artificial intelligence technology, AR (augmented reality) technology has made significant progress. At present, AR technology can not only seamlessly superimpose virtual information into the real world to provide users with a richer visual experience, but also achieve deep integration with the real environment through intelligent recognition, tracking and interaction technology.

[0003] In the existing AR scene gesture interaction algorithm, it is difficult to perform dynamic gesture recognition using only key point information, and dynamic gesture recognition requires powerful computing devices. At present, most mobile devices in AR scenes are low-computing devices. Therefore, how to perform dynamic gesture recognition on low-computing devices is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a gesture recognition interaction method, device, equipment, storage medium and product, aiming to solve the technical problem of dynamic gesture recognition on low-computing power devices.

[0005] To achieve the above objectives, the present application proposes a gesture recognition interaction method, which is applied to an AR device, and the method includes:

[0006] Acquire single-frame image data including gesture actions in real time;

[0007] Inputting the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features;

[0008] Determine multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data;

[0009] Inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action;

[0010] Based on the target gesture action, an interaction response corresponding to the target gesture action is triggered.

[0011] In one embodiment, the step of inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action includes:

[0012] When continuous single-frame image data greater than a preset number of frames include the same gesture action, the same gesture action is identified as the target gesture action.

[0013] In one embodiment, the hand shape feature extraction model includes a backbone network, and the step of inputting the single frame image data into the hand shape feature extraction model to obtain the single frame hand shape feature includes:

[0014] The single-frame image data is input into the backbone network, and the single-frame hand shape features are obtained through feature extraction.

[0015] In one embodiment, the step of determining the multi-frame sequence features based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data respectively comprises:

[0016] The single-frame hand shape features corresponding to the continuously acquired single-frame image data are respectively input into the time dimension encoder for time dimension encoding to obtain the multi-frame sequence features.

[0017] In one embodiment, the step of inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action includes:

[0018] Inputting the multi-frame sequence features into the gesture action discrimination model to obtain the predicted coordinates of the gesture key points;

[0019] Based on the predicted coordinates of the gesture key points, a target gesture action is determined.

[0020] In one embodiment, the step of triggering an interactive response corresponding to the target gesture action based on the target gesture action includes:

[0021] Acquire a calibration video image, and determine a mapping range of the AR display area according to the calibration video image;

[0022] Calculating a mapping range of the AR display area and a mapping relationship between the display area;

[0023] Based on the mapping relationship, determine a coordinate mapping result of the predicted coordinates of the gesture key points in the AR display area;

[0024] Based on the coordinate mapping result of the AR display area, an interactive response corresponding to the target gesture action and the predicted coordinates of the gesture key points is triggered.

[0025] In addition, to achieve the above purpose, the present application also proposes a gesture recognition interaction device, the gesture recognition interaction device comprising:

[0026] An image data acquisition module, used to acquire single-frame image data including gesture actions in real time;

[0027] A single-frame feature acquisition module, used for inputting the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features;

[0028] A multi-frame feature acquisition module, used to determine multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data;

[0029] A target gesture determination module, used for inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action;

[0030] The target gesture interaction module is used to trigger an interaction response corresponding to the target gesture action based on the target gesture action.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a gesture recognition interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the gesture recognition interaction method described above.

[0032] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition interaction method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition interaction method described above are implemented.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] By acquiring single-frame image data including gestures in real time, the single-frame image data is input into the hand shape feature extraction model to obtain single-frame hand shape features, and the multi-frame sequence features are determined based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data. The multi-frame sequence features are input into the gesture action discrimination model to obtain the target gesture action, and based on the target gesture action, the interactive response corresponding to the target gesture action is triggered. The process of obtaining the target gesture action through the gesture action discrimination model can realize direct image-level feature extraction, maximize the use of effective information, reduce the computing power of gesture interaction in AR scenarios, and enable the model to run in real time on a low-computing power platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 This is a flowchart of the first embodiment of the gesture recognition interaction method of the present application;

[0039] Figure 2 This is a schematic diagram of the MobileOne network structure of an embodiment of the present application;

[0040] Figure 3 This is a schematic diagram of the Transformer encoder structure of an embodiment of the present application;

[0041] Figure 4 This is a flow chart of the second embodiment of the gesture recognition interaction method of the present application;

[0042] Figure 5 This is a flowchart of the third embodiment of the gesture recognition interaction method of the present application;

[0043] Figure 6 This is a schematic diagram of the module structure of the gesture recognition interaction device according to an embodiment of the present application;

[0044] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the gesture recognition interaction method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] It should be noted that gestures are divided into static gestures and dynamic gestures. Static gesture recognition only classifies the hand shapes in a picture, such as classifying the "OK" gesture. When learning features, it only focuses on the spatial features of gestures. Compared with static gesture recognition, dynamic gestures are closer to people's expression habits. Generally, dynamic gesture recognition will have depth information input, which can know the 3D position of the hand and facilitate recognition. For images obtained by a monocular color camera, there is a lack of depth information, which increases the difficulty of recognition. In addition, dynamic gesture recognition requires powerful computing power equipment. Therefore, how to perform dynamic gesture recognition on low-computing power devices is a problem that needs to be solved urgently.

[0049] The present application provides a solution, which obtains single-frame image data including gestures in real time, inputs the single-frame image data into a hand shape feature extraction model, obtains single-frame hand shape features, and determines multi-frame sequence features based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data. The multi-frame sequence features are input into a gesture action discrimination model to obtain a target gesture action, and based on the target gesture action, an interactive response corresponding to the target gesture action is triggered. The process of obtaining the target gesture action through the gesture action discrimination model can realize direct image-level feature extraction, maximize the use of effective information, reduce the computing power of gesture interaction in AR scenarios, and enable the model to run in real time on a low-computing power platform.

[0050] Based on this, the embodiment of the present application provides a gesture recognition interaction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the gesture recognition interaction method of the present application.

[0051] In this embodiment, the gesture recognition interaction method includes steps S10 to S50:

[0052] Step S10, acquiring single-frame image data including gesture actions in real time.

[0053] It should be noted that gesture actions can be understood as the action of a user clicking on the AR display area when using an AR (Augmented Reality) device, including but not limited to two-finger pinching, single-finger clicking, four-finger sliding or single-finger sliding. The target single-frame image dataset can be understood as video image data that is identified as normal user interaction operations after preprocessing the video data.

[0054] In some embodiments, a terminal (e.g., an AR terminal) obtains single-frame image data captured in real time by a camera, including gesture interaction performed by a user through an AR device. Exemplarily, the camera may be integrated into an AR glasses device, and the obtained video data may be parsed by a computer device (the computer device may include a display screen), and single-frame image data may be obtained by parsing the video data of the gesture action.

[0055] Step S20, inputting the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features.

[0056] It should be noted that the hand shape feature extraction model includes a backbone network. The backbone network can be understood as a main network, which refers to a network for extracting features and is used to extract information from the image for use by subsequent networks. The backbone network can be a VGG model and a ResNet (residual network). Step S20 includes: inputting a single-frame image data into the backbone network, and obtaining a single-frame hand shape feature through feature extraction. For example, Figure 2 As shown, MobileOne is used as the backbone network of the model, and the preprocessed single-frame image data is sent as input to the convolution layer of the MobileOne model. The MobileOne model can be composed of multiple convolutional layers, ReLU activation functions, pooling layers, and fully connected layers. Exemplarily, the video frame image is scaled to 224x128 size and input into the network. The network passes through network block 1 (3 convolutional layers, downsampled 4 times) to obtain feature map F1, the network passes through network block 2 (8 convolutional layers, further downsampled 2 times) to obtain feature map F2, the network passes through network block 3 (12 convolutional layers, further downsampled 2 times) to obtain feature map F3, the network passes through network block 4 (2 convolutional layers, further downsampled 2 times) to obtain feature map F4, and the F1-F4 feature maps are pooled into 1x1 size and spliced ​​to fuse the deep and shallow features of the model, which is conducive to feature expression. The concatenated features are flattened and encoded into a 128-dimensional feature vector, with each dimension of size 1x2380. The final single-frame hand shape features are used as the input of the subsequent network.

[0057] Step S30, determining multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data.

[0058] It should be noted that step S30 includes: inputting the single-frame hand shape features corresponding to the continuously acquired single-frame image data into the time dimension encoder for time dimension encoding to obtain multi-frame sequence features.

[0059] The multi-frame sequence feature side can represent a feature representation that is constructed through feature fusion based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data, and can contain broader contextual information. It can be understood that each frame in the continuously acquired single-frame image data can obtain the corresponding single-frame hand shape features through step S20. MobileOne is used as the backbone network of the model, and two Transformer models are connected in parallel at the output end of the backbone network as the head network. Each head network is finally connected to a fully connected layer as the output layer. The output of the positioning head network is a 4-element vector, which respectively represents the upper left vertex coordinates and the lower right vertex coordinates of the palm positioning box. The output of the key point head network is a 42-element vector, which represents the coordinates of the 21 key points of the whole hand. Figure 3 As shown in FIG. 1 , the current frame features and the 128-dimensional features of the previous 19 historical frames are input into a 2-layer Transformer encoder for time dimension encoding. After Transformer encoding, multi-frame sequence features are obtained.

[0060] Step S40: input the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action.

[0061] It should be noted that the gesture action discrimination model can be a pre-trained machine learning model that can receive images containing gestures as input and is based on a Transformer structure that encodes multi-frame sequence features according to the time dimension to obtain the action type of each frame.

[0062] Step S50: triggering an interactive response corresponding to the target gesture action based on the target gesture action.

[0063] Exemplarily, the interactive response occurs at the position in the AR display area to which the predicted coordinates of the gesture key points are mapped. For example, the predicted coordinates of the gesture key points are the fingertip coordinates of the finger (for example, the index fingertip), and the interactive response of the single-finger poking action occurs at the position in the AR display area to which the predicted coordinates of the fingertip are mapped. It should be noted that the AR display area refers to the virtual information display area created by augmented reality (AR) technology in the physical world. In this area, virtual objects can be integrated with the scenes of the real world, which can provide users with a more immersive experience. In an AR device (such as AR glasses), the AR display area can be any part or all of the field of view of the device. Coordinate mapping can be a process of converting the predicted coordinates of the gesture key points from the camera image coordinate system to the AR display area coordinate system. The coordinate mapping result is the specific position of the gesture key points in the AR display area, so that the virtual object can be accurately adjusted and interactively fed back according to the user's gesture. The interactive response can include operations such as moving, rotating, scaling, selecting, activating, and other virtual objects, as well as related feedback effects such as sound and animation.

[0064] In this embodiment, by acquiring single-frame image data including gestures in real time, the single-frame image data is input into a hand shape feature extraction model to obtain single-frame hand shape features, and based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data, multi-frame sequence features are determined. The multi-frame sequence features are input into a gesture action discrimination model to obtain a target gesture action, and based on the target gesture action, an interactive response corresponding to the target gesture action is triggered. The process of obtaining the target gesture action through the gesture action discrimination model can realize direct image-level feature extraction, maximize the use of effective information, reduce the computing power of gesture interaction in AR scenarios, and enable the model to run in real time on a low-computing power platform.

[0065] It should be noted that, in order to improve the accuracy of gesture recognition, a specific implementation method for preventing accidental touches is proposed.

[0066] Specifically, step S10 includes: when continuous single-frame image data greater than a preset number of frames include the same gesture action, identifying the same gesture action as a target gesture action.

[0067] Exemplarily, if more than three consecutive frames are identified as the same gesture action, the gesture action may be regarded as the target gesture action.

[0068] In one embodiment, step S20 further includes: when the interval between the currently collected frame number and the last frame number input into the hand shape feature extraction model is greater than the preset interval frame number, returning to the step of acquiring single-frame image data including gesture action in real time. Exemplarily, after completing the current frame action prediction, the target single-frame image data can be re-collected after four frames.

[0069] In this embodiment, streaming reasoning can be performed through sliding windows (inference is performed every 4 frames). By setting a preset number of interval frames (such as four frames in the example), the system does not process every frame, but selectively processes key frames. This can greatly reduce the amount of calculation, especially when processing high-resolution videos or when computing resources are limited, and can significantly improve processing speed and efficiency. Through sliding window processing, the system can improve processing speed and reduce latency while ensuring prediction accuracy, thereby enhancing user experience.

[0070] Reference Figure 4 , Figure 4 This is a flow chart of the second embodiment of the gesture recognition interaction method of the present application. Figure 1 The first embodiment shown is a second embodiment of the gesture recognition interaction method of the present application.

[0071] In the second embodiment, the step S40 includes:

[0072] Step S401: input multi-frame sequence features into a gesture action discrimination model to obtain gesture key point prediction coordinates.

[0073] It should be noted that step S401 may include: inputting multi-frame sequence features into the positioning head network to obtain palm positioning results corresponding to the multi-frame sequence features; obtaining hand shape features from the multi-frame sequence features according to the palm positioning results (wherein the single-frame hand shape features of each frame can be determined by step S20, for example); inputting the hand shape features into the key point head network to obtain the predicted coordinates of the gesture key points.

[0074] Exemplarily, fine feature extraction can be performed on the palm ROI area on the multi-frame sequence features, including dividing the ROI area into several small grids (bins), using an interpolation method (such as bilinear interpolation) in each grid to calculate the pixel value at any position, and performing a pooling operation (such as maximum pooling) on ​​the pixel values ​​in each grid to obtain a feature vector of a fixed size.

[0075] In some embodiments, the gesture features are input into the key nod network to obtain the key point prediction coordinates corresponding to the continuously acquired single-frame image data. Based on the key point prediction coordinates corresponding to the continuously acquired single-frame image data, the gesture key point prediction coordinates are obtained. Exemplarily, when the end of the single-finger poking action is detected, the key point prediction coordinates corresponding to this time are determined to be the finger key point prediction coordinates. For example, 20 frames of image data are continuously acquired, and 20 groups of key point prediction coordinates can be obtained by inputting the hand shape features into the key nod network. It can be understood that since the frame image data is continuously acquired, the 20 groups of key point prediction coordinates also change continuously. When a change in the change trend is detected at any group of key point prediction coordinates in the 20 groups of key point prediction coordinates, it is confirmed that the single-finger poking action here is over. For example, during the single-finger poking action, the key point prediction coordinates move approximately in a straight line direction. When a change in direction occurs at and after a certain group of key point prediction coordinates or the position remains approximately unchanged, it is considered that a change in the change trend has occurred.

[0076] Step S402: determining the target gesture action based on the predicted coordinates of the gesture key points.

[0077] Exemplarily, the predicted coordinates of gesture key points include the palm position prediction and gesture action prediction for each frame. When predicting the palm position and gesture action of a single frame, a static gesture recognition method can be used. Exemplarily, MobileOne is used as the backbone network of the model. The output end of the backbone network is connected in parallel with two Transformer models as the head network. Each head network is finally connected to a fully connected layer as the output layer. The output of the positioning head network is a 4-element vector, which respectively represents the coordinates of the upper left vertex and the lower right vertex of the palm positioning box. The output of the key point head network is a 42-element vector, which represents the coordinates of the 21 key points of the whole hand. Figure 4 As shown in the figure, the current frame features and the 128-dimensional features of the previous 19 historical frames are input into a 2-layer Transformer encoder for time dimension encoding. After Transformer encoding, the action prediction and palm position prediction for each frame are output.

[0078] In this embodiment, the multi-frame sequence features are input into the gesture action discrimination model to obtain the predicted coordinates of the gesture key points, and the target gesture action is determined based on the predicted coordinates of the gesture key points. Through time dimension encoding, the dynamic changes of gestures can be captured more comprehensively, thereby improving the accuracy of gesture action discrimination. The feature extraction capability of the backbone network ensures that the key information extracted from the image is rich and accurate enough, providing a solid foundation for subsequent predictions. The introduction of frame feature information enables the system to better cope with sudden changes or occlusions of gestures, thereby improving the robustness of the system.

[0079] Reference Figure 5 , Figure 5 This is a flow chart of the third embodiment of the gesture recognition interaction method of the present application. Figure 2 The second embodiment shown provides a third embodiment of the gesture recognition interaction method of the present application.

[0080] In the third embodiment, the step S50 includes:

[0081] Step S501: Acquire a calibration video image, and determine a mapping range of an AR display area according to the calibration video image.

[0082] It should be noted that the step of determining the mapping range of the AR display area based on the calibrated video image includes: traversing each frame image in the calibrated video image; identifying the finger edge in each frame image, and tracking the movement trajectory of the finger along the edge of the display area through edge matching between consecutive frames; and determining the mapping range of the display area based on the movement trajectory.

[0083] For example, the edge of the finger in each frame can be identified by applying an edge detection algorithm (such as Canny, Sobel, etc.), and the movement trajectory of the finger along the edge of the display area can be tracked by edge matching between consecutive frames. This may require the use of optical flow, feature point tracking, or other video tracking techniques. The tracked edge points are aggregated in chronological order to form the edge contour of the display area, and the edge contour is fitted into a rectangle (or other shapes according to actual needs) using methods such as polygon fitting or least squares method, and the length, width, and center point coordinates of the fitted shape are calculated. These parameters will be used for subsequent coordinate mapping.

[0084] For example, you can turn on the AR glasses recording function, move your index finger along the edge of the display area, and then export the video. The movement range of the finger in the image is recorded to determine the mapping range of the AR display area.

[0085] Step S502 : calculating a mapping range of the AR display area and a mapping relationship of the AR display area to obtain a coordinate mapping.

[0086] It should be noted that coordinate mapping can be understood as calibrating the camera imaging and the image of the AR display area. Exemplarily, based on the movement trajectory of the finger in the video, the area corresponding to the AR display area in the camera image (for example, the calibrated video image) is determined as the mapping range of the AR display area. Then the ratio of the length and width of the two areas is calculated, and a linear transformation matrix is ​​constructed based on the calculated length-to-width ratio. The matrix is ​​used to convert the coordinate system in the camera image into the coordinate system on the AR display area. By adjusting the edge detection parameters, tracking algorithm parameters, or parameters of the linear transformation matrix, the predicted coordinates of the finger key points are calibrated, and the calibration results can be understood as the calibrated predicted coordinates of the finger key points.

[0087] Step S503: Based on the coordinate mapping, determine the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area.

[0088] For example, the coordinates of the calibrated finger key points can be predicted to realize the interaction between the gesture and the AR display area. It is understandable that the mapping algorithm can also be integrated into the application of AR glasses to realize real-time hand key point mapping and interaction.

[0089] In this embodiment, by acquiring the calibration video image and determining the mapping range of the display area, the accurate position of the predicted coordinates of the finger key points in the AR display area can be ensured. This precise mapping provides a reliable basis for subsequent interactive responses. The mapping range of the AR display area and the ratio of the AR display area are calculated, and the predicted coordinates of the finger key points are calibrated according to the ratio, which helps to eliminate the coordinate offset caused by the change in the size of the display area, further improve the accuracy of the interaction, and trigger the corresponding interactive response based on the calibration results according to the coordinate mapping results of the predicted coordinates of the finger key points in the AR display area, so that users can interact with the AR environment through natural gestures, thereby improving the fluency of the interaction.

[0090] In one embodiment, the training process of the gesture action recognition model may include: obtaining single-frame image data including gesture actions, and annotating the palm position and finger key point coordinates in each frame of the image frame by frame to generate a training data set; inputting the training data set into the gesture action recognition model to obtain predicted values ​​of the palm position and finger key point coordinates; based on the training data set, calculating the loss function according to the true value and the predicted value, if the loss function is less than a preset loss threshold, it is determined that the gesture action recognition model has converged; if the loss function is greater than or equal to the preset loss threshold, then updating the parameters of the gesture action recognition model through the back propagation algorithm according to the gradient of the loss function, and returning to the step of inputting the training data set into the gesture action recognition model.

[0091] Exemplarily, model training can be performed offline, and the model can be trained using classic forward reasoning and gradient back propagation algorithms. During the training process, each frame of the image is sent to the backbone network to obtain frame features, and then the features of the past 20 frames are spliced ​​to obtain sequence features, which are sent to the positioning head network and the key point network to obtain the palm positioning coordinates and key point coordinates predicted by the model. Use CIOU loss to supervise the training of palm positioning, use the ROI Align method to determine the corresponding area of ​​the palm area on the backbone output feature according to the positioning coordinates, and send the features of the area to the key point network for inference to obtain the prediction results of the key points. Based on the prediction results of the key points, determine the target gesture action. Use the cross entropy loss function for supervised training, and train a total of 100 iterations.

[0092] In this embodiment, a high-quality training data set can be generated by annotating the coordinates of the palm position and finger key points in the video data frame by frame. This fine annotation ensures that the model can access samples that are as real as possible during the learning process, thereby improving the accuracy of the model in the prediction stage. During the training process, by constantly comparing the actual value with the predicted value and calculating the error based on the loss function, the model can gradually adjust its internal parameters to reduce this error. When the loss function is less than the preset threshold, it indicates that the model has learned the distribution law of the training data well, thereby improving its accuracy in judging the target gesture action.

[0093] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the gesture recognition interaction method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0094] This application also provides a gesture recognition interaction device, please refer to Figure 6 , the gesture recognition interaction device comprises:

[0095] An image data acquisition module 10, used for acquiring single-frame image data including gesture actions in real time;

[0096] A single-frame feature acquisition module 20 is used to input the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features;

[0097] A multi-frame feature acquisition module 30, used to determine multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data;

[0098] A target gesture determination module 40 is used to input the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action;

[0099] The target gesture interaction module 50 is used to trigger an interaction response corresponding to the target gesture action based on the target gesture action.

[0100] The gesture recognition interaction device provided by the present application adopts the gesture recognition interaction method in the above embodiment, which can solve the technical problem of dynamic gesture recognition on low computing power devices. Compared with the prior art, the beneficial effects of the gesture recognition interaction device provided by the present application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, and the other technical features in the gesture recognition interaction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0101] The present application provides a gesture recognition interaction device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture recognition interaction method in the above-mentioned embodiment 1.

[0102] Reference below Figure 7 , which shows a schematic diagram of the structure of a gesture recognition interaction device suitable for implementing the embodiment of the present application. The gesture recognition interaction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The gesture recognition interaction device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0103] like Figure 7 As shown, the gesture recognition interaction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the gesture recognition interaction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the gesture recognition interaction device to communicate with other devices wirelessly or by wire to exchange data. Figure 7The gesture recognition interaction device with various systems is shown in the figure, but it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0104] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0105] The gesture recognition interaction device provided by the present application adopts the gesture recognition interaction method in the above embodiment, which can solve the technical problem of dynamic gesture recognition on low computing power devices. Compared with the prior art, the beneficial effects of the gesture recognition interaction device provided by the present application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, and the other technical features in the gesture recognition interaction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0106] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0108] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the gesture recognition interaction method in the above-mentioned embodiment.

[0109] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0110] The computer-readable storage medium may be included in the gesture recognition interaction device; or may exist independently without being assembled into the gesture recognition interaction device.

[0111] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the gesture recognition interaction device, the gesture recognition interaction device: acquires single-frame image data including gesture actions in real time; inputs the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features; determines multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data; inputs the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action; and triggers an interactive response corresponding to the target gesture action based on the target gesture action.

[0112] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0115] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned gesture recognition interaction method, and can solve the technical problem of dynamic gesture recognition on low-computing power devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the gesture recognition interaction method provided in the above-mentioned embodiment, and will not be repeated here.

[0116] The present application also provides a computer program product, including a computer program, which implements the steps of the gesture recognition interaction method as described above when executed by a processor.

[0117] The computer program product provided by this application can solve the technical problem of dynamic gesture recognition on low-computing power devices. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, which will not be repeated here.

[0118] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A gesture recognition interaction method, characterized in that: Applied to an AR device, the method includes: Acquire single-frame image data including gesture actions in real time; Inputting the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features; Determine multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data; Inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action; Based on the target gesture action, an interaction response corresponding to the target gesture action is triggered.

2. The method according to claim 1, characterized in that The step of inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action comprises: When continuous single-frame image data greater than a preset number of frames include the same gesture action, the same gesture action is identified as the target gesture action.

3. The method according to claim 1, characterized in that The hand shape feature extraction model includes a backbone network, and the step of inputting the single frame image data into the hand shape feature extraction model to obtain the single frame hand shape feature includes: The single-frame image data is input into the backbone network, and the single-frame hand shape features are obtained through feature extraction.

4. The method according to claim 1, characterized in that The step of determining the multi-frame sequence features based on the single-frame hand shape features corresponding to the continuously acquired single-frame image data respectively comprises: The single-frame hand shape features corresponding to the continuously acquired single-frame image data are respectively input into the time dimension encoder for time dimension encoding to obtain the multi-frame sequence features.

5. The method according to claim 1, characterized in that The step of inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action further includes: Inputting the multi-frame sequence features into the gesture action discrimination model to obtain the predicted coordinates of the gesture key points; Based on the predicted coordinates of the gesture key points, a target gesture action is determined.

6. The method according to claim 5, characterized in that The step of triggering an interactive response corresponding to the target gesture action based on the target gesture action includes: Acquire a calibration video image, and determine a mapping range of the AR display area according to the calibration video image; Calculating a mapping range of the AR display area and a mapping relationship between the display area; Based on the mapping relationship, determine a coordinate mapping result of the predicted coordinates of the gesture key points in the AR display area; Based on the coordinate mapping result of the AR display area, an interactive response corresponding to the target gesture action and the predicted coordinates of the gesture key points is triggered.

7. A gesture recognition interaction device, characterized in that: The device comprises: An image data acquisition module, used to acquire single-frame image data including gesture actions in real time; A single-frame feature acquisition module, used for inputting the single-frame image data into a hand shape feature extraction model to obtain single-frame hand shape features; A multi-frame feature acquisition module, used to determine multi-frame sequence features based on single-frame hand shape features corresponding to the continuously acquired single-frame image data; A target gesture determination module, used for inputting the multi-frame sequence features into a gesture action discrimination model to obtain a target gesture action; The target gesture interaction module is used to trigger an interaction response corresponding to the target gesture action based on the target gesture action.

8. A gesture recognition interaction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the gesture recognition interaction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition interaction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition interaction method according to any one of claims 1 to 6 are implemented.

Images