Gesture recognition interaction method and device, equipment, storage medium and product

Abstract

The invention discloses a gesture recognition interaction method and device, equipment, a storage medium and a product, and relates to the technical field of man-machine interaction, and the method comprises the steps: determining a finger key point prediction coordinate based on a palm positioning result and a finger key point prediction model through obtaining single-frame image data including a single finger stamp action in real time, the palm positioning result is obtained by inputting single-frame image data into the finger key point prediction model for calculation, and according to a coordinate mapping result of the finger key point prediction coordinate in the AR display area, triggering an interaction response corresponding to the single finger stamp action. Through interaction between the single-finger stamp and the display control, the convenience and intuition of interaction are greatly improved. Palm positioning and finger key point prediction are fused into one model to be completed at a time, so that the operand is reduced while the intuition and accuracy of gesture recognition interaction are improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to gesture recognition interaction methods, devices, equipment, storage media and products. Background Art

[0002] With the rapid development of artificial intelligence technology, AR (augmented reality) technology has made significant progress. At present, AR technology can not only seamlessly superimpose virtual information into the real world to provide users with a richer visual experience, but also achieve deep integration with the real environment through intelligent recognition, tracking and interaction technology.

[0003] Among the existing gesture interaction algorithms for AR scenarios, some designs require the user to make a fist, use the thumb as a pointer, project a ray on the display screen to simulate cursor movement, and then implement click operations by pinching the thumb (such as the method used by Pico4). Another method is to interact with the controls on the display screen through the coordinated pinching action of the eyes, index finger and thumb (such as the implementation in AVP). However, these methods lack intuitiveness, resulting in low positioning accuracy in gesture interaction recognition.

[0004] Therefore, how to improve the accuracy of gesture recognition interaction is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present application is to provide a gesture recognition interaction method, device, equipment, storage medium and product, aiming to solve the technical problem of low positioning accuracy in gesture interaction recognition.

[0006] To achieve the above objectives, the present application proposes a gesture recognition interaction method, which is applied to an AR device, and the method includes:

[0007] Acquire single-frame image data including single-finger poking action in real time;

[0008] Determine the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is calculated by inputting the single frame image data into the finger key point prediction model;

[0009] According to the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area, an interactive response corresponding to the single-finger poking action is triggered.

[0010] In one embodiment, the finger key point prediction model includes a backbone network, a positioning head network and a key point head network. The step of determining the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is obtained by inputting the single frame image data into the finger key point prediction model for calculation, includes:

[0011] Inputting the single-frame image data into the backbone network, and obtaining single-frame image features by forward reasoning;

[0012] Determine multi-frame sequence features based on single-frame image features respectively corresponding to the continuously acquired single-frame image data;

[0013] Inputting the multi-frame sequence features into the positioning head network to obtain a palm positioning result corresponding to the multi-frame sequence features;

[0014] Acquire palm features from the multi-frame sequence features according to the palm positioning result;

[0015] The palm features are input into the key point network to obtain the predicted coordinates of the finger key points.

[0016] In one embodiment, the step of inputting the palm features into the key point network to obtain predicted coordinates of finger key points includes:

[0017] Inputting the palm features into the key point network to obtain the predicted coordinates of the key points corresponding to the continuously acquired single frame image data;

[0018] Based on the predicted coordinates of the key points respectively corresponding to the continuously acquired single-frame image data, the predicted coordinates of the key points of the finger are obtained.

[0019] In one embodiment, the step of acquiring palm features from the multi-frame sequence features according to the palm positioning result includes:

[0020] Mapping the palm positioning result to the multi-frame sequence feature, and determining a palm region of interest on the multi-frame sequence feature;

[0021] The palm region of interest is divided into grids, and features are extracted in the grids to obtain palm features.

[0022] In one embodiment, the step of triggering an interactive response corresponding to the single-finger poking action according to a coordinate mapping result of the predicted coordinates of the finger key point in the AR display area includes:

[0023] Acquire a calibration video image, and determine a mapping range of the AR display area according to the calibration video image;

[0024] Calculating a mapping range of the AR display area and a mapping relationship of the AR display area to obtain a coordinate mapping;

[0025] Based on the coordinate mapping, a coordinate mapping result of the predicted coordinates of the finger key point in the AR display area is determined.

[0026] In one embodiment, the step of determining the mapping range of the AR display area according to the calibrated video image includes:

[0027] Traversing each frame of the calibrated video image;

[0028] Identify the fingertip position in each frame of the image, and track the movement trajectory of the finger along the edge of the display area;

[0029] A mapping range of the AR display area is determined according to the motion trajectory.

[0030] In addition, to achieve the above purpose, the present application also proposes a gesture recognition interaction device, the gesture recognition interaction device comprising:

[0031] A data acquisition module, used for acquiring single-frame image data including a single-finger poking action in real time;

[0032] A coordinate prediction module, used to determine the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is calculated by inputting the single frame image data into the finger key point prediction model;

[0033] The gesture interaction module is used to trigger an interactive response corresponding to the single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a gesture recognition interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the gesture recognition interaction method described above.

[0035] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition interaction method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition interaction method described above are implemented.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] Acquire single-frame image data including single-finger poking actions in real time, and determine the predicted coordinates of the finger key points based on the palm positioning results and the finger key point prediction model, wherein the palm positioning results are calculated by inputting the single-frame image data into the finger key point prediction model. Trigger the interactive response corresponding to the single-finger poking action based on the coordinate mapping results of the predicted coordinates of the finger key points in the AR display area. Interacting with the display control through single-finger poking greatly improves the convenience and intuitiveness of the interaction. Integrating palm positioning and finger key point prediction into one model at one time can improve the intuitiveness and accuracy of gesture recognition interaction, while also reducing the amount of calculation and inference time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a flowchart of the first embodiment of the gesture recognition interaction method of the present application;

[0042] Figure 2 This is a flow chart of the second embodiment of the gesture recognition interaction method of the present application;

[0043] Figure 3 This is a schematic diagram of the MobileOne network structure of the embodiment of the present application;

[0044] Figure 4 This is a flowchart of the third embodiment of the gesture recognition interaction method of the present application;

[0045] Figure 5 This is a schematic diagram of the module structure of the gesture recognition interaction device according to an embodiment of the present application;

[0046] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the gesture recognition interaction method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] In the existing AR scene gesture interaction algorithms, some designs require users to make a fist, use their thumb as a pointer, project a ray on the display screen to simulate cursor movement, and then perform click operations by pinching the thumb (such as the method used by Pico4). Another way is to interact with the controls on the display screen through the coordinated pinching of the eyes, index finger, and thumb (such as the implementation in AVP). These methods are not intuitive enough, resulting in low accuracy in gesture interaction recognition and positioning. At the same time, the calculation and reasoning require a relatively large amount of computing power, which may affect the battery life and performance of AR devices worn all day.

[0051] The present application provides a solution, which determines the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model by acquiring single-frame image data including the single-finger poking action in real time, wherein the palm positioning result is calculated by inputting the single-frame image data into the finger key point prediction model, and triggers the interactive response corresponding to the single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key points in the AR display area. The interaction with the display control by single-finger poking greatly improves the convenience and intuitiveness of the interaction. The integration of palm positioning and finger key point prediction into one model can improve the intuitiveness and accuracy of gesture recognition interaction while reducing the amount of calculation and reasoning time.

[0052] Based on this, the embodiment of the present application provides a gesture recognition interaction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the gesture recognition interaction method of the present application.

[0053] In this embodiment, the gesture recognition interaction method includes steps S10 to S30:

[0054] Step S10, acquiring single-frame image data including a single-finger poking action in real time.

[0055] It should be noted that the single-finger poking action can be understood as an action of a user clicking on an AR display area when using an AR (Augmented Reality) device.

[0056] In some embodiments, a terminal (e.g., an AR terminal) obtains single-frame image data including a single-finger poking action captured by a camera in real time. Exemplarily, the terminal obtains video data including a single-finger poking action captured by a camera. The camera can be integrated on an AR glasses device, and the obtained video data can be parsed by a computer device (the computer device can include a display screen), and the single-frame image data is obtained by parsing the video data of the single-finger poking action.

[0057] Step S20, determining the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model.

[0058] In some embodiments, the palm positioning result is calculated by inputting single-frame image data into a finger key point prediction model.

[0059] It should be noted that the finger key point prediction model can be a pre-trained machine learning model that can receive an image containing a gesture as input and output the predicted coordinates of specific key points in the gesture. These key points usually include fingertips, knuckles, etc. The palm positioning result can be understood as the positioning of the palm by the model during the finger key point prediction process, that is, identifying the position of the palm in the camera image (e.g., single frame image data). The predicted coordinates of the finger key points are mapped to the AR display area for subsequent interactive responses.

[0060] Step S30, triggering an interactive response corresponding to a single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key points in the AR display area. It can be understood that the interactive response occurs at the position mapped to the predicted coordinates of the finger key points in the AR display area. For example, the predicted coordinates of the finger key points are the fingertip coordinates of the finger (for example, the index fingertip), and the interactive response of the single-finger poking action occurs at the position mapped to the predicted coordinates of the fingertip in the AR display area. It should be noted that the AR display area refers to the virtual information display area created by augmented reality (AR) technology in the physical world. In this area, virtual objects can be integrated with the scenes of the real world, which can provide users with a more immersive experience. In an AR device (such as AR glasses), the AR display area can be any part or all of the field of view of the device. Coordinate mapping can be a process of converting the predicted coordinates of the finger key points from the camera image coordinate system to the AR display area coordinate system, and the coordinate mapping result is the specific position of the finger key points in the AR display area, so that the virtual object can be accurately adjusted and interactively fed back according to the user's gesture. The interactive response can include operations such as moving, rotating, scaling, selecting, activating, and other virtual objects, as well as related feedback effects such as sound and animation.

[0061] In this embodiment, a single-frame image data including a single-finger poking action is acquired in real time, and the predicted coordinates of the finger key points are determined based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is calculated by inputting the single-frame image data into the finger key point prediction model. The above interaction with the display control through single-finger poking greatly improves the convenience and intuitiveness of the interaction, and by integrating palm positioning and finger key point prediction into one model, the intuitiveness and accuracy of gesture recognition interaction can be improved.

[0062] It should be noted that the finger key point prediction model includes a backbone network, a positioning head network, and a key point head network. The backbone network can be understood as a main network, which refers to a feature extraction network used to extract information from the image for use by the subsequent network. The backbone network can be a VGG model and a ResNet (residual network).

[0063] Reference Figure 2 , Figure 2 This is a flow chart of the second embodiment of the gesture recognition interaction method of the present application. Figure 1 The first embodiment shown is a second embodiment of the gesture recognition interaction method of the present application.

[0064] In the second embodiment, the step S20 includes:

[0065] Step S201, inputting single-frame image data into the backbone network, and obtaining single-frame image features by forward reasoning.

[0066] For example, Figure 3 As shown, MobileOne is used as the backbone network of the model, and the preprocessed single-frame image data is sent as input to the convolution layer of the MobileOne model. The MobileOne model can be composed of multiple convolutional layers, ReLU activation functions, pooling layers, and fully connected layers. Exemplarily, the video frame image is scaled to 224x128 size and input into the network. The network passes through network block 1 (3 convolutional layers, downsampled 4 times) to obtain feature map F1, the network passes through network block 2 (8 convolutional layers, further downsampled 2 times) to obtain feature map F2, the network passes through network block 3 (12 convolutional layers, further downsampled 2 times) to obtain feature map F3, the network passes through network block 4 (2 convolutional layers, further downsampled 2 times) to obtain feature map F4, and the F1-F4 feature maps are pooled into 1x1 size and spliced ​​to fuse the deep and shallow features of the model, which is conducive to feature expression. The concatenated features are flattened and encoded into a 128-dimensional feature vector, with each dimension of size 1x2380, and the final single-frame image features are used as the input of the subsequent network.

[0067] Step S202, determining multi-frame sequence features based on single-frame image features corresponding to the continuously acquired single-frame image data.

[0068] Step S203: input the multi-frame sequence features into the positioning head network to obtain the palm positioning results corresponding to the multi-frame sequence features.

[0069] It should be noted that the multi-frame sequence feature side can represent a feature representation that can contain more extensive context information and is constructed by feature fusion based on the single-frame image features corresponding to the continuously acquired single-frame image data. It can be understood that each frame in the continuously acquired single-frame image data can obtain the corresponding single-frame image features through step S201.

[0070] The localization head network can be a neural network layer or module specially designed for target localization. It receives sequence features as input and outputs the localization result of the target. Exemplarily, the output of the localization head network is a 4-element vector, which respectively represents the coordinates of the upper left vertex and the lower right vertex of the palm localization box.

[0071] Step S204, obtaining palm features from the multi-frame sequence features according to the palm positioning result.

[0072] It should be noted that step S204 includes: mapping the palm positioning result to the multi-frame sequence feature and determining the palm region of interest on the multi-frame sequence feature; dividing the palm region of interest into grids and performing feature extraction in the grids to obtain palm features.

[0073] Exemplarily, fine feature extraction can be performed on the palm ROI area on the multi-frame sequence features, including dividing the ROI area into several small grids (bins), using an interpolation method (such as bilinear interpolation) in each grid to calculate the pixel value at any position, and performing a pooling operation (such as maximum pooling) on ​​the pixel values ​​in each grid to obtain a feature vector of a fixed size.

[0074] Step S205: input the palm features into the key point network to obtain the predicted coordinates of the finger key points.

[0075] It should be noted that step S205 includes: dividing the palm region of interest into grids, and extracting features in the grids to obtain palm features. Then, the palm features are input into the key nod network to obtain the predicted coordinates of the finger key points.

[0076] In some embodiments, the palm features are input into the key point head network to obtain the key point prediction coordinates corresponding to the continuously acquired single frame image data. Based on the key point prediction coordinates corresponding to the continuously acquired single frame image data, the finger key point prediction coordinates are obtained. When the end of the single-finger poking action is detected, the key point prediction coordinates corresponding to this time are determined to be the finger key point prediction coordinates. For example, 20 frames of image data are continuously acquired, and 20 groups of key point prediction coordinates can be obtained by inputting the palm features into the key point head network. It can be understood that since the frame image data is continuously acquired, the 20 groups of key point prediction coordinates also change continuously. When a change in the change trend is detected at any group of key point prediction coordinates in the 20 groups of key point prediction coordinates, it is confirmed that the single finger poking action here is over. For example, during the single finger poking action, the key point prediction coordinates move approximately in a straight line direction. When a change in direction occurs at and after a certain group of key point prediction coordinates or the position remains approximately unchanged, it is considered that a change in the change trend has occurred.

[0077] It should be noted that the keypoint network can be a neural network layer or module specially designed to detect key points in an image (such as human joints, fingertips, etc.). It receives palm features as input and outputs predicted coordinates of finger key points.

[0078] Exemplarily, MobileOne is used as the backbone network of the model, and two Transformer models are connected in parallel at the output end of the backbone network as the head network. Each head network is finally connected to a fully connected layer as the output layer. The output of the key point head network can be a 42-element vector, representing the coordinates of the 21 key points of the whole hand.

[0079] This embodiment inputs single-frame image data into the backbone network, uses forward reasoning to obtain single-frame image features, determines sequence features based on single-frame image features, and inputs the positioning head network according to the sequence features to obtain positioning results, obtains palm features from the sequence features according to the positioning results, and inputs the palm features into the key point head network to obtain the predicted coordinates of the finger key points. By inputting single-frame image data into the backbone network, rich spatial features can be efficiently extracted from the single-frame image. By extracting and fusing sequence features, the system can better cope with these complex scenes and improve the robustness of detection.

[0080] Reference Figure 4 , Figure 4 This is a flow chart of the third embodiment of the gesture recognition interaction method of the present application. Figure 2 The second embodiment shown provides a third embodiment of the gesture recognition interaction method of the present application.

[0081] In the third embodiment, the step S30 includes:

[0082] Step S301: Acquire a calibration video image, and determine a mapping range of an AR display area according to the calibration video image.

[0083] It should be noted that the step of determining the mapping range of the AR display area based on the calibrated video image includes: traversing each frame image in the calibrated video image; identifying the fingertip position in each frame image, and tracking the movement trajectory of the finger along the edge of the display area; and determining the mapping range of the display area based on the movement trajectory.

[0084] For example, you can turn on the AR glasses recording function, move your index finger along the edge of the display area, and then export the video. The movement range of the finger in the image is recorded to determine the mapping range of the AR display area.

[0085] Step S302 : calculating a mapping range of the AR display area and a mapping relationship of the AR display area to obtain a coordinate mapping.

[0086] It should be noted that coordinate mapping can be understood as calibrating the camera imaging and the image of the AR display area. Exemplarily, based on the movement trajectory of the finger in the video, the area corresponding to the AR display area in the camera image (for example, the calibrated video image) is determined as the mapping range of the AR display area. Then the ratio of the length and width of the two areas is calculated, and a linear transformation matrix is ​​constructed based on the calculated length-to-width ratio. The matrix is ​​used to convert the coordinate system in the camera image into the coordinate system on the AR display area. By adjusting the tracking algorithm parameters or the parameters of the linear transformation matrix, the predicted coordinates of the finger key points are calibrated, and the calibration results can be understood as the calibrated predicted coordinates of the finger key points.

[0087] Step S303: Based on the coordinate mapping, determine the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area.

[0088] For example, the coordinates of the calibrated finger key points can be predicted to realize the interaction between the gesture and the AR display area. It is understandable that the mapping algorithm can also be integrated into the application of AR glasses to realize real-time hand key point mapping and interaction.

[0089] In this embodiment, by acquiring the calibration video image and determining the mapping range of the display area, the accurate position of the predicted coordinates of the finger key points in the AR display area can be ensured. This precise mapping provides a reliable basis for subsequent interactive responses. The mapping range of the AR display area and the ratio of the AR display area are calculated, and the predicted coordinates of the finger key points are calibrated according to the ratio, which helps to eliminate the coordinate offset caused by the change in the size of the display area, further improve the accuracy of the interaction, and trigger the corresponding interactive response based on the calibration results according to the coordinate mapping results of the predicted coordinates of the finger key points in the AR display area, so that users can interact with the AR environment through natural gestures, thereby improving the fluency of the interaction.

[0090] In one embodiment, the training process of the finger key point prediction model may include: obtaining video data of a single-finger poking action, and annotating the palm position and finger key point coordinates in each frame of the image frame by frame to generate a training data set; inputting the training data set into the finger key point prediction model to obtain predicted values ​​of the palm position and the finger key point coordinates; based on the training data set, calculating the loss function according to the true value and the predicted value, if the loss function is less than a preset loss threshold, determining that the finger key point prediction model has converged; if the loss function is greater than or equal to the preset loss threshold, updating the parameters of the finger key point prediction model through a back propagation algorithm according to the gradient of the loss function, and returning to the step of inputting the training data set into the finger key point prediction model.

[0091] Exemplarily, model training can be performed offline, and the model can be trained using classic forward reasoning and gradient back propagation algorithms. During the training process, each frame of the image is sent to the backbone network to obtain frame features, and then the features of the past 20 frames are spliced ​​to obtain sequence features, which are sent to the positioning head network and the key point head network to obtain the palm positioning coordinates and key point coordinates predicted by the model. Use CIOU loss to supervise the training of palm positioning, use the ROI Align method to determine the corresponding area of ​​the palm area on the backbone output feature according to the positioning coordinates, and send the features of the area to the key point head network for inference to obtain the prediction results of the key points. Use RLE loss to supervise the training of key points, and train a total of 100 iterations.

[0092] In this embodiment, a high-quality training data set can be generated by annotating the coordinates of the palm position and finger key points in the video data frame by frame. This fine annotation ensures that the model can access samples that are as real as possible during the learning process, thereby improving the accuracy of the model in the prediction stage. During the training process, by constantly comparing the actual value with the predicted value and calculating the error based on the loss function, the model gradually adjusts its internal parameters through reverse gradient propagation to reduce this error. When the loss function is less than the preset threshold, it indicates that the model has learned the distribution law of the training data well, thereby improving the accuracy of its prediction of finger key points.

[0093] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the gesture recognition interaction method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0094] This application also provides a gesture recognition interaction device, please refer to Figure 5 , the gesture recognition interaction device comprises:

[0095] The data acquisition module 10 is used to acquire single-frame image data including a single-finger poking action in real time;

[0096] A coordinate prediction module 20, for determining predicted coordinates of finger key points based on a palm positioning result and a finger key point prediction model, wherein the palm positioning result is calculated by inputting the single frame image data into the finger key point prediction model;

[0097] The gesture interaction module 30 is used to trigger an interactive response corresponding to the single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area.

[0098] The gesture recognition interaction device provided by the present application adopts the gesture recognition interaction method in the above embodiment, which can solve the technical problem of low positioning accuracy in gesture interaction recognition. Compared with the prior art, the beneficial effects of the gesture recognition interaction device provided by the present application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, and other technical features in the gesture recognition interaction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0099] The present application provides a gesture recognition interaction device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture recognition interaction method in the above-mentioned embodiment 1.

[0100] Reference below Figure 6, which shows a schematic diagram of the structure of a gesture recognition interaction device suitable for implementing the embodiment of the present application. The gesture recognition interaction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The gesture recognition interaction device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0101] like Figure 6 As shown, the gesture recognition interaction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the gesture recognition interaction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the gesture recognition interaction device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The gesture recognition interaction device with various systems is shown in the figure, but it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0102] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0103] The gesture recognition interaction device provided by the present application adopts the gesture recognition interaction method in the above embodiment, which can solve the technical problem of low positioning accuracy in gesture interaction recognition. Compared with the prior art, the beneficial effects of the gesture recognition interaction device provided by the present application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, and other technical features in the gesture recognition interaction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0106] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the gesture recognition interaction method in the above-mentioned embodiment.

[0107] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0108] The computer-readable storage medium may be included in the gesture recognition interaction device; or may exist independently without being assembled into the gesture recognition interaction device.

[0109] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the gesture recognition interaction device, the gesture recognition interaction device: obtains single-frame image data including a single-finger poking action in real time; determines the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, and the palm positioning result is calculated by inputting the single-frame image data into the finger key point prediction model; triggers an interactive response corresponding to the single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key points in the AR display area.

[0110] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0113] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned gesture recognition interaction method, and can solve the technical problem of low positioning accuracy in gesture interaction recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the gesture recognition interaction method provided in the above-mentioned embodiment, and will not be repeated here.

[0114] The present application also provides a computer program product, including a computer program, which implements the steps of the gesture recognition interaction method as described above when executed by a processor.

[0115] The computer program product provided by the present application can solve the technical problem of low positioning accuracy in gesture interaction recognition. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the gesture recognition interaction method provided by the above embodiment, which will not be repeated here.

[0116] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A gesture recognition interaction method, characterized in that: Applied to an AR device, the method includes: Real-time acquisition of single-frame image data including single-finger poking action; Determine the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is calculated by inputting the single frame image data into the finger key point prediction model; According to the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area, an interactive response corresponding to the single-finger poking action is triggered.

2. The method according to claim 1, characterized in that The finger key point prediction model includes a backbone network, a positioning head network and a key point head network. The step of determining the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is obtained by inputting the single frame image data into the finger key point prediction model for calculation, includes: Inputting the single-frame image data into the backbone network, and obtaining single-frame image features by forward reasoning; Determine multi-frame sequence features based on single-frame image features respectively corresponding to the continuously acquired single-frame image data; Inputting the multi-frame sequence features into the positioning head network to obtain a palm positioning result corresponding to the multi-frame sequence features; Acquire palm features from the multi-frame sequence features according to the palm positioning result; The palm features are input into the key point network to obtain the predicted coordinates of the finger key points.

3. The method according to claim 2, characterized in that The step of inputting the palm features into the key nod network to obtain predicted coordinates of finger key points includes: Inputting the palm features into the key point network to obtain the predicted coordinates of the key points corresponding to the continuously acquired single frame image data; The predicted coordinates of the key points of the finger are obtained based on the predicted coordinates of the key points corresponding to the continuously acquired single-frame image data.

4. The method according to claim 2, characterized in that The step of acquiring palm features from the multi-frame sequence features according to the palm positioning result comprises: Mapping the palm positioning result to the multi-frame sequence feature, and determining a palm region of interest on the multi-frame sequence feature; The palm region of interest is divided into grids, and features are extracted in the grids to obtain palm features.

5. The method according to any one of claims 1 to 4, characterized in that The step of triggering an interactive response corresponding to the single-finger poking action according to a coordinate mapping result of the predicted coordinates of the finger key point in the AR display area includes: Acquire a calibration video image, and determine a mapping range of the AR display area according to the calibration video image; Calculating a mapping range of the AR display area and a mapping relationship of the AR display area to obtain a coordinate mapping; Based on the coordinate mapping, a coordinate mapping result of the predicted coordinates of the finger key point in the AR display area is determined.

6. The method according to claim 5, characterized in that The step of determining the mapping range of the AR display area according to the calibrated video image includes: Traversing each frame of the calibrated video image; Identify the fingertip position in each frame of the image, and track the movement trajectory of the finger along the edge of the display area; A mapping range of the AR display area is determined according to the motion trajectory.

7. A gesture recognition interaction device, characterized in that: The device comprises: A data acquisition module, used for acquiring single-frame image data including a single-finger poking action in real time; A coordinate prediction module, used to determine the predicted coordinates of the finger key points based on the palm positioning result and the finger key point prediction model, wherein the palm positioning result is calculated by inputting the single frame image data into the finger key point prediction model; The gesture interaction module is used to trigger an interactive response corresponding to the single-finger poking action according to the coordinate mapping result of the predicted coordinates of the finger key point in the AR display area.

8. A gesture recognition interaction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the gesture recognition interaction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition interaction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition interaction method according to any one of claims 1 to 6 are implemented.

Images