Gesture control method and device, equipment, storage medium and program product

Abstract

The invention provides a gesture control method and device, equipment, a storage medium and a program product, and the gesture control method comprises the steps: obtaining a plurality of target images containing a hand, and generating an air gesture track based on the plurality of target images; matching the air gesture track with the template track to obtain a target template track successfully matched with the air gesture track; wherein the template track is a track drawn in advance based on a plurality of preset points; and obtaining a target instruction corresponding to the target template track, and controlling the target device to execute an operation corresponding to the target instruction. The non-contact gesture control method and the non-contact gesture control device can improve the sensitivity and the accuracy of non-contact gesture control.

Description

Technical Field

[0001] The present application relates to the field of gesture recognition technology, and in particular to a gesture control method, device, equipment, storage medium and program product. Background Art

[0002] In recent years, with the emergence of smart hardware devices, users have a growing demand for more intuitive, natural and immersive interactive experiences. Contactless gesture interaction based on computer vision provides a possible solution.

[0003] Currently, common non-contact gesture recognition mainly includes the recognition of specific hand shapes, such as dynamic and static gesture recognition, and the recognition of finger movements, such as imitating contact touch screen gesture operations through finger pinching, sliding, tapping, separating and closing. After the non-contact gesture is recognized, the command control corresponding to the non-contact gesture is triggered.

[0004] However, the sensitivity and accuracy of currently common non-contact gesture controls are low. Summary of the invention

[0005] The present application provides a gesture control method, apparatus, device, storage medium and program product to improve the sensitivity and accuracy of non-contact gesture control.

[0006] According to a first aspect of an embodiment of the present application, a gesture control method is provided, including:

[0007] Acquire multiple target images including hands, and generate air gesture trajectories based on the multiple target images;

[0008] Matching the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory; wherein the template trajectory is a trajectory pre-drawn based on a plurality of preset points;

[0009] Obtain a target instruction corresponding to the target template trajectory, and control the target device to execute an operation corresponding to the target instruction.

[0010] Optionally, generating an air gesture trajectory based on the multiple target images includes:

[0011] identifying a target hand from a plurality of said target images;

[0012] Acquire the position of the target point in the target hand from the plurality of target images;

[0013] An air gesture trajectory is generated based on the positions of the target points in the multiple target images and the acquisition moments of the multiple target images.

[0014] Optionally, the identifying a target hand from the plurality of target images comprises:

[0015] From the multiple target images, a hand whose distance from the center point of the target image is less than a preset distance threshold and whose area is greater than a preset area threshold is determined as a target hand.

[0016] Optionally, before acquiring the position of the target point in the target hand from the plurality of target images, the method further comprises:

[0017] Acquire the positions of key points in the target hand from a plurality of the target images;

[0018] Based on the positions of the key points in the target hand, it is determined that the target hand performs a target action.

[0019] Optionally, the target point includes a pinch point between a thumb and an index finger;

[0020] The step of determining that the target hand performs a target action based on the position of the key point in the target hand includes:

[0021] Determining distances between key points in the target hand based on positions of the key points in the target hand;

[0022] Based on the distances between the key points in the target hand, it is determined that the thumb and index finger in the target hand perform a pinching action.

[0023] Optionally, acquiring a plurality of target images containing hands comprises:

[0024] When a hand is detected in the image, a plurality of target images including the hand are acquired at a first preset frame rate;

[0025] When it is detected that there is no hand in the image, detecting whether there is a hand in the image at a second preset frame rate;

[0026] The second preset frame rate is lower than the first preset frame rate.

[0027] Optionally, matching the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory includes:

[0028] generating a plurality of air gesture trajectory process images based on the air gesture trajectory according to the collection time of each point in the air gesture trajectory;

[0029] Acquire a plurality of template trajectory process images corresponding to the template trajectory, wherein the template trajectory process images are obtained according to the drawing order of each of the preset points in the template trajectory;

[0030] Matching each of the air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the air gesture trajectory;

[0031] Among them, the number of the air gesture trajectory process images corresponding to one air gesture trajectory is equal to the number of the template trajectory process images corresponding to one template trajectory.

[0032] Optionally, matching each of the air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the air gesture trajectory includes:

[0033] Inputting each of the mid-air gesture trajectory process images and each of the template trajectory process images into a temporal trajectory matching model, and obtaining a target template trajectory that successfully matches the mid-air gesture trajectory through the temporal trajectory matching model;

[0034] Among them, the time series trajectory matching model is trained based on sample data, and the sample data includes each sample generation trajectory process image corresponding to the sample generation trajectory, and each template trajectory process image corresponding to the template trajectory. The sample generation trajectory is a virtually generated trajectory, and the sample generation trajectory process image is obtained according to the generation time of each feature point in the sample generation trajectory.

[0035] Optionally, the method further comprises:

[0036] Based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided equally in time to generate each sample generation trajectory process image;

[0037] Alternatively, based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided into unequal time periods to generate each sample generation trajectory process image;

[0038] Alternatively, the position of each feature point in the sample generation trajectory is shifted within a preset range, and based on the shifted sample generation trajectory, each sample generation trajectory process image is generated;

[0039] Alternatively, the images of the trajectory process of each sample generation are arranged in a time-ordered order, and the images of the trajectory process of each template are arranged in a time-ordered order;

[0040] Alternatively, the images of the trajectory process of each sample generation are arranged in a time-based order from the last to the first, and the images of the trajectory process of each template are arranged in a time-based order from the last to the first;

[0041] Alternatively, a target sample generation trajectory process image in each sample generation trajectory process image is replaced by a sample generation trajectory process image preceding the target sample generation trajectory process image.

[0042] Optionally, the template trajectory is a trajectory pre-drawn based on nine preset points in the nine-square grid.

[0043] According to a second aspect of an embodiment of the present application, a gesture control device is provided, including:

[0044] A generating unit, configured to acquire a plurality of target images including a hand, and generate an air gesture trajectory based on the plurality of target images;

[0045] A matching unit, used for matching the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory; wherein the template trajectory is a trajectory pre-drawn based on a plurality of preset points;

[0046] A control unit is used to obtain a target instruction corresponding to the target template trajectory and control the target device to execute an operation corresponding to the target instruction.

[0047] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;

[0048] The memory is connected to the processor and is used to store programs;

[0049] The processor is used to implement the gesture control method as described in the first aspect by running the program in the memory.

[0050] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the gesture control method as described in the first aspect is implemented.

[0051] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the gesture control method as described in the first aspect.

[0052] In the present application, the template trajectory is a trajectory pre-drawn based on multiple preset points. The user can draw the template trajectory based on multiple preset points according to his own habits and preferences, which increases the types of template trajectories, thereby increasing the types of instructions corresponding to the template trajectory, and expanding richer gesture instructions. Moreover, the user draws the template trajectory based on multiple preset points according to his own habits and preferences, which reduces the learning cost of gesture actions and reduces the burden of gesture memory on the user. Moreover, multiple target images containing hands are obtained, and the air gesture trajectory is generated based on the multiple target images, without limiting the starting point and end point of the air gesture trajectory. Moreover, the air gesture trajectory is a recognized gesture trajectory, not a specific gesture, and is basically not affected by the user's hand shape difference and the scene. The air gesture trajectory and the template trajectory are matched to obtain a target template trajectory that successfully matches the air gesture trajectory. The template trajectory is a trajectory pre-drawn based on multiple preset points. The two trajectories are matched, which improves the sensitivity and accuracy of non-contact gesture recognition. The target instruction corresponding to the target template trajectory is obtained, and the target device is controlled to perform the operation corresponding to the target instruction. On the basis of improving the sensitivity and accuracy of non-contact gesture recognition, the sensitivity and accuracy of non-contact gesture control are further improved, and the user's interactive experience is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of an implementation environment involved in an embodiment of the present application;

[0055] Figure 2 A flowchart of a gesture control method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a process for generating air gesture trajectories based on multiple target images provided in an embodiment of the present application;

[0057] Figure 4 A flowchart of a gesture control method provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of multiple target point selection methods provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of a template trajectory definition based on a nine-square grid provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of a registration rule based on a template trajectory of a nine-square grid provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of a process of step 202 provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of multiple implementation methods of matching an air gesture trajectory with a template trajectory provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of a gesture recognition based on a nine-square grid provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram of the structure of a gesture control device provided in an embodiment of the present application;

[0065] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The current gesture recognition solutions are mainly divided into contactless gesture recognition and contact gesture recognition.

[0067] Contactless gesture recognition mainly includes the following two categories:

[0068] The first category is gesture recognition based on specific hand shapes.

[0069] Gesture recognition based on specific hand shapes, such as dynamic and static gesture recognition, will define standard actions. Different gesture actions can trigger corresponding command operations, but due to individual differences among users, it has certain memory and learning costs.

[0070] Due to the convenience of contactless gesture interaction, gesture recognition based on specific hand shapes is widely used in human-computer interaction scenarios. Traditional gesture recognition based on specific hand shapes is often considered a classification task, which is basically within 30 types. This type of technology shows wide application potential in human-computer interaction, but it still faces many challenges, including complex backgrounds, lighting changes, hand occlusion, and differences between users.

[0071] Regarding gesture recognition based on specific hand shapes, relevant studies have shown that the more categories of gesture recognition there are, the more serious the misrecognition between categories will be. More gesture categories will also bring higher learning costs and memory burdens to users.

[0072] The second category is gesture recognition based on finger movements

[0073] Gesture recognition based on finger movements is also a common way of contactless interaction, especially in augmented reality (AR) and virtual reality (VR). Finger movements are often used for fine interactions due to their natural movement patterns, such as pinching, sliding, tapping, separating and closing fingers. Finger movement recognition imitates contact touch screen gesture operations through operations such as clicking, pinching and dragging between fingers, which is often more in line with the user's natural habits and achieves different interactive feedback through different gesture movement amplitudes. Finger interaction has been studied in the contact field for a long time, but another important area of ​​its application is mid-air gestures. Although gesture recognition based on finger movements is a natural way of interaction that conforms to human intuition, its application scenarios and gesture definitions are still relatively fixed, and only work in some scenarios that require refined operations.

[0074] A common contact gesture recognition is stroke-based gesture recognition. Stroke-based gesture recognition is a common interaction method in touch screen technology, which provides users with greater freedom. This technology mostly relies on contact with the screen. The gesture trajectory is obtained by sampling points by touching the screen. The recognition of the gesture trajectory often depends on the matching of trajectory points. Researchers mainly conduct related research from two aspects: traditional machine learning feature extraction and matching and model-based trajectory matching. However, machine learning-based and matching-based algorithms are very robust to noise points. In recent years, there have also been works that introduce deep learning models to improve gesture matching effects. Due to factors such as limited sampling frequency and visual errors, there are currently few works that apply stroke-based gesture recognition to non-contact gesture recognition. Non-contact stroke-based gesture recognition has greater challenges. Stroke-based gesture recognition is more common in contact gestures, but it has not been extended to non-contact scenarios.

[0075] In order to improve the sensitivity and accuracy of non-contact gesture control, the present application provides a gesture control method, device, equipment, storage medium and program product.

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] Exemplary Implementation Environment

[0078] The gesture control method according to the embodiment of the present application can be executed by any electronic device equipped with a camera, and the electronic device equipped with a camera can be a television, a desktop computer, a laptop computer, a refrigerator, an air conditioner, a smart speaker, a vehicle-mounted device, etc., and multiple target images containing hands are directly obtained through the electronic device equipped with a camera, and an air gesture trajectory is generated based on the multiple target images, and gesture control is performed based on the air gesture trajectory. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0079] Please refer to Figure 1 , Figure 1 A schematic diagram of an implementation environment involved in an embodiment of the present application. Figure 1 The method includes a desktop computer equipped with a camera, which directly collects multiple target images including hands through the desktop computer equipped with a camera, generates air gesture trajectories based on the multiple target images, and performs gesture control based on the air gesture trajectories.

[0080] Exemplary Methods

[0081] See also Figure 2 In an exemplary embodiment, a gesture control method is provided. Figure 2 As shown, the process of the gesture control method mainly includes:

[0082] Step 201 , obtaining a plurality of target images including hands, and generating air gesture trajectories based on the plurality of target images.

[0083] The coherence of the mid-air gesture trajectory requires the system to have a high processing frame rate and stable and accurate prediction of the trajectory points. Therefore, accurate and efficient hand positioning and hand key point prediction are the key to improving the interaction effect. This application combines a large amount of public data with multi-view and hand gesture collection data, and customizes a lightweight hand detection model and hand key point prediction model to locate the hand position and multiple key points of the hand. Even in a standard CPU environment, it can maintain a processing rate of 20 frames per second.

[0084] In some embodiments, acquiring multiple target images containing hands in step 201 includes: when hands are detected in the image, acquiring multiple target images containing hands at a first preset frame rate; when hands are detected not in the image, detecting whether there are hands in the image at a second preset frame rate; wherein the second preset frame rate is less than the first preset frame rate.

[0085] For example, the first preset frame rate is 20 frames per second, and the second preset frame rate is 1 frame per second. When a hand is detected in the image, multiple target images containing the hand are acquired at 20 frames per second, thereby improving the coherence of the air gesture trajectory generated based on the multiple target images, and improving the accuracy of gesture recognition. When no hand is detected in the image, the image is detected at 1 frame per second to reduce the power consumption of the electronic device in the interactive state. When a hand is detected in the image again, multiple target images containing the hand are acquired at 20 frames per second, thereby improving the coherence of the air gesture trajectory generated based on the multiple target images, and improving the accuracy of gesture recognition.

[0086] When hands are detected in the image, multiple target images containing hands are acquired at a higher first preset frame rate, thereby improving the coherence of the air gesture trajectories generated based on the multiple target images and the accuracy of gesture recognition; when no hands are detected in the image, a lower second preset frame rate is used to detect whether there are hands in the image, thereby reducing the power consumption of the electronic device in an interactive state, thereby achieving both improving the coherence of the air gesture trajectories generated based on the multiple target images and reducing the power consumption of the electronic device in an interactive state.

[0087] In some other embodiments, acquiring multiple target images including the hand in step 201 includes: acquiring multiple target images including the hand at a first preset frame rate. Acquiring multiple target images including the hand at a fixed higher first preset frame rate improves the coherence of the air gesture trajectory generated based on the multiple target images and improves the accuracy of gesture recognition.

[0088] In some embodiments, Figure 3 As shown, in step 201, generating an air gesture trajectory based on multiple target images includes:

[0089] Step 301, identifying a target hand from a plurality of target images.

[0090] To ensure the continuity and accuracy of hand movements, the hands detected in each frame of the video stream are tracked.

[0091] In some embodiments, step 301 includes: from multiple target images, determining a hand whose distance from the center point of the target image is less than a preset distance threshold and whose hand area is greater than a preset area threshold as a target hand.

[0092] A dynamic priority selection strategy is adopted to select the hand that is close to the center point of the target image and has a larger area as the main command hand, that is, the target hand, which reduces the interference of multiple hands in the target image, improves the accuracy of the air gesture trajectory generated based on multiple target images, and improves the accuracy of gesture recognition. This strategy not only ensures the smoothness and accuracy of the interaction, but also realizes the real-time update of each frame of input data, providing a basis for lightweight hand key point prediction. Moreover, by dynamically selecting the target hand, the processing speed of gesture recognition is improved, and even in a standard CPU (Central Processing Unit) environment, a processing rate of 20 frames per second can be maintained.

[0093] In some other embodiments, step 301 includes: from a plurality of target images, determining a hand whose hand area is greater than a preset area threshold as a target hand.

[0094] Step 302, obtaining the position of the target point in the target hand from multiple target images.

[0095] In some embodiments, the position of the target point in the target hand may be directly obtained from the plurality of target images.

[0096] In other embodiments, Figure 4 As shown, before step 302, the gesture control method further includes:

[0097] Step 401, obtaining the positions of key points in the target hand from multiple target images.

[0098] In an exemplary embodiment, the positions of key points in the target hand are obtained from multiple target images, and the key points of the hand can be predicted by a hand key point prediction model. For example, 12 key points of the hand can be predicted. The processing speed of gesture recognition is improved through lightweight hand key point prediction, and a processing rate of 20 frames per second can be maintained even in a standard CPU (Central Processing Unit) environment.

[0099] In an exemplary embodiment, when the target point includes a pinch point between the thumb and the index finger, a large amount of data of the pinch state is incorporated into the prediction of the hand key points for optimization.

[0100] Step 402, based on the positions of the key points in the target hand, determine that the target hand performs the target action.

[0101] First, the positions of key points in the target hand are obtained from multiple target images, and based on the positions of the key points in the target hand, it is determined that the target hand performs the target action, and then the positions of the target points in the target hand are obtained from the multiple target images. Compared with directly obtaining the positions of the target points in the target hand from multiple target images, the processing speed of gesture recognition can be improved.

[0102] In some embodiments, Figure 5 As shown, there are many ways to select the target point, including but not limited to the following methods:

[0103] Method 1

[0104] like Figure 5 As shown in (a) in FIG. 1 , the target point includes the pinch point of the thumb and the index finger.

[0105] Method 2

[0106] like Figure 5 As shown in (b) in FIG. 1 , the target point includes the tip of the index finger.

[0107] Method 3

[0108] like Figure 5 As shown in (c) in FIG. 1 , the target point includes the center point of the fist after clenching the fist.

[0109] The pinching action of the index finger and thumb is a more natural way of finger interaction, and it is not easy to cause self-occlusion in the visual imaging process. Figure 5 (a) The gesture trajectory generation method based on the index finger and thumb pinching, Figure 5 (b) The gesture trajectory generation method based on the index finger and Figure 5 (c) The gesture trajectory generation method based on fist clenching. Figure 5 (a) The gesture trajectory generation method based on pinching the index finger and thumb is more natural and the effect is more stable.

[0110] In some embodiments, the target point includes a pinch point between the thumb and the index finger; step 402 includes: determining the distance between the key points in the target hand based on the position of the key points in the target hand; and determining that the thumb and the index finger in the target hand have performed a pinching action based on the distance between the key points in the target hand.

[0111] In an exemplary embodiment, based on the distance between key points in the target hand, it is determined that the thumb and index finger in the target hand have performed a pinching action. When the distance between the fingertip of the index finger and the fingertip of the thumb is less than a preset value, it is determined that the thumb and index finger in the target hand have performed a pinching action. When the distance between the fingertip of the index finger and the fingertip of the thumb is greater than or equal to a preset value, it is determined that the thumb and index finger in the target hand are moving away, and the trajectory is considered to have stopped when the thumb and index finger in the target hand are moving away. The user forms a complete hand trajectory by pinching the thumb and index finger - moving the hand to draw a trajectory - releasing the thumb and index finger. This action judgment method significantly improves the stability and naturalness of gesture trajectory generation.

[0112] In other embodiments, the target point includes the point where the tip of the index finger is located; step 402 includes: determining the distance between the key points in the target hand based on the positions of the key points in the target hand; and determining that the target hand has performed an index finger extending action based on the distance between the key points in the target hand.

[0113] Step 303: Generate an air gesture trajectory based on the positions of the target points in the multiple target images and the acquisition times of the multiple target images.

[0114] In an exemplary embodiment, step 303 may be to draw the trajectory of the air gesture based on the positions of the target points in the multiple target images in the order of the acquisition moments of the multiple target images.

[0115] The target hand is identified from multiple target images, the position of the target point in the target hand is obtained from the multiple target images, and the air gesture trajectory is generated based on the position of the target point in the multiple target images and the acquisition time of the multiple target images, thereby improving the accuracy of the air gesture trajectory generated based on the multiple target images and improving the accuracy of gesture recognition.

[0116] In other embodiments, generating an air gesture trajectory based on multiple target images in step 201 includes: obtaining the positions of key points in the hand from multiple target images; determining the positions of the target points in the hand based on the positions of the key points in the hand; generating an air gesture trajectory based on the positions of the target points in the multiple target images and the acquisition time of the multiple target images. Without identifying the target hand, directly identifying the positions of the key points in the hand in the multiple target images, and determining the positions of the target points in the hand based on the positions of the key points in the hand, the occurrence of missing the positions of the target points in the hand is reduced.

[0117] Step 202 : Match the mid-air gesture trajectory with the template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory.

[0118] The template trajectory is a trajectory pre-drawn based on multiple preset points.

[0119] In some embodiments, the template trajectory may be a trajectory pre-drawn based on nine preset points, or may be a trajectory pre-drawn based on sixteen preset points. The present application does not limit the number of preset points.

[0120] The template trajectory is a trajectory pre-drawn based on multiple preset points. Users can draw the template trajectory based on multiple preset points according to their own habits and preferences, which increases the types of template trajectories, and then increases the types of instructions corresponding to the template trajectories, and expands more abundant gesture instructions. Moreover, users draw template trajectories based on multiple preset points according to their own habits and preferences, which reduces the learning cost of gesture actions and alleviates the user's burden on gesture memory.

[0121] In some embodiments, the template trajectory is a trajectory pre-drawn based on nine preset points in the nine-square grid.

[0122] In the contact gesture interaction with the touch screen, the nine-square grid is a natural, convenient and personalized way of interaction. This application extends the interaction method of the nine-square grid to non-contact air gestures. Through real-time state detection of the hands by computer vision technology, trajectory generation and matching with no limit on starting and ending points can be achieved. This interaction method aims to alleviate the problems of high learning cost of gesture categories in traditional gesture interaction and the lack of scene definitions corresponding to the current refined finger movements, and to improve the naturalness and robustness of gesture interaction.

[0123] The interactive mode of the Nine-square grid is a well-known interactive mode. By increasing the possibility of pattern combinations, it effectively improves the security of matching and supports users to customize unlock patterns according to their own habits and preferences. By coding the points of the Nine-square grid row by row from 1 to 9, the trajectory can be encoded into a combination of different digital sequences, with a total of 389,112 trajectory groups.

[0124] In some embodiments, Figure 6 The figure shows a schematic diagram of the template trajectory definition based on the Nine-square grid. The Nine-square grid unlocking has three features: 1. The Nine-square grid path passes through at least four points; 2. The path does not allow skipping points that must be passed during the trajectory; 3. If the middle point is occupied, it can be skipped. Figure 6 In each of the nine-square grids in (a), (b), and (c), the first row from left to right is 1, 2, and 3, the second row from left to right is 4, 5, and 6, and the third row from left to right is 7, 8, and 9. Figure 6 In (a), only three points, 1, 2, and 9, are passed, which does not meet the requirements. Figure 6 The trajectory in (b) skips 4 points when moving from point 7 to point 1, which does not meet the requirements. Figure 6 (c) meets the requirements. Since the trajectory is triggered from 5 o'clock, it can be skipped when passing through 5 o'clock again from 4 o'clock to 6 o'clock.

[0125] The custom gesture interaction method proposed in this application refers to the design of the nine-square grid, and limits the movement trajectory of the action hand in the air within the rules of the nine-square grid. Specifically, the user first registers the familiar nine-square grid trajectory on the interactive page, and the registered gesture trajectory will be used as part of the template base library. Then perform gesture actions in the air, and match the gesture trajectory with the pre-registered nine-square grid trajectory. Whether in theory or in practical application, the types of gestures supported by this interaction method have been significantly improved. This application believes that different gesture trajectories can be defined as different gesture categories, which improves the richness and safety of gesture actions.

[0126] In some embodiments, Figure 7 As shown, it is a schematic diagram of the registration rule based on the template trajectory of the nine-square grid. Since this application aims to expand the contact nine-square grid gesture trajectory to the non-contact scenario, the non-contact interaction cannot obtain the actual size of the canvas, that is, Figure 7 Although the solutions of (a), (b), and (c) are different, they have the same hand trajectory. In order to eliminate the influence of trajectory size and starting point on contactless interaction, this application regularizes the trajectory into a standard nine-square grid space based on the coordinates of the poles of the trajectory (top, bottom, left, and right). It can be considered as a match between two images (the image of the air gesture determined based on the pole coordinates of the air gesture trajectory, and the original nine-square grid image). For example, the air gesture trajectory is Figure 7 One of (a), (b), and (c) can be considered as (c) after standardization. Figure 7 Although the solutions of (a), (b), and (c) are different, the images after normalization are the same. It is not necessary to adjust the size of the two images to be consistent. The final comparison is whether the shapes of the trajectories are consistent. Similarly, the same settings are added in the registration stage, that is, after the user registers Figure 7 After (a), no registration is allowed Figure 7 (b) or (c), Figure 7 (d) is still a registrable track. For example, in the template library, according to the preset rules, Figure 7 (a), (b), and (c) are combined into one gesture. If the user has created Figure 7 In (a), the user creates the corresponding instruction again. Figure 7 When another instruction corresponding to (b) is executed, the user will be prompted that the Figure 7 (b) The instructions corresponding to gestures of the same shape.

[0127] When the user is using it, when the trajectory generated by the main target hand is close to the registered nine-square grid template, the command can be triggered. The user pinches with two fingers and moves continuously in space. After the trajectory is drawn, release the two fingers to generate a complete trajectory.

[0128] For the generated continuous trajectory, this application proposes a time-series trajectory matching algorithm to improve robustness.

[0129] In some embodiments, Figure 8 As shown, step 202 includes:

[0130] Step 801 , generating a plurality of air gesture trajectory process images based on the air gesture trajectory according to the collection time of each point in the air gesture trajectory.

[0131] Step 802: Acquire multiple template trajectory process images corresponding to the template trajectory.

[0132] The template trajectory process image is obtained according to the drawing order of each preset point in the template trajectory.

[0133] Step 803 : Match each of the mid-air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the mid-air gesture trajectory.

[0134] The number of aerial gesture trajectory process images corresponding to an aerial gesture trajectory is equal to the number of template trajectory process images corresponding to a template trajectory.

[0135] The number of aerial gesture trajectory process images corresponding to an aerial gesture trajectory is equal to the number of template trajectory process images corresponding to a template trajectory, which facilitates the matching between each aerial gesture trajectory process image and each template trajectory process image and avoids the situation where poor matching accuracy occurs due to data imbalance.

[0136] For situations where the trajectories may be similar but the action sequences are different, the timing information is very important. We should fully consider the intermediate process information of the trajectory, match the process images of each aerial gesture trajectory with the process images of each template trajectory, and obtain the target template trajectory that successfully matches the aerial gesture trajectory, which can alleviate the confusion problem of the same pattern in different sequences.

[0137] In some embodiments, step 803 includes:

[0138] Input each aerial gesture trajectory process image and each template trajectory process image into the temporal trajectory matching model, and obtain the target template trajectory that successfully matches the aerial gesture trajectory through the temporal trajectory matching model;

[0139] Among them, the time series trajectory matching model is trained based on sample data, and the sample data includes each sample generation trajectory process image corresponding to the sample generation trajectory, and each template trajectory process image corresponding to the template trajectory. The sample generation trajectory is a virtually generated trajectory, and the sample generation trajectory process image is obtained according to the generation time of each feature point in the sample generation trajectory.

[0140] This application constructs possible gesture trajectories into a trajectory base library and matches the generated gesture trajectories with the base library. During the training process, the sample generated trajectory is a virtually generated trajectory, and no actual data collection is required. It only needs to be matched by virtually generating image trajectories, which greatly reduces the data cost. Powerful generalization capabilities can be achieved without relying on real data collection.

[0141] In an exemplary embodiment, a retrieval scheme is integrated into the dynamic gesture recognition process, and the generated trajectory is drawn into a process image in stages according to the time sequence, and the distance between the same gesture trajectory is shortened and the distance between different gesture trajectories is extended in the feature dimension through comparative learning, and the corresponding gesture is obtained using the retrieval scheme. For example, the 389,112 possible solutions of the nine-square grid trajectory are divided into training sets and test sets, of which 5,120 sets are used as test sets and the remaining 383,992 sets are used as training sets.

[0142] In some other embodiments, step 803 includes: calculating the similarity between each air gesture trajectory process image and the corresponding template trajectory process image, and obtaining the target template trajectory that successfully matches the air gesture trajectory based on the similarity.

[0143] In some embodiments, the processing methods for generating trajectory process images of each sample include but are not limited to the following methods:

[0144] Method 1

[0145] Based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided equally according to time to generate each sample generation trajectory process image.

[0146] For example, the trajectory is divided into multiple sequential trajectories at equal intervals according to the number of trajectory points after deduplication to perform image sequence matching.

[0147] Method 2

[0148] Based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided into unequal time periods to generate each sample generation trajectory process image.

[0149] During the training process, the randomness of the time segmentation points is increased to simulate the trajectories of gestures with different speeds, improve the robustness, and improve the generalization of the matching model. Time jitter is achieved mainly by generating time series data in an unequal manner for the trajectory sequence that was originally divided into equal parts in time. The key points of each group of trajectories are randomly jittered to enrich the types of samples and simulate the real trajectory drawing scene.

[0150] Method 3

[0151] The position of each feature point in the sample generation trajectory is offset within a preset range, and based on the offset sample generation trajectory, each sample generation trajectory process image is generated.

[0152] Spatial jitter is implemented. The key points of each set of trajectories are randomly jittered to enrich the types of samples and simulate the real trajectory drawing scene.

[0153] Method 4

[0154] The images of the trajectory process of each sample generation are arranged in the order of time from the earliest to the latest, and the images of the trajectory process of each template are arranged in the order of time from the earliest to the latest.

[0155] Method 5

[0156] The images of the trajectory process of each sample are arranged in the order of time from the last to the first, and the images of the trajectory process of each template are arranged in the order of time from the last to the first.

[0157] Reverse order augmentation is to reverse the order of the same track to be matched during training, and also reverse the order of the labels of the registered track. In the process of training the matching model, adding reverse order augmentation can improve the model's ability to learn sequences.

[0158] Method 6

[0159] The target sample generation trajectory process image in each sample generation trajectory process image is replaced by the sample generation trajectory process image preceding the target sample generation trajectory process image.

[0160] Temporal dropout is to randomly drop one of the temporal sequences of the matching sequential trajectory during training and use the trajectory of the previous moment instead. Adding random dropout of the intermediate process pattern simulates the gesture trajectory of different speeds, improves robustness, and improves the generalization of the matching model.

[0161] It should be noted that, of the above processing methods, only one processing method can be executed, or at least two processing methods can be combined. Of course, conflicting processing methods cannot be combined.

[0162] Different virtual data enhancements are performed on the time series trajectory matching algorithm. The above enhancement methods improve the matching effect on the virtual test set, and at the same time improve the robustness of the time series trajectory matching algorithm for actual scene applications.

[0163] In some embodiments, Fig. 9 As shown, there are multiple implementations of matching the air gesture trajectory with the template trajectory in step 202, including but not limited to the following methods:

[0164] Method 1

[0165] Fig. 9 In (a), Template Trajectory refers to template trajectory, Generated Trajectory refers to generated trajectory, Encoder refers to encoder, share refers to sharing, and Cosine Similarity refers to cosine similarity. Fig. 9 As shown in (a) in the figure, it is a matching scheme based on a single trajectory image. The template trajectory is input into the first encoder to obtain the vector corresponding to the template trajectory. The generated trajectory is input into the second encoder to obtain the vector corresponding to the generated trajectory. The parameters of the first encoder and the second encoder are shared. The cosine similarity between the vector corresponding to the template trajectory and the vector corresponding to the generated trajectory is calculated, and then the target template trajectory that successfully matches the aerial gesture trajectory is obtained.

[0166] Method 2

[0167] Fig. 9 In (b), Template Trajectory refers to template trajectory, Generated Trajectory refers to generated trajectory, Encoder refers to encoder, share refers to sharing, Cosine Similarity refers to cosine similarity, Decoder refers to decoder, and Corner Detection refers to corner detection. Fig. 9 As shown in (b), it is a matching scheme based on trajectory corner detection. The template trajectory is input into the first encoder to obtain the vector corresponding to the template trajectory. The generated trajectory is input into the second encoder to obtain the vector corresponding to the generated trajectory. The parameters of the first encoder and the second encoder are shared. The cosine similarity between the vector corresponding to the template trajectory and the vector corresponding to the generated trajectory is calculated to obtain the target template trajectory that successfully matches the aerial gesture trajectory. Corner point detection is performed on the vector generated by the generated trajectory. The position of the corner point (key point in the generated trajectory) is obtained based on the corner point detection. Adding the corner point detection part when calculating the loss during training can improve the effect of the vector generated by the generated trajectory. The template trajectory does not need to be corner-detected. The template trajectory itself is set with corner point labels. Corner point detection is only used during training, and it is not required when the model is actually used.

[0168] Method 3

[0169] Fig. 9In (c), Template Temporal Trajectory refers to the template trajectory sequence based on time sequence, Generated Temporal Trajectory refers to the generated trajectory sequence based on time sequence, Encoder refers to the encoder, share refers to sharing, Cosine Similarity refers to cosine similarity, Decoder refers to the decoder, and CornerDetection refers to corner detection. Fig. 9 As shown in (c) in the figure, it is a time-series trajectory matching algorithm. The time-series-based template trajectory sequence is input into the first encoder to obtain the vector corresponding to the time-series-based template trajectory sequence. The time-series-based generated trajectory sequence is input into the second encoder to obtain the vector corresponding to the time-series-based generated trajectory sequence. The parameters of the first encoder and the second encoder are shared. The cosine similarity between the vector corresponding to the time-series-based template trajectory sequence and the vector corresponding to the time-series-based generated trajectory sequence is calculated, and then the target template trajectory that successfully matches the air gesture trajectory is obtained. Corner point detection is performed on the vector corresponding to the time-series-based generated trajectory sequence. The position of the corner point (key point in the generated trajectory) is obtained according to the corner point detection. Adding the corner point detection part when calculating the loss during training can improve the effect of the vector corresponding to the time-series-based generated trajectory sequence. The time-series-based template trajectory sequence does not need to be corner-detected, and the template trajectory itself is set with corner point labels. Corner point detection is only used during training, and it is not required when the model is actually used. The corner point detection scheme is an additional auxiliary task and is optional. Adding the corner point detection scheme can further improve the effect, and it can also be deleted. Fig. 9 The corner point detection part in (c).

[0170] After testing, we got Fig. 9 The matching accuracy of (a), (b), and (c) in Fig. 9 The matching accuracy of the matching scheme based on a single trajectory image in (a) is 72.67%. Fig. 9 The matching accuracy of the matching solution based on trajectory corner detection (b) is 86.14%. Fig. 9 The matching accuracy of the timing trajectory matching algorithm in (c) is 97.19%. Fig. 9 The matching accuracy of the timing trajectory matching algorithm (c) is the highest.

[0171] Step 203: obtain the target instruction corresponding to the target template trajectory, and control the target device to execute the operation corresponding to the target instruction.

[0172] In an exemplary embodiment, Fig.10 , which is a schematic diagram of gesture recognition based on a nine-square grid. Fig.10In the system, users can register the nine-square template trajectory that suits their habits through contact interaction and associate it with specific functional instructions. In actual applications, the system uses visual sensors to track the user's hand movements and accurately locate the finger positions to locate the hand area and key points. The user generates the mid-air gesture trajectory by pinching the index finger and thumb and moving the hand. Subsequently, the system matches these real-time captured gesture trajectories with the user's pre-registered gesture templates. Once the match is successful, the corresponding functional instruction is triggered.

[0173] Through contact interaction on smart terminals, users pre-define and register personalized nine-square pattern gesture trajectories, and map them to specific functional instructions. A nine-square gesture trajectory registration and recognition method is proposed, which defines the action mode generated by gesture trajectories, reduces the learning cost of gesture actions, and alleviates individual action differences. It conforms to user habits and can reduce the burden of gesture memory for users. By designing the nine-square pattern to guide users to register personalized gestures, more abundant gesture instructions can be expanded through the registration and recognition of finger trajectories.

[0174] Using the timing trajectory matching algorithm, the system compares the gesture trajectory generated by the user in real time with the pre-registered gesture template. Once the trajectory matches successfully, the system immediately activates the functional instructions associated with the gesture. Since the generated gesture trajectory may be non-standard, this case uses refined hand positioning and an efficient matching algorithm based on timing trajectories to generate virtual trajectory data, which greatly reduces data costs, improves the robustness of the matching algorithm, and improves the matching sensitivity and accuracy of the drawn gesture trajectory with the registered template. Improve the sensitivity and accuracy of gestures in actual application scenarios, thereby improving user experience.

[0175] By designing a nine-square grid pattern to guide users to register personalized gestures, combined with hand tracking and time trajectory matching algorithms, natural, stable and efficient gesture recognition and command execution are promoted. While maintaining the accuracy and response speed of gesture commands, the user's interactive experience is significantly improved and the user's sense of participation is increased. This also provides new ideas for the interaction paradigm in the field of gesture recognition.

[0176] In summary, in the present application, the template trajectory is a trajectory pre-drawn based on multiple preset points. Users can draw template trajectories based on multiple preset points according to their own habits and preferences, which increases the types of template trajectories, and then increases the types of instructions corresponding to the template trajectories, and expands richer gesture instructions. Moreover, users draw template trajectories based on multiple preset points according to their own habits and preferences, which reduces the learning cost of gesture actions and reduces the burden of gesture memory on users. Moreover, multiple target images containing hands are obtained, and air gesture trajectories are generated based on multiple target images. The starting point and end point of the air gesture trajectory are not limited. Moreover, the air gesture trajectory is a recognized gesture trajectory, not a specific gesture, and is basically not affected by the hand shape differences of users and scenes. The air gesture trajectory is matched with the template trajectory to obtain a target template trajectory that successfully matches the air gesture trajectory. The template trajectory is a trajectory pre-drawn based on multiple preset points. The two trajectories are matched, which improves the sensitivity and accuracy of contactless gesture recognition. The target instructions corresponding to the target template trajectory are obtained, and the target device is controlled to perform the operations corresponding to the target instructions. On the basis of improving the sensitivity and accuracy of non-contact gesture recognition, the sensitivity and accuracy of non-contact gesture control are further improved, and the user's interactive experience is significantly improved.

[0177] Exemplary Devices

[0178] Accordingly, the embodiment of the present application also provides a gesture control device, such as Fig.11 As shown, the gesture control device includes:

[0179] A generating unit 1101 is used to obtain a plurality of target images including a hand, and generate an air gesture trajectory based on the plurality of target images;

[0180] A matching unit 1102 is used to match the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory; wherein the template trajectory is a trajectory pre-drawn based on a plurality of preset points;

[0181] The control unit 1103 is used to obtain the target instruction corresponding to the target template trajectory, and control the target device to execute the operation corresponding to the target instruction.

[0182] Optionally, the generating unit 1101 includes:

[0183] A recognition subunit, used for recognizing a target hand from a plurality of the target images;

[0184] A first acquisition subunit, configured to acquire the position of a target point in the target hand from a plurality of the target images;

[0185] The first generating subunit is used to generate an air gesture trajectory based on the position of the target point in the multiple target images and the acquisition time of the multiple target images.

[0186] Optionally, the identification subunit is specifically used to determine, from the multiple target images, a hand whose distance from the center point of the target image is less than a preset distance threshold and whose hand area is greater than a preset area threshold as a target hand.

[0187] Optionally, the generating unit 1101 further includes:

[0188] A processing subunit is used to obtain the positions of key points in the target hand from multiple target images before obtaining the positions of target points in the target hand from multiple target images; based on the positions of the key points in the target hand, determine that the target hand performs a target action.

[0189] Optionally, the target point includes a pinch point between a thumb and an index finger;

[0190] The processing subunit is specifically used to determine the distance between the key points in the target hand based on the positions of the key points in the target hand; based on the distance between the key points in the target hand, determine that the thumb and index finger in the target hand have performed a pinching action.

[0191] Optionally, the generating unit 1101 includes:

[0192] A detection subunit is used to acquire multiple target images containing hands at a first preset frame rate when a hand is detected in the image; and to detect whether there is a hand in the image at a second preset frame rate when no hand is detected in the image; wherein the second preset frame rate is less than the first preset frame rate.

[0193] Optionally, the matching unit 1102 includes:

[0194] A second generating subunit is used to generate a plurality of air gesture trajectory process images based on the air gesture trajectory according to the collection time of each point in the air gesture trajectory;

[0195] A second acquisition subunit is used to acquire a plurality of template trajectory process images corresponding to the template trajectory, wherein the template trajectory process images are obtained according to the drawing order of each of the preset points in the template trajectory;

[0196] A matching subunit, used for matching each of the air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the air gesture trajectory;

[0197] Among them, the number of the air gesture trajectory process images corresponding to one air gesture trajectory is equal to the number of the template trajectory process images corresponding to one template trajectory.

[0198] Optionally, a matching subunit is specifically used to input each of the air gesture trajectory process images and each of the template trajectory process images into a temporal trajectory matching model, and obtain a target template trajectory that successfully matches the air gesture trajectory through the temporal trajectory matching model;

[0199] Among them, the time series trajectory matching model is trained based on sample data, and the sample data includes each sample generation trajectory process image corresponding to the sample generation trajectory, and each template trajectory process image corresponding to the template trajectory. The sample generation trajectory is a virtually generated trajectory, and the sample generation trajectory process image is obtained according to the generation time of each feature point in the sample generation trajectory.

[0200] Optionally, the gesture control device further includes a processing unit, and the processing unit is specifically used for:

[0201] Based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided equally in time to generate each sample generation trajectory process image;

[0202] Alternatively, based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided into unequal time periods to generate each sample generation trajectory process image;

[0203] Alternatively, the position of each feature point in the sample generation trajectory is shifted within a preset range, and based on the shifted sample generation trajectory, each sample generation trajectory process image is generated;

[0204] Alternatively, the images of the trajectory process of each sample generation are arranged in a time-ordered order, and the images of the trajectory process of each template are arranged in a time-ordered order;

[0205] Alternatively, the images of the trajectory process of each sample generation are arranged in a time-based order from the last to the first, and the images of the trajectory process of each template are arranged in a time-based order from the last to the first;

[0206] Alternatively, a target sample generation trajectory process image in each sample generation trajectory process image is replaced by a sample generation trajectory process image preceding the target sample generation trajectory process image.

[0207] Optionally, the template trajectory is a trajectory pre-drawn based on nine preset points in the nine-square grid.

[0208] The gesture control device provided in this embodiment belongs to the same application concept as the gesture control method provided in the above embodiments of this application, can execute the gesture control method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the gesture control method. For technical details not fully described in this embodiment, please refer to the specific processing content of the gesture control method provided in the above embodiments of this application, and will not be repeated here.

[0209] The functions implemented by the above generation unit 1101, matching unit 1102 and control unit 1103 may be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0210] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0211] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0212] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0213] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0214] Exemplary Electronic Devices

[0215] An embodiment of the present application provides an electronic device, see Fig.12 As shown, the device includes:

[0216] Memory 200 and processor 210;

[0217] The memory 200 is connected to the processor 210 and is used to store programs;

[0218] The processor 210 is used to implement the gesture control method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0219] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0220] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0221] A bus may include a pathway that transfers information between components of a computer system.

[0222] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0223] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0224] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0225] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0226] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0227] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0228] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any gesture control method provided in the above embodiments of the present application.

[0229] Exemplary computer program products and storage media

[0230] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the gesture control method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0231] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0232] In addition, the embodiment of the present application may also be a storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the gesture control method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0233] Step 201 , obtaining a plurality of target images including hands, and generating air gesture trajectories based on the plurality of target images.

[0234] Step 202 : Match the mid-air gesture trajectory with the template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory.

[0235] The template trajectory is a trajectory pre-drawn based on multiple preset points.

[0236] Step 203: obtain the target instruction corresponding to the target template trajectory, and control the target device to execute the operation corresponding to the target instruction.

[0237] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0238] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0239] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0240] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0242] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0244] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0245] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0246] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0247] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture control method, characterized in that: include: Acquire multiple target images including hands, and generate air gesture trajectories based on the multiple target images; Matching the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory; wherein the template trajectory is a trajectory pre-drawn based on a plurality of preset points; Obtain a target instruction corresponding to the target template trajectory, and control the target device to execute an operation corresponding to the target instruction.

2. The gesture control method according to claim 1, characterized in that: The generating of the air gesture trajectory based on the plurality of target images comprises: identifying a target hand from a plurality of said target images; Acquire the position of the target point in the target hand from the plurality of target images; An air gesture trajectory is generated based on the positions of the target points in the multiple target images and the acquisition moments of the multiple target images.

3. The gesture control method according to claim 2, characterized in that: The identifying the target hand from the plurality of target images comprises: From the multiple target images, a hand whose distance from the center point of the target image is less than a preset distance threshold and whose area is greater than a preset area threshold is determined as a target hand.

4. The gesture control method according to claim 2, characterized in that: Before acquiring the position of the target point in the target hand from the plurality of target images, the method further includes: Acquire the positions of key points in the target hand from a plurality of the target images; Based on the positions of the key points in the target hand, it is determined that the target hand performs a target action.

5. The gesture control method according to claim 4, characterized in that: The target point includes the pinch point of the thumb and index finger; The step of determining that the target hand performs a target action based on the position of the key point in the target hand includes: Determining distances between key points in the target hand based on positions of the key points in the target hand; Based on the distances between the key points in the target hand, it is determined that the thumb and index finger in the target hand perform a pinching action.

6. The gesture control method according to claim 1, characterized in that: The acquiring of multiple target images containing hands comprises: When a hand is detected in the image, a plurality of target images including the hand are acquired at a first preset frame rate; When it is detected that there is no hand in the image, detecting whether there is a hand in the image at a second preset frame rate; The second preset frame rate is lower than the first preset frame rate.

7. The gesture control method according to claim 1, characterized in that: The step of matching the mid-air gesture trajectory with the template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory includes: generating a plurality of air gesture trajectory process images based on the air gesture trajectory according to the collection time of each point in the air gesture trajectory; Acquire a plurality of template trajectory process images corresponding to the template trajectory, wherein the template trajectory process images are obtained according to the drawing order of each of the preset points in the template trajectory; Matching each of the air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the air gesture trajectory; Among them, the number of the air gesture trajectory process images corresponding to one air gesture trajectory is equal to the number of the template trajectory process images corresponding to one template trajectory.

8. The gesture control method according to claim 7, characterized in that: The step of matching each of the air gesture trajectory process images with each of the template trajectory process images to obtain a target template trajectory that successfully matches the air gesture trajectory includes: Inputting each of the mid-air gesture trajectory process images and each of the template trajectory process images into a temporal trajectory matching model, and obtaining a target template trajectory that successfully matches the mid-air gesture trajectory through the temporal trajectory matching model; Among them, the time series trajectory matching model is trained based on sample data, and the sample data includes each sample generation trajectory process image corresponding to the sample generation trajectory, and each template trajectory process image corresponding to the template trajectory. The sample generation trajectory is a virtually generated trajectory, and the sample generation trajectory process image is obtained according to the generation time of each feature point in the sample generation trajectory.

9. The gesture control method according to claim 8, characterized in that: The method further comprises: Based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided equally in time to generate each sample generation trajectory process image; Alternatively, based on the generation time of each feature point in the sample generation trajectory, the sample generation trajectory is divided into unequal time periods to generate each sample generation trajectory process image; Alternatively, the position of each feature point in the sample generation trajectory is shifted within a preset range, and based on the shifted sample generation trajectory, each sample generation trajectory process image is generated; Alternatively, the images of the trajectory process of each sample generation are arranged in a time-ordered order, and the images of the trajectory process of each template are arranged in a time-ordered order; Alternatively, the images of the trajectory process of each sample generation are arranged in a time-based order from the last to the first, and the images of the trajectory process of each template are arranged in a time-based order from the last to the first; Alternatively, a target sample generation trajectory process image in each sample generation trajectory process image is replaced by a sample generation trajectory process image preceding the target sample generation trajectory process image.

10. The gesture control method according to any one of claims 1 to 9, characterized in that: The template trajectory is a trajectory pre-drawn based on nine preset points in the nine-square grid.

11. A gesture control device, characterized in that: include: A generating unit, configured to acquire a plurality of target images including a hand, and generate an air gesture trajectory based on the plurality of target images; A matching unit, used for matching the mid-air gesture trajectory with a template trajectory to obtain a target template trajectory that successfully matches the mid-air gesture trajectory; wherein the template trajectory is a trajectory pre-drawn based on a plurality of preset points; A control unit is used to obtain a target instruction corresponding to the target template trajectory and control the target device to execute an operation corresponding to the target instruction.

12. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the gesture control method according to any one of claims 1 to 10 by running the program in the memory.

13. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the gesture control method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to perform the gesture control method according to any one of claims 1 to 10.

Images