Method and system for recognizing gestures

Abstract

The claimed technical solution relates generally to the field of image processing, and more particularly to a method and system for recognizing gestures. The technical result of the claimed solution is an increase in the accuracy of recognition of user gestures. This technical result is achieved using a method for recognizing gestures which is implemented by at least one computing device and includes the steps of: obtaining at least two images of a user within a given period of time; identifying the region of the user's hand on each of the obtained images with the aid of a first neural network-based machine learning model trained to recognize user hand regions; recognizing the class of gesture on each obtained image with the aid of a second neural network-based machine learning model trained to recognize static gestures; forming a sequence of gestures; determining the gesture type from said sequence of gestures; and recognizing the user's gesture.

Description

METHOD AND SYSTEM FOR GESTURE RECOGNITION AREA OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of image processing, and in particular to a method and system for recognizing user gestures. LEVEL OF TECHNOLOGY

[0002] At present, the technologies of user interaction with computing devices and various systems using gestures have become widespread. Thus, such technologies are used in the automotive industry, smart home systems, video conferencing services, display devices, etc. A hand gesture can be used to interact with an electronic device without the help of any additional device. Hand gesture recognition technology is based on detecting user gestures using image capture tools and recognizing the received gesture by processing the image to interpret the recognition results into a target command. Thus, a recognized hand gesture based on computer vision can be accepted by a computer system to perform a human-machine interaction operation, such as an on / off command, an interface scrolling command, a control command, etc.

[0003] However, despite the widespread use of this technology, there are a number of problems that affect the accuracy and speed of hand gesture recognition, and, as a result, the accuracy of control of electronic devices and / or vehicles. Thus, in response to an inaccurately recognized gesture, an unintended command of the system and / or device may be executed. In addition, the speed of gesture recognition may also affect the accuracy of system control. Also, another significant problem is the ability to recognize gestures in difficult environmental conditions, such as poor lighting / glare, a large distance of the subject from the camera, etc.

[0004] Thus, a number of solutions aimed at recognizing gestures are known from the state of the art.

[0005] A keypoint detector from Google is known. This method is based on searching for the palm instead of determining the position and size of the entire hand. Once the palm is recognized, the finger movements are analyzed separately. The method recognizes the image of the hand and assigns coordinates corresponding to the location of the joints and fingers. Once the position of the hand and fingers is determined, the gesture is compared to the sign language database. As a result, a fast way of tracking and recognizing gestures is provided.

[0006] The disadvantages of this solution include the impossibility of explicit gesture classification due to the use of a keypoint-based approach and low gesture recognition accuracy in extreme environmental conditions. In addition, this solution is not intended for recognizing dynamic gestures due to the impossibility of explicit gesture localization, since this method uses a keypoint-based approach.

[0007] Also, from the state of the art, gesture recognition algorithms are known based on solving the problem of gesture classification.

[0008] The disadvantage of these solutions is the low accuracy of gesture recognition when the subject (the person showing the gesture) is moving away from the camera, due to the excessive amount of “extra” context.

[0009] Common drawbacks of existing solutions include the lack of an efficient and accurate gesture recognition method when capturing images in difficult environments, such as poor lighting / glare, large subject-to-camera distance, etc. Also, such a solution should provide high gesture recognition speed while maintaining gesture recognition accuracy. In addition, the method should be able to recognize both static gestures and dynamic gestures. DISCLOSURE OF INVENTION

[0010] The claimed technical solution proposes a new approach to user gesture recognition. This solution uses an approach that allows user gesture recognition when capturing an image with aggressive parameters of the surrounding scene, with high accuracy. [UN] This solves the technical problem of recognizing user gestures.

[0012] The technical result achieved by solving this problem is an increase in the accuracy of user gesture recognition.

[0013] An additional technical result that appears when solving the above problem is the ability to recognize static and dynamic user gestures.

[0014] The specified technical results are achieved by implementing a method for recognizing gestures, performed by at least one computing device, and comprising the steps of: a) obtaining at least two images of a user over a specified time interval; b) selecting an area of the user's hand in each obtained image using a first machine learning model based on a neural network trained to recognize areas of users' hands, wherein, during training, the following is performed: i. augmenting each image of the first training data set by changing the image parameters; iii. combining the images obtained in step i; iii. normalizing the augmented data set; c) recognizing a gesture class in each obtained image using a second machine learning model based on a neural network trained to recognize static gestures, wherein, during training, the following is performed: i.augmenting each image of the second training data set by changing the parameters of the image; j. normalizing the augmented data set; d) generating a sequence of gestures determined in step c); e) determining the type of gesture from the sequence of gestures obtained in step d), wherein, during the determination, the following is performed: i. determining the class of the first and last gesture from the sequence of gestures; j. determining the position of the first and last gesture in space; f) recognizing the user gesture based on the data obtained in step e).

[0015] In one particular embodiment of the method, recognition of the user's hand area is carried out by selecting multiple areas of the hand in the image.

[0016] In another particular embodiment of the method, the area of the user's hand is determined based on the selection of the best selected area of the user's hand in each image by comparing the selected areas of the hand for the specified image with each other.

[0017] In another particular embodiment of the method, at least two images are obtained using a photo / video recording device.

[0018] In another particular embodiment of the method, the photo / video recording device is a video camera.

[0019] In another particular embodiment of the method, the image parameters represent at least one of the following parameters: brightness, contrast, size, orientation.

[0020] In another particular embodiment of the method, combining images is the formation of a mosaic image from at least four images.

[0021] In another particular embodiment of the method, the determination of the class of the first and last gesture from a sequence of gestures is performed in a specified time period.

[0022] In another particular embodiment of the method, the specified time period does not exceed thirty frames.

[0023] In another particular embodiment of the method, determining the position of the first and last gesture in space additionally includes determining the distance between the first and last gesture of the sequence.

[0024] In another particular embodiment, the method additionally comprises the step of performing a target action corresponding to the recognized gesture.

[0025] In addition, the stated technical results are achieved through a gesture recognition system containing: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the gesture recognition method. BRIEF DESCRIPTION OF DRAWINGS

[0026] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings.

[0027] Fig. 1 illustrates a block diagram of the implementation of the claimed method.

[0028] Fig. 2 illustrates an example of hand gesture class recognition.

[0029] Fig. 3 illustrates an example of a generated dataset with reasoned images.

[0030] Fig. 4 illustrates an example of an initial and final static gesture in gesture type recognition.

[0031] Fig. 5 illustrates an example of the implementation of the non-maximal object suppression algorithm.

[0032] Fig. 6 illustrates an example of the general appearance of a computing device that ensures the implementation of the claimed solution. IMPLEMENTATION OF THE INVENTION

[0033] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0034] A model in machine learning (ML) is a set of artificial intelligence methods, the characteristic feature of which is not a direct solution to a problem, but learning in the process of applying solutions to many similar problems.

[0035] Augmentation is the process of modifying input data to increase or increase the complexity of the original data set.

[0036] A batch is a small package or set of data processed at once during one iteration of a model's operation.

[0037] The claimed technical solution offers a new approach that ensures increased gesture recognition accuracy when capturing images with a surrounding scene containing interference, such as poor lighting / glare, a large distance of the subject to the camera, etc. In addition, the claimed technical solution provides the ability to recognize static and dynamic gestures by using an algorithm for determining the gesture type. Also, this kind of solution ensures high gesture recognition speed while maintaining gesture recognition accuracy.

[0038] The claimed technical solution can be implemented on a computer, in the form of an automated information system (AIS) or a machine-readable medium containing instructions for performing the above-mentioned method.

[0039] The technical solution can also be implemented as a distributed computer system or computing device.

[0040] In this solution, the term “system” refers to a computer system, a computer (electronic computer), a numerical control (CNC), a PLC (programmable logic controller), computerized control systems, and any other devices capable of performing a given, clearly defined sequence of computing operations (actions, instructions).

[0041] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs) /

[0042] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices, such as random access memory (RAM) and / or read-only memory (ROM). ROM may include, but is not limited to, hard disk drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[0043] A program is a sequence of instructions intended for execution by a computer's control unit or command processing device.

[0044] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, encoding and decoding texts, filtering, ranking, translating texts into a dialog system, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the Python programming language, C++, Java, Python, various libraries (e.g., MFC; Microsoft Foundation Classes), etc. The instructions that perform the processes described in this solution may be transmitted either by both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[0045] Fig. 1 shows a block diagram of a method 100 for recognizing gestures, which is disclosed step by step in more detail below. Said method 100 consists of performing steps aimed at processing images obtained from image capturing devices. The processing is typically performed by a system, for example, a system 200, which can also represent, for example, a server, a computer, a mobile device, a computing device, etc. Elements of the system 200 are disclosed in more detail in Fig. 6.

[0046] The claimed technical solution can be used for gesture recognition in a human-machine interaction system. Human-machine interaction systems can be understood as interaction with an electronic device, such as a mobile computing device, a portable device, a display device, a computer, a laptop, a vehicle, smart home devices, devices equipped with a camera, etc. In addition, in another particular embodiment, the claimed technical solution can be used to control videoconferencing services. Thus, by using the said solution in videoconferencing services, it is possible to send commands by hand gestures to the said service. Thus, control commands can be commands to turn off / on the microphone, send a response to a message, etc.The specified application of the claimed technical solution, in particular, may be intended for people with hearing and speech defects and, accordingly, solve the problem of interaction of such people with the specified devices.

[0047] A gesture recognition system may be used as a system implementing the said method 100. The gesture recognition system may represent or be part of a system, such as a server, computer, intended for gesture recognition.

[0048] At step 110, a system such as system 200 receives at least two images of the user within a given time period.

[0049] Thus, at the specified step 110, for example, using image capture means, at least two images of the user are obtained over a specified time period.

[0050] The image capturing means may be, for example, a photo / video camera, an optical sensor, etc. Furthermore, in one particular embodiment, the images may be obtained, for example, from a database storing images captured by the capturing devices. In another particular embodiment, the input images may be obtained via data transmission channels, such as the Internet.

[0051] The specified time interval in the claimed solution should be understood as the time interval during which continuous capture of the scene by image capture devices occurs. Thus, in one particular embodiment, the time interval may be equal to, for example, 20, 30 frames, etc. In another particular embodiment, the time interval may be equal to 1 second, 0.5 seconds, etc.

[0052] Thus, at the specified step 110, a plurality of successive images captured by the device camera during a specified time period can be obtained.

[0053] Next, at step 120, the user's hand area is selected in each obtained image using the first machine learning model based on a neural network trained to recognize user hand areas, and, during training, the following is performed: augmentation of each image of the first training data set by changing the image parameters; combining images, normalization of the augmented data set.

[0054] At said step 120, at least two images obtained at step 110 are processed by a system, such as system 200, to extract an area of the user's hand in each image.

[0055] Thus, the specified step 120 can be performed by means of a hand localization detector built on the basis of a machine learning model trained to recognize areas of users' hands. As a machine learning model implementing the function of detecting areas of hands, a machine learning model based on the YOLOv7-HaGRID neural network, etc., can be used, without limitation.

[0056] Detection of user hand areas can be performed using detection based on framing rectangles. Thus, the specified approach is based on the selection of objects in the image by framing rectangles (boxes). The specified approach is part of the algorithms computer vision. In addition, this detection approach is used for localization through a regression task, to determine the position of an object in space.

[0057] Thus, one of the features of the declared technical solution is the division of the task of localization and classification of user hand gestures into two parallel processes. This feature provides the ability to continuously monitor the position of the user's hand through localization of the regression task to determine the position of the object in space and classification to search for the desired class of the object by a separate neural network. Due to the fact that these tasks can be solved in parallel, the overall functionality of the error in training neural networks is the sum of two components - the localization error and the classification error. Together, they allow training the neural network to search for an object of the desired class in the image. In addition, due to the implementation of such an approach to gesture recognition, the ability to recognize dynamic gestures is ensured due to the high processing speed, sincethe localization task requires much less computing resources, which, accordingly, provides the ability to continuously track the position of the user's hand. Also, one of the problems solved by the declared technical solution is the impossibility of recognizing dynamic gestures from static gestures, since video is used for dynamic gestures instead of images, i.e. such a problem is usually solved without the localization stage - only by classifying frame sequences.

[0058] To train this machine learning model and the gesture recognition machine learning model (the second machine learning model), we used a single large and heterogeneous dataset, which includes 552,992 high-resolution images, each belonging to one of 18 gesture classes: call, dislike, fist, four, like, mute, ok, one, palm, peace, peace inverted, rock, stop, stop inverted, three, three2, two, two inverted.

[0059] Each image represents one of 34,730 unique subjects (people who show a gesture), and the class of the image is determined entirely by the gesture shown. All images in the dataset are assigned a a markup containing the coordinates of a rectangle framing the hand with a gesture and labeled with one of the 18 classes. In just over 123,000 images, the subject's other hand, not showing a gesture, is also in the frame - such images correspond to a markup containing the coordinates of two framing rectangles. The rectangle corresponding to the inactive hand belongs to the 19th class "not a gesture". The number of images belonging to a particular class varies depending on the class, but is always in the range [30,121; 32,224].

[0060] All 18 gesture classes were selected for the purpose of recognition for smart home control, however, as is obvious to a person skilled in the art, they are not language-specific and serve a semiotic functional role. Such gestures help to convey information between people and are used in human-computer interaction. The relatively small lexicon compared to some other gesture data sets is intended to reduce the complexity of the gesture recognition system and reduce unnecessary cognitive load for the user of the smart home device. The selected gesture classes are also the most used and known to most.

[0061] In addition, in addition to gestures that communicate information, gestures that can manipulate objects are also needed to control electronic devices. These include various dynamic gestures, such as swipes: left, right, down, and up. With their help, users can manipulate objects on the screen (for example, by scrolling through tabs).

[0062] The dataset is the most heterogeneous in the relevant domain in terms of such characteristics as the number of unique subjects equal to 34,730; their age and gender; the number of different scenes; lighting conditions fluctuating up to extreme; the distance of subjects from the camera (from 0.5 to 4 meters).

[0063] The images were collected and labelled with a high degree of accuracy - each image was checked for compliance and labelled with framing rectangles by at least three trained people.

[0064] So, to train the first machine learning model for hand localization, i.e. to train the detector, for example, based on the YOLOv7tiny neural network on the specified dataset, the following sequence of actions was performed.

[0065] All gesture class data from the training dataset is replaced with the class “hand”.

[0066] To train this neural network, the entire training sample, consisting of 509,323 images of different classes, and the test sample, consisting of 43,669 images, were used. Before being fed to the neural network input, the images go through several successive stages of automatic transformation.

[0067] As mentioned above, one of the significant issues affecting the accuracy of gesture recognition is the extreme surrounding scene of the user in the image. As stated above, the extreme surrounding scene is an image captured in poor lighting and / or an image with highlights and / or an image in which the user's hand is far from the device's camera. To mitigate the influence of these parameters of the surrounding scene on the accuracy of gesture recognition, the training dataset was augmented with the following images.

[0068] Since the data set almost always has one person in the center in one image, in order for the model trained on such data to be able to recognize gestures in photos where there are several people or the gesture is on the frame border, several images were combined into one. For this, a mosaic of 4 images with different changes in brightness, contrast, and size was created. It is worth noting that if you skip this transformation, the model will remember the described specifics of the collected data and will work poorly on real data.

[0069] Next, the images were normalized.

[0070] This process involves bringing the pixel values of an image to a specific range, which is necessary for training models. Color images have an original pixel range of [0, 255], which are represented as 8-bit unsigned numbers. To bring images to the range of [-1, 1], it is necessary to calculate the mean and standard deviation for all pixels in the dataset. After calculation, the mean is subtracted from each pixel in the image and divided by the standard deviation:

[0071] The transformed images were then combined into batches — this is the process of combining several final examples that have undergone a preliminary automatic transformation (augmentation, normalization, image resizing) into one set for processing by the model in one pass. It allows optimizing the training speed since the model can process several examples simultaneously, which speeds up the training process and increases the efficiency of using computer resources. For training the detectors, the batch size was set to 16 — thus, 16 images were transformed into a single tensor of dimension (16, c, h, w), where c is the number of channels, h is the height of the image, w is its width. Then, the batches were fed to the neural network model for training. An example of a formed batch is shown in Fig. 3.

[0072] To evaluate the results of model training, a threshold value of the mAP (mean Average Precision) metric was used. If the metric value is too high (more than 0.9), this may indicate overtraining of the model on training data, which entails a low ability to find objects in real images. To prevent the loss of the generalization ability of the model, a threshold was set in the range [0.8, 0.9], reaching which led to a stop in training.

[0073] For each class, the arithmetic mean of the accuracy obtained for each predicted object was calculated. Accuracy is the proportion of correctly predicted objects relative to all objects that were predicted by the neural network. The mAP metric is the arithmetic mean of the accuracy of all classes that the neural network detects on the test sample. where C is the total number of classes; where D is the results of object detection; where r is the decision threshold; Rges where TP is the number of correctly predicted objects and FN is the number of required objects not predicted by the model.

[0074] The results of the trained detector for the specified metric were tAP = 74.8

[0075] Thus, by training the hand localization detector on an augmented data set, high accuracy of hand recognition in an image is ensured in conditions of noise in the surrounding scene, such as poor lighting, distance from the camera, etc. Also, by using the first machine learning model based on a neural network only for recognizing the position of the hand in the image, the possibility of continuous hand tracking is ensured.

[0076] Thus, at step 120, the selection of the user's hand area is performed on each received image.

[0077] Next, method 100 proceeds to step 130.

[0078] At the specified stage 130, the gesture type is recognized on each received image using a second machine learning model based on a neural network trained to recognize static gestures, and, during the training, the following is performed: augmentation of each image of the second training data set by changing the parameters of the image; normalization of the augmented data set.

[0079] At said step 130, in one particular embodiment, the gesture classifier receives cropped image fragments captured at step 120 to determine the class of the gesture. Thus, Fig. 2 shows an example of recognized hand gesture classes.

[0080] In another particular embodiment, the classifier was trained to predict one of 24 static gestures, containing 18 static gestures and 4 static gestures characterizing dynamic gestures. Thus, gesture classes can be such gestures as: “bell”, “dislike”, “fist”, “four”, “like”, “mute”, “ok”, “one”, “palm”, “victory”, “victory expanded”, “goat”, “stop”, “stop expanded”, “three”, “three2”, “two”, “two expanded”, “palm left”, “palm right”, “palm down”, “palm up”.

[0081] The dataset given above was used to train the classification neural network. In one particular embodiment, the gesture classifier can be built on the basis of the LeNet neural network. The following sequence of actions was performed to train the said classifier.

[0082] To prepare the dataset, in one particular implementation, images of hands were cut from the training dataset using bounding boxes.

[0083] Next, a similar approach was used to train the model as was used to train the detector at step 120, with a few differences - the images were not assembled into mosaics, but were fed one at a time to the network input, and the palm, stop, stop inverted classes were augmented with 90- and 180-degree rotations to classify hand_left, hand right, hand down gestures.

[0084] To evaluate the results of model training, the F1 metric value (the harmonic mean between recall and accuracy) was used. Precision • Recall F-, = 2 • - , Precision + Recall where Recall (recall) demonstrates the ability of the algorithm to detect a given class in general, and Precision (accuracy) is the ability to distinguish this class from other classes.

[0085] For the F1 metric, it is also necessary to set a threshold value in the range [0.8, 0.9], reaching which would lead to a stop in training.

[0086] As a result of classifier training, the specified classifier showed the value of the metric F1 = 91.8.

[0087] Thus, at step 130, recognition of gesture classes is performed on the obtained images.

[0088] Since the model often identifies several hypotheses with high confidence (the model assigns a confidence value to each prediction) pointing to the same object, in one particular embodiment, the Non-Maximum Suppression (NMS) method is used to handle such cases, which allows only one bounding box to be left, as shown in Fig. 5. However, it must be taken into account that two or more objects of the same class may be present in one image. Predictions with low confidence can be discarded using an additional threshold value t s= 0.35, this speeds up the method, because it compares each bounding box with all other boxes of the same class. The method can be represented as a formula: NMS B, S, t loU ) = { ■. Vb j€ B, i Ф / ; IoU b b bj) < t IoU ,- S t > Sj}, where B is all the bounding boxes of the class, S is the network's confidence in the class of the bounding box, t / ot / = 0.65 - threshold for preventing duplicate bounding boxes. IoU vA, B) 7 = —, лив' where A and B are the bounding frames.

[0089] Since the thresholds t s = 0.35 and t IoU = 0.65 are generally accepted and widely used, we also decided to use them.

[0090] Method 100 then proceeds to step 140.

[0091] At step 140, a sequence of gestures determined at step 130 is generated.

[0092] Thus, at said step 140, the images are combined into a sequential set. It is worth noting that said set contains the images in the order in which they were captured at step 110. In addition, in one particular embodiment, the images may be assigned serial numbers to perform parallel processing.

[0093] Thus, the specified step 140 is necessary for further recognition of the gesture type. In this solution, the gesture type should be understood as the presence or absence of a change in the position of the gesture in space in a given time period. Thus, in one particular embodiment, two types of gestures can be accepted: a dynamic gesture and a static gesture. A dynamic gesture can represent, for example, a swipe to the side, implemented by changing the position of the user's hand in a horizontal position both from left to right and vice versa, scrolling of interface elements up / down, implemented by the user by changing the position of the hand in a vertical position, etc. A static gesture can be a stop command gesture, implemented with an open palm facing the camera, a clenched fist gesture, etc.

[0094] Next, at step 150, the type of gesture is determined from the sequence of gestures obtained at step d), and, during the determination, the following is carried out: determining the class of the first and last gesture from the sequence of gestures; determining the position of the first and last gesture in space.

[0095] To recognize dynamic gestures from static images, the present technical solution uses a dynamic gesture recognition algorithm. Thus, a dynamic gesture consists of a start gesture and a start gesture. end, for example, the “swipe left” gesture (swipe an interface element, horizontal scrolling) consists of the initial “hand right” and the final “hand left”.

[0096] To check the correctness of the dynamic gesture, we developed an algorithm with a queue, which includes the following steps:

[0097] In the first step, the sequence of gestures obtained in step 140 is fed to the image processing unit. As indicated above, the sequence can be formed in a given time period. Thus, in one particular embodiment, the final gesture should be recognized no more than N = 20-30 frames after the initial gesture.

[0098] Next, the spatial position of the initial and final gestures is determined. Thus, the specified step can be performed by determining the coordinates of the framing frames (boxes) with which the specified gestures were recognized. For example, at the specified step, it is determined whether the initial and final gestures are on the same vertical or horizontal border.

[0099] In addition, in another particular embodiment, the position of the gestures relative to the captured scene is determined. Thus, in order to recognize a gesture as dynamic, the start gesture and the end gesture must be spaced relative to each other by at least one box width (for horizontal) or box height (for vertical). This step can also be performed based on the coordinates of the framing frames.

[0100] In addition, in another particular embodiment, to improve the quality of recognition and increase accuracy, 2-3 start and end gestures in a queue can be determined. Such a feature can eliminate false triggering of the gesture type classifier in the event of an unintentional movement of the user's hand and / or an unintentional capture of the user's hand that is not showing a gesture.

[0101] Accordingly, if the position of the start gesture and the end gesture coincide and the gestures are located in the same coordinates, then the gesture type is defined as static. If the start gesture differs from the end gesture, then the gesture type is defined as dynamic.

[0102] So, let us consider the indicated algorithm using the example of Fig. 4. The indicated Fig. 4 shows an example of dynamic gestures.

[0103] Thus, if the queue contains the initial gesture "hand on the right", and the final gesture is classified as "hand on the left", then the gesture type is recognized as dynamic. Moreover, it is worth noting that if the initial and final gestures are not spaced apart, then the gesture will be recognized as static. This feature is due to the fact that when a static gesture is shown by the user, the camera can capture the moment of lowering or bending the user's hand, which, accordingly, will lead to the formation of another class of final gesture in the sequence.

[0104] Accordingly, if during a given time interval the initial and final gestures coincide, then the system 200 recognizes a static gesture, for example, the “stop” gesture of an open palm, etc.

[0105] Thus, based on the specified algorithm, the type of gesture shown by the user is determined.

[0106] After the specified step 150 is completed, the queue is cleared and begins to be filled again with static gestures. The specified features allow a dynamic gesture to be recognized with a high degree of reliability and acceptable quality.

[0107] Next, at step 160, the user's gesture is recognized based on the data obtained at step 150.

[0108] At step 160, based on the type of user gesture obtained at step 150, the user gesture is recognized.

[0109] Thus, at step 160, the system 200 determines the class of the gesture entered by the user and can interpret the specified class of the gesture into a corresponding command. [IT] For example, in response to recognizing a gesture input corresponding to the class "stop", the gesture recognition system 200 can perform a target action corresponding to the specified class. In another particular embodiment, the recognized gesture in the form of a command can be sent, for example, over a communication network, to an electronic device.

[0111] Now let's look at one example of the implementation of the declared technical solution.

[0112] In one particular embodiment, the implementation of the claimed method 100 can be performed by a gesture recognition system, such as system 200. The said system comprises various functional units, which in the sets form a single system for converting the input image into recognized gestures. A color image is fed to the input of system 200, which goes to the preprocessing unit. It consists of three nodes - preliminary image conversion to the desired size, filling the image with a background (zero pixels) and normalization. In the process of converting the image, the image size is specified by the requirements for the input extension of the neural network. Typical sizes at the input of the neural network can be, for example, 224 x 224, 256 x 256, 300 x 300, 320 x 320, etc. pixels in width and height. After converting the size, image filling is applied, which adds zeros (black background) along the edges of the image, if necessary. This allows you to make all the images of the same size for further combining them into a pack (batch).In the final stage, the transformation block applies image normalization, which performs mathematical transformations on the image pixel values and converts them into the range from -1 to 1.

[0113] Next, the image is fed to the input of the processing unit, which consists of three main nodes.

[0114] A detector that is pre-trained for the localization task. Together with the NMS block, the detector outputs the coordinates of all recognized hands in the image. Then, using the predicted frame, a fragment of the found hand is cut out of the image and fed to the classifier input.

[0115] The classifier predicts one of 24 static gestures (the 18 are supplemented by “hand left”, “hand right”, “hand down”). The classifier model produces prediction probabilities for each gesture class. To remove less confident answers, a threshold device is placed after the neural network, which processes all hypotheses (predictions of objects) and leaves only the best gesture class;

[0116] a dynamic gesture recognition unit configured to recognize a gesture type based on the algorithm described in step 150.

[0117] Accordingly, after processing the received images of the user, the system recognizes the user's gesture as a command.

[0118] Fig. 6 shows an example of a general view of a computing system 200 that ensures the implementation of the claimed method or is part of a computer system, for example, a server, a personal computer, part of a computing cluster, processing the necessary data for implementing the claimed technical solution.

[0119] In general, the system 200 comprises components such as: one or more processors 201, at least one memory 202, data storage means 203, input / output interfaces 204, I / O facility 205, network communication facility 206, which are connected via a universal bus.

[0120] The processor 201 performs the basic computing operations necessary for processing the data when performing the method 100. The processor 201 executes the necessary machine-readable instructions contained in the RAM 202.

[0121] Memory 202 is usually implemented as RAM and contains the necessary software logic to provide the required functionality.

[0122] Data storage means 203 can be implemented in the form of HDD, SSD disks, RAID array, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. Means 203 allow long-term storage of various types of information, such as images of user gestures, etc.

[0123] To organize the operation of the components of the 200 system and to organize the operation of external connected devices, various types of I / O interfaces are used. 204. The choice of appropriate interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0124] The choice of interfaces 204 depends on the specific implementation of system 200, which can be implemented on the basis of a wide range of devices, for example, a personal computer, mainframe, laptop, server cluster, thin client, smartphone, server, etc.

[0125] The following may be used as I / O data means 205: keyboard, joystick, display (touch display), monitor, touch display, touchpad, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[0126] Network interaction means 206 are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, a WLAN / Wi-Fi module, a Bluetooth module, a BLE module, an NFC module, IrDa, an RFID module, a GSM modem, etc. With the help of means 205, data exchange is organized between, for example, a system 200, represented in the form of a server, and a user's computing device, on which the received data can be displayed via a wired or wireless data transmission channel, for example, a WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[0127] The specific selection of elements of system 200 for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0128] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology. Thus, the scope of the present technical solution is limited only by the scope of the attached formula.

Claims

FORMULA 1. A method for recognizing gestures, performed by at least one computing device, and comprising the steps of: a) obtaining at least two images of a user over a given time interval; b) selecting a region of the user's hand in each obtained image using a first machine learning model based on a neural network trained to recognize regions of users' hands, wherein, during training, the following is performed: i. augmenting each image of the first training data set by changing the image parameters; ii. combining the images obtained in step i; iii. normalizing the augmented data set; c) recognizing a gesture class in each obtained image using a second machine learning model based on a neural network trained to recognize static gestures, wherein, during training, the following is performed: i.augmenting each image of the second training data set by changing the parameters of the image; ii. normalizing the augmented data set; d) generating a sequence of gestures determined in step c); e) determining the type of gesture from the sequence of gestures obtained in step d), wherein, during the determination, the following is performed: i. determining the class of the first and last gesture from the sequence of gestures; ii. determining the position of the first and last gesture in space; f) recognizing the user gesture based on the data obtained in step e).

2. The method according to claim 1, characterized in that the recognition of the user's hand area is carried out by selecting a plurality of hand areas in the image.

3. The method according to item 2, characterized in that the area of the user's hand is determined based on the selection of the best selected area of the user's hand on each image by comparing the selected areas of the hand for the specified image with each other.

4. The method according to claim 1, characterized in that at least two images are obtained by means of a photo / video recording device.

5. The method according to paragraph 2, characterized in that the photo / video recording device is a video camera.

6. The method according to claim 1, characterized in that the image parameters represent at least one of the following parameters: brightness, contrast, size, orientation.

7. The method according to claim 1, characterized in that combining images is the formation of a mosaic image from at least four images.

8. The method according to item 1, characterized in that the determination of the class of the first and last gesture from the sequence of gestures is performed in a specified time period.

9. The method according to item 8, characterized in that the specified time period does not exceed thirty frames.

10. The method according to claim 1, characterized in that determining the position of the first and last gesture in space additionally includes determining the distance between the first and last gesture of the sequence. I. The method according to paragraph 1, characterized in that it additionally contains the stage of performing a target action corresponding to the recognized gesture.

12. A gesture recognition system comprising: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the method according to any one of claims 1-11.

Images