System and Method for Recognizing Hand Gesture based on Camera Vision

Abstract

The present invention relates to a gesture-based user interface, and more specifically, to a system and method for recognizing hand pointer gestures and hand mouse gestures using a MediaPipe Hands Model. According to the present invention, a hand is detected using a MediaPipe Hands Model and a static pose is detected using feature points, and hand pointer gestures and hand mouse gestures are recognized according to the detected dynamic gestures. Furthermore, various static poses are detected using feature points (landmarks) which are the output values ​​of the MediaPipe Hands Model, and dynamic gestures are detected through the movement and change of static poses, thereby providing good intuitiveness and convenience.

Description

Technology Field

[0001] The present invention relates to a gesture-based user interface, and more specifically, to a camera vision-based hand gesture recognition system and method using a mediapipe hands model. Background Technology

[0002] User interfaces are evolving in a direction of convergence and intelligence that eliminates the point of contact between humans and machines and autonomously grasps human intentions; among these advancements, fully inputting user commands through intuitive and natural gestures has been a long-cherished goal.

[0003] As a natural method of communication and information transmission, hands are widely used as input devices in digital devices where traditional input devices such as keyboards or mice cannot be used.

[0004] To automate information transmission using hands, a method for recognizing information expressed by hands is required, which is called hand expression recognition. Hand expression recognition is divided into two types: hand pose recognition, based on the static form of the hand, and hand gesture recognition, based on the dynamic movement of the hand.

[0005] Recently, as Google’s MediaPipe has emerged as the center of cross-platform frameworks in the field of pose estimation, various hand expression recognition technologies based on the MediaPipe Hands Model are being proposed.

[0006] For example, various hand expression recognition methods have been proposed, such as the implementation of hand mouse functions using monocular cameras, finger digit recognition, handwriting recognition, hand space touch, and sign language recognition. However, among these functions, it is difficult to find excellent implementation examples of user-friendly and intuitive user interfaces, such as dialing or scrolling functions. Prior art literature

[0007] Published Patent 2014-0028064 (March 7, 2014) The problem to be solved

[0008] The present invention is designed to solve the aforementioned problem and aims to provide a system and method for recognizing hand pointer gestures and hand mouse gestures using a MediaPipe Hands Model, which detects various static poses using landmarks that are output values ​​of the MediaPipe Hands Model and detects dynamic gestures through the movement and change of static poses.

[0009] In addition, the purpose is to propose a system and method for recognizing hand pointer gestures and hand mouse gestures using a media pipe hands model that offers good intuitiveness and convenience. means of solving the problem

[0010] A hand gesture recognition system for detecting whether fingers are spread or bent using hand model feature points (landmarks of hands model) detected based on camera vision according to the present invention, wherein a hand mouse posture including a mouse cursor posture, a left-click posture, and a right-click posture is defined by a combination of finger contact and spread, and spread and bending of each finger based on hand model feature points, and the hand mouse posture is detected using hand model feature points detected from each input frame, the system includes a hand mouse posture recognition unit that recognizes the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of the hand model feature points as the mouse cursor coordinate and displays it if necessary, wherein in a specific section of the introduction of a continuous frame list composed of a preset number of frames, the number of frames including the hand mouse cursor posture is greater than or equal to a preset first mouse cursor posture occupancy ratio, and thereafter in a specific section, the number of frames including either the left-click posture or the right-click posture is greater than or equal to a preset first click posture occupancy ratio, and if the current frame is a frame including any one click posture, the mouse cursor coordinate of the current frame and any one It includes a hand mouse gesture recognition unit that recognizes a left-click or right-click command corresponding to a single click posture. Effects of the invention

[0011] According to one embodiment of the present invention, by using landmarks, which are output values ​​of the MediaPipe Hands Model, to detect various static poses and to detect dynamic gestures through the movement and change of static poses, thereby detecting and recognizing mouse gestures based on the MediaPipe Hands Model, which has good intuitiveness and convenience, it is possible to perform a virtual smart home device control simulation through a hand gesture recognition method. Brief explanation of the drawing

[0012] FIG. 1 is a diagram showing the configuration of a camera vision-based hand gesture recognition system according to one embodiment of the present invention. Figure 2 is a diagram illustrating the operation of the hand mouse posture recognition unit. FIG. 3 is a diagram illustrating bending determination using two vectors in a finger of a camera vision-based hand gesture recognition system according to one embodiment of the present invention. Figure 4 is a diagram illustrating a hand mouse gesture recognition operation. Figure 5 is a diagram showing the continuous operation of hand mouse gesture input. FIG. 6 is a drawing showing a hand mouse posture according to one embodiment of the present invention. FIG. 7 is a diagram illustrating the configuration of a double-click gesture recognition unit of a camera vision-based hand gesture recognition system according to an embodiment of the present invention. Figure 8 is a diagram illustrating the operation of a double-click gesture recognition unit. FIG. 9 is a diagram showing the configuration for recognizing hand mouse and hand point functions of a camera vision-based hand gesture recognition system according to one embodiment of the present invention. FIG. 10 is a diagram showing a configuration for recognizing a double-click gesture function in a hand mouse and hand point function of a camera vision-based hand gesture recognition system according to an embodiment of the present invention. Figure 11 is a diagram illustrating the operation of the hand point posture recognition unit. FIG. 12 is a drawing showing a hand point posture according to an embodiment of the present invention, and FIG. 13 is a diagram showing a contact reference value (min_len1) according to one embodiment of the present invention. Figure 14 is a diagram illustrating a hand point gesture recognition operation. FIG. 15 is a drawing showing a hand dial position according to one embodiment of the present invention. Figure 16 is a diagram showing a case where a NONE value occurred due to misrecognition during hand point pose sequence input. Figure 17 is a diagram showing the travel distance reference value (min_len2). Figure 18 is a diagram showing the continuous operation of hand mouse gesture input. Figure 19 is a diagram showing the angle representation in a hand dial gesture. FIG. 20 is a graph showing the hand gesture recognition rate of a camera vision-based hand gesture recognition system according to one embodiment of the present invention. FIG. 21 shows a virtual smart home device control menu using a camera vision-based hand gesture recognition system proposed in this embodiment. Figure 22 is a diagram showing an increase to TV channel 2 using a hand dial gesture. FIG. 23 is a diagram for schematically explaining the operation of a camera vision-based hand gesture recognition system according to one embodiment of the present invention. Specific details for implementing the invention

[0013] The specific structural or functional descriptions presented in the embodiments of the present invention are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms. Furthermore, it should not be interpreted as being limited to the embodiments described herein, but should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0014] Meanwhile, in the present invention, terms such as "first" and / or "second" may be used to describe various components, but said components are not limited to said terms. For the sole purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0015] In one embodiment of the camera vision-based hand gesture recognition system according to the present invention, we propose a hierarchical hand posture model based on the position and shape of fingers for hand dial and mouse gesture recognition, and a hand dial gesture model that dynamically applies the recognition results of this model. For hand posture recognition, we intend to use the open-source MediaPipe Hands model as a basis, and to hierarchically configure a model representing finger states and a model representing hand dial movements through this.

[0016] In this embodiment, a static model for hand dial and mouse gesture recognition is constructed using Google's MediaPipe. MediaPipe is a cross-platform framework designed to process various data through pipeline processing. MediaPipe provides models of faces, bodies, hands, etc., trained for use in pose estimation.

[0017] In this embodiment, a hand pose model is implemented using 21 landmarks provided by Media Pipe in relation to hand joints. Since the extracted landmarks cannot be directly used for hand pose recognition, finger poses used to represent hand poses are defined through the relative positions of the landmarks, and hand poses are defined through the finger poses. Through this hierarchical hand pose representation method, various hand poses can be abstractly defined, and a complete hand gesture recognition model can be constructed by using the estimated poses together with a dynamic model.

[0018] FIG. 1 is a diagram showing the configuration of a camera vision-based hand gesture recognition system according to one embodiment of the present invention.

[0019] As illustrated in FIG. 1, the camera vision-based hand gesture recognition system according to the present embodiment detects whether fingers are extended or bent using landmarks of hands model detected based on camera vision, and the hand gesture recognition system (10) includes a hand mouse posture recognition unit (110) and a hand mouse gesture recognition unit (120).

[0020] FIG. 2 is a diagram for explaining the operation of a hand mouse posture recognition unit. As shown in FIG. 2, the hand mouse posture recognition unit (110) defines (S2) a hand mouse posture including a mouse cursor posture, a left click posture, and a right click posture based on hand model feature points, and detects (S4) the hand mouse posture using hand model feature points detected from each input frame, and recognizes (S6) the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of hand model feature points as the mouse cursor coordinate, and displays (S8) if necessary.

[0021] The hand model landmarks are the landmarks of Google's MediaPipe Hands Model, and the fingers of the landmarks are the fingers of either the left or right hand, or the fingers of both hands.

[0022] FIG. 3 is a diagram illustrating bending determination using two vectors in a finger of a camera vision-based hand gesture recognition system according to one embodiment of the present invention.

[0023] Figure 3 shows the locations of 21 landmarks of the media pipe hands model. Here, PIP (proximal interphalangeal joint) refers to the location of the proximal interphalangeal joint, DIP (distal interphalangeal joint) refers to the location of the distal interphalangeal joint, and MCP (metacarpophalangeal joint) refers to the location of the metacarpophalangeal joint.

[0024] Therefore, INDEX_FINGER_PIP is a feature point of the proximal interphalangeal joint of the index finger, MIDDLE_FINGER_DIP is a feature point of the distal interphalangeal joint of the middle finger, and INDEX_FINGER_MCP is a feature point of the metacarpal joint of the middle finger.

[0025] Furthermore, as IP (interphalangeal joint) refers to the location of the 'interphalangeal joint' and CMC (carpometacarpal joint) refers to the location of the 'carpometacarpal joint,' THUMB_IP signifies the characteristic point of the thumb's interphalangeal joint, and THUMB_CMC signifies the characteristic point of the thumb's carpometacarpal joint location. Additionally, TIP refers to the area around the fingertips, such as the tips of the middle and index fingers; typically, it refers to a characteristic point located at the tip of the finger bone rather than the tip of the fingernail. WRIST refers to the characteristic point of the wrist joint.

[0026] In this embodiment, point symmetry refers to a symmetry in which a figure completely overlaps the original figure when rotated 180° around a point that is the center of symmetry.

[0027] The simplest way to determine whether a finger is stretched or bent is to [use] a specific finger When, This is a method of comparing the coordinate values ​​of the end part, TIP, and the first joint, DIP (distal interphalangeal). However, an algorithm that detects whether the finger is spread by simply comparing TIP and DIP has difficulty recognizing when the hand is rotated so that the fingertips face the floor. To solve this, in this embodiment, specific feature points of the finger are designated, and based on these, two 3D vectors are used to detect whether the finger is spread.

[0028] The camera vision-based hand gesture recognition system according to the present embodiment detects whether a finger is extended using the joint angles of the finger. As shown in FIG. 3, a specific landmark of the finger is designated as C, and the landmarks above and below connected to C are designated as x and y, respectively. Subsequently, a 3D vector extending from C to x A 3D vector pointing to y in C Find .

[0029] To find the angle (d) between two vectors, the dot product of the vectors is used as in Equation (1). Afterwards, by referring to the change in angle of each finger joint, the index finger, middle finger, and ring finger are judged to be bent if the angle is less than 100 degrees, and the thumb and pinky finger are judged to be bent if the angle is less than 150 degrees.

[0030] (Mathematical Formula 1)

[0031]

[0032] The hand gesture recognition system (10) determines whether the fingers are extended or bent by calculating the finger joint angles using the coordinates of the hand model feature points. The angle of the finger joints is determined by the angle between the line segment connecting the proximal interphalangeal joint feature point of each finger to the distal interphalangeal joint feature point and the line segment connecting the proximal interphalangeal joint feature point of the finger to the metacarpophalangeal joint feature point. For the index, middle, and ring fingers, if the angle is 100° or more, it is determined to be extended, otherwise it is determined to be bent; for the ring finger, if the angle is 150° or more, it is determined to be extended, otherwise it is determined to be bent.

[0033] Here, the hand gesture recognition system (10) determines that in the case of the thumb, if either of the angle between the line segment connecting the feature point of the thumb's end at the feature point of the thumb's interphalangeal joint and the line segment connecting the feature point of the thumb's metacarpophalangeal joint and the line segment connecting the feature point of the thumb's metacarpophalangeal joint and the line segment connecting the feature point of the thumb's metacarpophalangeal joint and the feature point of the metacarpophalangeal joint is 150° or more, it is determined as unfolded, and otherwise, it is determined as bent.

[0034] A hand mouse gesture recognition unit (120) according to one embodiment of the present invention recognizes the mouse cursor coordinates of the current frame and a left click or right click command corresponding to the click pose, if the number of frames including a hand mouse cursor pose in a specific section of an introduction section of a continuous frame list composed of a preset number of frames is greater than or equal to a preset first mouse cursor pose occupancy ratio, and thereafter the number of frames including one of a left click pose and a right click pose in a specific section is greater than or equal to a preset first click pose occupancy ratio, and the current frame is a frame including one of the click poses.

[0035] Figure 4 is a diagram illustrating a hand mouse gesture recognition operation.

[0036] The hand mouse gesture recognition unit (120) satisfies the condition that the number of frames containing a mouse cursor posture in the first half of the continuous frame list is greater than or equal to the first mouse cursor posture occupancy ratio in the first half of the continuous frame list when the continuous frame list is filled with a preset number of frames (S2), and if the current frame is a frame containing either a left click posture or a right click posture (S4), and if the number of frames containing either a click posture in the last quarter of the continuous frame list is greater than or equal to the first click posture occupancy ratio in the first half of the continuous frame list (S6), it recognizes the left click or right click command corresponding to the click posture and the mouse cursor coordinates of the current frame (S8) and utilizes them as user interface commands when necessary.

[0037] Figure 5 is a diagram showing the continuous operation of hand mouse gesture input.

[0038] As illustrated in Fig. 5, the reason for checking the mouse cursor mode posture during the first half of the consecutive frames is to confirm that there was a finger movement for a left click or a right click after moving the mouse pointer to a desired location in the mouse cursor mode. And the reason for checking the click posture during the last quarter of the frames is to confirm what kind of click posture there was if there was a movement in the mouse cursor mode.

[0039] In addition, the hand mouse gesture recognition unit resets the continuous frame list to recognize the next command of the user interface command, and then repeats the process of re-recognizing the hand mouse gesture.

[0040] At this time, the preset number of frames is twice the number of frames per second, and the mouse cursor posture is a posture in which the index and middle fingers are spread out and all other fingers are bent using hand model feature points detected in each input frame, the left click posture is a posture in which the middle finger is extended, the index finger is bent and all other fingers are bent using hand model feature points, and the right click posture is a posture in which the index finger is extended, the middle finger is bent and all other fingers are bent using hand model feature points.

[0041] In the hand mouse gesture recognition unit (120), the first mouse cursor posture occupancy ratio is 60% and the first click posture occupancy ratio is 50%, and the mouse cursor coordinates are coordinates obtained by point symmetry of the wrist joint feature points based on the midpoint between the metacarpophalangeal joint feature points of the index and middle fingers, which are feature points whose values ​​change relatively small when the fingers are bent.

[0042] FIG. 6 is a diagram showing a hand mouse posture according to an embodiment of the present invention. The hand mouse posture of a camera vision-based hand gesture recognition system according to an embodiment of the present invention is described as follows.

[0043] Hand mouse posture refers to a static hand position for input that allows left-clicking and right-clicking similar to a real mouse. To use left-clicking and right-clicking of the mouse as gestures, it is divided into three postures: mouse cursor mode, left-click, and right-click.

[0044] As shown in Fig. 6(a), the shape of the hand with the middle and index fingers spread out, similar to using an actual mouse, signifies entry into mouse cursor mode. In mouse cursor mode, bending the index finger as shown in Fig. 6(b) is recognized as a left click, and bending the middle finger as shown in Fig. 6(c) is recognized as a right click. The mouse point, which is the position of the mouse cursor, is determined by point-symmetricing the coordinates of the wrist (WRIST) with respect to the midpoint between the index and middle finger MCPs, which are characteristic points whose values ​​do not change significantly when the fingers are bent.

[0045] FIG. 7 is a diagram illustrating the configuration of a double-click gesture recognition unit of a camera vision-based hand gesture recognition system (10) according to one embodiment of the present invention.

[0046] In a specific section of the introduction of a continuous frame list, the number of frames including a hand mouse cursor posture is greater than or equal to a preset second mouse cursor posture occupancy ratio, and thereafter, in a specific section, the number of frames including either a left click posture or a right click posture is greater than or equal to a preset second click posture occupancy ratio, and thereafter, the number of frames including a hand mouse cursor posture is greater than or equal to a preset third mouse cursor posture occupancy ratio, and thereafter, the number of frames including any one click posture is greater than or equal to a preset third click posture occupancy ratio, and if the current frame is a frame including any one click posture, a double-click gesture recognition unit (130) that recognizes a left double-click or right double-click command corresponding to the mouse cursor coordinates of the current frame and any one click posture is further included between the hand mouse posture recognition unit and the hand mouse gesture recognition unit.

[0047] Figure 8 is a diagram illustrating the operation of a double-click gesture recognition unit.

[0048] As illustrated in FIG. 8, the double-click gesture recognition unit (130) satisfies the condition (S2) that the number of frames including a mouse cursor posture in the first half of 2 / 5 of the continuous frame list is greater than or equal to the first half of 2 / 5 of the continuous frame list, and if the current frame is a frame including either a left-click posture or a right-click posture (S4), and if the condition is satisfied that the number of frames including a click posture in the second half of 3 / 5 of the continuous frame list is greater than or equal to the first half of 2 / 5 of the continuous frame list, then the number of frames including a click posture is greater than or equal to the first half of 2 / 5 of the continuous frame list, then the number of frames including a click posture is greater than or equal to the first half of 2 / 5 of the continuous frame list, then the number of frames including a hand mouse cursor posture is greater than or equal to the first half of 2 / 53 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 3 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 3 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 2 / 5 of 3 / The mouse cursor coordinates of the current frame are recognized and used as user interface commands if necessary. Here, to recognize the next command of the user interface commands, the continuous frame list is reset, and the process of re-recognizing the double-click gesture is repeated (S14).

[0049] Here, the preset second mouse cursor pose occupancy ratio is 60%, the preset third mouse cursor pose occupancy ratio is 5%, the preset second click pose occupancy ratio is 30%, and the preset third click pose occupancy ratio is 50%.

[0050] FIG. 9 is a diagram showing a configuration for recognizing hand mouse and hand pointer functions of a camera vision-based hand gesture recognition system according to an embodiment of the present invention. FIG. 10 is a diagram showing a configuration for recognizing a double-click gesture function of a hand mouse and hand pointer functions of a camera vision-based hand gesture recognition system according to an embodiment of the present invention.

[0051] As illustrated in FIG. 9, the configuration of FIG. 1 includes a hand point posture recognition unit (140) and a hand point gesture recognition unit (150). As illustrated in FIG. 10, the hand gesture recognition system is configured to include a hand mouse and a hand point function, as well as a double-click gesture function. Here, details regarding the hand mouse posture recognition unit and the hand mouse gesture recognition unit are omitted as they are mentioned in FIG. 1.

[0052] Figure 11 is a diagram illustrating the operation of the hand point posture recognition unit.

[0053] The hand point posture recognition unit (140) defines a hand point posture using hand model feature points, a combination of contact and spread of fingers and spread and bending of each finger, so as not to overlap with the mouse cursor posture, left click posture, and right click posture (S2), and detects the hand point posture using hand model feature points detected from each input frame (S4), recognizes the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of hand model feature points as the hand point coordinate (S6), and displays the hand point coordinate if necessary (S8).

[0054] Here, the hand point pose is a pose in which the index and middle fingers are open and touching, and all other fingers are bent, using hand model feature points detected in each input frame, and the hand point coordinates are the coordinates of either the end feature point of the index finger or the end feature point of the middle finger.

[0055] At this time, the hand point posture recognition unit determines that the index finger and the middle finger are in contact if the distance between the feature points at the tip of the index finger and the tip of the middle finger is less than or equal to a preset contact threshold value, and recognizes the hand point coordinates as the coordinates of the feature point at the tip of the index finger.

[0056] The hand point posture recognition unit (140) according to the present embodiment determines that the index finger and the middle finger are spread apart when determining the mouse cursor posture, if the distance between the feature points of the tip of the index finger and the tip of the middle finger exceeds a preset contact reference value.

[0057] Here, the preset contact threshold is the distance between the tip feature point of the middle finger and the distal interphalangeal joint feature point, which varies according to camera zoom in / out.

[0058] FIG. 12 is a drawing showing a hand point posture according to one embodiment of the present invention, and FIG. 13 is a drawing showing a contact reference value (min_len1) according to one embodiment of the present invention.

[0059] The hand point posture is a hand shape designed to receive dynamic hand gestures in the up, down, left, and right directions as input. As shown in Fig. 12, the condition for the hand point posture is the touch between the two fingers when the rest of the fingers are folded and only the index and middle fingers are extended. The contact between the two fingers is investigated using the TIP distance between the middle and index fingers. That is, after calculating the distance between the two TIPs, it is compared with min_len1; if the distance is smaller than this, it is determined that the two fingers are in contact. At this time, considering that the finger length varies depending on the camera zoom in / out, the contact criterion value (min_len1) for the hand point posture uses the relative distance (distance between the middle finger's DIP and TIP) that can be obtained within the hand.

[0060] The hand point gesture recognition unit (150) is configured to determine the distance and direction of movement of a hand point by comparing the coordinates obtained from the hand point coordinates of any one frame among the introductory frames of the continuous frame list or a combination of the hand point coordinates, and the coordinates obtained from the hand point coordinates of any one frame among the end frames of the continuous frame list or a combination of the hand point coordinates, when the number of frames including the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio.

[0061] FIG. 14 is a diagram illustrating a hand point gesture recognition operation. As shown in FIG. 14,

[0062] The hand point gesture recognition unit satisfies the condition that when the continuous frame list reaches a preset number of frames, the number of frames containing the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio (S2). If the current frame is a frame containing the hand point posture, the hand point coordinates of the leading frame among the frames containing the hand point posture in the continuous frame list are compared with the hand point coordinates of the current frame to determine the movement direction with the maximum distance gap among upward, downward, leftward, and rightward directions (S4). If the condition that the maximum distance gap is greater than or equal to the preset distance gap reference value is satisfied, the movement direction and the maximum distance gap are recognized as the movement direction and movement distance of the hand point, respectively (S6), and utilized as a user interface command if necessary. After resetting the continuous frame list to recognize the next command of the user interface command, the process of re-recognizing the hand point gesture (S8) is repeated.

[0063] Here, the preset hand point pose occupancy ratio is 80%, and the preset distance gap reference value is the distance between the wrist joint feature point and the metacarpophalangeal joint feature point of the index finger within the current frame, which varies according to camera zoom in / out.

[0064] In addition, the continuous frame list is loaded with each current frame as the last frame of the list whenever it is input, and when the list reaches a preset number of frames, the leading frames of the list are deleted one by one and the current frames are loaded as the last frames, thereby updating the list so that the latest frames up to the preset number are stored, and when a user interface command is recognized, the entire list is reset to recognize the next user interface command, and this process is repeated.

[0065] The camera vision-based hand gesture recognition system according to the present embodiment further includes a user interface unit that utilizes as a gesture-based user command any one of the following: a user interface command received through camera vision-based hand gesture recognition from either the left or right hand, a user interface command received through camera vision-based hand gesture recognition from both hands, and a command created by combining user interface commands received through camera vision-based hand gesture recognition from each of the hands.

[0066] At least one of the user interface commands is audibly expressed through Text-to-Speech (TTS) when necessary, and the TTS can utilize pyttsx3, a Python text-to-speech library.

[0067] In this embodiment, to suppress the adverse effects of noise during the hand point gesture recognition process, the coordinates obtained from the combination of hand point coordinates of the introductory frames are coordinates obtained by median filtering the hand point coordinates of the introductory frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers, and the coordinates obtained from the combination of hand point coordinates of the end frames are coordinates obtained by median filtering the hand point coordinates of the end frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers.

[0068] A camera vision-based hand gesture recognition system according to another embodiment of the present invention may be configured by independently configuring the aforementioned hand point posture recognition unit and hand point gesture recognition unit separately from the hand mouse posture recognition unit and the hand mouse gesture recognition unit.

[0069] A camera vision-based hand gesture recognition system according to another embodiment of the present invention may further include a configuration that performs hand dial posture recognition and hand dial gesture recognition functions.

[0070] Accordingly, the hand dial position is explained as follows.

[0071] FIG. 15 is a drawing showing a hand dial position according to one embodiment of the present invention.

[0072] The hand dial position is a hand shape used to rotate the hand like a dial button to receive an angle increase / decrease (+ / -) value as an input value for the increase / decrease of the dial scale. As shown in FIG. 15, the hand dial position is in a state where the thumb is extended in the hand point position, and when the hand dial gesture mode is detected, a blue line connecting the index finger's TIP and the thumb's TIP is displayed to show the change in angle on the screen.

[0073] The dynamic gestures recognized by the camera vision-based hand gesture recognition system according to one embodiment of the present invention are described as follows.

[0074] Dynamic hand gestures are determined based on the previously obtained static poses and are ultimately utilized as hand gesture inputs. The method for detecting dynamic gestures involves using OpenCV to sequentially store static pose sequences from the recent n_frames by designating n×2 as n_frames when the camera's frame rate per second is n. That is, dynamic gestures are continuously detected by checking for changes in the static pose sequences starting from when the number of stored frames reaches n_frames. In this paper, a total of three dynamic gestures—hand pointer gestures, hand mouse gestures, and hand dial gestures—are detected using the previously obtained static poses.

[0075] When the green point generated when the hand point posture is taken as shown in Fig. 12 is moved, the up, down, left, and right movements are identified and received as input sequence data. The conditions for detecting the hand point gesture are as follows.

[0076] Condition 1. The static position of the current frame is the hand-pointing position.

[0077] Condition 2. The proportion of hand point poses among all n_frames is 80% or more.

[0078] Condition 3. The distance difference between the first point and the n_frame-th point is greater than or equal to min_len2.

[0079] Looking at Condition 2, more than 80% of the total frames must be input in the hand-pointing position to be recognized as a hand-pointing gesture. This is to buffer errors occurring in the MediaPipe Hands Model and to detect dynamic gestures with a higher probability.

[0080] Figure 16 is a diagram showing a case where a NONE value occurred due to misrecognition during hand point pose sequence input.

[0081] When the Hands model is in operation, there are often cases where it is detected differently from reality due to intermittent misrecognition. In such cases, the static pose sequence for n_frame is stored as shown in Fig. 16, but problems arise where misrecognition occurs partially during n_frame, leading to misjudgment of dynamic gestures or failure to detect the motion gesture itself. To solve this, it is determined that sufficient hand point poses have been input and recognized as point gestures only when a certain percentage, i.e., 80% or more, of the entire static pose sequence consists of hand point poses.

[0082] Figure 17 is a diagram showing the travel distance reference value (min_len2).

[0083] In condition 3, a value called min_len2 is used to measure whether the point has moved. In hand point gestures, the hand point gesture is recognized as input when it moves a distance greater than min_len2. As shown in FIG. 8, the reference value for the movement distance of the hand point gesture (min_len2) is set to the distance between the wrist coordinate WRIST and the index finger MCP so that it varies according to camera zoom in / out.

[0084] Hand mouse gestures detect left or right clicks in mouse cursor mode and trigger a mouse click at the mouse pointer location within the image. The conditions for triggering such a hand mouse click are as follows.

[0085] Condition 1. The static stance of the current frame is the left-click or right-click stance.

[0086] Condition 2. The static pose ratio of the mouse cursor mode during n_frame / 2 frames is 60% or more.

[0087] Condition 3. The ratio of left-click or right-click poses during the last n_frame / 4 frames is 50% or more.

[0088] Looking at conditions 2 and 3, the first half of the total n_frames checks the static pose of the mouse cursor mode, and the last quarter checks the static pose of the click. The reason for detecting hand mouse gestures through these conditions can be seen in Fig. 18.

[0089] Figure 18 is a diagram showing the continuous operation of hand mouse gesture input.

[0090] The reason for checking the mouse cursor mode posture during the first half of the consecutive frames is to confirm whether there was a finger movement for a left click or a right click after moving the mouse pointer to the desired location while in mouse cursor mode. And the reason for checking the click posture during the last quarter of the frames is to confirm what kind of click posture that movement was if there was a movement while in mouse cursor mode.

[0091] Figure 19 is a diagram showing the angle representation in a hand dial gesture.

[0092] The hand dial gesture designates the hand dial posture as the starting angle and converts the angle increase / decrease (+ / -) according to the direction of rotation in 30-degree increments each time the hand is rotated, receiving it as the input value for the increase / decrease of the dial scale. As shown in FIG. 19, when the user takes the hand dial posture, the center is calculated as the center between the index finger TIP and the thumb TIP, and the starting angle (start_deg) of Equation (2) is calculated using the center and the index finger TIP.

[0093] (Mathematical Formula 2)

[0094]

[0095] Subsequently, the same process is performed on the hand dial position in the continuously input frames to obtain the current angle and compare it with the starting angle. Whenever the angle change exceeds +30 degrees, the dial scale is increased by 1, and conversely, whenever it exceeds -30 degrees, the dial scale is decreased by 1. Meanwhile, if the angle change exceeds +60 degrees, the dial scale is increased by 2, and conversely, if it exceeds -60 degrees, the dial scale is decreased by 2. That is, whenever the angle change exceeds 30 degrees or 60 degrees, the angle increase / decrease (+ / -) according to the direction of rotation is converted into 30-degree units and used as the input value for the increase / decrease of the dial scale.

[0096] FIG. 20 is a graph showing the hand gesture recognition rate of a camera vision-based hand gesture recognition system according to an embodiment of the present invention. In this embodiment, a computer simulation was performed to control a virtual smart home device in an environment using JetBrains PyCharm, Google MediaPipe Hands, OpenCV 4.6, Logitech C930e WebCam, and NVIDIA GTX 1070Ti GPU.

[0097] To measure the recognition rate of input data generation using hand gestures in computer simulation, the experimenter input data using gestures 100 times. Figure 20 is a graph showing the hand gesture recognition rate when performing up / down / left / right movement of the hand point gesture, angle increase / decrease (+ / -) of the hand dial gesture, and left click and right click of the hand mouse gesture.

[0098] Data can be entered with good overall accuracy because the current static position of the hand is displayed on the screen as dots and lines, allowing the user to input accurately. However, since only a single monocular camera (webcam) is used to detect the hand, accurate detection may be difficult if the hand moves away from the front of the camera. A typical example is a mouse click, where accuracy is lower than other input methods because the hand tilts downward when the fingers are folded during the clicking process.

[0099] FIG. 21 shows a virtual smart home device control menu using a camera vision-based hand gesture recognition system proposed in this embodiment.

[0100] In this embodiment, a simulation was performed focusing on a scenario in which a virtual smart home device is controlled using the proposed hand gesture recognition method. FIG. 21 is a menu implemented using the hand gesture recognition method of the present invention, showing a menu that performs a total of five functions: Light, Memo, TV, Air Cleaner, and Exit.

[0101] In the initial state, the user inputs 'down' or 'up' using hand pointer gestures to turn the main menu on or off. The main menu can be activated or deactivated through the 'down' and 'up' inputs, respectively. Subsequently, the user selects the desired menu (Light, Memo, TV, Air Cleaner, Exit) using left or right inputs. For example, the TV menu controls the channels and volume of the TV connected to the smart home. When the TV menu is selected from the main menu, the TV volume and menu options are displayed. When the user inputs an angle increase or decrease (+ / -) based on the rotation direction using the dial positions of the left and right hands, respectively, to control the volume and channels, this is converted into 30-degree increments and received as the input value for the dial scale.

[0102] FIG. 22 is a diagram showing an increase of 2 TV channels using a hand dial gesture. FIG. 22 is an example of an instance where the TV channel is increased by 2 by increasing by more than +60 degrees using a right-hand dial gesture.

[0103] In this embodiment, a hand gesture-based user interface technology was developed using the MediaPipe Hands model, and a computer simulation was performed to control a virtual smart home device. The proposed hand gesture recognition method detects hands using the MediaPipe Hands model and detects static postures using feature points, and finally detects dynamic gestures as input data by hierarchically detecting changes in static postures. Based on this, recognition methods for hand point gestures, hand mouse gestures, and hand dial gestures were proposed, and their usefulness was examined through a simulation of controlling a virtual smart home device. Since there are many people who are unfamiliar with intricate postures and gestures such as sign language, and complex finger joint expressions increase fatigue and reduce intuitiveness, a hand gesture recognition method with excellent intuitiveness and simple expression was devised in the mouse gesture recognition system using the MediaPipe Hands model according to one embodiment of the present invention.

[0104] FIG. 23 is a diagram schematically illustrating the operation of a camera vision-based hand gesture recognition system according to an embodiment of the present invention. As shown in FIG. 23, a camera vision-based hand gesture recognition system for recognizing hand pointer gestures and hand mouse gestures using a mediapipe hand model receives an image input (S1) and detects a hand using a mediapipe hand model (S2). It determines whether a hand has been detected, and if a hand is detected (S3), it detects the finger posture according to the folding of the fingers (S4). Subsequently, it detects the static posture of the hand (S5), and determines whether a specific static posture of the hand has been detected; if detected (S6), it detects a hand pointer gesture, a hand mouse gesture, and a hand dial gesture according to the static posture detection result (S7). Next, it generates input data through the dynamic gesture of the hand (S8). Subsequently, depending on whether the Esc key is pressed, the procedure proceeds to the termination or image input step (S9).

[0105] According to one embodiment of the present invention, by using landmarks, which are output values ​​of the MediaPipe Hands Model, to detect various static poses and to detect dynamic gestures through the movement and change of static poses, thereby detecting and recognizing hand dial and mouse gestures based on the MediaPipe Hands Model, which has good intuitiveness and convenience, it is possible to perform a virtual smart home device control simulation through a hand gesture recognition method.

[0106] The present invention described above is not limited by the aforementioned embodiments and attached drawings, and it will be obvious to those skilled in the art that various substitutions, modifications, and changes are possible within the scope of the technical concept of the present invention.

Claims

Claim 1 A hand gesture recognition system that detects whether fingers are extended or bent using hand model feature points (landmarks of hands model) detected based on camera vision, wherein, when a hand mouse posture including a mouse cursor posture, a left-click posture, and a right-click posture is defined by a combination of finger contact and separation and extension and bending of each finger based on the hand model feature points, the hand mouse posture is detected using hand model feature points detected from each input frame, and the hand mouse posture is recognized as a mouse cursor coordinate or a coordinate obtained from a combination of the hand model feature points, and displayed if necessary, the system includes a hand mouse posture recognition unit that, in a specific section of the introduction of a continuous frame list composed of a preset number of frames, the number of frames including the hand mouse cursor posture is greater than or equal to a preset first mouse cursor posture occupancy ratio, and thereafter, in a specific section, the number of frames including either the left-click posture or the right-click posture is greater than or equal to a preset first click posture occupancy ratio, and if the current frame is a frame including the one click posture, the mouse cursor coordinate of the current frame and the one A camera vision-based hand gesture recognition system comprising a hand mouse gesture recognition unit that recognizes left-click or right-click commands corresponding to a click posture, wherein the continuous frame list is loaded as the last frame of the list whenever each current frame is input, and when the list reaches the number of preset frames, the leading frame of the list is deleted one by one and the current frame is loaded as the last frame, thereby updating the list so that the number of latest frames equal to the number of preset frames is stored, and when a user interface command is recognized, the process of resetting everything to recognize the next user interface command is repeated. Claim 2 A camera vision-based hand gesture recognition system according to claim 1, wherein the hand model feature points are landmarks of Google’s MediaPipe Hands Model, and the fingers of the feature points are fingers of either the left hand or the right hand, or fingers of both hands. Claim 3 In claim 1, the degree of extension and flexion of the finger is determined by calculating the finger joint angle using the coordinates of the hand model feature points, wherein the angle of the finger joint is determined such that for the index, middle, and ring fingers, if the angle between the line segment connecting the proximal interphalangeal joint feature point of each finger to the distal interphalangeal joint feature point and the line segment connecting the proximal interphalangeal joint feature point of that finger to the metacarpophalangeal joint feature point is 100° or greater, it is determined as extension and otherwise as flexion; for the ring finger, if the angle is 150° or greater, it is determined as extension and otherwise as flexion; and in the case of the thumb, the angle between the line segment connecting the thumb interphalangeal joint feature point to the thumb tip feature point and the angle between the interphalangeal joint feature point of the thumb A camera vision-based hand gesture recognition system characterized by determining that if either of the angle between line segments connecting feature points of the metacarpophalangeal joint, the angle between the line segment connecting the feature point of the metacarpophalangeal joint of the thumb to the feature point of the interphalangeal joint of the thumb, and the angle between the line segment connecting the feature point of the metacarpophalangeal joint to the feature point of the carpometacarpal joint is 150° or greater, it is determined as unfolded, and otherwise, it is determined as flexed. Claim 4 A camera vision-based hand gesture recognition system according to claim 1, wherein the hand mouse gesture recognition unit satisfies the condition that, when the continuous frame list is filled with the preset number of frames, the number of frames including the mouse cursor posture in the first half of the continuous frame list is greater than or equal to the preset first mouse cursor posture occupancy ratio, and the current frame is a frame including either the left click posture or the right click posture, and when the number of frames including the click posture in the last quarter of the continuous frame list is greater than or equal to the preset first click posture occupancy ratio, the left click or right click command corresponding to the click posture and the mouse cursor coordinates of the current frame are recognized and utilized as user interface commands if necessary, and after resetting the continuous frame list to recognize the next command of the user interface command, the hand mouse gesture is re-recognized. Claim 5 A camera vision-based hand gesture recognition system according to claim 1, wherein the number of preset frames is twice the number of frames per second, the mouse cursor posture is a posture in which the index and middle fingers are spread out and all other fingers are bent using the hand model feature points detected in each input frame, the left click posture is a posture in which the middle finger is extended, the index finger is bent and all other fingers are bent using the hand model feature points, and the right click posture is a posture in which the index finger is extended, the middle finger is bent and all other fingers are bent using the hand model feature points. Claim 6 A camera vision-based hand gesture recognition system according to claim 4, wherein the preset first mouse cursor posture occupancy ratio is 60% and the preset first click posture occupancy ratio is 50%, and the mouse cursor coordinates are coordinates obtained by point symmetry of the wrist joint feature points with respect to the midpoint between the metacarpophalangeal joint feature points of the index and middle fingers, which are feature points whose values ​​change relatively small when the fingers are bent. Claim 7 A camera vision-based hand gesture recognition system according to claim 1, wherein in a specific section of the introduction of the continuous frame list, the number of frames including the hand mouse cursor posture is greater than or equal to a preset second mouse cursor posture occupancy ratio, and thereafter, in a specific section, the number of frames including either the left click posture or the right click posture is greater than or equal to a preset second click posture occupancy ratio, and thereafter, the number of frames including the hand mouse cursor posture is greater than or equal to a preset third mouse cursor posture occupancy ratio, and thereafter, the number of frames including any one click posture is greater than or equal to a preset third click posture occupancy ratio, and if the current frame is a frame including any one click posture, a double-click gesture recognition unit that recognizes a left double-click or right double-click command corresponding to the mouse cursor coordinates of the current frame and any one click posture is further included between the hand mouse posture recognition unit and the hand mouse gesture recognition unit. Claim 8 In claim 7, the double-click gesture recognition unit satisfies the condition that, when the continuous frame list reaches the preset number of frames, the number of frames including the mouse cursor posture in the first half of 2 / 5 of the continuous frame list is greater than or equal to the preset second mouse cursor posture occupancy ratio, and the current frame is a frame including either the left click posture or the right click posture; if the condition that the number of frames including the one click posture in the second half of 3 / 5 of the continuous frame list is greater than or equal to the preset second click posture occupancy ratio is satisfied, and the number of frames including the one click posture is greater than or equal to the preset second click posture occupancy ratio, and subsequently the number of frames including the hand mouse cursor posture is greater than or equal to the preset third mouse cursor posture occupancy ratio, and subsequently the number of frames including the one click posture is greater than or equal to the preset third click posture occupancy ratio, and the current frame is a frame including the one click posture, it recognizes the left double-click or right double-click command corresponding to the one click posture and the mouse cursor coordinates of the current frame, and if necessary, as a user interface command A camera vision-based hand gesture recognition system characterized by utilizing, resetting the continuous frame list to recognize the next command of the above user interface command, and then re-recognizing a double-click gesture. Claim 9 A camera vision-based hand gesture recognition system according to claim 8, characterized in that the preset second mouse cursor pose occupancy ratio is 60%, the preset third mouse cursor pose occupancy ratio is 5%, the preset second click pose occupancy ratio is 30%, and the preset third click pose occupancy ratio is 50%. Claim 10 A camera vision-based hand gesture recognition system according to claim 5, characterized in that the mouse cursor posture is determined to be separated between the index finger and the middle finger if the distance between the feature points of the tip of the index finger and the tip of the middle finger exceeds a preset contact threshold value. Claim 11 A camera vision-based hand gesture recognition system according to claim 1, further comprising a hand point posture recognition unit that detects a hand point posture defined such that it does not overlap with the mouse cursor posture, the left click posture, and the right click posture by a combination of contact and separation of fingers and the spreading and bending of each finger using the hand model feature points, and recognizes the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of the hand model feature points as a hand point coordinate and displays it if necessary; and further comprising a hand point gesture recognition unit that, if the number of frames containing the hand point posture in the continuous frame list is greater than or equal to a preset hand point posture occupancy ratio, compares the hand point coordinate of any one frame in the introductory part of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates with the hand point coordinate of any one frame in the end part of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates to determine the movement distance and movement direction of the hand point. Claim 12 A camera vision-based hand gesture recognition system according to claim 11, wherein the hand point posture is a posture in which the index finger and middle finger are in an open state and touching each other, and all other fingers are bent, using the hand model feature points detected in each input frame, and the hand point coordinates are the coordinates of either the end feature point of the index finger or the end feature point of the middle finger. Claim 13 A camera vision-based hand gesture recognition system according to claim 12, wherein if the distance between the end feature points of the index finger and the end feature point of the middle finger is less than or equal to a preset contact threshold value, the index finger and the middle finger are determined to be in contact, and the hand point coordinates are the coordinates of the end feature point of the index finger. Claim 14 A camera vision-based hand gesture recognition system according to claim 10 or 13, characterized in that the preset contact reference value is the distance between the end feature point of the middle finger and the distal interphalangeal joint feature point so as to vary according to camera zoom in / out. Claim 15 In claim 11, the hand point gesture recognition unit is characterized by: when the continuous frame list reaches the preset number of frames, satisfying the condition that the number of frames containing the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio, and when the current frame is a frame containing the hand point posture, comparing the hand point coordinates of the leading frame among the frames containing the hand point posture in the continuous frame list with the hand point coordinates of the current frame to obtain the movement direction having the maximum distance gap among upward, downward, leftward, and rightward directions; and when the condition that the maximum distance gap is greater than or equal to a preset distance gap reference value is satisfied, recognizing the movement direction and the maximum distance gap as the movement direction and movement distance of the hand point, respectively, and utilizing them as user interface commands when necessary, and resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing the hand point gesture. Claim 16 A camera vision-based hand gesture recognition system according to claim 15, characterized in that the above-mentioned preset hand point posture occupancy ratio is 80%, and the above-mentioned preset distance gap reference value is the distance between the wrist joint feature point and the metacarpophalangeal joint feature point of the index finger within the current frame, such that it varies according to camera zoom in / out. Claim 17 A camera vision-based hand gesture recognition system characterized in that, in order to suppress the adverse effects of noise during the hand point gesture recognition process, the coordinate obtained from the combination of hand point coordinates of the introductory frames is a coordinate obtained by median filtering the hand point coordinates of the introductory frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers, and the coordinate obtained from the combination of hand point coordinates of the end frames is a coordinate obtained by median filtering the hand point coordinates of the end frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers. Claim 18 delete Claim 19 A camera vision-based hand gesture recognition system characterized in that, in any one of claims 4, 8, and 15, at least one of the user interface commands is audibly expressed through Text-to-Speech (TTS) when necessary. Claim 20 A camera vision-based hand gesture recognition system according to claim 19, characterized in that the text-to-speech (TTS) utilizes pyttsx3, a Python text-to-speech library. Claim 21 A camera vision-based hand gesture recognition system characterized by further including, in any one of claims 1 to 13, 15, 16, and 17, a user interface unit that utilizes as a gesture-based user command any one of the following: a user interface command received through camera vision-based hand gesture recognition from either the left hand or the right hand, a user interface command received through camera vision-based hand gesture recognition from both hands, and a command created by combining the user interface commands received through camera vision-based hand gesture recognition from each of the hands. Claim 22 A gesture recognition system for detecting whether fingers are extended or bent using hand model feature points (landmarks of hands model) detected based on camera vision, comprising: a hand point posture recognition unit that detects a hand point posture defined by a combination of finger contact, separation, and extension and bending of each finger using the hand model feature points detected in each input frame, recognizes the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of the hand model feature points as a hand point coordinate, and displays it if necessary; and a hand point gesture recognition unit that, if the number of frames containing the hand point posture among a continuous frame list composed of a preset number of frames is greater than or equal to a preset hand point posture occupancy ratio, compares the hand point coordinate of any one frame among the introductory frames of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates with the hand point coordinate of any one frame among the endpost frames of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates to determine the movement distance and direction of movement of the hand point, wherein the continuous frame list is formed whenever each current frame is input at that time A camera vision-based hand gesture recognition system characterized by a process in which the last frame of a list is loaded, and when the list reaches the number of frames set, the leading frame of the list is deleted one by one and the current frame is loaded as the last frame, thereby updating the system so that the latest frames up to the number of frames set are stored, and when a user interface command is recognized, all frames are reset to recognize the next user interface command. Claim 23 A camera vision-based hand gesture recognition system according to claim 22, characterized in that the hand model feature points are landmarks of Google’s MediaPipe Hands Model, and the fingers are fingers of either the left hand or the right hand, or fingers of both hands. Claim 24 In Clause 22, the degree of extension and flexion of the finger is determined by calculating the finger joint angle using the coordinates of the hand model feature points; the angle of the finger joint is such that the index, middle, and ring fingers are determined to be extended if the angle between the line segment connecting the proximal interphalangeal joint feature point of each finger to the distal interphalangeal joint feature point and the line segment connecting the proximal interphalangeal joint feature point to the metacarpophalangeal joint feature point is 100° or greater, and otherwise determined to be flexed; the ring finger is determined to be extended if the angle is 150° or greater, and otherwise determined to be flexed; and in the case of the thumb, the angle between the line segment connecting the thumb's interphalangeal joint feature point to the thumb's tip feature point and the angle between the interphalangeal joint feature point and the thumb's A camera vision-based hand gesture recognition system characterized by determining that if either of the angle between line segments connecting feature points of the metacarpophalangeal joint, the angle between the line segment connecting the feature point of the metacarpophalangeal joint of the thumb to the feature point of the interphalangeal joint of the thumb, and the angle between the line segment connecting the feature point of the metacarpophalangeal joint to the feature point of the carpometacarpal joint is 150° or greater, it is determined as unfolded, and otherwise, it is determined as flexed. Claim 25 A camera vision-based hand gesture recognition system according to claim 22, wherein the hand point posture is a posture in which the index finger and middle finger are in an open state and touching, and all other fingers are bent, using the hand model feature points detected in each input frame, and the hand point coordinates are the coordinates of either the end feature point of the index finger or the end feature point of the middle finger. Claim 26 A camera vision-based hand gesture recognition system according to claim 25, wherein if the distance between the tip feature points of the index finger and the tip feature point of the middle finger in the hand point posture is less than or equal to a preset contact threshold value, the index finger and the middle finger are determined to be in contact, and the hand point coordinates are the coordinates of the tip feature point of the index finger. Claim 27 A camera vision-based hand gesture recognition system according to claim 26, characterized in that the above-mentioned preset contact reference value is the distance between the end feature point of the middle finger and the distal interphalangeal joint feature point so as to vary according to camera zoom in / out. Claim 28 In claim 22, the hand point gesture recognition unit is characterized by: when the continuous frame list reaches the preset number of frames, satisfying the condition that the number of frames containing the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio, and if the current frame is a frame containing the hand point posture, comparing the hand point coordinates of the leading frame among the frames containing the hand point posture in the continuous frame list with the hand point coordinates of the current frame to obtain the movement direction having the maximum distance gap among upward, downward, leftward, and rightward directions; and if the condition that the maximum distance gap is greater than or equal to a preset distance gap reference value is satisfied, recognizing the movement direction and the maximum distance gap as the movement direction and movement distance of the hand point, respectively, and utilizing them as user interface commands when necessary; and resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing the hand point gesture. Claim 29 A camera vision-based hand gesture recognition system according to claim 28, wherein the above-mentioned preset number of frames is twice the frame per second, the above-mentioned preset hand point pose occupancy ratio is 80%, and the above-mentioned preset distance gap reference value is the distance between the wrist joint feature point and the metacarpophalangeal joint feature point of the index finger within the current frame so as to vary according to camera zoom in / out. Claim 30 A camera vision-based hand gesture recognition system characterized in that, in order to suppress the adverse effects of noise during the hand point gesture recognition process, the coordinate obtained from the combination of hand point coordinates of the introductory frames is a coordinate obtained by median filtering the hand point coordinates of the introductory frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers, and the coordinate obtained from the combination of hand point coordinates of the end frames is a coordinate obtained by median filtering the hand point coordinates of the end frames of the continuous frame list in the horizontal and vertical directions, respectively, or by summing and averaging the coordinates after removing outliers. Claim 31 delete Claim 32 A camera vision-based hand gesture recognition system according to claim 28, characterized in that the above user interface command is audibly expressed through Text-to-Speech (TTS) when necessary. Claim 33 A camera vision-based hand gesture recognition system according to claim 32, characterized in that the text-to-speech (TTS) utilizes pyttsx3, a Python text-to-speech library. Claim 34 A camera vision-based hand gesture recognition system characterized by further including, in any one of claims 22 to 30 and 33, a user interface unit that utilizes as a gesture-based user command any one of the following: a user interface command received through camera vision-based hand gesture recognition from either the left hand or the right hand, a user interface command received through camera vision-based hand gesture recognition from both hands, and a command created by combining the user interface commands received through camera vision-based hand gesture recognition from each of both hands. Claim 35 A method using a hand gesture recognition system that detects whether fingers are spread or bent using hand model feature points (landmarks of hands model) detected based on camera vision, wherein the hand gesture recognition system defines a hand mouse posture including a mouse cursor posture, a left-click posture, and a right-click posture based on the hand model feature points and detects the hand mouse posture using hand model feature points detected from each input frame, and the hand mouse posture is then recognized as a mouse cursor coordinate or a coordinate obtained from a combination of the hand model feature points and displayed if necessary;A hand gesture recognition method based on camera vision, wherein the hand gesture recognition system includes a hand mouse gesture recognition step in which, in a specific section of the introduction of a continuous frame list composed of a preset number of frames, the number of frames including the hand mouse cursor posture is greater than or equal to a preset first mouse cursor posture occupancy ratio, and thereafter, in a specific section, the number of frames including either the left click posture or the right click posture is greater than or equal to a preset first click posture occupancy ratio, and if the current frame is a frame including the one click posture, the hand mouse gesture recognition system recognizes the mouse cursor coordinates of the current frame and a left click or right click command corresponding to the one click posture; wherein the continuous frame list is loaded as the last frame of the list whenever each current frame is input, and when the list reaches the preset number of frames, the leading frame of the list is deleted one by one and the current frame is loaded as the last frame, so that the latest frames equal to the preset number of frames are stored as a result, and when a user interface command is recognized, the process of resetting everything to recognize the next user interface command is repeated. Claim 36 A camera vision-based hand gesture recognition method according to claim 35, characterized in that the hand model feature points are landmarks of Google’s MediaPipe Hands Model, and the fingers are fingers of either the left hand or the right hand, or fingers of both hands. Claim 37 In claim 35, the hand gesture recognition system determines whether the finger is extended or bent by calculating the finger joint angle using the coordinates of the hand model feature points; the angle of the finger joint is determined by the angle between the line segment connecting the proximal interphalangeal joint feature point of each finger to the distal interphalangeal joint feature point and the line segment connecting the proximal interphalangeal joint feature point of that finger to the metacarpophalangeal joint feature point; for the index, middle, and ring fingers, if the angle is 100° or greater, it is determined as extended, and otherwise as bent; for the ring finger, if the angle is 150° or greater, it is determined as extended, and otherwise as bent; and for the thumb, the angle between the line segment connecting the thumb interphalangeal joint feature point to the thumb tip feature point and the interphalangeal joint feature point A camera vision-based hand gesture recognition method characterized by determining that if either of the angle between line segments connecting feature points of the metacarpophalangeal joint of the thumb, the angle between the line segment connecting the feature point of the metacarpophalangeal joint of the thumb to the feature point of the interphalangeal joint of the thumb, and the angle between the line segment connecting the feature point of the metacarpophalangeal joint to the feature point of the carpometacarpal joint is 150° or greater, it is determined as unfolded, and otherwise, it is determined as bent. Claim 38 In claim 35, the hand mouse gesture recognition step is characterized by: when the continuous frame list reaches the preset number of frames, satisfying the condition that the number of frames including the mouse cursor posture in the first half of the continuous frame list is greater than or equal to the preset first mouse cursor posture occupancy ratio, and when the current frame is a frame including either the left click posture or the right click posture, satisfying the condition that the number of frames including the one click posture in the last quarter of the continuous frame list is greater than or equal to the preset first click posture occupancy ratio, recognizing the left click or right click command corresponding to the one click posture and the mouse cursor coordinates of the current frame and utilizing them as user interface commands if necessary, and resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing the hand mouse gesture. Claim 39 A camera vision-based hand gesture recognition method according to claim 35, wherein the number of preset frames is twice the number of frames per second, the mouse cursor posture is a posture in which the index and middle fingers are spread out and all other fingers are bent using the hand model feature points detected in each input frame, the left click posture is a posture in which the middle finger is extended, the index finger is bent and all other fingers are bent using the hand model feature points, and the right click posture is a posture in which the index finger is extended, the middle finger is bent and all other fingers are bent using the hand model feature points. Claim 40 A camera vision-based hand gesture recognition method according to claim 38, wherein the preset first mouse cursor posture occupancy ratio is 60% and the preset first click posture occupancy ratio is 50%, and the mouse cursor coordinates are coordinates obtained by point symmetry of the wrist joint feature points with respect to the midpoint between the metacarpophalangeal joint feature points of the index and middle fingers, which are feature points whose values ​​do not change significantly when the fingers are bent. Claim 41 In claim 35, a camera vision-based hand gesture recognition method is characterized by further including a double-click gesture recognition step between the hand mouse posture recognition step and the hand mouse gesture recognition step, wherein, in a specific section of the introduction of the continuous frame list, the number of frames including the hand mouse cursor posture is greater than or equal to a preset second mouse cursor posture occupancy ratio, thereafter the number of frames including either the left click posture or the right click posture is greater than or equal to a preset second click posture occupancy ratio, thereafter the number of frames including the hand mouse cursor posture is greater than or equal to a preset third mouse cursor posture occupancy ratio, and thereafter the number of frames including any one click posture is greater than or equal to the preset third click posture occupancy ratio, and if the current frame is a frame including any one click posture, the double-click gesture recognition step recognizes a left double-click or right double-click command corresponding to the mouse cursor coordinates of the current frame and any one click posture. Claim 42 In claim 41, the double-click gesture recognition step comprises: satisfying the condition that, when the continuous frame list reaches the preset number of frames, the number of frames including the mouse cursor posture in the first half of 2 / 5 of the continuous frame list is greater than or equal to the preset second mouse cursor posture occupancy ratio, and the current frame is a frame including either the left click posture or the right click posture; satisfying the condition that the number of frames including the one click posture in the latter half of 3 / 5 of the continuous frame list is greater than or equal to the preset second click posture occupancy ratio; and if the number of frames including the one click posture is greater than or equal to the preset second click posture occupancy ratio, and subsequently the number of frames including the hand mouse cursor posture is greater than or equal to the preset third mouse cursor posture occupancy ratio, and subsequently the number of frames including the one click posture is greater than or equal to the preset third click posture occupancy ratio, and the current frame is a frame including the one click posture, recognizing the left double-click or right double-click command corresponding to the one click posture and the mouse cursor coordinates of the current frame, and if necessary, as a user interface command A camera vision-based hand gesture recognition method characterized by utilizing, resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing a double-click gesture. Claim 43 A camera vision-based hand gesture recognition method according to claim 42, characterized in that the preset second mouse cursor pose occupancy ratio is 60%, the preset third mouse cursor pose occupancy ratio is 5%, the preset second click pose occupancy ratio is 30%, and the preset third click pose occupancy ratio is 50%. Claim 44 A camera vision-based hand gesture recognition method according to claim 39, characterized in that the mouse cursor posture is determined to be separated when the distance between the tip of the index finger and the tip of the middle finger exceeds a preset contact threshold value. Claim 45 A camera vision-based hand gesture recognition method according to claim 35, wherein if the hand gesture recognition system detects a hand point posture defined by the combination of contact and separation of fingers and the spreading and bending of each finger using the hand model feature points, such that it does not overlap with the mouse cursor posture, the left click posture, and the right click posture, the hand point posture recognition step recognizes the coordinate of any one of the hand model feature points or the coordinate obtained from a combination of the hand model feature points as the hand point coordinate and displays it if necessary; and if the number of frames containing the hand point posture in the continuous frame list is greater than or equal to a preset hand point posture occupancy ratio, the hand point gesture recognition system compares the hand point coordinate of any one frame in the introductory part of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates with the hand point coordinate of any one frame in the end part of the continuous frame list or the coordinate obtained from a combination of such hand point coordinates to obtain the movement distance and movement direction of the hand point. Claim 46 A camera vision-based hand gesture recognition method according to claim 45, wherein the hand point posture is a posture in which the index finger and middle finger are in an open state and touching each other, and all other fingers are bent, using the hand model feature points detected in each input frame, and the hand point coordinates are the coordinates of either the end feature point of the index finger or the end feature point of the middle finger. Claim 47 A camera vision-based hand gesture recognition method according to claim 46, wherein if the distance between the end feature points of the index finger and the end feature point of the middle finger is less than or equal to a preset contact threshold value, the index finger and the middle finger are determined to be in contact, and the hand point coordinates are the coordinates of the end feature point of the index finger. Claim 48 A camera vision-based hand gesture recognition method according to claim 44 or 47, characterized in that the preset contact reference value is the distance between the end feature point of the middle finger and the distal interphalangeal joint feature point so as to vary according to camera zoom in / out. Claim 49 In claim 45, the hand point gesture recognition step is characterized by: when the continuous frame list reaches the preset number of frames, satisfying the condition that the number of frames containing the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio, and if the current frame is a frame containing the hand point posture, comparing the hand point coordinates of the leading frame among the frames containing the hand point posture in the continuous frame list with the hand point coordinates of the current frame to obtain the movement direction having the maximum distance gap among upward, downward, leftward, and rightward directions; and if the condition that the maximum distance gap is greater than or equal to a preset distance gap reference value is satisfied, recognizing the movement direction and the maximum distance gap as the movement direction and movement distance of the hand point, respectively, and utilizing them as user interface commands if necessary; and resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing the hand point gesture. Claim 50 A camera vision-based hand gesture recognition method according to claim 49, characterized in that the preset hand point pose occupancy ratio is 80%, and the preset distance gap reference value is the distance between the wrist joint feature point and the metacarpophalangeal joint feature point of the index finger within the current frame, so as to vary according to camera zoom in / out. Claim 51 In claim 35, to suppress the adverse effects of noise during the hand point gesture recognition process, the coordinate obtained from the combination of hand point coordinates of the introductory frames is a coordinate obtained by median filtering the hand point coordinates of the introductory frames of the continuous frame list in the horizontal and vertical directions, or a coordinate obtained by summing and averaging after removing outliers, and the coordinate obtained from the combination of hand point coordinates of the end frames is a coordinate obtained by median filtering the hand point coordinates of the end frames of the continuous frame list in the horizontal and vertical directions, or a coordinate obtained by summing and averaging after removing outliers, characterized in that it is a camera vision-based hand gesture recognition method. Claim 52 delete Claim 53 A camera vision-based hand gesture recognition method characterized in that, in any one of claims 38, 42, and 49, at least one of the user interface commands is audibly expressed through Text-to-Speech (TTS) when necessary. Claim 54 A camera vision-based hand gesture recognition method according to claim 53, characterized in that the text-to-speech (TTS) utilizes pyttsx3, a Python text-to-speech library. Claim 55 A camera vision-based hand gesture recognition method according to any one of claims 35 to 47, 49, 50, and 51, further comprising the step of performing a user interface in which the hand gesture recognition system utilizes as a gesture-based user command any one of the following: a user interface command received from either the left or right hand through camera vision-based hand gesture recognition, a user interface command received from both hands through camera vision-based hand gesture recognition, and a command created by combining the user interface commands received from each of the two hands through camera vision-based hand gesture recognition. Claim 56 A method using a hand gesture recognition system that detects whether fingers are spread or bent using hand model feature points (landmarks of hands model) detected based on camera vision, wherein when the hand gesture recognition system detects a hand point posture defined as a combination of fingers touching and spreading, and each finger spreading and bending, using the hand model feature points detected in each input frame, a hand point posture recognition step of recognizing the coordinate of any one of the hand model feature points or a coordinate obtained from a combination of the hand model feature points as a hand point coordinate and displaying it if necessary; A hand point gesture recognition method based on camera vision, wherein if the number of frames including the hand point pose among a continuous frame list composed of a preset number of frames is greater than or equal to a preset number of hand point pose occupancy ratio, the hand point gesture recognition step calculates the distance and direction of movement of the hand point by comparing the hand point coordinates of any one frame among the introductory frames of the continuous frame list or a combination of such hand point coordinates with the hand point coordinates of any one frame among the end frames of the continuous frame list or a combination of such hand point coordinates; wherein the continuous frame list is loaded as the outermost frame of the list whenever each current frame is input, and when the list reaches the preset number of frames, the leading frame of the list is deleted one by one and the current frame is loaded as the outermost frame, thereby updating the list so that the latest frames equal to the preset number of frames are stored, and when a user interface command is recognized, the process of resetting everything to recognize the next user interface command is repeated. Claim 57 A camera vision-based hand gesture recognition method according to claim 56, characterized in that the hand model feature points are landmarks of Google’s MediaPipe Hands Model, and the fingers are fingers of either the left hand or the right hand, or fingers of both hands. Claim 58 In claim 56, the hand gesture recognition system determines whether the finger is extended or bent by calculating the finger joint angle using the coordinates of the hand model feature points; the angle of the finger joint is such that the index, middle, and ring fingers are determined as extended if the angle between the line segment connecting the proximal interphalangeal joint feature point of each finger to the distal interphalangeal joint feature point and the line segment connecting the proximal interphalangeal joint feature point of that finger to the metacarpophalangeal joint feature point is 100° or greater, and as bent otherwise; the ring finger is determined as extended if the angle is 150° or greater, and as bent otherwise; and in the case of the thumb, the angle between the line segment connecting the interphalangeal joint feature point of the thumb to the tip feature point of the thumb and the angle between the interphalangeal joint feature point of the thumb A camera vision-based hand gesture recognition method characterized by determining that if either of the angle between line segments connecting feature points of the metacarpophalangeal joint, the angle between the line segment connecting the feature point of the metacarpophalangeal joint of the thumb to the feature point of the interphalangeal joint of the thumb, and the angle between the line segment connecting the feature point of the metacarpophalangeal joint to the feature point of the carpometacarpal joint is 150° or greater, it is determined as unfolded, and otherwise, it is determined as bent. Claim 59 A camera vision-based hand gesture recognition method according to claim 56, wherein the hand point posture is a posture in which the index finger and middle finger are in an open state and touching each other, and all other fingers are bent, using the hand model feature points detected in each input frame, and the hand point coordinates are the coordinates of either the end feature point of the index finger or the end feature point of the middle finger. Claim 60 A camera vision-based hand gesture recognition method according to claim 59, wherein if the distance between the tip feature points of the index finger and the tip feature point of the middle finger in the hand point posture is less than or equal to a preset contact threshold value, the index finger and the middle finger are determined to be in contact, and the hand point coordinates are the coordinates of the tip feature point of the index finger. Claim 61 A camera vision-based hand gesture recognition method according to claim 60, characterized in that the above-mentioned preset contact reference value is the distance between the end feature point of the middle finger and the distal interphalangeal joint feature point so as to vary according to camera zoom in / out. Claim 62 In claim 56, the hand point gesture recognition step is characterized by: when the continuous frame list reaches the preset number of frames, satisfying the condition that the number of frames containing the hand point posture in the continuous frame list is greater than or equal to the preset hand point posture occupancy ratio, and if the current frame is a frame containing the hand point posture, comparing the hand point coordinates of the leading frame among the frames containing the hand point posture in the continuous frame list with the hand point coordinates of the current frame to obtain the movement direction having the maximum distance gap among upward, downward, leftward, and rightward directions; and if the condition that the maximum distance gap is greater than or equal to a preset distance gap reference value is satisfied, recognizing the movement direction and the maximum distance gap as the movement direction and movement distance of the hand point, respectively, and utilizing them as user interface commands if necessary; and resetting the continuous frame list to recognize the next command of the user interface command, and then re-recognizing the hand point gesture. Claim 63 A camera vision-based hand gesture recognition method according to claim 62, wherein the preset number of frames is twice the number of frames per second, the preset hand point pose occupancy ratio is 80%, and the preset distance gap reference value is the distance between the wrist joint feature point and the metacarpophalangeal joint feature point of the index finger within the current frame, such that it varies according to camera zoom in / out. Claim 64 In claim 56, to suppress the adverse effects of noise during the hand point gesture recognition process, the coordinate obtained from the combination of hand point coordinates of the introductory frames is a coordinate obtained by median filtering the hand point coordinates of the introductory frames of the continuous frame list in the horizontal and vertical directions, or a coordinate obtained by summing and averaging after removing outliers, and the coordinate obtained from the combination of hand point coordinates of the end frames is a coordinate obtained by median filtering the hand point coordinates of the end frames of the continuous frame list in the horizontal and vertical directions, or a coordinate obtained by summing and averaging after removing outliers, characterized in that it is a camera vision-based hand gesture recognition method. Claim 65 delete Claim 66 A camera vision-based hand gesture recognition method according to claim 62, characterized in that the user interface command is audibly expressed through Text-to-Speech (TTS) when necessary. Claim 67 A camera vision-based hand gesture recognition method according to claim 66, characterized in that the text-to-speech (TTS) utilizes pyttsx3, a Python text-to-speech library. Claim 68 A camera vision-based hand gesture recognition method characterized by further including the step of performing a user interface that utilizes as a gesture-based user command any one of the following: a user interface command received through camera vision-based hand gesture recognition from either the left hand or the right hand, a user interface command received through camera vision-based hand gesture recognition from both hands, and a command created by combining the user interface commands received through camera vision-based hand gesture recognition from each of both hands.

Images