Man-machine interaction technology only depending on single finger

Abstract

The present disclosure relates to a man-machine interaction system for an interaction device, comprising: an image acquisition unit configured to acquire a hand image including a hand of a user; the hand detection part is configured to determine the position of a key point of the hand in the hand image; and the interaction execution part is configured to calculate the relative instantaneous speed of a fingertip key point of a predetermined finger of the user relative to a reference point according to the position of the key point, and judge the intention of the user to control the interaction equipment according to the relative instantaneous speed. In addition, the invention further relates to a corresponding man-machine interaction method, electronic equipment, a computer readable medium and a computer program product. According to the invention, the intention of the user to control the interactive device can be identified only by depending on the bending of a single finger.

Description

Technical Field

[0001] The present disclosure relates to human-computer interaction technology that relies solely on a single finger, and more specifically, to a human-computer interaction system and method that relies solely on the bending of a single finger to identify user intent. Furthermore, the present disclosure also relates to related electronic devices, computer-readable media, and computer program products. Background Art

[0002] Currently, with the widespread adoption of digital cameras, image and video-based gesture recognition has become a key research area in computer vision. Many traditional approaches for dynamic gesture recognition have already been proposed, including hidden Markov models (HMMs), finite state machines (FSMs), dynamic Bayesian networks (DBNs), artificial neural networks (ANNs), and combined neural networks. These traditional approaches often require manually defined features, such as scale-invariant feature transforms (SIFTs), scale-invariant acceleration features (SURFs), histograms of oriented gradients (HOGs), and Fourier descriptors for hand shape descriptions, as well as gesture motion information descriptions based on optical flow and motion tracking. Methods requiring manually defined features have significant limitations, as they often require prior knowledge, experience, extensive manual adjustments, and a large and diverse number of training samples. Furthermore, the recognition rate of deep learning models is significantly affected by variations in the speed and direction of dynamic gestures, as well as hand size.

[0003] For example, patent document CN 110490165 A discloses a dynamic gesture recognition method based on a combined neural network. This method has the problems of complex tracking gesture sequences, a heavy gesture recognition model, and can only achieve gesture tracking.

[0004] Patent document CN 111813321 A discloses a method for air gesture control. This type of method has the problem that one gesture can only correspond to one function. Therefore, air gestures will increase as the number of control types increases, and the gestures are numerous and the operation range is large, which increases the user's learning cost and usage fatigue, and thus gradually gives up the desire for air gesture interaction. This type of gesture recognition is performed on multiple categories of gestures. Therefore, as the number of gestures increases, the number of categories will also increase. The accuracy of accurately identifying a gesture decreases, especially when a similar gesture category is added as a new category and needs to be identified, the recognition accuracy of the model will drop significantly. This type of gesture recognition interaction solution has high maintenance costs and is difficult to update and upgrade.

[0005] On the other hand, with the continuous development of machine vision technology and artificial intelligence technology, object position prediction is an important research direction in the fields of computer vision, machine learning and artificial intelligence. It has important value in many application scenarios such as gesture control, autonomous driving, robot navigation, video surveillance, motion analysis, etc.

[0006] In the prior art, there are mainly three types of location prediction.

[0007] First, prediction methods based on physical models: This method primarily uses information about the physical properties of an object, such as mass, velocity, and acceleration, to build a dynamic model for prediction. However, this method relies on a precise understanding of the object's characteristics and environmental factors, and has difficulty handling complex and uncertain dynamic environments.

[0008] Second, there are prediction methods based on statistical learning. These methods, including Kalman filtering and particle filtering, build probabilistic models for prediction by learning and analyzing historical data on object motion. However, these methods can experience performance degradation when dealing with nonlinear, high-dimensional, and noisy data.

[0009] Third, prediction methods based on deep learning: In recent years, deep learning has made significant progress in predicting the movement and position of objects. Deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) can automatically learn the complex patterns and regularities of object motion from large amounts of data. However, deep learning methods typically require large amounts of labeled data, and the models' interpretability and generalization capabilities still need to be improved.

[0010] With the development of the research field of object motion position prediction, although the above-mentioned prediction methods in the prior art have their own advantages, there is still a need to further overcome the problems existing therein.

[0011] It is necessary to predict the object movement speed data identified by the recognition and control system, which will help reduce response delays and thus improve user experience. Summary of the Invention

[0012] Technical issues

[0013] In view of the above-mentioned deficiencies of most existing gesture interaction technologies, the purpose of the present disclosure is to provide a human-computer interaction technology that only relies on the bending of a single finger to recognize the user's intention.

[0014] Technical solutions to technical problems

[0015] According to a first aspect of the present disclosure, a human-computer interaction system for an interactive device is provided, comprising: an image acquisition unit configured to acquire a hand image containing a user's hand; a hand detection unit configured to determine the positions of key points of the hand in the hand image; and an interaction execution unit configured to calculate the relative instantaneous velocity of the fingertip key point of the user's predetermined finger relative to a reference point based on the position of the key point, and to determine the user's intention to control the interactive device based on the relative instantaneous velocity.

[0016] According to a second aspect of the present disclosure, a human-computer interaction method for an interactive device is provided, which includes the following steps: acquiring a hand image including a user's hand; determining the positions of key points of the hand in the hand image; calculating the relative instantaneous speed of the fingertip key points of the user's predetermined fingers relative to a reference point based on the positions of the key points; and judging the user's intention to control the interactive device based on the relative instantaneous speed.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory; and a computer program stored in the memory, wherein the computer program implements the above-mentioned human-computer interaction method when executed by the processor.

[0018] According to a fourth aspect of the present disclosure, a computer-readable medium storing a computer program is provided, wherein the computer program implements the above-mentioned human-computer interaction method when executed by a processor.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, which implements the above-mentioned human-computer interaction method when executed by a processor.

[0020] Beneficial effects

[0021] Different from traditional gesture control and recognition, the human-computer interaction technology according to the present disclosure utilizes the user's most direct and comfortable gestures to accurately and effectively identify the user's intentions, thereby improving the human-computer interaction experience.

[0022] Furthermore, the gesture recognition algorithm used in the human-computer interaction technology disclosed herein is independent of neural networks and does not require any third-party hardware. Hand tracking can be achieved instantly simply by extending a finger, and effective tracking can be achieved even on non-dedicated devices such as home appliances within a short period of time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A human-computer interaction system and its application scenario according to the first embodiment of the present disclosure are shown.

[0024] Figure 2The hand key points determined by the hand detection unit in the hand image are shown.

[0025] Figure 3 A flow chart of a human-computer interaction method according to a second embodiment of the present disclosure is shown.

[0026] Figure 4 An electronic device according to a third embodiment of the present disclosure is shown.

[0027] Figure 5 The flowchart of the position prediction method based on the most recent motion feature according to another embodiment of the present invention is shown.

[0028] Figure 6 A schematic diagram showing an optical mouse control system according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] The present disclosure is further described below with reference to the specific embodiments shown in the accompanying drawings so that those skilled in the art can better understand and implement the present disclosure. However, the embodiments are not intended to limit the present disclosure. The embodiments of the present disclosure and the technical features of each embodiment may be combined with each other unless there is a conflict.

[0030] [First Form]

[0031] <First embodiment>

[0032] Figure 1 The human-computer interaction system 100 and its application scenario according to the first embodiment of the present disclosure are shown. Figure 1 In the embodiment, the human-computer interaction system 100 is a separate component and is in communication with the interactive device 150. However, the human-computer interaction system 100 may also be a part of the interactive device 150, for example, the interactive device 150 implements the functions of the human-computer interaction system 100 by executing a computer program stored in a memory through a processor.

[0033] The interactive device 150 may be, for example, a television, an advertising screen, an in-vehicle computer, a personal computer, or XR glasses.

[0034] like Figure 1 As shown, the human-computer interaction system 100 includes an image acquisition unit 110 , a hand detection unit 120 and an interaction execution unit 130 .

[0035] The operations and functions of each component of the human-computer interaction system 100 will be described in detail below.

[0036] [Image Acquisition Unit 110]

[0037] Image acquisition unit 110 is connected to an external camera 140 to acquire a hand image containing the user's hand, captured by camera 140. According to the human-computer interaction technology disclosed herein, camera 140 can be a common RGB camera that can capture still or moving images containing the hand. Image acquisition unit 110 stores the received hand image in a memory (not shown) of human-computer interaction system 100 and / or directly outputs it to downstream components, such as hand detection unit 120.

[0038] In addition, the interactive device 150 may obtain the hand image captured by the camera 140 from the camera 140 or the image acquisition unit 110 to display it on its display device in real time.

[0039] although Figure 1 The camera 140 is shown as an external component of the human-computer interaction system 110, but the camera 140 may also be a part of the human-computer interaction system 100 or a part of the interaction device 150 according to the present disclosure. For example, in the human-computer interaction system 100, the camera 140 may be connected to the image acquisition unit 110 or integrated in the image acquisition unit 110.

[0040] However, optionally, because images captured by ordinary RGB cameras may contain a lot of noise under certain conditions (such as dark light, backlight, etc.), the image acquisition unit 110 may pre-process the hand image before outputting the acquired hand image to the hand detection unit 120 to reduce the noise in the hand image and make the hand image smoother, thereby enhancing the hand features in the hand image. For example, the human-computer interaction technology according to the present disclosure uses a Gaussian filter represented by the following expression (1):

[0041]

[0042] Among them, I smoothed (i, j) represents the preprocessed pixel value of the hand image at the pixel position (i, j), I(ik, jl) represents the initial pixel value of the hand image at the pixel position (ik, jl), k and l represent the offset of the center of the Gaussian filter relative to the current pixel position (i, j), and represents the weight of the Gaussian function, which determines the weight of each pixel so that pixels farther from the center have smaller weights.

[0043] [Hand detection unit 120]

[0044] The hand detection unit 120 can determine the hand key points and their positions based on the hand image received from the image acquisition unit 110, whether it has undergone the above-mentioned preprocessing or not. According to the human-computer interaction technology disclosed in the present invention, the number of hand key points can be, for example, 21, but can also be other numbers, such as 14, 22, etc.

[0045] like Figure 2 As shown in FIG, the 21 hand key points are wrist key point 0, thumb first to fourth key points 1 to 4, index finger first to fourth key points 5 to 8, middle finger first to fourth key points 9 to 12, ring finger first to fourth key points 13 to 16, and little finger first to fourth key points 17 to 20. In this article, the first key point of each finger is also referred to as the base key point of the corresponding finger, and the fourth key point of each finger is referred to as the fingertip key point of the corresponding finger.

[0046] The process of determining the key points of the hand by the hand detection unit 120 will be described below.

[0047] First, the hand detection unit 120 uses a relevant hand detection algorithm to determine the extent of the hand region (i.e., the area within the bounding box that exactly surrounds the entire hand) in the hand image. Hand detection algorithms are well known in the art, and those skilled in the art can select them based on their needs and application scenarios. Therefore, they will not be described in detail here.

[0048] For example, the hand detection unit 120 can determine that the coordinate range of the hand area in the hand image is (x0, y0, w, h), where (x0, y0) is the coordinate of the upper left corner pixel of the hand area, and w and h are the width and height of the hand area, respectively.

[0049] Then, the hand detection unit 120 can extract the hand region from the hand image according to the coordinate range of the hand region in the hand image. For example, the extraction of the hand region can be performed according to the following expression (2):

[0050] I and (i, j) = I (i + x0, j + y0) ... (2)

[0051] Among them, I hand (i, j) is the pixel value of the hand area at the pixel position (i, j), and satisfies 0≤i≤w and 0≤j≤h.

[0052] Due to the limited resolution of ordinary RGB cameras, the extracted hand region often has low resolution and may even have "afterimages" during high-speed hand movement. Furthermore, if the background of the extracted hand region contains patterns with colors similar to skin color, the accuracy of hand key point calculations will be significantly affected. To address this, image enhancement processing can optionally be performed on the hand region. This image enhancement processing includes convolution, logarithmic transformation, and normalization.

[0053] The convolution process is used to filter out specific frequencies in the frequency domain of the hand area to enhance the image quality. Here, the convolution process can be expressed by the following expression (3):

[0054] G(u,v)=F(u,v)·H(u,v)……(3)

[0055] Where G(u,v) is the convolution result, F(u,v) is the Fourier transform of the hand region, and H(u,v) is the Fourier transform of the convolution kernel. The convolution kernel determines the filtering characteristics of the convolution process. The convolution result G(u,v) can be inverse Fourier transformed to obtain the convolved hand region.

[0056] The logarithmic transformation process is used to reduce the contrast of the hand area. The logarithmic transformation process can be expressed by the following expression (4):

[0057] I log (i,j)=C·log(1+I hand (i,j))……(4)

[0058] Among them, I log (i, j) is the pixel value of the hand region at position (i, j) after the logarithmic transformation, and C is a constant for adjusting the contrast.

[0059] Normalization is used to ensure that the pixel values (e.g., RGB, YUV, or HSV values) of the hand area are within a reasonable range. The normalization can be expressed by the following expression (5):

[0060]

[0061] Among them, I normalized (i, j) is the normalized pixel value of the hand region at position (i, j), μ is the mean of the hand region, and σ is the standard deviation of the hand region.

[0062] Then, the hand detection unit 120 may determine the positions of the key points of the hand in the hand image that has undergone image enhancement processing or has not undergone image enhancement processing.

[0063] Specifically, the hand detection unit 120 rotates and / or flips the hand image to a suitable angle so that the hand faces upward and the palm faces the direction of the lens. Thereafter, the hand detection unit 120 inputs the hand image into the hand key point neural network and obtains the coordinate information points of the hand key points. For example, in order to facilitate the deployment of the human-computer interaction technology according to the present disclosure and reduce the computing power requirements, the backbone architecture of the hand key point neural network can adopt a lightweight MobieNet-V2 network and use multiple layers of 1×1 convolution. In order to make reasoning faster and more accurate, the network converts the convolution features into N×1 dimensional features through a fully connected layer, and uses a one-dimensional neural network layer as a regression layer to regress the coordinate information points of the hand key points. Finally, the hand detection unit 120 restores the hand image to its state before rotation, thereby obtaining the coordinate information points of the hand key points before the image is rotated.

[0064] [Interaction Execution Unit 130]

[0065] The interaction execution unit 130 can receive the positions of the key points of the hand from the hand detection unit 120, and identify or judge the user's intention (such as interaction willingness, interaction intention, etc.) based on these positions to facilitate the interaction between the user and the interactive device.

[0066] Hand tracking

[0067] The interaction execution unit 130 can control the position of the cursor displayed on the interactive device based on the position of a hand key point, so that the cursor is always displayed at a position corresponding to a point on the hand (e.g., the tip of the index finger), thereby achieving hand tracking of the cursor. For example, the cursor position can be controlled based on any of the 21 key points, such as wrist key point 0.

[0068] However, for a common RGB camera 140 with high noise, hand tracking of the cursor may be jittery, seriously affecting the interactive experience. To solve the cursor jitter problem, the interaction execution unit 130 can adjust the cursor movement sensitivity based on, for example, the instantaneous speed of wrist key point 0.

[0069] Specifically, in any image frame, assuming that the coordinates of the wrist key point are W, the interaction execution unit 130 can calculate the instantaneous velocity V of the wrist key point in the image frame according to the following expression (6): inst :

[0070]

[0071] Wherein, Δt is the duration of one image frame, and i is the sequence number of the image frame.

[0072] On the one hand, when the instantaneous velocity V of the wrist key point inst Greater than or equal to the threshold V δWhen the wrist moves significantly, the cursor sensitivity S is kept at the standard sensitivity S N On the other hand, when the instantaneous velocity V of the wrist key point inst Less than the threshold V δ When the wrist is considered to have no obvious movement or just noise, the sensitivity S at this time is the standard sensitivity S N k times the average instantaneous velocity V avg With threshold V δ Therefore, the cursor sensitivity S can be adjusted according to the following expression (7):

[0073]

[0074] Among them, V δ is defined to be equal to the minimum possible instantaneous velocity of the wrist keypoint, k is a decimal between 0 and 1, and n is a positive number.

[0075] To eliminate the influence of interfering factors such as noise, the interaction execution unit 130 can preferably adjust the cursor sensitivity based on the average instantaneous velocity of wrist key point 0. In this case, according to the above expression, the interaction execution unit 130 can calculate the instantaneous velocity of the wrist key point in two adjacent image frames and calculate the arithmetic average of these two instantaneous velocities as the average instantaneous velocity of wrist key point 0. In this example, two adjacent image frames are used to calculate the average instantaneous velocity of wrist key point 0, but three or more adjacent image frames can also be used.

[0076] The dynamic sensitivity adjustment method adopted can not only effectively suppress noise, but also retain the effect that when the user moves his hand slowly, the cursor will also move slowly, instead of being completely still and affecting the user experience.

[0077] User interaction intention identification

[0078] The interaction execution unit 130 may identify the user's interaction intention based on the orientation of the user's palm and / or the extension degree of specific fingers, and wake up the human-computer interaction system accordingly.

[0079] For example, the user can control the index finger flexibly and effortlessly, and thus the index finger may be preferably selected as the specific finger.

[0080] For example, if the back of the user's hand is not facing the camera and / or the index finger is in an extended state, the interaction execution unit 130 can determine that the user has the intention to interact, and thus activate the interaction function of the human-computer interaction system 100, thereby allowing the user to interact with the interactive device.

[0081] For example, the interaction execution unit 130 may identify the direction of the back of the hand according to the position of the key points of the hand. If the back of the hand is not facing the camera, it may be considered that the user has the intention to interact with the interactive device.

[0082] For example, the interaction execution unit 130 can use the positions of the four key points of the index finger to calculate the straightness of the connecting line between them and thereby identify the degree of extension of the index finger. Regardless of the user's gesture, as long as the index finger is extended or straightened, such as opening the palm, making the number "1" gesture, or waving, the index finger can be identified as being in an extended state.

[0083] If the above situation does not exist, for example, the back of the user's hand is facing the camera, the palm is naturally hanging down, the palm is naturally put down, or the palm disappears from the field of view of the camera, the interaction execution unit 130 can determine that the user does not have or no longer has the intention to interact, and therefore turns off the interaction function of the human-computer interaction system 100.

[0084] User intention judgment

[0085] The interaction execution unit 130 can determine the user's intention, such as the interaction intention, based on the relative instantaneous velocity of the key point of the fingertip of the specific finger relative to the reference point. The user's interaction intention may include the user's desire to perform a press operation, a release operation, or a click operation on the interactive device.

[0086] For example, the index finger is relatively flexible, so the index finger tip key point 8 can be preferably used as the fingertip key point of a specific finger. The index finger base key point 5 can be selected as the reference point, but other key points can also be selected, such as the wrist key point 0 or the base key point of other fingers.

[0087] The calculation of the relative instantaneous velocity of the index finger tip key point 8 relative to the index finger base key point 5 can be based on the projection of the displacement vector difference between the index finger tip key point 8 and the index finger base key point 5 in two adjacent image frames in the direction from the index finger tip key point 8 to the index finger base key point 5.

[0088] Specifically, in any image frame, assume that the coordinate of the key point 8 of the index finger tip is F, and the coordinate of the key point 5 of the index finger base is W. In this case, in this image frame, the relative instantaneous velocity V of the key point 8 of the index finger tip is r It can be calculated according to the following expression (8):

[0089]

[0090] Where i is the sequence number of the image frame, and α is the displacement vector difference The angle between the direction from the key point 8 of the index finger tip to the key point 5 of the index finger base. Note that the relative instantaneous speed V rThe sign of is used to judge the movement of the index finger, that is, when it is less than zero, it means that the index finger makes an extension movement, and when it is greater than zero, it means that the index finger makes a bending movement.

[0091] In order to determine whether the user's interaction intention is "press" or "release", the relative instantaneous speed V of the key point 8 of the index finger tip can be used to determine whether the user's interaction intention is "press" or "release". r To directly determine whether the user's index finger is bending or stretching.

[0092] For example, the interaction execution unit 130 can calculate the relative instantaneous velocity V of the key point 8 of the index finger tip according to ... r The absolute amount and sign of the touch gesture are used to determine the user's interaction intention, that is, whether the user wants to perform a "press" or "release" operation on the interactive device.

[0093] If the relative instantaneous velocity V of the key point 8 of the index finger tip r Greater than zero and the absolute amount is greater than the press action threshold The interaction execution unit 130 determines that the user's index finger makes a "bending" action, and sends an instruction for a "pressing" operation to the interaction device 150 .

[0094] If the relative instantaneous velocity V of the key point 8 of the index finger tip r Less than zero and the absolute value is greater than the release operation threshold The interaction execution unit 130 determines that the user's index finger makes a "stretching" action, and sends an instruction for a "relaxing" operation to the interaction device 150 .

[0095] If the relative instantaneous velocity V of the key point 8 of the index finger tip r in and , the interaction execution unit 130 determines whether the user's index finger maintains the previous "bend" or "stretch" action, and sends an instruction for maintaining the previous operation to the interaction device 150.

[0096] The above process can be summarized as the following expression (9):

[0097]

[0098] here, It is defined based on the minimum possible speed of a press operation. is defined in terms of the minimum possible speed of the release operation and can be equal to

[0099] Preferably, to eliminate the influence of interference factors such as noise, the interaction execution unit 130 may also determine the user's intention based on the average relative instantaneous velocity of the index fingertip key point 8. For example, the average relative instantaneous velocity of the index fingertip key point 8 may be calculated as the arithmetic average of the relative instantaneous velocities of the index fingertip key point 8 in two or more adjacent image frames.

[0100] In addition, if the interaction execution unit 130 recognizes the “press” operation and the “release” operation successively within a predetermined time (eg, 140 milliseconds), it can be determined that the user has completed a valid “click” operation.

[0101] The interaction execution unit 130 may also use the bending or extension of the finger joints to determine whether the user's index finger performs a "press" operation or a "release" operation, or to determine whether the user performs a "click" operation. In principle, the bending or extension of the index finger can be defined as any one or more of the three joints of the index finger, namely, the proximal interphalangeal joint, the distal interphalangeal joint, and the metacarpophalangeal joint, bending or extending within a predetermined time. In the process of identifying whether the joints are bent or extended, since the bending or extension amplitudes of the three joints are different under the same bending or extension action of the index finger, different bending or extension amplitude thresholds can be specified for them to determine whether they are bent or extended.

[0102] Specifically, using geometric methods, it is possible to identify whether the respective finger joints are flexed or extended based on the changes in the relative positions of some or all of the four key points of the index finger associated with the three index finger joints relative to a reference point within a predetermined time. If the reference point is the wrist key point, this identification can be performed using the changes in the relative positions of some or all of the key points of the index finger relative to the wrist key point within a predetermined time. If the reference point is the key point of the base of the index finger, this identification can be performed using the changes in the relative positions of some or all of the key points of the index finger, excluding the key point of the base of the index finger, relative to the key point of the base of the index finger within a predetermined time.

[0103] For example, if any one or more of the three joints of the index finger bends or extends within a predetermined time (e.g., 140 milliseconds), the interaction execution unit 130 determines that the user's index finger has performed a "press" or "release" operation and sends a "press" or "release" instruction to the interactive device. Similarly, if the interaction execution unit 130 recognizes a "press" operation and then a "release" operation within a predetermined time (e.g., 140 milliseconds), it can be determined that the user has completed a valid "click" operation.

[0104] During operation, the user only needs to move the index finger slightly within a predetermined time to complete an effective click. This is because the movement of the index finger will inevitably cause one or more of the three joints of the index finger to bend, and then return to the state before bending.

[0105] Changes in the cursor graphic representation

[0106] The interaction execution unit 130 may enable the cursor of the interaction device to have different graphic representations according to different recognized user intentions.

[0107] For example, in the case of a "press" operation, the cursor graphic is displayed as a fist. In the case of a "release" operation, the cursor graphic is displayed as an open palm. In the case of a "click" operation, the cursor graphic is displayed as a finger. Therefore, in order to provide feedback to the user's operation, the interactive device changes the cursor graphic representation when the user clicks or drags.

[0108] Continuous click protection

[0109] The interactive execution unit 130 also has a continuous click protection function. The interactive execution unit 130 can implement the continuous click protection function by filtering out multiple "click" operations in a short period of time to reduce accidental touches. Normally, the time it takes for a person to bend his fingers twice in a row is between 140 and 400 milliseconds. Therefore, the minimum interval time between double-clicks can be set to, for example, 100 milliseconds, 120 milliseconds, or 140 milliseconds. That is, the interactive execution unit 130 will identify two or more "clicks" that occur within the shortest time interval as erroneous operations or impossible situations, and therefore will not issue any instructions to the interactive device.

[0110] In summary, as mentioned above, in the gesture interaction of the existing technology, what bothers users the most is the numerous types of gestures and posture standard requirements, which makes the user's learning cost high and very tiring to use. In addition, the existing technology uses various deep learning models to recognize a large number of gestures, which leads to high computing power requirements and heavy deployment. In contrast, according to the interaction technology disclosed in the present invention, users only need to spontaneously and instinctively use their index fingers to click to interact with the interactive device. The clicking process includes a slight bending and rebound of the index finger joint, which can achieve effective recognition and response. The user has almost no learning cost, and long-term interaction will not cause obvious fatigue to the user.

[0111] <Second embodiment>

[0112] Figure 3 FIG. 2 is a flow chart of a human-computer interaction method 200 according to a second embodiment of the present disclosure.

[0113] First, in step 210, an RGB camera (e.g. Figure 1The camera 140 shown in FIG. 1 obtains a hand image including the user's hand.

[0114] In this step, before determining the hand key points, the hand image can be preprocessed to reduce noise in the hand image and make the hand image smoother, thereby enhancing the hand features in the hand image. For example, the preprocessing can use a Gaussian filter represented by the above expression (1).

[0115] Next, in step 220 , the positions (eg, pixel coordinates) of the hand key points are determined in the hand image.

[0116] First, a hand detection algorithm is used to extract the hand region (i.e., the region just within the bounding box containing the entire hand) from the hand image. Optionally, the aforementioned image enhancement processing can be performed on the hand region. Finally, a hand keypoint neural network can be used to locate the hand keypoints in the hand image.

[0117] Next, in step 230, the position of the cursor displayed on the interactive device can be controlled based on the position of a hand key point (e.g., wrist key point 0), so that the cursor is always displayed at, for example, the tip of the index finger, thereby achieving hand tracking of the cursor. During the hand tracking process, to address cursor jitter, the cursor movement sensitivity can be adjusted based on, for example, the instantaneous velocity of wrist key point 0.

[0118] Next, in step 240 , the user's interaction intention may be determined based on the orientation of the user's palm and / or the extent of extension of specific fingers.

[0119] If the user has no intention to interact, step 240 returns the human-computer interaction method to step 230 to continue to perform hand tracking of the cursor.

[0120] On the other hand, if the user intends to interact, step 240 causes the human-computer interaction method to transition to step 250. In step 250, the user's intention, such as interaction intention, can be determined based on the relative instantaneous velocity of the key point of the fingertip of a specific finger relative to the reference point. The user's interaction intention may include the user's desire to perform a press, release, or click operation on the interactive device.

[0121] Furthermore, in step 250, the interactive device's cursor is given a different graphical representation based on the determined user intent. For example, in the case of a "press" operation, the cursor is displayed as a fist. In the case of a "release" operation, the cursor is displayed as an open palm. In the case of a "click" operation, the cursor is displayed as a finger. Thus, to provide feedback to the user's operation, the interactive device changes the cursor's graphical representation when the user clicks or drags.

[0122] In addition, in step 250, a continuous click protection function can be implemented by filtering out multiple "click" operations within a short period of time to reduce accidental touches. Generally, the time it takes for a person to bend their finger twice in a row is between 140 and 400 milliseconds. Therefore, the minimum interval between double-clicks can be set to, for example, 100 milliseconds, 120 milliseconds, or 140 milliseconds. In other words, the interaction execution unit 130 will identify two or more "clicks" occurring within the shortest time interval as an erroneous operation or an impossible situation, and therefore will not issue any instructions to the interactive device.

[0123] Next, in step 260, a human-computer interaction operation is performed according to the user intention determined in step 250. After the human-computer interaction operation is performed, the process returns to step 230, or the user operation is terminated when necessary.

[0124] Thus, the entire interaction process is completed.

[0125] <Third embodiment>

[0126] Figure 4 An electronic device according to a fourth embodiment of the present disclosure is shown.

[0127] like Figure 4 As shown, the electronic device 300 according to the third embodiment of the present disclosure includes a memory 310 , a processor 320 , and a computer program 311 stored in the memory 310 .

[0128] For example, the electronic device 300 may be a personal computer, a portable device, a microcomputer on a mobile object, etc. For example, the mobile object may be an airplane, a ship, a car, an electric car, an unmanned aerial vehicle, a robot, etc.

[0129] The processor 320 includes, for example, a data processing unit having a program execution function such as a CPU or a GPU, and implements the human-computer interaction method according to the above embodiment by executing the computer program 311 stored in the memory 310 .

[0130] <Fourth embodiment>

[0131] The present disclosure also relates to a computer-readable medium in which the above-mentioned computer program is stored.

[0132] The computer program may be executed by a data processor having a program execution function, such as a CPU or a GPU, to implement the human-computer interaction method according to the above-described embodiment.

[0133] <Fifth embodiment>

[0134] The present disclosure also relates to a computer program product, which may include the computer program described above.

[0135] [Second Form]

[0136] The present invention also relates to a position prediction method based on recent motion features and a gesture recognition and control system, and in particular, to a position prediction method capable of reducing response delay and a gesture recognition and control system using the position prediction method.

[0137] Combine Figure 5 , taking the implementation status of the position prediction method in the gesture recognition control system as an example to explain in detail.

[0138] The gesture recognition and control system may include a gesture recognition unit, a data acquisition unit, a motion trend determination unit, a processing unit, and a display carrier such as a screen. When the gesture recognition and control system is in operation, the data acquisition unit acquires the motion speed data identified by the gesture recognition unit and stores it in a memory as needed. The motion trend determination unit then extracts the most recent motion features and determines the motion trend of the speed data. The processing unit then makes a prediction based on the motion trend determination result. Specifically, the position prediction method based on the most recent motion features may include the following steps:

[0139] Step 1: Obtain the identified motion speed data, select n frames through the sliding window method to form the speed data matrix V, and initialize the algorithm parameters, including the following process:

[0140] (1) Obtain N frames of recognized motion speed data, assuming that the data are sampled at uniform time intervals;

[0141] (2) Setting a sliding window to limit the extraction range of the most recent motion features, obtaining n frames of motion speed data for prediction, and forming a speed data matrix V of size n × 1;

[0142] (3) Initializing the jog velocity threshold μ, which is the maximum value of the noise in the identified velocity data and needs to be determined based on the specific identification unit; when the motion velocity is close to the noise amplitude, the noise will interfere with the identified user's movement, which is called jog and requires special treatment, as detailed in the following steps; and

[0143] (4) Initialize the prediction superposition coefficient η, which is used to adjust the prediction strength.

[0144] Step 2: Determine the motion trend of the velocity data to classify the motion trend into acceleration, deceleration, and slowing processes. There are two methods for determining motion trends: singular value decomposition (SVD) or acceleration discrimination. For example, when using the acceleration discrimination method to determine motion trends, the following process is involved:

[0145] (1) Calculate the displacement data based on the velocity data matrix V. Assuming A is the acceleration data matrix and V is the velocity data matrix, the acceleration data per unit time can be obtained by discrete difference, as shown in the following formula:

[0146] A[n]=V[n]-V[n-1]

[0147] (2) Determine the sum of the elements of the acceleration data matrix A The relationship with the easing speed threshold μ is used to determine the motion trend, where:

[0148] The current movement trend is an accelerating process;

[0149] The current movement trend is decelerating; and

[0150] The current movement trend is a slow motion process.

[0151] When the singular value decomposition method is used to determine the movement trend, the following process is included:

[0152] (1) Perform singular value decomposition on the velocity data matrix V to decompose it into singular values, left singular vectors, and right singular vectors, as shown in the following formula:

[0153] V=U∑W *

[0154] Where U is the left singular vector matrix, which contains the distribution of motion patterns over time; Σ is the diagonal singular value matrix, which contains the importance of the data, where the singular values are arranged by size, and larger singular values correspond to the main motion trends; W * is the right singular vector matrix containing the distribution of motion modes in terms of velocity; and

[0155] (2) Right singular vector matrix W * Each column corresponds to a right singular vector, and its mathematical meaning can be understood as the weight distribution of a motion mode on the speed;

[0156] The main motion patterns and anomalies are identified by observing the relative sizes of the singular values and the patterns of the singular vectors. Acceleration and deceleration motion trends usually correspond to larger singular values and obvious patterns in the left and right singular vectors, while easing motion trends correspond to smaller singular values or abnormal patterns in the left and right singular vectors.

[0157] In step 3, if the motion trend is determined to be a slow motion process, the discrete Fourier transform and adaptive filtering are combined to extract small displacements in the slow motion process, and the extracted velocity data matrix V is obtained, and then the step 4 described below is performed; if the motion trend is determined to be an acceleration or deceleration process, the step 4 described below is directly performed. Step 3 includes the following process:

[0158] (1) For the velocity data matrix V as one-dimensional time domain data, the noise in the slowing process has a great influence on the real motion velocity data. Therefore, the velocity data is converted to the frequency domain through discrete Fourier transform (DFT). Here, the fast Fourier transform algorithm is used to obtain the frequency domain data X[k], where k represents the frequency index. First, the time domain data is decomposed and the N-point DFT is decomposed into Point DFT, and then recursively decompose, the purpose of this step is to reduce the computational complexity; secondly, perform transformation calculations, and convert the time domain data into frequency domain data based on the transformation mapping, as shown in the following formula:

[0159]

[0160] Finally, the frequency domain data is merged, and the solutions to the decomposed subproblems are combined into the solution to the original problem. The merging process requires multiplying the rotation factor to merge the frequency domain data and obtain the frequency domain data X[k].

[0161] (2) Adaptively filter the one-dimensional frequency domain data to obtain filtered frequency domain data Y[k]. The goal of the adaptive filter is to adjust the parameters of the filter according to the characteristics of the motion speed data to optimally remove interference. Specifically, first, initialize the parameters of the adaptive filter, which may include the filter weight W[k] and the adaptability parameter σ; second, use the adaptive filter to update the frequency domain data. The update rule of the adaptive filter can adopt the least mean square (LMS) algorithm, as shown in the following formula:

[0162] W[k](n+1)=W[k](n)+σe[n]X[k] *

[0163] Where W[k] is the filter weight, σ is the adaptability parameter, e[n] is the error signal, X[k] * is the conjugate of X[k];

[0164] Finally, the updated filter parameters are applied to the frequency domain data X[k] to obtain the filtered frequency domain data Y[k]. This can be achieved through point-by-point multiplication, as shown in the following formula:

[0165] Y[k]=W[k]·X[k]

[0166] (3) Perform inverse Fourier transform on the filtered frequency domain data Y[k] to obtain the time domain data V[n] after noise removal, as shown in the following formula:

[0167]

[0168] Step 4: Calculate the predicted superposition value Δv and the predicted value v according to the predetermined superposition curve prediction , including the following processes:

[0169] (1) Determine a prediction superposition curve f for describing the mapping relationship between the prediction superposition value and each element of the motion speed data V[n]. In an actual system, this curve usually needs to be determined based on the actual application scenario. The following formula is an example of a superposition curve:

[0170] f(v)=a3v 3 +a2v 2 +a1v+a0

[0171] The values of the coefficients of the above polynomials need to be determined experimentally;

[0172] (2) According to the superposition curve, the latest motion speed data V[n] is superimposed with the prediction superposition value Δv to calculate the prediction value v prediction :

[0173] Δv=f(V[n]),v prediction =V[n]+ηΔv

[0174] Afterwards, the predicted value is transmitted to the screen or other display carrier to obtain the predicted position, thereby reducing response delay, and the current frame prediction is completed. If new motion speed data is obtained from the system, jump to step 2; otherwise, the prediction process ends.

[0175] The prediction effect of this embodiment is shown in the following Table 1. Table 1 shows the average cursor movement delay in the non-prediction state and the prediction state, as well as the percentage of delay reduction in the prediction state compared to the non-prediction state:

[0176] Table 1

[0177]

[0178] It can be seen from Table 1 that the prediction method of the present invention can effectively reduce delay.

[0179] The present invention is not limited to the above-mentioned embodiments, and can also be applied to position prediction of an optical mouse control system, for example. Figure 6As shown in the left figure, when the user suddenly accelerates the mouse to move from the position indicated by the black arrow at time t0 to the position indicated by the black arrow at time t1, due to the system response delay, the cursor on the screen that was originally moving at a low speed does not follow the mouse to the corresponding position indicated by the dotted arrow, but is in a certain intermediate position indicated by the white arrow.

[0180] like Figure 6 As shown in the right figure, when the user suddenly decelerates and moves the mouse from the position indicated by the black arrow at time t0 to the position indicated by the black arrow at time t1, due to the system response delay, the cursor on the screen that was originally moving at high speed does not follow the mouse to the corresponding position indicated by the dotted arrow, but is at a certain right position indicated by the white arrow.

[0181] According to the present invention, by applying a technology similar to the above-mentioned gesture recognition control system, after data processing, the movement speed data of the mouse is predicted. Figure 6 The white arrow in the middle of the left figure gives the offset to the right and the Figure 6 The white arrow on the right side of the right figure is given an offset to the left, which can make the cursor be at the corresponding position shown by the dotted arrow and consistent with the position where the user moves the mouse, thus reducing the system's response delay and improving the user experience.

[0182] As in the first aspect, the second aspect of the present invention may also relate to an electronic device, a computer program, and a computer-readable storage medium, which will not be described again here.

[0183] As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it will be apparent that those skilled in the art may modify or replace these embodiments without departing from the gist of the present disclosure. That is, the present disclosure has been disclosed in the form of examples and should not be interpreted in a limiting sense. In order to determine the gist of the present disclosure, the claims should be considered.

[0184] Note that the present disclosure can have the following configurations.

[0185] [First Form]

[0186] (1) A human-computer interaction system for an interactive device, comprising:

[0187] an image acquisition unit configured to acquire a hand image including a hand of a user;

[0188] a hand detection unit configured to determine positions of key points of the hand in the hand image; and

[0189] An interaction execution unit is configured to calculate the relative instantaneous velocity of the fingertip key point of the user's predetermined finger relative to a reference point based on the position of the key point, and determine the user's intention to control the interaction device based on the relative instantaneous velocity.

[0190] (2) The human-computer interaction system according to (1), wherein:

[0191] The interaction execution unit is further configured to determine the direction of the back of the user's hand and / or the extension degree of the predetermined finger according to the position of the key point, and

[0192] The interaction execution unit identifies whether the user has a desire to interact with the interaction device based on the direction of the back of the user's hand and / or the degree of extension of the predetermined finger.

[0193] (3) The human-computer interaction system according to (2), wherein:

[0194] The interaction execution unit determines the intention of the user when recognizing that the user has an intention to interact.

[0195] (4) The human-computer interaction system according to any one of (1) to (3), wherein:

[0196] The key points also include key points of other fingers except the predetermined finger, and

[0197] The interaction execution unit calculates only the relative instantaneous velocity of the fingertip key point with respect to the reference point.

[0198] (5) The human-computer interaction system according to any one of (1) to (4), wherein:

[0199] The intention includes that the user wants to perform a press operation, a release operation, or a click operation on the interactive device.

[0200] (6) The human-computer interaction system according to any one of (1) to (5), wherein:

[0201] The predetermined finger is the index finger, and the reference point is a key point at the base of the index finger.

[0202] (7) The human-computer interaction system according to any one of (1) to (6), wherein:

[0203] The calculation of the relative instantaneous velocity is based on the projection of the displacement vector difference between the fingertip key point and the reference point in adjacent image frames in the direction from the fingertip key point to the reference point.

[0204] (8) The human-computer interaction system according to any one of (1) to (7), wherein:

[0205] The interaction execution unit is configured to control the interaction device according to a position of one of the key points so that the interaction device displays a cursor at a position corresponding to the fingertip of the predetermined finger in the hand image.

[0206] (9) The human-computer interaction system according to (8), wherein:

[0207] The interaction executing unit is configured to cause the cursor to have different graphic representations depending on the determined intention of the user.

[0208] (10) The human-computer interaction system according to (8) or (9), wherein:

[0209] The interaction execution unit is configured to calculate the instantaneous speed of the one hand key point according to the position of the key point, and dynamically adjust the sensitivity of the cursor according to the instantaneous speed.

[0210] (11) The human-computer interaction system according to any one of (5) to (10), wherein:

[0211] The interaction execution unit is configured to determine multiple click operations recognized within a predetermined time interval as erroneous operations.

[0212] (12) The human-computer interaction system according to any one of (1) to (11), wherein:

[0213] The image acquisition unit is further configured to perform pre-processing on the acquired hand image to increase features of the hand.

[0214] (13) The human-computer interaction system according to any one of (1) to (12), wherein:

[0215] The hand detection unit is further configured to perform a hand region extraction process and an image enhancement process before determining the position of the key point of the hand.

[0216] The hand region extraction process is used to extract the hand region from the hand image.

[0217] The image enhancement processing includes one or more of a convolution processing for filtering the frequency domain of the hand area, a logarithmic transformation processing for adjusting the contrast of the hand area and highlighting the hand details in the hand area, and a normalization processing for normalizing the hand area.

[0218] (14) The human-computer interaction system according to any one of (1) to (13), wherein:

[0219] The interactive device is a television, an advertising screen, an in-vehicle computer, a personal computer or XR glasses.

[0220] (15) A human-computer interaction method for an interactive device, comprising the following steps:

[0221] obtaining a hand image including a user's hand;

[0222] Determining positions of key points of the hand in the hand image;

[0223] Calculating the relative instantaneous velocity of the fingertip key point of the user's predetermined finger relative to the reference point based on the position of the key point; and

[0224] The user's intention to control the interactive device is determined according to the relative instantaneous speed.

[0225] (16) The human-computer interaction method according to (15), further comprising, before the step of determining the user's intention to control the interactive device:

[0226] Determine the direction of the back of the user's hand and / or the extension degree of the predetermined finger according to the position of the key point, and

[0227] Whether the user is willing to interact with the interactive device is identified based on the direction of the back of the user's hand and / or the extension degree of the predetermined finger.

[0228] (17) The human-computer interaction method according to (15) or (16), wherein:

[0229] The key points also include key points of other fingers except the predetermined finger, and

[0230] Only the relative instantaneous velocity of the fingertip key point relative to the reference point is calculated.

[0231] (18) The human-computer interaction method according to any one of (15) to (17), wherein:

[0232] The intention includes that the user wants to perform a press operation, a release operation, or a click operation on the interactive device.

[0233] (19) The human-computer interaction method according to any one of (15) to (18), wherein:

[0234] The predetermined finger is the index finger, and the reference point is a key point at the base of the index finger.

[0235] (20) The human-computer interaction method according to any one of (15) to (19), wherein:

[0236] The calculation of the relative instantaneous velocity is based on the projection of the displacement vector difference between the fingertip key point and the reference point in adjacent image frames in the direction from the fingertip key point to the reference point.

[0237] (21) The human-computer interaction method according to any one of (15) to (20), further comprising:

[0238] The interactive device is controlled according to the position of one of the key points, so that the interactive device displays a cursor at a position corresponding to the fingertip of the predetermined finger in the hand image.

[0239] (22) The human-computer interaction method according to (21), wherein the step of displaying the cursor further comprises:

[0240] The cursor is given different graphic representations according to the determined intention of the user.

[0241] (23) The human-computer interaction method according to (21) or (22), wherein the step of displaying the cursor further comprises:

[0242] The instantaneous speed of the one key point of the hand is calculated according to the position of the key point, and the sensitivity of the cursor is dynamically adjusted according to the instantaneous speed.

[0243] (24) The human-computer interaction method according to any one of (18) to (23), further comprising:

[0244] Multiple click operations recognized within a predetermined time interval are determined to be erroneous operations.

[0245] (25) The human-computer interaction method according to any one of (15) to (24), wherein the step of acquiring the hand image comprises:

[0246] Preprocessing for increasing features of the hand is performed on the acquired hand image.

[0247] (26) The human-computer interaction method according to any one of (15) to (25), wherein the step of determining the position of the key point of the hand comprises:

[0248] Before determining the positions of the key points of the hand, a hand region extraction process and an image enhancement process are performed.

[0249] The hand region extraction process is used to extract the hand region from the hand image.

[0250] The image enhancement processing includes one or more of a convolution processing for filtering the frequency domain of the hand area, a logarithmic transformation processing for adjusting the contrast of the hand area and highlighting the hand details in the hand area, and a normalization processing for normalizing the hand area.

[0251] (27) The human-computer interaction method according to any one of (15) to (26), wherein:

[0252] The interactive device is a television, an advertising screen, an in-vehicle computer, a personal computer or XR glasses.

[0253] (28) An electronic device comprising:

[0254] processor;

[0255] Memory; and

[0256] A computer program stored in the memory, wherein the computer program implements the human-computer interaction method as described in any one of (15) to (27) when executed by the processor.

[0257] (29) A computer-readable medium storing a computer program, wherein the computer program, when executed by a processor, implements the human-computer interaction method as described in any one of (15) to (27).

[0258] (30) A computer program product, which, when executed by a processor, implements the human-computer interaction method as described in any one of (15) to (27).

[0259] [Second Form]

[0260] (31) A method for predicting a position based on a recent motion feature comprises the following steps:

[0261] Step 1, obtaining the identified motion speed data to form a speed data matrix V;

[0262] Step 2: determining the motion trend of the speed data to classify the motion trend into acceleration, deceleration, and easing processes;

[0263] Step 3: If the motion trend is determined to be a slow motion process, extract the small displacements in the slow motion process to obtain the extracted velocity data matrix V, and then proceed to step 4 described below; if the motion trend is determined to be an acceleration or deceleration process, proceed directly to step 4 described below; and

[0264] Step 4: Calculate the predicted superposition value Δv and the predicted value v according to the predetermined superposition curve prediction , thus obtaining the predicted position.

[0265] This optimizes the delay of the recognition and control system and achieves the effect of reducing delay.

[0266] (32) The position prediction method according to (31), wherein step 1 comprises the following process:

[0267] (1) Obtain motion speed data of N frames;

[0268] (2) Setting a sliding window to limit the extraction range of the most recent motion features, obtaining n frames of motion speed data for prediction, and forming a speed data matrix V of size n × 1;

[0269] (3) Initializing the slow speed threshold μ, which is the maximum value of the identified speed data noise; and

[0270] (4) Initialize the prediction superposition coefficient η, which is used to adjust the prediction strength.

[0271] In this way, the problem of data selection and parameter calculation is solved, and the effect of supporting dynamic adjustment of parameters is achieved.

[0272] (33) The position prediction method according to (31) or (32), wherein step 2 comprises the following process:

[0273] (1) Calculate the displacement data based on the velocity data matrix V. Assuming A is the acceleration data matrix and V is the velocity data matrix, the acceleration data per unit time is obtained by discrete difference, as shown in the following formula:

[0274] A[n]=V[n]-V[n-1]

[0275] (2) Determine the sum of the elements of the acceleration data matrix A The relationship with the easing speed threshold μ is used to determine the motion trend, where:

[0276] The current movement trend is an accelerating process;

[0277] The current movement trend is decelerating; and

[0278] The current movement trend is a slow motion process.

[0279] In this way, the problem of judging the movement trend is solved, and the effect of judging the movement trend quickly and easily is achieved.

[0280] (34) The position prediction method according to (31) or (32), wherein step 2 comprises the following process:

[0281] (1) Perform singular value decomposition on the velocity data matrix V to decompose it into singular values, left singular vectors, and right singular vectors, as shown in the following formula:

[0282] V=U∑W *

[0283] Where U is the left singular vector matrix, which contains the distribution of motion patterns over time; Σ is the diagonal singular value matrix, which contains the importance of the data, where the singular values are arranged by size, and larger singular values correspond to the main motion trends; W * is the right singular vector matrix containing the distribution of motion modes in terms of velocity; and

[0284] (2) Right singular vector matrix W * Each column corresponds to a right singular vector, and its mathematical meaning is understood as the weight distribution of a motion mode on the velocity;

[0285] The main motion patterns and anomalies are identified by observing the relative sizes of the singular values and the patterns of the singular vectors. Acceleration and deceleration motion trends correspond to larger singular values and obvious patterns in the left and right singular vectors, while easing motion trends correspond to smaller singular values or abnormal patterns in the left and right singular vectors.

[0286] In this way, the problem of judging the movement trend is solved, and the effect of decomposing the movement pattern and speed distribution and accurately judging the movement trend is achieved.

[0287] (35) The position prediction method according to any one of (31) to (34), wherein step 3 comprises the following process:

[0288] (1) For the velocity data matrix V as one-dimensional time domain data, the velocity data is converted to the frequency domain by discrete Fourier transform to obtain frequency domain data X[k], where k represents the frequency index;

[0289] (2) performing adaptive filtering on the one-dimensional frequency domain data to obtain filtered frequency domain data Y[k]; and

[0290] (3) Perform inverse Fourier transform on the filtered frequency domain data Y[k] to obtain the time domain data V[n] after noise removal.

[0291] In this way, the problem of inaccurate recognition during the deceleration process is solved, and the effect of separating and eliminating noise and then extracting small displacements is achieved.

[0292] (36) The position prediction method according to any one of (31) to (35), wherein step 4 includes the following process:

[0293] (1) determining a prediction superposition curve f for describing a mapping relationship between a prediction superposition value and each element of the motion speed data V[n]; and

[0294] (2) According to the superposition curve, the latest motion speed data V[n] is superimposed with the prediction superposition value Δv to calculate the prediction value V prediction .

[0295] In this way, for the problem of determining the measured value of the next frame speed, a relatively accurate prediction value is calculated and dynamic adjustment of parameters is supported.

[0296] (37) The position prediction method according to (35), wherein the conversion of velocity data between the time domain and the frequency domain by Fourier transform comprises the following process:

[0297] (1) Decompose the time domain data and decompose the N-point DFT into Point DFT, then recursively decompose;

[0298] (2) Perform transformation calculations to obtain frequency domain data based on the transformation mapping of time domain data, as shown in the following formula:

[0299]

[0300] (3) merging the frequency domain data to obtain the frequency domain data X[k]; and

[0301] (4) Perform inverse Fourier transform on the filtered frequency domain data Y[k] to obtain the time domain data V[n] using the following formula:

[0302]

[0303] In this way, the technical problem of separating noise from real motion is solved by converting from time domain to frequency domain and then separating noise from real motion.

[0304] (38) The position prediction method according to (35), wherein the adaptive filtering of the one-dimensional frequency domain data comprises the following process:

[0305] (1) Initializing the parameters of the adaptive filter, which include the filter weight W[k] and the adaptability parameter σ;

[0306] (2) An adaptive filter is used to update the frequency domain data. The updating rule of the adaptive filter adopts the least mean square (LMS) algorithm, as shown in the following formula:

[0307] W[k](n+1)=W[k](n)+σe[n]X[k] *

[0308] Where W[k] is the filter weight, σ is the adaptability parameter, e[n] is the error signal, X[k] * is the conjugate of X[k]; and

[0309] (3) Apply the updated filter parameters to the frequency domain data X[k] to obtain the filtered frequency domain data Y[k], as shown in the following formula:

[0310] Y[k]=W[k]·X[k].

[0311] In this way, the technical problem of unstable separated noise is solved, and the effect of adaptive, dynamic separation and noise filtering is achieved.

[0312] (39) A gesture recognition control system configured to implement the position prediction method according to any one of (31) to (38).

[0313] (40) A gesture recognition and control system comprising:

[0314] A gesture recognition unit configured to recognize a user's gesture action;

[0315] a data acquisition unit configured to acquire the identified motion speed data to form a speed data matrix V;

[0316] a motion trend determination unit configured to determine the motion trend of the speed data to classify the motion trend into acceleration, deceleration, and easing processes;

[0317] A processing unit is configured to calculate the prediction superposition value Δv and the prediction value v according to the judgment result of the motion trend prediction , thus obtaining the predicted position.

[0318] (41) An electronic device comprising:

[0319] at least one processor; and

[0320] At least one memory storing a computer program, which, when executed by the at least one processor, implements the method according to any one of (31) to (38).

[0321] (42) A computer program that, when executed by a computer, causes the computer to perform the steps of the method according to any one of (31) to (38).

[0322] (43) A computer-readable storage medium having stored thereon the computer program according to (42).

[0323] According to the second embodiment of the present invention, the position prediction method and gesture recognition and control system utilize N frames of recognized motion velocity data, a motion trend determination algorithm, an algorithm for extracting small displacements during slow motion, and a pre-determined experimental overlay curve to predict the next frame's velocity and, in turn, the predicted position. This technology, provided continuous motion velocity data is available, can be successfully applied to gesture recognition and control systems, effectively reducing response latency and improving the user experience.

[0324] In addition, the second aspect of the present invention can also achieve the following technical effects.

[0325] 1. Realized the discrimination of the motion state of the control operation of the gesture recognition control system.

[0326] 2. Predict different motion trends and achieve targeted optimization of different states of the gesture recognition and control system.

[0327] 3. Reduced the user experience delay of the gesture recognition control system. Under the 30fps gesture recognition control system, the delay can be reduced by about 33ms.

[0328] 4. For gesture recognition and control systems, flexible prediction intensity control can be achieved, and the corresponding algorithm can be customized according to actual scenario needs.

[0329] 5. The separation of noise and small movements is achieved, and the gesture recognition control system can correctly identify that the user is moving slowly, improving the sensitivity of the gesture recognition control system from the user's perspective.

[0330] It will be apparent to those skilled in the art that the present disclosure is not limited to the details of the exemplary embodiments described above and that the present disclosure can be implemented in other specific forms without departing from the spirit or essential characteristics of the present disclosure. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present disclosure is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0331] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A human-computer interaction system for an interactive device, comprising: an image acquisition unit configured to acquire a hand image including a hand of a user; a hand detection unit configured to determine positions of key points of the hand in the hand image; as well as An interaction execution unit is configured to calculate the relative instantaneous velocity of the fingertip key point of the user's predetermined finger relative to a reference point based on the position of the key point, and determine the user's intention to control the interaction device based on the relative instantaneous velocity.

2. The human-computer interaction system according to claim 1, wherein: The interaction execution unit is further configured to determine the direction of the back of the user's hand and / or the extension degree of the predetermined finger according to the position of the key point, and The interaction execution unit identifies whether the user has a desire to interact with the interaction device based on the direction of the back of the user's hand and / or the degree of extension of the predetermined finger.

3. The human-computer interaction system according to claim 2, wherein: The interaction execution unit determines the intention of the user when recognizing that the user has an intention to interact.

4. The human-computer interaction system according to claim 1, wherein: The key points also include key points of other fingers except the predetermined finger, and The interaction execution unit calculates only the relative instantaneous velocity of the fingertip key point with respect to the reference point.

5. The human-computer interaction system according to claim 1, wherein: The intention includes that the user wants to perform a press operation, a release operation, or a click operation on the interactive device.

6. The human-computer interaction system according to claim 1, wherein: The predetermined finger is the index finger, and the reference point is a key point at the base of the index finger.

7. The human-computer interaction system according to claim 1, wherein: The calculation of the relative instantaneous velocity is based on the projection of the displacement vector difference between the fingertip key point and the reference point in adjacent image frames in the direction from the fingertip key point to the reference point.

8. The human-computer interaction system according to claim 1, wherein: The interaction execution unit is configured to control the interaction device according to a position of one of the key points so that the interaction device displays a cursor at a position corresponding to the fingertip of the predetermined finger in the hand image.

9. The human-computer interaction system according to claim 8, wherein: The interaction executing unit is configured to cause the cursor to have different graphic representations depending on the determined intention of the user.

10. The human-computer interaction system according to claim 8, wherein: The interaction execution unit is configured to calculate an instantaneous speed of the key point according to the position of the key point, and dynamically adjust the sensitivity of the cursor according to the instantaneous speed.

11. The human-computer interaction system according to claim 5, wherein: The interaction execution unit is configured to determine multiple click operations recognized within a predetermined time interval as erroneous operations.

12. The human-computer interaction system according to claim 1, wherein: The image acquisition unit is further configured to perform pre-processing on the acquired hand image to increase features of the hand.

13. The human-computer interaction system according to claim 1, wherein: The hand detection unit is further configured to perform a hand region extraction process and an image enhancement process before determining the position of the key point of the hand. The hand region extraction process is used to extract the hand region from the hand image. The image enhancement processing includes one or more of a convolution processing for filtering the frequency domain of the hand area, a logarithmic transformation processing for adjusting the contrast of the hand area and highlighting the hand details in the hand area, and a normalization processing for normalizing the hand area.

14. The human-computer interaction system according to claim 1, wherein: The interactive device is a television, an advertising screen, an in-vehicle computer, a personal computer or XR glasses.

15. A human-computer interaction method for an interactive device, comprising the following steps: obtaining a hand image including a user's hand; Determining positions of key points of the hand in the hand image; Calculating the relative instantaneous velocity of the fingertip key point of the user's predetermined finger relative to the reference point based on the position of the key point; and The user's intention to control the interactive device is determined according to the relative instantaneous speed.

16. The human-computer interaction method according to claim 15, further comprising, before the step of determining the user's intention to control the interactive device: Determine the direction of the back of the user's hand and / or the extension degree of the predetermined finger according to the position of the key point, and Whether the user is willing to interact with the interactive device is identified based on the direction of the back of the user's hand and / or the extension degree of the predetermined finger.

17. The human-computer interaction method according to claim 15, wherein: The key points also include key points of other fingers except the predetermined finger, and Only the relative instantaneous velocity of the fingertip key point relative to the reference point is calculated.

18. The human-computer interaction method according to claim 15, wherein: The intention includes that the user wants to perform a press operation, a release operation, or a click operation on the interactive device.

19. The human-computer interaction method according to claim 15, wherein: The predetermined finger is the index finger, and the reference point is a key point at the base of the index finger.

20. The human-computer interaction method according to claim 15, wherein: The calculation of the relative instantaneous velocity is based on the projection of the displacement vector difference between the fingertip key point and the reference point in adjacent image frames in the direction from the fingertip key point to the reference point.

21. The human-computer interaction method according to claim 15, further comprising: The interactive device is controlled according to the position of one of the key points, so that the interactive device displays a cursor at a position corresponding to the fingertip of the predetermined finger in the hand image.

22. The human-computer interaction method according to claim 21, wherein: The step of displaying the cursor further comprises: The cursor is given different graphic representations according to the determined intention of the user.

23. The human-computer interaction method according to claim 21, wherein: The step of displaying the cursor further comprises: The instantaneous speed of the key point is calculated according to the position of the key point, and the sensitivity of the cursor is dynamically adjusted according to the instantaneous speed.

24. The human-computer interaction method according to claim 18, further comprising: Multiple click operations recognized within a predetermined time interval are determined to be erroneous operations.

25. The human-computer interaction method according to claim 15, wherein: The step of acquiring the hand image comprises: Preprocessing for increasing features of the hand is performed on the acquired hand image.

26. The human-computer interaction method according to claim 15, wherein: The step of determining the position of the key points of the hand comprises: Before determining the positions of the key points of the hand, a hand region extraction process and an image enhancement process are performed. The hand region extraction process is used to extract the hand region from the hand image. The image enhancement processing includes one or more of a convolution processing for filtering the frequency domain of the hand area, a logarithmic transformation processing for adjusting the contrast of the hand area and highlighting the hand details in the hand area, and a normalization processing for normalizing the hand area.

27. The human-computer interaction method according to claim 15, wherein: The interactive device is a television, an advertising screen, an in-vehicle computer, a personal computer or XR glasses.

28. An electronic device comprising: processor; Memory; as well as A computer program stored in the memory, wherein when the computer program is executed by the processor, the computer program implements the human-computer interaction method according to any one of claims 15 to 27.

29. A computer-readable medium storing a computer program, wherein when the computer program is executed by a processor, the computer program implements the human-computer interaction method according to any one of claims 15 to 27.

30. A computer program product, wherein when the computer program is executed by a processor, the computer program implements the human-computer interaction method according to any one of claims 15 to 27.

Images