Interaction mode switching method driven by natural air gestures

Abstract

The invention discloses a natural air gesture-driven interaction mode switching method, which comprises the following steps that: when a user executes an interaction mode switching action, equipment acquires hand information of the user through an information acquisition device, and calculates depth information and standard deviation data of the hand of the user relative to the equipment by utilizing a sliding window at each moment; wherein the hand of the user moves along a front-back axis perpendicular to the frontal plane of the human body; judging whether the depth information of the hand of the user at the current moment and the standard deviation data meet an asymmetric preset condition or not; wherein the preset conditions specifically include the size relation between the depth information and the first threshold value and the second threshold value and the size relation between the standard deviation data and the third threshold value; and determining whether the equipment is in an interactive mode or a non-interactive mode at the current moment according to a preset rule. According to the invention, the switching of the interaction modes can be quickly and naturally realized, and the operation experience and the interaction efficiency of the user are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular relates to an interaction mode switching method driven by natural mid-air gestures. Background Art

[0002] With the continuous development of science and technology, more and more human-computer interaction systems are being used in our lives. Human-computer interaction systems are usually composed of hardware and software, allowing users to interact with devices through intuitive interfaces, such as eye movement control, voice recognition, and gesture control. Effective human-computer interaction systems focus on user needs and adopt user-friendly design and feedback mechanisms to ensure smooth operation and ease of use. With the advancement of technology, human-computer interaction systems are moving towards more natural interaction methods, such as virtual reality, augmented reality, and cross-device interaction, to further optimize user experience and improve efficiency.

[0003] In most human-computer interaction systems, in order to adapt to various application scenarios and system fluency, multiple interaction modes often need to be switched; and how to switch between interaction modes has always been a difficult problem. Traditional interaction mode switching methods often require specific devices or require users to make complicated gestures, which not only has great deficiencies in versatility and interaction efficiency, but also increases the physical and mental burden on users.

[0004] Most of the interaction modes in the current existing technologies are still limited to special application scenarios, and are not very universal, and have their own shortcomings; for example, patents CN114270295A and KR20220016274A are in the virtual reality scene, and the user makes fixed gestures or moves the hand to a specified position to switch the interaction mode. This method consumes a lot of physical energy, and it is not natural and convenient for users to use; the method for entering and exiting a remote system proposed in patent US20230110890A1 uses head position, activation gestures, thresholds, etc. to switch the interaction mode. The reliability and stability of the system are improved by using methods such as time value, but the overly cumbersome operation will increase the memory burden of the user; Patent CN218599956U and Patent JP2022554219A both use voice input to complete the switching of the interaction mode. Although this method is fast, it is not applicable in some noisy or public occasions due to the limitations of voice recognition itself; Patent US10545900B2 and Patent US10628183B2 use the method of changing the state of the fixed device to switch the interaction mode, which has great deficiencies in portability and flexibility. In summary, there is currently no method that can be used naturally, conveniently, and widely for switching the interaction mode of the human-computer interaction system. Summary of the invention

[0005] In order to solve the problems existing in the prior art when switching interaction modes, the present invention provides an interaction mode switching method driven by natural air gestures, so as to quickly and naturally switch the interaction modes and improve the user's operating experience and interaction efficiency.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] A natural air gesture driven interactive mode switching method, comprising:

[0008] When the user performs an interactive mode switching action, the device obtains the user's hand information through the information collection device, and uses a sliding window to calculate the depth information and standard deviation data of the user's hand relative to the device at each moment; wherein the user's hand moves along the front-to-back axis perpendicular to the frontal plane of the human body;

[0009] Determine whether the depth information and standard deviation data of the user's hand at the current moment meet the asymmetric preset condition; wherein the asymmetric preset condition is specifically the magnitude relationship between the depth information and the first threshold and the second threshold and the magnitude relationship between the standard deviation data and the third threshold;

[0010] Determine whether the device is in interactive mode or non-interactive mode at the current moment according to preset rules.

[0011] Furthermore, if the information collection device cannot directly collect the depth information of the hand, the hand width is used as the depth information of the hand.

[0012] Furthermore, the calculating the depth information and standard deviation data of the user's hand relative to the device using the sliding window at each moment includes:

[0013] At any time, the formula for calculating the user's hand depth information is as follows:

[0014]

[0015] Where y is the position of the hand at a certain moment on the front-back axis, n is the size of the sliding window, and x is the position of the hand at a certain moment on the front-back axis. i is the depth information of the hand from the device at the i-th moment in the sliding window;

[0016] The standard deviation is calculated as follows:

[0017]

[0018] Where σ(X) represents the standard deviation of the hand depth information at a certain moment, Represents the mean of the hand depth information.

[0019] Furthermore, the first threshold and the second threshold are determined by:

[0020] Step 2.1, using the device, projector and computer to conduct the Wizard of Oz experiment, the computer is used as the task system of the experiment, the screen in the task system is displayed to the user by the projector, and the user performs the image manipulation task in front of the device through gestures; during the experiment, the device is used to collect the user's hand data; according to the guidance of the experimenter, different functions are performed on the images in the task system by the hands; when the user is unable to continue to perform the task due to hand physiological constraints or device reasons during the execution of any function, the experimenter will give a voice prompt, and the user needs to drag the palm toward the body at the first time, and the experimenter will switch the task system to non-interactive mode; after the user readjusts the hand posture and pushes the palm away from the body, the experimenter will switch the task system to interactive mode, and the user can continue to perform the interactive task;

[0021] Step 2.2, the hand data includes the key point coordinates of the user's hand, the hand width is obtained based on the key point coordinates, and the hand width is used to replace the depth information of the user's hand; a mapping relationship between the hand width and the depth information is established, and curve fitting is performed to obtain a fitting curve;

[0022] Step 2.3, establishing a density map based on the user's hand depth information, estimating the probability density function of the depth information in the interactive mode and the non-interactive mode through the kernel density curve and performing fitting;

[0023] Step 2.4, for the density map of depth information, select the hand width data corresponding to the time when the interactive mode data accounts for the largest proportion and the non-interactive mode data accounts for the smallest proportion from the map as the parameter s of the dynamic weight function;

[0024] The inverse exponential function is used as the dynamic weight function. Different depth information has different weights. Each depth information is used as the segmentation point. The left side of the segmentation point is determined as a non-interactive mode, and the right side of the segmentation point is determined as an interactive mode:

[0025]

[0026] Where W(x) is the weight for calculating the probability of switching the interactive mode at a certain depth, x is the depth information of the user's hand at a certain moment, e is a natural constant, and d is an empirical parameter;

[0027] When it is determined that the error rate of the interactive mode and non-interactive mode judgment is less than the first set value, the distance between the hand and the device is the first threshold a; starting from the first threshold a, until a certain position, if the probabilities of false positive and false negative that meet the judgment are less than the second set value, the position is used as the second threshold b.

[0028] Furthermore, the formula for kernel density estimation is:

[0029]

[0030] Where F(x) is the estimated density at a certain depth information, n is the number of hand data collected during the experiment, h is the bandwidth used to control the kernel function, X is the specific depth information, that is, the point whose probability density function value is to be estimated, and x i is the i-th depth information, is a uniform kernel function, which is defined as follows:

[0031]

[0032] Furthermore, the third threshold is determined by:

[0033] Step 2.5, during the experiment, for the user's hand width data at different times, use a sliding window to calculate the standard deviation of the hand width; construct a hand width-standard deviation graph, whose vertical axis is the user's hand width at each time, and the horizontal axis is the standard deviation of the hand width calculated at each time;

[0034] From the graph, select a horizontal line that is parallel to the horizontal axis, greater than the corresponding standard deviation in all interactive modes, has an intersection with the corresponding standard deviation interval in all non-interactive modes, and is closest to the horizontal axis as the threshold line. The value of the vertical axis corresponding to the threshold line is determined as the third threshold c.

[0035] Furthermore, the first setting value is 5%, the second setting value is 10%, and the value of the empirical parameter d is 0.01-0.03.

[0036] Further, the determining, according to a preset rule, whether the device is in the interactive mode or the non-interactive mode at the current moment includes:

[0037] In the human-computer interaction system of the device, three states are defined: interactive state, non-interactive state, and transition state;

[0038] The standard deviation of the hand in the sliding window at the current moment is greater than c as event one, the hand depth information calculated by the sliding window for N consecutive times is greater than b as event two, the standard deviation of the hand in the sliding window at the current moment is less than c as event three, the hand depth information calculated by the sliding window for N consecutive times is less than a as event four, the hand depth information calculated by the sliding window at the current moment is greater than b as event five, and the hand depth information calculated by the sliding window at the current moment is less than a as event six, and the judgment is made:

[0039] If the current moment is in the interactive state, event one is satisfied, indicating that the user is switching from the interactive state to the transitional state; if event two is satisfied, it indicates that the user is switching to the non-interactive state; if the current moment is in the transitional state, event three is satisfied, indicating that the user is switching to the non-interactive state; event six is ​​satisfied, indicating that the user is switching to the interactive state; if the current moment is in the non-interactive state, event four is satisfied, indicating that the user is switching to the interactive state, and event five is satisfied, indicating that the user is switching to the transitional state;

[0040] In the interactive state, the human-computer interaction system of the device enters the interactive mode, at which time the device will respond to the user's gesture input to achieve interaction; in the transition state and non-interactive state, the system enters the non-interactive mode, at which time the device will block the user's gesture input.

[0041] A device, the device having an information collection device;

[0042] When the user performs an interactive mode switching action, the device obtains the user's hand information through the information collection device, and uses a sliding window to calculate the depth information and standard deviation data of the user's hand relative to the device at each moment; wherein the user's hand moves along the front-to-back axis perpendicular to the frontal plane of the human body;

[0043] The device determines whether the depth information and standard deviation data of the user's hand at the current moment meet the preset conditions; wherein the preset conditions are specifically the magnitude relationship between the depth information and the first threshold and the second threshold and the magnitude relationship between the standard deviation data and the third threshold;

[0044] The device determines whether the device is in interactive mode or non-interactive mode at the current moment according to a preset rule.

[0045] An interactive mode switching device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the natural air gesture-driven interactive mode switching method is implemented.

[0046] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the natural air gesture-driven interactive mode switching method is implemented.

[0047] Compared with the prior art, the present invention has the following technical features:

[0048] The present invention provides a natural air gesture for switching interactive modes in a human-computer interaction system; the user switches between interactive mode and non-interactive mode by dragging the hand toward the human body and pushing the hand in the opposite direction. By defining the distance that the user drags the hand, the reduction in interaction efficiency caused by too long movement of the hand is avoided, and at the same time, the erroneous recognition of mode switching instructions caused by too short movement of the hand is reduced, thereby affecting normal interaction. The processing of the user's position information on the front-back axis perpendicular to the frontal plane of the human body, coupled with the definition of three system states, enables the user to switch the interactive mode stably and quickly. This technology has extremely broad application prospects in a variety of application scenarios, especially in cutting-edge fields such as remote control, virtual reality, and augmented reality. It will greatly improve the user's interaction efficiency, bring users a more efficient, convenient and pleasant operating experience, and further promote the technology development and application innovation in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the application state of the method of the present invention;

[0050] Figure 2 A schematic diagram of a gesture motion plane provided by the present invention;

[0051] Figure 3 A state diagram of a human-computer interaction system according to an embodiment of the present invention;

[0052] Figure 4 : is a diagram showing the relationship between the palm width and the depth distance in an embodiment of the present invention;

[0053] Figure 5 is a density map of depth information in an embodiment of the present invention;

[0054] Figure 6 Graph showing hand width-standard deviation in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The basic principle of the present invention is to use a gesture information capture device to obtain depth information of the user's hand interaction, provide a natural air gesture-driven gesture action, use the change of depth information to achieve switching between interactive mode and non-interactive mode, alleviate the problem of turning stones into gold in human-computer interaction, and reduce the user's unconscious input.

[0056] The present invention provides a natural air gesture driven interactive mode switching method, which specifically includes the following steps:

[0057] Step 1: When the user performs an interactive mode switching action, the device obtains the user's hand information through the information collection device, and uses a sliding window to calculate the depth information y and standard deviation data σ(X) of the user's hand relative to the device at each moment; wherein the user's hand moves along the front-to-back axis perpendicular to the frontal plane of the human body.

[0058] The device in this solution may refer to any smart device, such as a smart phone, a projection device, or the like that can interact with user gestures; the information collection device refers to a device carried by the device for obtaining depth information of the user's hand, such as a camera, a sensor, etc.

[0059] In this scheme, the depth information y of the hand is used as the basis for judgment. Therefore, if the information acquisition device cannot directly acquire the depth information of the hand, the depth information can be replaced by other information. For example, when the hand moves along the front and rear axes, the width of the hand in the image will change. Therefore, the change in the hand width can be used to replace the change in depth.

[0060] At any time t, the formula for calculating the user's hand depth information is as follows:

[0061]

[0062] Where y is the position of the hand at a certain moment on the front-back axis, n is the size of the sliding window, and x is the position of the hand at a certain moment on the front-back axis. i It is the depth information of the hand from the device at the i-th moment in the sliding window. This solution adopts mean filtering to process it, mainly considering the noise data in the hand interaction process, which can smooth the noise interference in the interaction process.

[0063] The standard deviation is calculated as follows:

[0064]

[0065] Where σ(X) represents the standard deviation of the hand depth information at a certain moment, Represents the mean of the hand depth information.

[0066] The calculation of standard deviation data in this solution is mainly based on the consideration that the depth information will also change when the user performs normal interaction tasks. Relying solely on the depth information to determine the switching of the interaction mode is prone to cause wrong judgment.

[0067] Step 2, determine whether the depth information y and standard deviation data σ(X) of the user's hand at the current moment meet the asymmetric preset conditions; wherein the asymmetric preset conditions are specifically the size relationship between the depth information y and the first threshold a and the second threshold b, and the size relationship between the standard deviation data σ(X) and the third threshold c.

[0068] In this solution, the first threshold a, the second threshold b and the third threshold c in the preset condition are determined by experiments designed for the device, as follows:

[0069] Step 2.1, using the device, projector and computer to conduct the Wizard of Oz experiment, the computer serves as the task system of the experiment, the screen in the task system is displayed to the user by the projector, and the user performs the image manipulation task by gestures in front of the device; during the experiment, the device is used to collect the user's hand data; according to the guidance of the experimenter, the image in the task system is rotated, scaled and moved respectively by the yaw rotation movement of the hand, the distance change between the thumb and the index finger, and the translation movement of the hand; when the user is unable to continue to perform the task due to physiological constraints of the hand or equipment reasons during the execution of any of the above functions, the experimenter will give a voice prompt, and the user needs to drag the palm toward the body as soon as possible, at which time the experimenter will switch the task system to non-interactive mode; after the user readjusts the hand posture and pushes the palm away from the body, the experimenter will switch the task system to interactive mode, at which time the user can continue to perform the interactive task.

[0070] Step 2.2, in order to facilitate data processing, the hand data includes the key point coordinates of the user's hand, and the hand width is obtained based on the key point coordinates, and the hand width is used to replace the depth information of the user's hand; according to the interactive mode and non-interactive mode, the hand data collected at each moment is marked with corresponding labels.

[0071] Among them, the extraction of the coordinates of the key points of the hand can be done with the help of existing tools, such as the mediapipe framework; the calculation method of the hand width X is as follows:

[0072]

[0073] Among them, x1 and y1 are the two-dimensional coordinates of the base joint of the index finger of the palm in the image captured by the device, and x2 and y2 are the two-dimensional coordinates of the base joint of the little finger of the palm in the image captured by the mobile device.

[0074] In order to map the hand width to the real depth information, multiple test measurements were conducted to establish the mapping relationship between the hand width and the depth information, and curve fitting was performed. The fitted curve expression is as follows:

[0075] X=αY 3 +βY 2 +γY+δ

[0076] Among them, α, β, γ and δ are fitting parameters, Y represents the hand width, and X represents the depth information of the hand relative to the device.

[0077] Since the hand width changes linearly on the front-rear axis, the depth information of the user's hand at any time can be inferred using the fitting curve of the above formula.

[0078] Step 2.3: A density map is established based on the user's hand depth information, and the probability density function of the depth information in the interactive mode and the non-interactive mode is estimated and fitted through the kernel density curve; the formula for kernel density estimation is:

[0079]

[0080] Where F(x) is the estimated density at a certain depth information, n is the number of hand data collected during the experiment, h is the bandwidth used to control the kernel function, X is the specific depth information, that is, the point whose probability density function value is to be estimated, and x i is the i-th depth information, is a uniform kernel function, which is defined as follows:

[0081]

[0082] Step 2.4, for the density map of depth information, select the hand width data corresponding to the time when the interactive mode data accounts for the largest proportion and the non-interactive mode data accounts for the smallest proportion from the map as the parameter s of the dynamic weight function.

[0083] The dynamic weight function in this scheme specifically adopts an inverse exponential function. The closer the user's hand is to the device, the more likely the user is in interactive mode, and the lower the weight for determining interactive mode switching is. The farther the user's hand is from the device, the more likely the user is in non-interactive mode, and the higher the weight for determining interactive mode switching is. Therefore, different depth information has different weights. Each depth information is used as a segmentation point, and the areas covered by the two kernel density estimation curves of the interactive mode and the non-interactive mode on the left and right sides of the segmentation point are calculated, which is used as the basis for calculating the determination probability. The left side of the segmentation point is determined to be non-interactive mode, and the right side of the segmentation point is determined to be interactive mode.

[0084]

[0085] Where W(x) is the weight for calculating the probability of switching the interactive mode at a certain depth, x is the depth information of the user's hand at a certain moment, e is a natural constant, and d is an empirical parameter, which is 0.01-0.03 in this solution.

[0086] Based on the above calculation method, when the error rate of interactive mode and non-interactive mode judgment is less than the first set value, the distance between the hand and the device is the first threshold a; starting from the first threshold a, until a certain position, if the probability of false positive and false negative judgment is less than the second set value, the position is used as the second threshold b; in this solution, the first set value is 5% and the second set value is 10%.

[0087] Step 2.5, during the experiment, for the user's hand width data at different times, use a sliding window to calculate the standard deviation of the hand width; construct a hand width-standard deviation graph, whose vertical axis is the user's hand width at each time, and the horizontal axis is the standard deviation of the hand width calculated at each time;

[0088] From the graph, select a horizontal line that is parallel to the horizontal axis, greater than the corresponding standard deviation in all interactive modes, has an intersection with the corresponding standard deviation interval in all non-interactive modes, and is closest to the horizontal axis as the threshold line. The value of the vertical axis corresponding to the threshold line is determined as the third threshold c.

[0089] Step 3: Determine whether the device is in interactive mode or non-interactive mode at the current moment according to a preset rule.

[0090] In the human-computer interaction system of the device, three states are defined: interactive state, non-interactive state, and transition state to avoid the switching of interactive modes caused by system misidentification, which leads to system confusion; multiple events are defined, and the system is switched between the three states by using the events that occur, which enhances the stability of the system. The interactive state corresponds to the normal interactive mode, and the user can input information and complete the corresponding command operation; the non-interactive state and the transition state both represent the non-interactive mode, and any input from the user will be blocked by the system at this time. In the non-interactive state, it is difficult for the system to directly switch to the interactive state. It takes a period of time for the user's hand data to meet the threshold conditions before the state is switched. This is to ensure that the user will not be mistakenly switched to the interactive state due to some accidental actions in the non-interactive state. The transition state is also a non-interactive mode, but compared with the non-interactive state, the user will switch to the interactive state as long as the trigger condition of the corresponding interactive mode switching threshold is met. This fast and convenient method can better ensure that the user's normal interactive behavior will not be disturbed. In the interactive state, the threshold condition of the standard deviation is met to switch to the transition state, and the continuous multiple frames of data meet the threshold of the interactive mode switching to switch to the non-interactive state.

[0091] The standard deviation of the hand in the sliding window at the current moment is greater than c as event one, the hand depth information calculated by the sliding window for N consecutive times is greater than b as event two, the standard deviation of the hand in the sliding window at the current moment is less than c as event three, the hand depth information calculated by the sliding window for N consecutive times is less than a as event four, the hand depth information calculated by the sliding window at the current moment is greater than b as event five, and the hand depth information calculated by the sliding window at the current moment is less than a as event six, and the judgment is made:

[0092] If the current moment is in the interactive state and event one is satisfied, it indicates that the user is switching from the interactive state to the transitional state; if event two is satisfied, it indicates that the user is switching to the non-interactive state; if the current moment is in the transitional state, event three is satisfied and the user is switching to the non-interactive state; event six is ​​satisfied and the user is switching to the interactive state; if the current moment is in the non-interactive state, event four is satisfied and the user is switching to the interactive state; event five is satisfied and the user is switching to the transitional state.

[0093] In the interactive state, the human-computer interaction system of the device enters the interactive mode, at which time the device will respond to the user's gesture input to achieve interaction; in the transition state and non-interactive state, the system enters the non-interactive mode, at which time the device will block the user's gesture input.

[0094] Here, N is a preset number of times, and in this embodiment, its value is 5, which can be set according to specific needs.

[0095] Example:

[0096] In one embodiment of the present invention, when performing the Wizard of Oz experiment in step 2.1, the device uses an MHA-AL00 mobile phone, an EPSON CBW32CBW32 projector and a computer with a windows system; the mobile phone has a front camera with 30 frames per second, and the user is 20 cm in front of the device. The hand key point capture program developed using the mediapipe framework is run on the device, which can capture the coordinates of 21 key points of the user's hand; the hand key point coordinates are composed of 3 floating point numbers, the first two dimensions are the x and y axis coordinates in the camera coordinate system, and the third dimension is the front and rear position relationship of the hand key point relative to the wrist; the device will collect the user's hand key point coordinates at a rate of 30 frames per second; the user can use hand gesture changes to control the image changes on the large screen; the interactive tasks are divided into three types, namely rotating the image, scaling the image, and moving the image. During the task execution, the user needs to listen to the prompts given by the experimenter, a red border will appear on the large screen, and a sound will be emitted; within 4.5 seconds after the prompt is issued, the user makes an interactive mode switching action according to the prompt, which is considered a successful task, and there will also be successful voice and visual prompts. If the corresponding action is not completed within the specified time or is not successfully recognized by the system, it is considered a task failure, and a voice and visual prompt of failure will appear.

[0097] In step 2.2, the result of curve fitting in this embodiment is as follows: Figure 4 As shown, in the fitting curve expression, α is -6233.006, β is 5046.269, γ is -1340.536, and δ is 133.054.

[0098] In step 2.3 and step 2.4, the density map of the depth information is established as follows: Figure 5 As shown, Figure 5 It can be seen that when the hand width data is greater than 0.13, that is, when the user's hand movement distance from the interactive device is less than 31 cm, the data in the interactive mode occupies the majority, and there is little data in the non-interactive mode at this time. Therefore, it is believed that the user's behavior is closer to the interactive mode, and the interactive mode switching will not occur at this time; starting from 31 cm away from the interactive device, the interactive mode data and the non-interactive mode data begin to intersect, and the closer the depth distance, the more likely the user's interactive behavior is to be in the interactive mode, otherwise it is more likely to be in the non-interactive mode; therefore, in this embodiment, the value of s is 0.13 and the value of d is 0.0132.

[0099] When the user's hand moves at a distance of 44 cm from the interactive device, the error rate of the interactive mode and non-interactive mode determined by the interactive data is only about 5%, which is the most balanced segmentation point; when the user's hand moves at a distance of 40 cm to 58 cm from the interactive plane, the probability of false positive and false negative determined at this time is less than 10%, which is an acceptable range. Through the processing of dynamic weights, the depth data range for switching between interactive mode and non-interactive mode is determined to ensure that the probability of false positive and false negative for interactive mode switching is less than 10%, avoiding the confusion of the two data caused by too small a threshold and affecting the normal interactive behavior, and also preventing the sensitivity of interactive mode switching caused by too large a threshold, thereby reducing the speed and efficiency of interactive mode switching; thus, the first threshold a related to the depth information y can be determined to be 40, and the second threshold b is 58.

[0100] In step 2.5, a sliding window of size 15 is used to calculate the standard deviation. When the user's hand makes gestures from near to far and from far to near, the standard deviation data of the hand information will fluctuate dramatically within a certain period of time. The specific data is as follows: Figure 6 As shown, it can be clearly seen that there is a clear distinction between the interactive mode and non-interactive mode data; the red and blue data above are the user's hand width data, of which blue represents the data in the interactive mode, red represents the data in the non-interactive mode, and the green data is the standard deviation data of the user's hand width in the sliding window; it can be clearly seen that the user performed three gestures from near to far and from far to near during the interactive task, corresponding to the switching of the interactive mode; although the standard deviation data of the user's hand width has a more obvious fluctuation, when the user drags the hand to the end closest to the body, the standard deviation data will return to the level of normal interaction due to the slowing of the action; therefore, in this example, the hand width corresponding to the selected threshold line is 0.01, that is, when the threshold line is selected, it is above the corresponding standard deviation in all interactive modes, and there is an intersection with the corresponding standard deviation interval in all non-interactive modes. Among all the horizontal lines that meet this condition, the horizontal line corresponding to 0.01 is the closest to the horizontal axis.

[0101] The accuracy of the natural mid-air gesture-driven switching method obtained by the method of the present invention is 99.07%, indicating that the scheme can accurately implement the switching of the interaction mode and reduce the system error caused by unconscious input.

[0102] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A natural air gesture driven interactive mode switching method, characterized in that: include: When the user performs an interactive mode switching action, the device obtains the user's hand information through the information collection device, and uses a sliding window to calculate the depth information and standard deviation data of the user's hand relative to the device at each moment; wherein the user's hand moves along the front-to-back axis perpendicular to the frontal plane of the human body; Determine whether the depth information and standard deviation data of the user's hand at the current moment meet the asymmetric preset condition; wherein the asymmetric preset condition is specifically the magnitude relationship between the depth information and the first threshold and the second threshold and the magnitude relationship between the standard deviation data and the third threshold; Determine whether the device is in interactive mode or non-interactive mode at the current moment according to preset rules.

2. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: If the information collection device cannot directly collect the depth information of the hand, the hand width is used as the depth information of the hand.

3. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: The method of calculating the depth information and standard deviation data of the user's hand relative to the device using the sliding window at each moment includes: At any time, the formula for calculating the user's hand depth information is as follows: Where y is the position of the hand at a certain moment on the front-back axis, n is the size of the sliding window, and x is the position of the hand at a certain moment on the front-back axis. i is the depth information of the hand from the device at the i-th moment in the sliding window; The standard deviation is calculated as follows: Where σ(X) represents the standard deviation of the hand depth information at a certain moment, Represents the mean of the hand depth information.

4. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: The method for determining the first threshold and the second threshold is: Step 2.1, using the device, projector and computer to conduct the Wizard of Oz experiment, the computer is used as the task system of the experiment, the screen in the task system is displayed to the user by the projector, and the user performs the image manipulation task in front of the device through gestures; during the experiment, the device is used to collect the user's hand data; according to the guidance of the experimenter, different functions are performed on the images in the task system by the hands; when the user is unable to continue to perform the task due to hand physiological constraints or device reasons during the execution of any function, the experimenter will give a voice prompt, and the user needs to drag the palm toward the body at the first time, and the experimenter will switch the task system to non-interactive mode; after the user readjusts the hand posture and pushes the palm away from the body, the experimenter will switch the task system to interactive mode, and the user can continue to perform the interactive task; Step 2.2, the hand data includes the key point coordinates of the user's hand, the hand width is obtained based on the key point coordinates, and the hand width is used to replace the depth information of the user's hand; a mapping relationship between the hand width and the depth information is established, and a curve fitting is performed to obtain a fitting curve; Step 2.3, establishing a density map based on the user's hand depth information, estimating the probability density function of the depth information in the interactive mode and the non-interactive mode through the kernel density curve and performing fitting; Step 2.4, for the density map of depth information, select the hand width data corresponding to the time when the interactive mode data accounts for the largest proportion and the non-interactive mode data accounts for the smallest proportion from the map as the parameter s of the dynamic weight function; The inverse exponential function is used as the dynamic weight function. Different depth information has different weights. Each depth information is used as the segmentation point. The left side of the segmentation point is determined as a non-interactive mode, and the right side of the segmentation point is determined as an interactive mode: Where W(x) is the weight for calculating the probability of switching the interactive mode at a certain depth, x is the depth information of the user's hand at a certain moment, e is a natural constant, and d is an empirical parameter; When it is determined that the error rate of the interactive mode and non-interactive mode judgment is less than the first set value, the distance between the hand and the device is the first threshold a; starting from the first threshold a, until a certain position, if the probabilities of false positive and false negative that meet the judgment are less than the second set value, the position is used as the second threshold b.

5. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: The formula for the kernel density estimation is: Where F(x) is the estimated density at a certain depth information, n is the number of hand data collected during the experiment, h is the bandwidth used to control the kernel function, X is the specific depth information, that is, the point whose probability density function value is to be estimated, and x i is the i-th depth information, is a uniform kernel function, which is defined as follows:

6. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: The method for determining the third threshold is: Step 2.5, during the experiment, for the user's hand width data at different times, use a sliding window to calculate the standard deviation of the hand width; construct a hand width-standard deviation graph, whose vertical axis is the user's hand width at each time, and the horizontal axis is the standard deviation of the hand width calculated at each time; From the graph, select a horizontal line that is parallel to the horizontal axis, greater than the corresponding standard deviation in all interactive modes, has an intersection with the corresponding standard deviation interval in all non-interactive modes, and is closest to the horizontal axis as the threshold line. The value of the vertical axis corresponding to the threshold line is determined as the third threshold c.

7. The natural air gesture driven interactive mode switching method according to claim 4, characterized in that: The first setting value is 5%, the second setting value is 10%, and the value of the empirical parameter d is 0.01-0.

03.

8. The natural air gesture driven interactive mode switching method according to claim 1, characterized in that: The determining, according to a preset rule, whether the device is in an interactive mode or a non-interactive mode at the current moment includes: In the human-computer interaction system of the device, three states are defined: interactive state, non-interactive state, and transition state; The standard deviation of the hand in the sliding window at the current moment is greater than c as event one, the hand depth information calculated by the sliding window for N consecutive times is greater than b as event two, the standard deviation of the hand in the sliding window at the current moment is less than c as event three, the hand depth information calculated by the sliding window for N consecutive times is less than a as event four, the hand depth information calculated by the sliding window at the current moment is greater than b as event five, and the hand depth information calculated by the sliding window at the current moment is less than a as event six, and the judgment is made: If the current moment is in the interactive state, event one is satisfied, indicating that the user is switching from the interactive state to the transitional state; if event two is satisfied, it indicates that the user is switching to the non-interactive state; if the current moment is in the transitional state, event three is satisfied, indicating that the user is switching to the non-interactive state; event six is ​​satisfied, indicating that the user is switching to the interactive state; if the current moment is in the non-interactive state, event four is satisfied, indicating that the user is switching to the interactive state, and event five is satisfied, indicating that the user is switching to the transitional state; In the interactive state, the human-computer interaction system of the device enters the interactive mode, at which time the device will respond to the user's gesture input to achieve interaction; in the transition state and non-interactive state, the system enters the non-interactive mode, at which time the device will block the user's gesture input.

9. A device having an information collection device, characterized in that: When the user performs an interactive mode switching action, the device obtains the user's hand information through the information collection device, and uses a sliding window to calculate the depth information and standard deviation data of the user's hand relative to the device at each moment; wherein the user's hand moves along the front-to-back axis perpendicular to the frontal plane of the human body; The device determines whether the depth information and standard deviation data of the user's hand at the current moment meet the preset conditions; wherein the preset conditions are specifically the magnitude relationship between the depth information and the first threshold and the second threshold and the magnitude relationship between the standard deviation data and the third threshold; The device determines whether the device is in interactive mode or non-interactive mode at the current moment according to a preset rule.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the natural air gesture driven interactive mode switching method according to any one of claims 1 to 8.

Images