A cross-device multi-scale cursor control method in mid-air gesture interaction

Abstract

The present invention discloses a cross-device multi-scale cursor control method for mid-air gesture interaction. First, an input device collects user hand motion information data and transmits it to a server via a message queue. After receiving the data transmitted by the input device, the server performs mean filtering on the hand information. Different gesture signals are collected according to the user's expected cursor movement type. After the server obtains the user's gesture signals of different motion types, it determines the user's intention based on the user's static gestures. The server generates corresponding instructions and accurately sends the instructions to the output device via a stable communication channel. After receiving the instructions sent by the server, the output device obtains the change value of the cursor movement, calculates the new cursor position, and implements the corresponding cursor movement function. The present invention can not only achieve extremely efficient cursor movement of the output device over a large range, but also accurately perform extremely fine fine-tuning operations.

Description

Technical Field

[0001] The present invention belongs to the field of gesture recognition technology, and in particular relates to a cross-device multi-scale cursor control method in mid-air gesture interaction. Background Art

[0002] With the continuous development of information technology, mid-air gesture interaction, as a natural form of human-computer interaction, has gradually become a research hotspot. Compared with traditional physical input devices such as keyboards and mice, mid-air gesture interaction is more intuitive and flexible, freeing users from the constraints of specific devices and allowing them to interact with the system using hand movements. Mid-air gesture interaction technology has shown great potential in applications such as large-screen displays, smart home control, virtual reality (VR), and augmented reality (AR).

[0003] Traditional mid-air gesture interactions are mostly used on a single device. However, in multi-device collaborative operation scenarios, especially cross-device interactions, users often require more flexible and precise control of the cursor position on their devices. However, current gesture recognition and control methods still have some limitations. In particular, in multi-device collaborative scenarios, achieving cross-scale cursor control on different devices still faces problems such as the limited capture range of input devices and insufficient cursor movement precision, which greatly affects the smoothness and accuracy of interactions.

[0004] Patent CN108776550A proposes a cursor positioning control device, control method and electronic device. The device includes a base and a control ball, and the movement and pressing of the control ball are controlled by fingers.

[0005] Patent CN113282222A provides a cursor control method and device for a mobile terminal equipped with an accelerometer. The method includes: when the terminal is in remote control mode, obtaining the accelerometer's values ​​on two coordinate axes, determining a target key value (the key value corresponding to the remote control's direction keys) based on the values, and controlling the cursor of a second terminal to move in the direction of the target key value. The device includes an acquisition module, a determination module, and a control module. Controlling cursor movement by controlling the terminal's posture avoids the problems of physical keys being easily damaged and inconvenient to operate.

[0006] Patent CN111045638A discloses a cursor control method, device, storage medium and terminal, which aims to use voice recognition technology to input operation instructions to a mouse with a microphone, replacing the user's hands to control the mouse cursor, thereby improving the accuracy and convenience of cursor control.

[0007] Patent CN115291733A discloses a cursor control method that uses a camera to capture user gesture displacement information to control cursor movement. The cursor movement is mainly determined by determining a first distance and a second distance based on the focus position, the position before the movement, and the position after the movement.

[0008] Patent CN108292362A describes a system, method, device, and computer-readable medium for gesture recognition. This technology creates descriptors by analyzing hand motion vectors and optical flow histograms, enabling gesture recognition and cursor control. The technology is invariant to scale, rotation, and hand shape, and can be applied to different image types, improving accuracy and performance.

[0009] Patent PCTCN2023135670 discloses a cursor control method, the main steps of which are: the first electronic device controls the cursor according to the instructions of the second electronic device, receives the first instruction to display the cursor in a static state, receives the second instruction to change and move the cursor between the static state and the pointing state, and receives the third instruction to play the target focus screen; the second electronic device generates instructions in response to user operations to control the movement and status display of the cursor, focus and list on the first electronic device.

[0010] Patent PCTCN2021080438 relates to a method for interacting with a display device and a terminal device, including the display device receiving a terminal device access request and establishing communication, generating and displaying a corresponding cursor, and controlling the cursor position according to cursor control instructions; the terminal device obtaining an access address, establishing a connection, and controlling the cursor; multiple display devices can display synchronously, and they are connected via a cloud server. The method also includes setting permissions for the terminal device.

[0011] Patent CN116048313A discloses a cursor control method involving contact gesture control. The main steps are: a cursor input device detects a user's touch operation from lifting to releasing the touch area of ​​the cursor input device; a display device is controlled to determine cursor position offset information for each cycle; and the cursor position offset information is reduced to a preset value. The display device then displays the cursor based on the preset value.

[0012] Patent CN113010130A discloses a long-distance large-screen interactive control method based on image streaming, including a large-screen end and a smartphone end. The movement of the rectangular cursor on the large-screen end is controlled by the built-in inertial sensor of the smartphone, and a screenshot of the content in the rectangular cursor area is transmitted to the smartphone end in real time. The touch control on the smartphone end confirms the selection of the target point, and then various interactive operations on the selected target point are realized through the inertial sensor control mode or the touch screen control mode.

[0013] Most current cursor control methods do not consider cross-device mid-air gesture interaction scenarios, nor do they take into account users' multi-scale cursor control needs. The aforementioned patents CN108776550A, CN113282222A, and CN111045638A use specific terminal devices, voice commands, and other methods to achieve efficient cursor control, and do not apply gesture control. Patents CN115291733A and CN108292362A both use gestures for cursor control, but each focuses on specific gesture recognition methods for gesture-controlled cursors, and does not involve cursor control in cross-device interaction. Patents PCTCN2023135670 and PCTCN2021080438 involve cursor control methods between multiple devices, but mostly use multiple devices as auxiliary devices, and do not design targeted cursor control methods based on the unique interaction capabilities of different devices. The cursor control methods mentioned in patents CN116048313A and CN113010130A are applicable to cross-device scenarios, but they are not applicable to contact gestures rather than mid-air gestures, and do not involve multi-scale cursor control methods.

[0014] Currently, no method can meet the multi-scale cursor control requirements of cross-device mid-air gesture interaction scenarios, namely, meeting the multi-scale requirements of efficient cursor control over a large range and precise cursor control over a small range. Therefore, a cursor control method that can adapt to multi-device mid-air gesture interaction scenarios is urgently needed to improve the user experience and interaction efficiency. Summary of the Invention

[0015] To overcome the shortcomings of the prior art, the present invention provides a cross-device multi-scale cursor control method for mid-air gesture interaction. First, an input device collects user hand motion information data and transmits it to a server via a message queue. After receiving the data transmitted by the input device, the server performs mean filtering on the hand information. Different gesture signals are collected based on the user's expected cursor movement type. After obtaining the user's gesture signals of different motion types, the server determines the user's intention based on the user's static gestures. The server generates corresponding instructions and accurately transmits the instructions to the output device via a stable communication channel. After receiving the instructions sent by the server, the output device obtains the change in cursor movement based on the information in the instructions, calculates the new cursor position, and implements the corresponding cursor movement function. The present invention not only enables highly efficient cursor movement of the output device over a wide range, but also enables precise and precise fine-tuning operations. During the interaction process, the user can switch between multi-scale cursor control methods based on their subjective judgment and control ability. If the user feels that the cursor control accuracy is insufficient, they can choose a small-scale fine-tuning operation. If the user feels that the cursor control speed is insufficient, they can choose a large-scale transfer operation. Due to the differences between interactive tasks and individuals, we do not make a quantitative distinction between the applicable scopes of the two control methods here to prevent rigid requirements from leading to a decrease in interaction efficiency.

[0016] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0017] Step 1: The input device collects user hand motion information data and transmits it to the server through the message queue;

[0018] Step 2: After receiving the data transmitted by the input device, the server performs mean filtering on the hand information;

[0019] Step 3: Collect different gesture signals based on the user's expected cursor movement type;

[0020] For large-scale cursor transfer operations, the hand rotational motion information is extracted; with the user's right wrist as the center, the overall pitch and roll angle changes of the user's right palm are calculated; and threshold filtering is then used to filter the information;

[0021] For small-scale fine-tuning of the cursor, the system extracts the user's hand translation information. During this process, based on a fixed coordinate system perpendicular to the ground, the system collects and calculates the translational motion changes of the root node of the user's right middle finger in three-dimensional space. Through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be determined, and its position changes at different times can be monitored and analyzed to obtain the translational motion trajectory and changes of the user's hand in three-dimensional space.

[0022] Step 4: After the server obtains the user's gesture signals of different motion types, it determines the user's intention based on the user's static gestures;

[0023] Gesture judgment is performed through a pre-trained neural network consisting of an input layer, two dropout layers, and two fully connected layers. The neural network inputs the user's gesture information and obtains the user's gesture intention, namely large-scale cursor transfer operations and small-scale cursor fine-tuning operations;

[0024] Step 5: After accurately analyzing the specific type of cursor movement and its corresponding change value, the server generates a corresponding instruction and accurately sends the instruction to the output device through a stable communication channel to ensure that the output device can receive the relevant information and respond in a timely manner;

[0025] Step 6: After receiving the instruction sent by the server, the output device obtains the change value of the cursor movement based on the information in the instruction; the output device reads the current position of the cursor, and then the output device calculates the current position and the change value of the corresponding movement direction to obtain a new cursor position; finally, the output device implements the corresponding cursor movement function based on this new cursor position, so that the cursor can be accurately moved to the specified position on the screen, thereby meeting the user's operation needs.

[0026] Preferably, the mean filtering is performed in the form of a sliding window for smoothing; the server selects a window of fixed size, which slides gradually on the data sequence; for the data in the window, the server calculates their average value and uses this average value as the filtering result of the data at the center of the window.

[0027] Preferably, the large-scale cursor transfer operation is based on a pre-designed CD function, with the speed of the hand rotating around three axes as a variable input to obtain the cdgain at the current moment; the difference between the palm pitch angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the vertical direction; the difference between the palm roll angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the horizontal direction.

[0028] Preferably, the small-range cursor fine-tuning operation is to calculate the physical movement distance of the hand at the current moment compared with the previous moment in the spatial coordinate system; according to a pre-designed CD function, the cdgain at the current moment is calculated; the difference in the vertical displacement of the palm between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the vertical direction; the difference in the horizontal displacement of the palm between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the horizontal direction; and the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

[0029] Preferably, when the input device collects user hand motion information data, the mobile device is required to be an Android phone with a front camera and the number of frames captured by the camera is more than 30 frames; a hand key point capture program developed using the mediapipe framework runs on the mobile device, which can capture the coordinates of 21 key points of the user's hand; the hand key point coordinates are composed of 3 floating-point numbers, the first two dimensions are the x and y axis coordinates in the camera coordinate system, and the third dimension is the front and rear position relationship of the hand key point relative to the wrist; the mobile device collects the user's hand key point coordinates at a rate of 30 frames per second.

[0030] The mobile device establishes a connection with the server, transmits the collected hand information in the form of a message queue, and specifies the switch type and queue name for transmitting the message.

[0031] Preferably, during the mean filtering, a smoothing operation is performed with a sliding window of 20 frames.

[0032] Preferably, the intention determination is to input the landmark coordinates of the user's right hand into the neural network, with the input data format being 21*2, i.e., the x-axis and y-axis coordinates of 21 key points of the right hand; the neural network outputs two types of results, representing a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation, respectively;

[0033] For large-scale cursor transfer operations, the user's hand rotation angle is collected. In order to calculate the user's hand rotation angle, the camera coordinate system is converted into the world coordinate system through the PnP algorithm. The conversion formula is:

[0034]

[0035] Where s is the scale factor, K is the 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is the 3*3 rotation matrix, and t is the 3*1 translation vector;

[0036] After obtaining the world coordinate system of the hand, calculate the rotation matrix of the hand relative to the camera coordinate system; the rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. rel Calculated by the following formula:

[0037]

[0038] in is the transpose of the first frame’s rotation matrix;

[0039] Then the relative rotation matrix R relConvert the rotation vector into the Rodrigues vector, and then convert it into the rotation angle;

[0040] Calculate the rotation angle between the two frames; if the rotation angle in any direction is greater than the threshold of 0.3, it is determined that the user has rotated in that direction; otherwise, the hand rotation signal in that direction is directly filtered out;

[0041] For small-scale fine-tuning operations, it is necessary to collect the difference in the user's hand displacement. First, the width of the user's palm must be calculated as a reference unit to block the interference caused by depth information. The specific formula is as follows:

[0042]

[0043] Wherein (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device.

[0044] After calculating the palm width, the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment is calculated. The difference in coordinates divided by the palm width gives the difference in the user's hand displacement between two adjacent frames.

[0045] For large-scale cursor control, a nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows:

[0046]

[0047] For small-scale cursor fine-tuning operations, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows:

[0048] Y(x)=0.6x

[0049] The obtained cdgain is multiplied by the physical distance of the hand displacement or the angle of hand rotation to obtain the distance the cursor should move; the server will send the corresponding instructions to the large-screen device; after receiving the instructions, the large-screen device will complete the execution of the corresponding cursor movement.

[0050] The beneficial effects of the present invention are as follows:

[0051] By comprehensively and effectively utilizing hand information preprocessing, multi-scale hand information capture, a flexible composite mapping mechanism, and a dynamically adjusted CD function, this invention not only enables highly efficient cursor movement of output devices over a wide range, but also enables precise and precise fine-tuning. This innovative approach significantly improves the efficiency of mid-air gesture interaction and greatly optimizes the user experience, making it smoother and more natural. This method cleverly combines the respective advantages of mid-air gesture input and output devices, enabling this cross-device interaction technology to perfectly adapt to more complex and diverse application scenarios. It fully integrates the interactive capabilities of multiple devices, effectively overcoming the limitations of traditional single-device interaction and providing users with richer and more diverse interaction options. Furthermore, this technology has extremely broad application prospects in a variety of scenarios, particularly in cutting-edge fields such as remote control, virtual reality, and augmented reality. It will greatly improve user operation precision and convenience, providing users with a more efficient, convenient, and enjoyable operating experience, further promoting technological development and application innovation in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the method of the present invention;

[0053] Figure 2 Flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and examples.

[0055] In the process of cross-device interaction, cursor control of the output device is an important interactive function. Users often have the need to move the cursor over a large range and fine-tune the cursor over a small range. For large-scale cursor movement, users need to achieve this function efficiently and quickly; for small-scale cursor fine-tuning, users need to achieve this function accurately. Since input devices in cross-device interaction often have problems such as limited perception range, traditional cursor control methods cannot meet the above two user needs. Therefore, this method proposes a cross-device multi-scale cursor control method suitable for air gesture interaction. In response to the above two user needs, two different gesture actions are used to implement the corresponding control methods, different gesture signals are collected for different control methods, different mapping functions are dynamically adjusted, and the cursor movement sensitivity is flexibly changed, thereby meeting the needs of efficient large-scale cursor movement and small-scale precise cursor fine-tuning.

[0056] Input devices accurately and comprehensively collect user hand motion data, covering various hand movement characteristics and details. This information is then reliably transmitted to the server through a rigorous and reliable message queue. During this process, the message queue ensures that messages arrive at the server in a consistent order, preventing out-of-order messages. Furthermore, it strives to ensure information integrity, minimizing or even eliminating message loss. This ensures that the server receives accurate and complete user hand motion data, providing a reliable foundation for subsequent processing and analysis.

[0057] After receiving data from the input device, the server performs mean filtering on the hand information. Specifically, this smoothing operation is performed using a sliding window. During this process, the server selects a fixed-size window that gradually slides across the data sequence. For the data within the window, the server calculates its average value and uses this average value as the filtering result for the data at the center of the window. This method effectively reduces noise and fluctuations in the hand information, making the data smoother and more stable, providing a more accurate and reliable data foundation for subsequent processing steps.

[0058] After smoothing the cursor, the server needs to collect different gesture signals based on the user's intended cursor movement type. For large-scale cursor transfers, hand rotation information must be extracted. Centered around the user's right wrist, the overall pitch and roll angles of the right hand are calculated. Because hand rotational motion can be accompanied by noise from other non-motion directions, threshold filtering is used to filter it.

[0059] For small-scale fine-tuning of the cursor, it is crucial to accurately extract the translational motion information of the user's hand. During this process, based on a fixed coordinate system perpendicular to the ground, the system accurately collects and calculates the translational motion changes of the root node of the user's right middle finger in three-dimensional space. Through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be accurately determined, and its position changes at different times can be monitored and analyzed in detail. In this way, the translational motion trajectory and changes of the user's hand in three-dimensional space can be accurately obtained, thereby providing an accurate basis and support for the small-scale fine-tuning of the cursor, ensuring that the cursor can be accurately adjusted within a small range.

[0060] After receiving gesture signals of different motion types, the server determines the user's intention based on the user's static gestures. This is done using a pre-trained neural network consisting of an input layer, two dropout layers, and two fully connected layers. The network takes the user's gesture information as input and determines the user's gesture intent, which can be either a large-scale cursor shift or a small-scale cursor fine-tuning operation.

[0061] For large-scale cursor shifts, a pre-designed CD function uses the hand's rotational speed around three axes as input variables to calculate the current cdgain. The vertical cursor change is calculated by multiplying the difference in the hand's pitch angle between the previous and current moments by the calculated cdgain. The horizontal cursor change is calculated by multiplying the difference in the hand's roll angle between the previous and current moments by the calculated cdgain.

[0062] For small-scale fine-tuning of the cursor, calculate the physical movement distance of the hand at the current moment compared to the previous moment in the spatial coordinate system. According to the pre-designed CD function, calculate the cdgain at the current moment. The difference in the vertical displacement of the palm of the hand between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change in the vertical direction of the cursor; the difference in the horizontal displacement of the palm of the hand between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change in the horizontal direction of the cursor. The distance between the user's hand and the input device will affect the displacement difference of the palm of the hand at two adjacent moments. In order to mask the error caused by the depth information of the user's hand, the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

[0063] After accurately analyzing the specific type of cursor movement and its corresponding change value, the server will quickly generate corresponding instructions and accurately send the instructions to the output device through a stable communication channel to ensure that the output device can receive relevant information in a timely manner and make accurate responses.

[0064] After receiving the command from the server, the output device accurately determines the change in cursor movement based on the information in the command. The output device reads the current cursor position and then calculates the change in the corresponding movement direction. In this way, the output device accurately determines the new cursor position. Finally, the output device implements the corresponding cursor movement function based on this new cursor position, allowing the cursor to accurately move to the specified position on the screen, thus meeting the user's operational requirements.

[0065] Corresponding high-availability cross-device air gesture interaction system:

[0066] The user uses the monocular camera of a mobile device to capture the user's hand information. The mobile device must be an Android phone with a front-facing camera that can capture more than 30 frames. A hand key point capture program developed using the mediapipe framework runs on the mobile device, which can capture the coordinates of 21 key points on the user's hand. The hand key point coordinates are composed of three floating-point numbers. The first two dimensions are the x and y coordinates in the camera coordinate system, and the third dimension is the front-to-back position of the hand key point relative to the wrist. The mobile device collects the coordinates of the user's hand key points at a rate of 30 frames per second.

[0067] The mobile device will establish a connection with the server and transmit the collected hand information in the form of a message queue, specifying the switch type and queue name for transmitting messages to prevent data loss, disorder and other problems.

[0068] The server sequentially retrieves the hand keypoint coordinate data from a pre-determined queue. It then applies a mean filter to the received hand data, smoothing it with a 20-frame sliding window to reduce interference caused by natural hand movement and mediapipe recognition errors.

[0069] After smoothing the data, the user's intent needs to be determined. The coordinates of the landmarks on the user's right hand are input into the neural network. The input data format is 21*2, representing the x-axis and y-axis coordinates of 21 key points on the right hand. The neural network outputs two types of results, representing large-scale cursor movement and small-scale fine-tuning cursor movement.

[0070] Next, we need to collect information about the user's gesture changes. For large-scale cursor transfer operations, we need to collect the angle of the user's hand rotation. In order to calculate the user's hand rotation angle, the camera coordinate system is converted into world coordinates through the PnP algorithm. The conversion formula is:

[0071]

[0072] Where s is a scale factor, K is a 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and t is a 3*1 translation vector.

[0073] After obtaining the world coordinate system of the hand, the rotation matrix of the hand relative to the camera coordinate system can be calculated. The rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. The relative rotation matrix R rel It can be calculated by the following formula:

[0074]

[0075] in is the transpose of the rotation matrix of the first frame.

[0076] Then the relative rotation matrix R rel Convert the rotation vector into a Rodrigues vector, and then convert it into the rotation angle. The Rodrigues function in OpenCV can achieve this. The Rodrigues vector is a vector that represents the rotation axis and the rotation angle, and its modulus is the rotation angle.

[0077] Through the above steps, the rotation angle between the two frames can be accurately calculated. If the rotation angle in any direction is greater than the threshold of 0.3, it is considered that the user has rotated in that direction; otherwise, the hand rotation signal in that direction is directly filtered out.

[0078] For small-scale fine-tuning operations, it is necessary to collect the difference in the user's hand displacement. First, the width of the user's palm must be calculated as a reference unit to block the interference caused by depth information. The specific formula is as follows:

[0079]

[0080] Wherein (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device.

[0081] After calculating the palm width, the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment is calculated. The difference in coordinates divided by the palm width gives the difference in the user's hand displacement between two adjacent frames.

[0082] For large-scale cursor control, a nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows:

[0083]

[0084] For small-range fine-tuning control, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows:

[0085] Y(x)=0.6x

[0086] Multiplying the resulting cdgain by the physical distance of the hand displacement or the angle of hand rotation yields the desired cursor movement distance. The server then sends the corresponding command to the large-screen device, which then executes the corresponding cursor movement.

[0087] Definition of cursor control information; for the rotation posture information of the hand, with the user's cursor controlled wrist as the center, calculate the overall pitch angle change and roll angle change of the user's palm. The pitch change of the palm controls the large-scale vertical movement of the cursor on the two-dimensional plane, and the roll change of the palm controls the large-scale horizontal movement of the cursor on the two-dimensional plane; for the spatial movement information of the hand, based on the fixed coordinate system perpendicular to the ground, collect and calculate the projection of the distance change of the root node of the middle finger of the user's cursor controlled hand in the three-dimensional space to the two-dimensional plane. The vertical distance change controls the small-scale vertical movement of the cursor on the two-dimensional plane, and the horizontal distance change controls the small-scale horizontal movement of the cursor on the two-dimensional plane.

[0088] During mid-air gesture interaction, the rich perception capabilities of some devices are used as input devices, and the rich information sharing capabilities of some devices are used as output devices. The input devices collect user gesture information, which is processed by the multi-scale mid-air gesture cursor control method, and the output devices execute the function corresponding to the gesture.

[0089] For the pre-processing of hand information, the hand information captured by the input device needs to be processed by mean filtering to smooth out interference caused by hand shaking and other reasons:

[0090]

[0091] Where g(x, y, z) is the hand information at a certain moment in three-dimensional space, m is the size of the sliding window, and f(x i ,y i ,z i ) is a plurality of hand information data within the sliding window;

[0092] When processing hand movements in one direction, threshold filtering is used to block interference from data in other directions:

[0093]

[0094] Among them, f(x) is the flag of whether to discard the data in a certain direction, x i is the hand data in that direction at a certain moment, x i-1 is the hand data in this direction at the previous moment.

[0095] Establish a mapping relationship between the physical input of the hand changes and the cursor movement distance on the output device. The mapping formula is:

[0096] ΔT=Δx·cdgain

[0097] Where Δx is the physical distance of hand displacement or the angle of hand rotation, ΔT is the distance the cursor moves, and cdgain is a dynamically adjusted coefficient. The CD function adapted to the two types of cursor movement mentioned in claim 1 is designed based on information such as the speed of the hand. The nonlinear CD function is suitable for efficient control of a large-scale cursor, while the linear CD function is suitable for precise control of a small-scale cursor:

[0098] Use the generalized logistic function to define the nonlinear CD function:

[0099]

[0100] Where Y(x) is the changing cdgain, x is the speed information of the hand change, and in this study, a=0, k=10, c=1, q=1, b=6.908, and m=0.5 are defined.

[0101] The linear CD function is:

[0102] Y(x)=ax+b

[0103] Where Y(x) is the changing cdgain, x is the speed information of the hand change, and in this study, a=0.6 and b=0 are defined.

[0104] The CD function adjusts the mapping between hand movements and screen cursor movement, enabling rapid targeting of the cursor during large-scale movements while providing highly precise control for smaller, more detailed operations. This function automatically adjusts the cursor's response sensitivity based on dynamic parameters such as the user's hand's speed, direction, and displacement, optimizing the control experience for different operating scenarios. This adaptive cursor mapping mechanism enables efficient and smooth cursor operation without frequent hand adjustments, making it suitable for complex tasks in cross-device gesture interactions.

[0105] An embodiment of the present invention provides a cursor control method for a mobile device as an input device and a large-screen device as an output device. The method includes collecting user hand data, preprocessing the data, determining the user's cursor movement intention, using a composite mapping mechanism, dynamically adjusting the CD mapping function, and sending and executing cursor movement instructions. This example can be applied in conference room scenarios, allowing participants to use their mobile phones to interact with the large-screen device through mid-air gestures, controlling the movement of the large-screen device's cursor. This method can also be expanded to implement gesture control of multimedia functions.

Claims

1. A cross-device multi-scale cursor control method in mid-air gesture interaction, characterized in that: The steps include: Step 1: The input device collects user hand motion information data and transmits it to the server through the message queue; Step 2: After receiving the data transmitted by the input device, the server performs mean filtering on the hand information; Step 3: Collect different gesture signals based on the user's expected cursor movement type; For large-scale cursor transfer operations, the hand rotational motion information is extracted; with the user's right wrist as the center, the overall pitch and roll angle changes of the user's right palm are calculated; and threshold filtering is then used to filter the information; For small-scale fine-tuning of the cursor, the system extracts the user's hand translation information. During this process, based on a fixed coordinate system perpendicular to the ground, the system collects and calculates the translational motion changes of the root node of the user's right middle finger in three-dimensional space. Through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be determined, and its position changes at different times can be monitored and analyzed to obtain the translational motion trajectory and changes of the user's hand in three-dimensional space. Step 4: After the server obtains the user's gesture signals of different motion types, it determines the user's intention based on the user's static gestures; Gesture judgment is performed through a pre-trained neural network consisting of an input layer, two dropout layers, and two fully connected layers. The neural network inputs the user's gesture information and obtains the user's gesture intention, namely large-scale cursor transfer operations and small-scale cursor fine-tuning operations; Step 5: After accurately analyzing the specific type of cursor movement and its corresponding change value, the server generates a corresponding instruction and accurately sends the instruction to the output device through a stable communication channel to ensure that the output device can receive the relevant information and respond in a timely manner; Step 6: After receiving the instruction sent by the server, the output device obtains the change value of the cursor movement based on the information in the instruction; the output device reads the current position of the cursor, and then the output device calculates the current position and the change value of the corresponding movement direction to obtain a new cursor position; finally, the output device implements the corresponding cursor movement function based on this new cursor position, so that the cursor can be accurately moved to the specified position on the screen, thereby meeting the user's operation needs.

2. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: The mean filter performs a smoothing operation in the form of a sliding window; the server selects a window of fixed size, which slides gradually on the data sequence; for the data in the window, the server calculates their average value and uses this average value as the filtering result of the data at the center of the window.

3. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: The large-scale cursor transfer operation is based on a pre-designed CD function, which uses the speed of the hand's rotation around three axes as a variable input to obtain the cdgain at the current moment; the difference in the palm pitch angle change between the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value in the vertical direction of the cursor; the difference in the palm roll angle change between the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value in the horizontal direction of the cursor.

4. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: The small-range cursor fine-tuning operation is to calculate the physical movement distance of the hand at the current moment compared to the previous moment in the spatial coordinate system; according to the pre-designed CD function, the cdgain at the current moment is calculated; the difference in the vertical displacement of the palm of the hand between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value in the vertical direction of the cursor; the difference in the horizontal displacement of the palm of the hand between the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value in the horizontal direction of the cursor; and the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

5. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: When the input device collects user hand motion information data, the mobile device is required to be an Android phone with a front-facing camera that can capture at least 30 frames. A hand key point capture program developed using the mediapipe framework runs on the mobile device and is capable of capturing the coordinates of 21 key points of the user's hand. The hand key point coordinates are composed of three floating-point numbers: the first two dimensions are the x and y coordinates in the camera coordinate system, and the third dimension is the front-to-back position of the hand key point relative to the wrist. The mobile device collects the user's hand key point coordinates at a rate of 30 frames per second. The mobile device establishes a connection with the server, transmits the collected hand information in the form of a message queue, and specifies the switch type and queue name for transmitting the message.

6. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: During the mean filtering, a smoothing operation is performed using a sliding window of 20 frames.

7. The cross-device multi-scale cursor control method in mid-air gesture interaction according to claim 1, characterized in that: The intention determination is to input the landmark coordinates of the user's right hand into the neural network. The input data format is 21*2, that is, the x-axis and y-axis coordinates of 21 key points on the right hand. The neural network outputs two types of results, representing large-scale cursor transfer operations and small-scale cursor fine-tuning operations. For large-scale cursor transfer operations, the user's hand rotation angle is collected. In order to calculate the user's hand rotation angle, the camera coordinate system is converted into the world coordinate system through the PnP algorithm. The conversion formula is: Where s is the scale factor, K is the 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is the 3*3 rotation matrix, and t is the 3*1 translation vector; After obtaining the world coordinate system of the hand, calculate the rotation matrix of the hand relative to the camera coordinate system; the rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. rel Calculated by the following formula: in is the transpose of the first frame’s rotation matrix; Then the relative rotation matrix R rel Convert the rotation vector into the Rodrigues vector, and then convert it into the rotation angle; Calculate the rotation angle between the two frames; if the rotation angle in any direction is greater than the threshold of 0.3, it is determined that the user has rotated in that direction; otherwise, the hand rotation signal in that direction is directly filtered out; For small-scale fine-tuning operations, it is necessary to collect the difference in the user's hand displacement. First, the width of the user's palm is calculated as a reference unit to block the interference caused by depth information. The specific formula is as follows: Where (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device; After calculating the palm width, calculate the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment. Divide the coordinate difference by the palm width to get the difference in the user's hand displacement between two adjacent frames. For large-scale cursor control, a nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows: For small-scale cursor fine-tuning operations, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows: Y(x)=0.6x The obtained cdgain is multiplied by the physical distance of the hand displacement or the angle of hand rotation to obtain the distance the cursor should move; the server will send the corresponding instructions to the large-screen device; after receiving the instructions, the large-screen device will complete the execution of the corresponding cursor movement.

Images