Cross-device multi-scale cursor control method in air gesture interaction

Abstract

The invention discloses a cross-device multi-scale cursor control method in air gesture interaction, which comprises the following steps: firstly, acquiring hand motion information data of a user by an input device, and transmitting the hand motion information data to a server in a message queue mode; after receiving the data transmitted by the input device, the server performs mean filtering processing on the hand information; acquiring different gesture signals according to a cursor movement type expected by a user; after acquiring the gesture signals of different motion types of the user, the server judges the intention of the user according to the static gesture of the user; the server generates a corresponding instruction and accurately sends the instruction to the output equipment through a stable communication channel; and after receiving the instruction sent by the server, the output device obtains the change value of cursor movement and solves a new cursor position, so that a corresponding cursor movement function is realized. According to the invention, not only can the cursor movement of the output equipment in a large range be extremely efficiently realized, but also extremely fine adjustment operation can be accurately carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture recognition, and in particular relates to a cross-device multi-scale cursor control method in air gesture interaction. Background Art

[0002] With the continuous development of information technology, mid-air gesture interaction, as a natural human-computer interaction method, has gradually become a research hotspot. Compared with traditional physical input devices such as keyboards and mice, mid-air gesture interaction is more intuitive and flexible. Users can get rid of the limitations of specific devices and use hand movements to interact with the system. In particular, in the fields of large-screen display, smart home control, virtual reality (VR) and augmented reality (AR), mid-air gesture interaction technology has shown great application potential.

[0003] Traditional mid-air gesture interactions are mostly used on a single device, but in multi-device collaborative operation scenarios, especially in cross-device interactions, users often need to control the cursor position on the device more flexibly and accurately. However, current gesture recognition and control methods still have some limitations, especially in multi-device collaborative scenarios. How to achieve cross-scale control of the cursor on different devices still faces problems such as limited capture range of input devices and insufficient cursor movement accuracy, which greatly affects the smoothness and accuracy of the interaction.

[0004] Patent CN108776550A proposes a cursor positioning control device, a control method and an electronic device. The device includes a base and a control ball, and the movement and pressing of the control ball are controlled by fingers.

[0005] Patent CN113282222A provides a cursor control method and device, which is applied to a mobile terminal equipped with an acceleration sensor. The method includes: when the terminal is in remote control mode, obtaining the values ​​of the acceleration sensor on two coordinate axes, determining the target key value (the key value corresponding to the direction key of the remote control) according to the value, and controlling the cursor of the second terminal to move in the direction of the target key value. The device includes an acquisition module, a determination module and a control module. By controlling the terminal posture to control the cursor movement, the problem of easy damage and inconvenience of operation of physical buttons is avoided.

[0006] Patent CN111045638A discloses a cursor control method, device, storage medium and terminal, which aims to use voice recognition technology to input operation instructions of a mouse with a microphone, replace the user's hands to control the mouse cursor, and improve the accuracy and convenience of cursor control.

[0007] Patent CN115291733A discloses a cursor control method, which uses a camera to capture user gesture displacement information to control the movement of the cursor. The movement of the cursor is mainly determined by determining the first distance and the second distance according to the focus position, the position before the movement and the position after the movement.

[0008] Patent CN108292362A describes a system, method, device and computer-readable medium for gesture recognition. The technology creates descriptors by analyzing the motion vectors and optical flow histograms of the hand to achieve gesture recognition and cursor control. It has the characteristics of being invariant to scale, rotation and hand shape, and can be applied to different image types, improving accuracy and performance.

[0009] Patent PCTCN2023135670 discloses a cursor control method, the main steps of which are: the first electronic device controls the cursor according to the instructions of the second electronic device, receives the first instruction to display the cursor in a static state, receives the second instruction to change and move the cursor between the static state and the pointing state, and receives the third instruction to play the target focus screen; the second electronic device generates instructions in response to user operations to control the movement and status display of the cursor, focus and list on the first electronic device.

[0010] Patent PCTCN2021080438 relates to an interactive method between a display device and a terminal device, including the display device receiving a terminal device access request and establishing communication, generating and displaying a corresponding cursor, and controlling the cursor position according to a cursor control instruction; the terminal device obtains an access address to establish a connection and control the cursor; multiple display devices can display synchronously, and they are connected through a cloud server. The method also includes setting permissions for the terminal device, etc.

[0011] Patent CN116048313A discloses a cursor control method, which involves the problem of contact gesture control. The main steps are that the cursor input device detects the user's operation of lifting and leaving the touch area of ​​the cursor input device, controls the display device to determine the cursor position offset information of each cycle, and reduces the cursor position offset information to a preset value. The display device displays the cursor based on the preset value.

[0012] Patent CN113010130A discloses a long-distance large-screen interactive control method based on image stream, including a large-screen end and a smart phone end. The movement of the rectangular cursor on the large-screen end is controlled by the built-in inertial sensor of the smart phone. The screenshot of the content in the rectangular cursor area is transmitted to the smart phone end in real time. The touch control on the smart phone end confirms the selection of the target point, and then various interactive operations on the selected target point are realized through the inertial sensor control mode or the touch screen control mode.

[0013] Most of the current cursor control methods do not consider cross-device air gesture interaction scenarios, and do not consider the user's multi-scale cursor control needs. The above-mentioned patents CN108776550A, CN113282222A, and CN111045638A use specific terminal devices, voice commands, etc. to achieve efficient cursor control, and do not apply gesture control; patents CN115291733A and CN108292362A both use gestures for cursor control, but each focuses on the specific gesture recognition method of gesture-controlled cursors, and does not involve cursor control in cross-device interaction; patents PCTCN2023135670 and PCTCN2021080438 involve cursor control methods between multiple devices, but more often use multiple devices as auxiliary devices, and do not combine the unique interaction capabilities of different devices to design targeted cursor control methods. The cursor control methods mentioned in patents CN116048313A and CN113010130A are applicable to cross-device scenarios, but they are not applicable to contact gestures rather than air gestures, and do not involve multi-scale cursor control methods.

[0014] Currently, there is no method that can meet the multi-scale cursor control requirements in cross-device air gesture interaction scenarios, that is, to meet the multi-scale requirements of large-scale efficient cursor control and small-scale precise cursor control. Therefore, there is an urgent need for a cursor control method that can adapt to multi-device air gesture interaction scenarios to improve the user's operating experience and interaction efficiency. Summary of the invention

[0015] In order to overcome the shortcomings of the prior art, the present invention provides a cross-device multi-scale cursor control method in air gesture interaction. First, the input device collects the user's hand motion information data and transmits it to the server through a message queue; after receiving the data transmitted by the input device, the server performs mean filtering on the hand information; different gesture signals are collected according to the user's expected cursor movement type; after the server obtains the user's gesture signals of different motion types, the user's intention is judged according to the user's static gesture; the server generates a corresponding instruction and accurately sends the instruction to the output device through a stable communication channel; after receiving the instruction sent by the server, the output device obtains the change value of the cursor movement according to the information in the instruction, calculates the new cursor position, and realizes the corresponding cursor movement function. The present invention can not only realize the cursor movement of the output device in a large range very efficiently, but also can accurately perform extremely fine fine-tuning operations. During the interaction process, the user can switch the multi-scale cursor control method according to his own subjective judgment and control ability. When the user feels that the accuracy of the cursor control is not enough, he can choose a small-range fine-tuning operation, and when the user feels that the speed of the cursor control is not enough, he can choose a large-range transfer operation. Due to the differences between interactive tasks and individuals, no quantitative distinction is made here between the applicable scopes of the two control methods to prevent rigid requirements from leading to a decrease in interactive efficiency.

[0016] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0017] Step 1: The input device collects the user's hand motion information data and transmits it to the server through the message queue;

[0018] Step 2: After receiving the data transmitted by the input device, the server performs mean filtering on the hand information;

[0019] Step 3: Collect different gesture signals according to the cursor movement type expected by the user;

[0020] For large-scale cursor transfer operations, the rotational motion information of the hand is extracted; with the user's right wrist as the center, the overall pitch angle change and roll angle change of the user's right palm are calculated; and then threshold filtering is used to filter the information;

[0021] For small-range fine-tuning operations of the cursor, the translational motion information of the user's hand is extracted; in this process, based on a fixed coordinate system perpendicular to the ground, the system collects and calculates the translational motion changes of the root node of the user's right middle finger in three-dimensional space; through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be determined, and its position changes at different times can be monitored and analyzed to obtain the translational motion trajectory and changes of the user's hand in three-dimensional space;

[0022] Step 4: After the server obtains the gesture signals of different motion types of the user, it determines the user's intention based on the user's static gestures;

[0023] Gesture judgment is performed through a pre-trained neural network, which consists of an input layer, two dropout layers, and two fully connected layers. The user's gesture information is input to obtain the user's gesture intention, that is, a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation;

[0024] Step 5: After accurately analyzing the specific type of cursor movement and its corresponding change value, the server generates a corresponding instruction and accurately sends the instruction to the output device through a stable communication channel to ensure that the output device can receive the relevant information in time and respond;

[0025] Step 6: After receiving the instruction sent by the server, the output device obtains the change value of the cursor movement according to the information in the instruction; the output device reads the current position of the cursor, and then the output device calculates the current position and the change value of the corresponding moving direction to obtain a new cursor position; finally, the output device implements the corresponding cursor movement function according to the new cursor position, so that the cursor can be accurately moved to the specified position on the screen, thereby meeting the user's operation requirements.

[0026] Preferably, the mean filtering is performed in the form of a sliding window for smoothing; the server selects a window of a fixed size, which slides gradually on the data sequence; for the data in the window, the server calculates their average value and uses this average value as the filtering result of the data at the center of the window.

[0027] Preferably, the large-scale cursor transfer operation is based on a pre-designed CD function, and the speed of the hand rotating around three axes is input as a variable to obtain the cdgain at the current moment; the difference between the palm pitch angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the vertical direction; the difference between the palm roll angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the horizontal direction.

[0028] Preferably, the small-range cursor fine-tuning operation is to calculate the physical movement distance of the hand at the current moment compared with the previous moment in the spatial coordinate system; according to the pre-designed CD function, the cdgain at the current moment is calculated; the difference in the vertical displacement of the palm at the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the vertical direction; the difference in the horizontal displacement of the palm at the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the horizontal direction; the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

[0029] Preferably, when the input device collects user hand motion information data, the mobile device is required to be an Android phone with a front camera and the number of frames captured by the camera is more than 30 frames; a hand key point capture program developed using the mediapipe framework is run on the mobile device, which can capture 21 key point coordinates of the user's hand; the hand key point coordinates are composed of 3 floating point numbers, the first two dimensions are the x and y axis coordinates in the camera coordinate system, and the third dimension is the front and rear position relationship of the hand key point relative to the wrist; the mobile device collects the user's hand key point coordinates at a rate of 30 frames per second.

[0030] The mobile device establishes a connection with the server, transmits the collected hand information in the form of a message queue, and specifies the switch type and queue name for transmitting the message.

[0031] Preferably, during the mean filtering, a smoothing operation is performed with a sliding window of 20 frames.

[0032] Preferably, the intention judgment is to input the landmark coordinates of the user's right hand into the neural network, and the input data format is 21*2, that is, the x-axis and y-axis coordinates of 21 key points of the right hand; the neural network outputs 2 types of results, representing a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation respectively;

[0033] For large-scale cursor transfer operations, the user's hand rotation angle is collected; in order to calculate the user's hand rotation angle, the camera coordinate system is converted into the world coordinate system through the PnP algorithm. The conversion formula is:

[0034]

[0035] Where s is the scale factor, K is the 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix, representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and t is a 3*1 translation vector;

[0036] After obtaining the world coordinate system of the hand, calculate the rotation matrix of the hand relative to the camera coordinate system; the rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. rel Calculated by the following formula:

[0037]

[0038] in is the transpose of the rotation matrix of the first frame;

[0039] Then the relative rotation matrix R relConvert to a rotation vector represented by a Rodrigues vector, and then convert to a rotation angle;

[0040] Calculate the rotation angle between the two frames; if the rotation angle in any direction is greater than the threshold value of 0.3, it is determined that the user has a rotation action in that direction; otherwise, directly filter out the hand rotation signal in that direction;

[0041] For small-scale fine-tuning operations, the difference in the user's hand displacement needs to be collected. First, the width of the user's palm must be calculated as a reference unit to shield the interference caused by depth information. The specific formula is as follows:

[0042]

[0043] Wherein (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device.

[0044] After calculating the palm width, the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment is calculated. The coordinate difference divided by the palm width is the difference in the user's hand displacement between two adjacent frames.

[0045] For the cursor control method with large-scale transfer, the nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows:

[0046]

[0047] For small-range cursor fine-tuning operations, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows:

[0048] Y(x)=0.6x

[0049] The obtained cdgain is multiplied by the physical distance of the hand displacement or the angle of the hand rotation to obtain the distance the cursor should move; the server will send the corresponding instruction to the large-screen device; after receiving the instruction, the large-screen device completes the movement of the corresponding cursor.

[0050] The beneficial effects of the present invention are as follows:

[0051] The present invention can not only realize the cursor movement of the output device in a large range very efficiently, but also accurately perform extremely fine fine-tuning operations by comprehensively and effectively utilizing the preprocessing of hand information, the capture of multi-scale hand information, the flexible composite mapping mechanism, and the dynamically adjusted CD function. This innovative measure significantly improves the efficiency of air gesture interaction, greatly optimizes the user experience, and makes it more smooth and natural during operation. The method cleverly combines the respective advantages of the input and output devices of air gestures, so that this cross-device interaction technology can perfectly adapt to more complex and diverse application scenarios, fully integrate the interaction capabilities of multiple devices, effectively overcome the limitations of traditional single device interaction, and provide users with richer and more diverse interaction options. In addition, this technology has extremely broad application prospects in a variety of application scenarios, especially in frontier fields such as remote control, virtual reality, and augmented reality. It will greatly improve the user's operation accuracy and convenience, bring users a more efficient, convenient and pleasant operation experience, and further promote the technical development and application innovation in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 Flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0055] In the process of cross-device interaction, cursor control of output devices is an important interactive function. Users often have the need to transfer the cursor over a large range and fine-tune the cursor over a small range. For large-range transfer of the cursor, users need to implement this function efficiently and quickly; for small-range fine-tuning of the cursor, users need to implement this function accurately. Since input devices often have problems such as limited perception range in cross-device interaction, traditional cursor control methods cannot meet the above two user needs. Therefore, this method proposes a cross-device multi-scale cursor control method suitable for air gesture interaction. In response to the above two user needs, two different gesture actions are used to implement the corresponding control methods, different gesture signals are collected for different control methods, different mapping functions are dynamically adjusted, and the cursor movement sensitivity is flexibly changed, thereby meeting the needs of efficient large-range transfer of the cursor and small-range precise fine-tuning of the cursor.

[0056] The input device will collect the user's hand movement information data in an all-round and accurate manner, which covers various hand movement characteristics and details. Subsequently, this information will be stably transmitted to the server through a rigorous and reliable message queue method. In this process, the message queue will ensure that the messages arrive at the server in a certain order to avoid disordered messages. At the same time, it will also try its best to ensure the integrity of the information, reduce or even avoid the loss of messages, so as to ensure that the server can receive accurate and complete user hand movement information data, providing a reliable basis for subsequent processing and analysis.

[0057] After receiving the data transmitted by the input device, the server will perform mean filtering on the hand information. Specifically, it will perform smoothing operations in the form of a sliding window. In this process, the server will select a fixed-size window that will slide gradually on the data sequence. For the data in the window, the server will calculate their average value and use this average value as the filtering result of the data at the center of the window. In this way, the noise and fluctuations in the hand information can be effectively reduced, making the data smoother and more stable, providing a more accurate and reliable data basis for subsequent processing steps.

[0058] After the server performs the smoothing operation, it needs to collect different gesture signals according to the type of cursor movement expected by the user. For large-scale cursor transfer operations, it is necessary to extract the rotational movement information of the hand. With the user's right wrist as the center, the overall pitch angle change and roll angle change of the user's right palm are calculated; because the rotational movement of the hand will be accompanied by noise data in other non-movement directions, threshold filtering is used for filtering.

[0059] For small-range fine-tuning of the cursor, it is crucial to accurately extract the translational motion information of the user's hand. In this process, based on a fixed coordinate system perpendicular to the ground, the system will accurately collect and calculate the translational motion changes of the root node of the user's right middle finger in three-dimensional space. Through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be accurately determined, and its position changes at different times can be monitored and analyzed in detail. In this way, the translational motion trajectory and changes of the user's hand in three-dimensional space can be accurately obtained, thereby providing an accurate basis and support for realizing small-range fine-tuning operations of the cursor, ensuring that the cursor can be accurately adjusted within a small range.

[0060] After the server receives the user's gesture signals of different motion types, it determines the user's intention based on the user's static gestures. Gesture judgment is performed through a pre-trained neural network, which consists of an input layer, two Dropout layers, and two fully connected layers. The user's gesture information is input to obtain the user's gesture intention, that is, a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation.

[0061] For large-scale cursor transfer operations, the speed of the hand rotating around the three axes is used as a variable input according to the pre-designed CD function to obtain the cdgain at the current moment. The difference between the palm pitch angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the vertical direction; the difference between the palm roll angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the horizontal direction.

[0062] For small-range fine-tuning operations of the cursor, calculate the physical movement distance of the hand at the current moment compared to the previous moment in the spatial coordinate system. According to the pre-designed CD function, calculate the cdgain at the current moment. The difference between the vertical displacement of the palm at the previous moment and the current moment, multiplied by the calculated cdgain, can get the change value of the cursor in the vertical direction; the difference between the horizontal displacement of the palm at the previous moment and the current moment, multiplied by the calculated cdgain, can get the change value of the cursor in the horizontal direction. The distance between the user's hand and the input device will affect the displacement difference of the palm at two adjacent moments. In order to shield the error caused by the user's hand depth information, the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

[0063] After accurately parsing the specific type of cursor movement and its corresponding change value, the server will quickly generate corresponding instructions and accurately send the instructions to the output device through a stable communication channel to ensure that the output device can receive relevant information in a timely manner and make accurate responses.

[0064] After receiving the instruction sent by the server, the output device will accurately obtain the change value of the cursor movement based on the information in the instruction. The output device reads the current position of the cursor, and then the output device calculates the current position and the change value of the corresponding moving direction. In this way, the output device can accurately calculate the new cursor position. Finally, the output device implements the corresponding cursor movement function based on this new cursor position, so that the cursor can accurately move to the specified position on the screen, thereby meeting the user's operation needs.

[0065] The corresponding highly available cross-device air gesture interaction system:

[0066] The user uses the monocular camera of the mobile device to capture the user's hand information. The mobile device is required to be an Android phone with a front camera and the number of frames captured by the camera can reach more than 30 frames. The hand key point capture program developed using the mediapipe framework runs on the mobile device, which can capture the coordinates of 21 key points of the user's hand. The hand key point coordinates are composed of 3 floating point numbers. The first two dimensions are the x and y axis coordinates in the camera coordinate system, and the third dimension is the front and back position relationship of the hand key point relative to the wrist. The mobile device will collect the user's hand key point coordinates at a rate of 30 frames per second.

[0067] The mobile device will establish a connection with the server, transmit the collected hand information in the form of a message queue, specify the switch type and queue name for transmitting the message, and prevent data loss, disorder and other problems.

[0068] The server takes out the coordinate data of the key points of the hand from the pre-set queue in order, performs mean filtering on the received hand data, and performs smoothing operations with a sliding window of 20 frames to reduce interference caused by natural hand shaking, mediapipe recognition errors, etc.

[0069] After smoothing the data, it is necessary to determine the user's intention. The landmark coordinates of the user's right hand are input into the neural network. The input data format is 21*2, that is, the x-axis and y-axis coordinates of the 21 key points of the right hand. The neural network outputs two types of results, representing large-scale transfer and small-scale fine-tuning of the cursor movement.

[0070] Next, we need to collect information about the user's gesture changes. For large-scale cursor transfer operations, we need to collect the angle of the user's hand rotation. In order to calculate the user's hand rotation angle, the camera coordinate system is converted into world coordinates through the PnP algorithm. The conversion formula is:

[0071]

[0072] Where s is a scale factor, K is a 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and t is a 3*1 translation vector.

[0073] After obtaining the world coordinate system of the hand, the rotation matrix of the hand relative to the camera coordinate system can be calculated. The rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. The relative rotation matrix R rel It can be calculated by the following formula:

[0074]

[0075] in is the transpose of the rotation matrix of the first frame.

[0076] Then the relative rotation matrix R rel Convert to a rotation vector represented by a Rodrigues vector, and then convert to a rotation angle. The Rodrigues function in OpenCV can achieve this. A Rodrigues vector is a vector representing the rotation axis and the rotation angle, and its modulus is the rotation angle.

[0077] Through the above steps, the rotation angle between two frames can be accurately calculated. If the rotation angle in any direction is greater than the threshold value 0.3, it is considered that the user has a rotation action in that direction; otherwise, the hand rotation signal in that direction is directly filtered out.

[0078] For small-scale fine-tuning operations, the difference in the user's hand displacement needs to be collected. First, the width of the user's palm must be calculated as a reference unit to shield the interference caused by depth information. The specific formula is as follows:

[0079]

[0080] Wherein (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device.

[0081] After calculating the palm width, the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment is calculated. The coordinate difference divided by the palm width is the difference in the user's hand displacement between two adjacent frames.

[0082] For the cursor control method with large-scale transfer, the nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows:

[0083]

[0084] For small-range fine-tuning control, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows:

[0085] Y(x)=0.6x

[0086] The obtained cdgain is multiplied by the physical distance of the hand displacement or the angle of hand rotation to get the distance the cursor should move. The server will send the corresponding command to the large screen device. After receiving the command, the large screen device completes the movement of the corresponding cursor.

[0087] Definition of cursor control information; for the rotation posture information of the hand, with the user's cursor controlled wrist as the center, calculate the overall pitch angle change and roll angle change of the user's palm. The pitch change of the palm controls the large-scale vertical movement of the cursor on the two-dimensional plane, and the roll change of the palm controls the large-scale horizontal movement of the cursor on the two-dimensional plane; for the spatial movement information of the hand, based on the fixed coordinate system perpendicular to the ground, collect and calculate the projection of the distance change of the root node of the middle finger of the user's cursor controlled hand in the three-dimensional space to the two-dimensional plane. The vertical distance change controls the small-scale vertical movement of the cursor on the two-dimensional plane, and the horizontal distance change controls the small-scale horizontal movement of the cursor on the two-dimensional plane.

[0088] In the process of air gesture interaction, the rich perception capabilities of some devices are used as input devices, and the rich information sharing capabilities of some devices are used as output devices; the input device collects user gesture information, which is processed by the multi-scale air gesture cursor control method, and the output device executes the function corresponding to the gesture.

[0089] For the preprocessing of hand information, the hand information captured by the input device needs to be processed by mean filtering to smooth out the interference caused by hand shaking and other reasons:

[0090]

[0091] Where g(x, y, z) is the hand information at a certain moment in three-dimensional space, m is the size of the sliding window, and f(x i ,y i ,z i ) is a plurality of hand information data in the sliding window;

[0092] When processing hand movements in one direction, in order to shield the interference of data in other directions, threshold filtering is used for filtering:

[0093]

[0094] Where f(x) is a flag indicating whether to discard data in a certain direction, x i is the hand data in this direction at a certain moment, x i-1 is the hand data in this direction at the previous moment.

[0095] Establish a mapping relationship between the physical input of the hand change and the cursor movement distance on the output device. The mapping formula is:

[0096] ΔT=Δx·cdgain

[0097] Wherein Δx is the physical distance of the hand displacement or the angle of the hand rotation, ΔT is the distance of the cursor movement, cdgain is a dynamically adjusted coefficient, and the CD function adapted to the two types of cursor movement mentioned in claim 1 is designed by information such as the speed of the hand change, wherein the nonlinear CD function is suitable for efficient control of a large-scale cursor, and the linear CD function is suitable for precise control of a small-scale cursor:

[0098] Use the generalized logistic function to define the nonlinear CD function:

[0099]

[0100] Where Y(x) is the changing cdgain, x is the speed information of the hand change, and in this study, a=0, k=10, c=1, q=1, b=6.908, and m=0.5 are defined.

[0101] The linear CD function is:

[0102] Y(x)=ax+b

[0103] Where Y(x) is the changing cdgain, x is the speed information of the hand change, and in this study, a=0.6 and b=0 are defined.

[0104] The CD function adjusts the mapping relationship between hand movement and screen cursor movement, so that the cursor can quickly reach the target area when moving in a large range, and provides highly precise control in fine operations in a small range. This function automatically adjusts the response sensitivity of the cursor according to dynamic parameters such as the user's hand movement speed, direction, and displacement, thereby optimizing the control experience in different operation scenarios. Through this adaptive cursor mapping mechanism, users can achieve efficient and smooth cursor operations without frequently adjusting hand movements, which is suitable for complex tasks in cross-device gesture interaction.

[0105] The embodiment of the present invention provides a cursor control method when a mobile device is used as an input device and a large-screen device is used as an output device, which includes: collecting user hand data, preprocessing data, judging the user's cursor movement intention, using a composite mapping mechanism, dynamically adjusting the CD mapping function, and sending and executing cursor movement instructions. This example can be applied to the scene of a conference room. Participants can use their mobile phones to interact with the large-screen device through air gestures to control the movement of the cursor of the large-screen device. On this basis, gesture control of multimedia functions can be realized.

Claims

1. A cross-device multi-scale cursor control method in air gesture interaction, characterized in that: The steps include: Step 1: The input device collects the user's hand motion information data and transmits it to the server through the message queue; Step 2: After receiving the data transmitted by the input device, the server performs mean filtering on the hand information; Step 3: Collect different gesture signals according to the cursor movement type expected by the user; For large-scale cursor transfer operations, the rotational motion information of the hand is extracted; with the user's right wrist as the center, the overall pitch angle change and roll angle change of the user's right palm are calculated; and then threshold filtering is used to filter the information; For small-range fine-tuning operations of the cursor, the translational motion information of the user's hand is extracted; in this process, based on a fixed coordinate system perpendicular to the ground, the system collects and calculates the translational motion changes of the root node of the user's right middle finger in three-dimensional space; through this fixed coordinate system, the position of the root node of the user's right middle finger in three-dimensional space can be determined, and its position changes at different times can be monitored and analyzed to obtain the translational motion trajectory and changes of the user's hand in three-dimensional space; Step 4: After the server obtains the gesture signals of different motion types of the user, it determines the user's intention based on the user's static gestures; Gesture judgment is performed through a pre-trained neural network, which consists of an input layer, two dropout layers, and two fully connected layers. The user's gesture information is input to obtain the user's gesture intention, that is, a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation; Step 5: After accurately analyzing the specific type of cursor movement and its corresponding change value, the server generates a corresponding instruction and accurately sends the instruction to the output device through a stable communication channel to ensure that the output device can receive the relevant information in time and respond; Step 6: After receiving the instruction sent by the server, the output device obtains the change value of the cursor movement according to the information in the instruction; the output device reads the current position of the cursor, and then the output device calculates the current position and the change value of the corresponding moving direction to obtain a new cursor position; finally, the output device implements the corresponding cursor movement function according to the new cursor position, so that the cursor can be accurately moved to the specified position on the screen, thereby meeting the user's operation requirements.

2. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: The mean filter performs a smoothing operation in the form of a sliding window; the server selects a window of a fixed size, which will slide gradually on the data sequence; for the data in the window, the server calculates their average value and uses this average value as the filtering result of the data at the center of the window.

3. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: The large-scale cursor transfer operation is based on a pre-designed CD function, which takes the speed of the hand rotating around three axes as a variable input to obtain the cdgain at the current moment; the difference between the palm pitch angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the vertical direction; the difference between the palm roll angle change at the previous moment and the current moment is multiplied by the obtained cdgain to obtain the change value of the cursor in the horizontal direction.

4. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: The small-range cursor fine-tuning operation is to calculate the physical movement distance of the hand at the current moment compared with the previous moment in the spatial coordinate system; according to the pre-designed CD function, the cdgain at the current moment is calculated; the difference in the vertical displacement of the palm at the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the vertical direction; the difference in the horizontal displacement of the palm at the previous moment and the current moment is multiplied by the calculated cdgain to obtain the change value of the cursor in the horizontal direction; the length from the root node of the user's index finger to the root node of the little finger is used as the reference unit of the user's palm displacement.

5. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: When the input device collects the user's hand motion information data, the mobile device is required to be an Android phone with a front camera and the number of frames captured by the camera is more than 30 frames; the hand key point capture program developed using the mediapipe framework is run on the mobile device, which can capture the coordinates of 21 key points of the user's hand; the hand key point coordinates are composed of 3 floating point numbers, the first two dimensions are the x and y axis coordinates in the camera coordinate system, and the third dimension is the front and rear position relationship of the hand key point relative to the wrist; the mobile device collects the user's hand key point coordinates at a rate of 30 frames per second; The mobile device establishes a connection with the server, transmits the collected hand information in the form of a message queue, and specifies the switch type and queue name for transmitting the message.

6. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: During the mean filtering, a smoothing operation is performed with a sliding window of 20 frames.

7. The cross-device multi-scale cursor control method in air gesture interaction according to claim 1, characterized in that: The intention judgment is to input the landmark coordinates of the user's right hand into the neural network. The input data format is 21*2, that is, the x-axis and y-axis coordinates of 21 key points of the right hand; the neural network outputs two types of results, representing a large-scale cursor transfer operation and a small-scale cursor fine-tuning operation; For large-scale cursor transfer operations, the user's hand rotation angle is collected; in order to calculate the user's hand rotation angle, the camera coordinate system is converted into the world coordinate system through the PnP algorithm. The conversion formula is: Where s is the scale factor, K is the 3*3 camera intrinsic parameter matrix, [R|t] is a 3*4 matrix, representing the rotation and translation transformation from the world coordinate system to the camera coordinate system, R is a 3*3 rotation matrix, and t is a 3*1 translation vector; After obtaining the world coordinate system of the hand, calculate the rotation matrix of the hand relative to the camera coordinate system; the rotation angle between the two frames is equal to the relative rotation angle from the rotation matrix R1 of the first frame to the rotation matrix R2 of the second frame. rel Calculated by the following formula: in is the transpose of the rotation matrix of the first frame; Then the relative rotation matrix R rel Convert to a rotation vector represented by a Rodrigues vector, and then convert to a rotation angle; Calculate the rotation angle between the two frames; if the rotation angle in any direction is greater than the threshold value of 0.3, it is determined that the user has a rotation action in that direction; otherwise, the hand rotation signal in that direction is directly filtered out; For small-range fine-tuning operations, it is necessary to collect the difference in the user's hand displacement. First, the width of the user's palm should be calculated as the reference unit to shield the interference caused by the depth information. The specific formula is as follows: Where (x1, y1) is the two-dimensional coordinate of the root node of the user's right index finger in the image frame captured by the mobile device, and (x2, y2) is the two-dimensional coordinate of the root node of the user's right little finger in the image frame captured by the mobile device; After calculating the palm width, calculate the difference between the root node coordinates of the user's right middle finger at the current moment and the coordinates at the previous moment. The coordinate difference divided by the palm width is the difference in the user's hand displacement between two adjacent frames. For the cursor control method with large-scale transfer, the nonlinear CD function is used to calculate cdgain, with the user's hand rotation speed as input. The specific formula is as follows: For small-range cursor fine-tuning operations, a linear CD function is used to calculate cdgain, with the user's hand displacement speed as input. The specific formula is as follows: Y(x)=0.6x The obtained cdgain is multiplied by the physical distance of the hand displacement or the angle of the hand rotation to obtain the distance the cursor should move; the server will send the corresponding instruction to the large-screen device; after receiving the instruction, the large-screen device completes the movement of the corresponding cursor.

Images