A control method and system for an intelligent mouse

Abstract

The present invention relates to a control method and system for an intelligent mouse. The method includes: waking up an eyeball monitoring model according to an eyeball detection start signal, determining the fixation point position of a user on a computer screen according to an eyeball tracking technology, and initially automatically moving the mouse pointer position according to the fixation point position; determining a gesture monitoring area according to the fixation point position and user position information; receiving a user's gesture within the gesture monitoring area, and further determining a mouse operation instruction according to the gesture. In the present invention, by starting the eyeball tracking technology, the fixation point position and the mouse pointer position are initially determined, and the gesture monitoring area is flexibly and accurately determined according to the fixation point position and the user position. On the basis of facilitating user operation, the gesture monitoring area of the present invention is conducive to improving the operation accuracy, effectively reducing the recognition error, receiving the user's gesture within the gesture monitoring area, and further determining the mouse operation instruction according to the gesture, which can effectively reduce the operation amount.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent mice, and specifically to a control method and system for an intelligent mouse. Background Art

[0002] With the continuous development of wireless technology, wireless connection technologies such as Bluetooth or wireless radio frequency technology are commonly adopted in intelligent mice. These technologies enable users to get rid of the bondage of cables and move and use the mouse more freely. At the same time, wireless connection technologies also improve the flexibility and portability of devices, making intelligent mice more suitable for scenarios such as mobile office and remote work.

[0003] Although the existing intelligent mice using gesture recognition to control the mouse do bring unprecedented convenience and innovation, there are also problems of misrecognition, resulting in low control accuracy. Summary of the Invention

[0004] The present invention provides a control method and system for an intelligent mouse to solve the technical problems mentioned in the above background art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A control method for an intelligent mouse, the method comprising:

[0007] Receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the fixation point position of the user on the computer screen according to eye tracking technology, and initially automatically moving the mouse pointer position according to the fixation point position;

[0008] Determining a gesture monitoring area according to the fixation point position and user position information;

[0009] Receiving a gesture of the user within the gesture monitoring area, and further determining a mouse operation instruction according to the gesture.

[0010] As a further technical solution of the present invention, the steps of receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the fixation point position of the user on the computer screen according to eye tracking technology, and initially automatically moving the mouse pointer position according to the fixation point position include:

[0011] Receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, and the eye monitoring model includes an eye tracking device;

[0012] Collect the eye movement data of the user based on the eye tracking device, calculate the user's line of sight direction using the line of sight estimation algorithm, combine the line of sight direction with the coordinate system of the computer screen, and calculate the fixation point position of the user on the screen;

[0013] Preliminarily and automatically move the position of the mouse pointer according to the fixation point position, so that the mouse pointer is at the fixation point position, and record the residence time of the mouse pointer at the fixation point position in real time. When the residence time exceeds the preset time threshold, turn off the eye tracking device so that the eye monitoring model is in a sleep state.

[0014] As a further technical solution of the present invention, the steps of determining the gesture monitoring area according to the fixation point position and the user position information include:

[0015] Obtain the position information of the eye tracking device as the user position information, and the position information of the eye tracking device is the position information relative to the computer screen;

[0016] Obtain the mapped position of the user position information on the computer screen (the user position information is in the form of coordinates in three-dimensional space, and the mapped position is in the form of two-dimensional coordinates of the screen, specifically referring to the corresponding point where the three-dimensional coordinates of the user relative to the computer screen are converted or mapped to the two-dimensional coordinate system of the screen);

[0017] Calculate the distance between the mapped position and the fixation point position;

[0018] Judge whether the distance is less than the first preset threshold. When the distance is less than the first preset threshold, construct a gesture monitoring area with a preset size with the fixation point position as the center point;

[0019] When the distance is not less than the first preset threshold, construct a gesture monitoring area with a preset size with the mapped position as the center point.

[0020] As a further technical solution of the present invention, the steps of receiving the user's gesture in the gesture monitoring area and further determining the mouse operation instruction according to the gesture include:

[0021] Capture the user's gesture in the gesture monitoring area through a camera, or use other sensors (such as an accelerometer, a gyroscope) to detect the movement and posture of the hand;

[0022] Classify the gesture to determine the gesture type. The gesture types include single-point gestures and multi-point gestures. The single-point gestures include click, long press, and swipe. The multi-point gestures include pinch, spread, rotation, and zoom;

[0023] Determine the mouse operation instruction according to the gesture type.

[0024] Another object of the present invention is to provide a control system for an intelligent mouse, the system comprising:

[0025] A preliminary mouse position movement module, configured to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine a fixation point position of the user on a computer screen according to eye tracking technology, and preliminarily and automatically move the mouse pointer position according to the fixation point position;

[0026] A gesture monitoring area determination module, configured to determine a gesture monitoring area according to the fixation point position and user position information;

[0027] A mouse operation instruction generation module, configured to receive a user's gesture within the gesture monitoring area and further determine a mouse operation instruction according to the gesture.

[0028] As a further technical solution of the present invention, the preliminary mouse position movement module comprises:

[0029] A monitoring model wake-up unit, configured to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, and the eye monitoring model includes an eye tracking device;

[0030] A fixation position determination unit, configured to collect user's eye movement data based on the eye tracking device, calculate the user's line of sight direction by using a line of sight estimation algorithm, combine the line of sight direction with the coordinate system of the computer screen, and calculate the fixation point position of the user on the screen;

[0031] A preliminary mouse movement unit, configured to preliminarily and automatically move the mouse pointer position according to the fixation point position, so that the mouse pointer is at the fixation point position, and record the residence time of the mouse pointer at the fixation point position in real time. When the residence time exceeds a preset time threshold, turn off the eye tracking device, so that the eye monitoring model is in a sleep state.

[0032] As a further technical solution of the present invention, the gesture monitoring area determination module comprises:

[0033] A position information acquisition unit, configured to acquire the position information of the eye tracking device as user position information;

[0034] A position mapping unit, configured to acquire the mapped position of the user position information on the computer screen;

[0035] A distance calculation unit, configured to calculate the distance between the mapped position and the fixation point position;

[0036] A distance first determination unit is configured to determine whether the distance is less than a first preset threshold. When the distance is less than the first preset threshold, a gesture monitoring area with a preset size is constructed with the fixation point position as the center point. The gesture monitoring area is generally square or circular.

[0037] A distance second determination unit is configured to, when the distance is not less than the first preset threshold, construct a gesture monitoring area with a preset size with the mapped position as the center point.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a control method and system for an intelligent mouse. In the present invention, the fixation point position and the mouse pointer position are initially determined by starting the eye tracking technology. On this basis, the gesture monitoring area is flexibly and accurately determined according to the fixation point position and the user position. The gesture monitoring area of the present invention is convenient for user operation and is conducive to improving the operation accuracy, effectively reducing the recognition error. The gestures of the user are received within the gesture monitoring area, and the mouse operation instruction is further determined according to the gestures, which can effectively reduce the amount of operation. In the present invention, the eye tracking technology is combined with the gesture recognition technology to achieve precise control of the intelligent mouse, effectively reducing the recognition error while reducing the amount of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a control method for an intelligent mouse.

[0040] Figure 2 It is a flowchart of initially moving the mouse pointer position in a control method for an intelligent mouse.

[0041] Figure 3 It is a flowchart of determining a gesture monitoring area in a control method for an intelligent mouse.

[0042] Figure 4 It is a structural block diagram of a control system for an intelligent mouse. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] As Figure 1 shown, an embodiment of the present invention provides a control method for an intelligent mouse. The method includes:

[0045] Step S100: Receive the eye detection start signal input by the user (which can be the user's gesture or a pre-set button operation), wake up the eye monitoring model according to the eye detection start signal, determine the fixation point position of the user on the computer screen according to the eye tracking technology, and initially automatically move the mouse pointer position according to the fixation point position. The eye monitoring model includes an eye tracking device, and the eye monitoring model is used to determine the fixation point position of the user on the computer screen;

[0046] Step S200: Determine the gesture monitoring area according to the fixation point position and the user position information;

[0047] Step S300: Receive the user's gesture in the gesture monitoring area and further determine the mouse operation instruction according to the gesture.

[0048] In this embodiment, the fixation point position and the mouse pointer position are initially determined by starting the eye tracking technology. On this basis, the gesture monitoring area is flexibly and accurately determined according to the fixation point position and the user position. The gesture monitoring area of the present invention is convenient for the user to operate and is conducive to improving the operation accuracy and effectively reducing the recognition error. Receive the user's gesture in the gesture monitoring area and further determine the mouse operation instruction according to the gesture, which can effectively reduce the operation amount. In the present invention, the eye tracking technology is combined with the gesture recognition technology to achieve precise control of the intelligent mouse, effectively reducing the recognition error on the basis of reducing the operation amount.

[0049] Please refer to Figure 2 , the steps of receiving the eye detection start signal input by the user, waking up the eye monitoring model according to the eye detection start signal, determining the fixation point position of the user on the computer screen according to the eye tracking technology, and initially automatically moving the mouse pointer position according to the fixation point position include:

[0050] Step S101: Receive the eye detection start signal input by the user (which can be the user's gesture or a pre-set button operation), wake up the eye monitoring model according to the eye detection start signal. The eye monitoring model includes an eye tracking device, and the eye tracking device is generally worn on the user's face;

[0051] Step S102: Collect the user's eye movement data based on the eye tracking device, calculate the user's line of sight direction using the line of sight estimation algorithm, combine the line of sight direction with the coordinate system of the computer screen, and calculate the fixation point position of the user on the screen. The fixation point position is in the form of two-dimensional coordinates. Among them, the line of sight estimation algorithm is the core of eye tracking technology. It uses the eye movement data collected by the eye tracking device and calculates the user's line of sight direction through complex algorithm processing. This process usually involves multiple steps such as the eye structure model, geometric feature extraction, and mapping from feature learning to the line of sight direction. Eye structure model: In some cases, the algorithm uses parameter information such as the geometric shape of the eye and the direction of the visual axis to assist in calculating the line of sight direction. Geometric feature extraction: The algorithm extracts eye appearance information or geometric features from the eye image, such as pupil position, corneal reflection, etc. Mapping from feature learning to the line of sight direction: Based on technologies such as deep learning, the algorithm learns the mapping relationship between these features and the line of sight direction of the human eye, thereby completing the estimation of the line of sight direction. After obtaining the user's line of sight direction, the next step is to combine this direction with the coordinate system of the computer screen to determine the fixation point position of the user on the screen. This process usually involves the following steps: Screen calibration: First, the screen needs to be calibrated to determine the position and orientation of the screen in three-dimensional space. Calculation of the intersection point of the line of sight and the screen: In the three-dimensional coordinate system, according to the user's line of sight direction and the position information of the screen, calculate the intersection point of the line of sight and the screen. This intersection point is the fixation point position of the user on the screen.

[0052] Step S103: Initially automatically move the mouse pointer position according to the fixation point position, so that the mouse pointer is at the fixation point position, and record the residence time of the mouse pointer at the fixation point position in real time. When the residence time exceeds the preset time threshold, turn off the eye tracking device to make the eye monitoring model enter the sleep state.

[0053] In this embodiment, through eye recognition, the amount of user gestures is reduced. For example, from the right side to the left side of the screen, the hand initially has to swing a large amplitude. First, use the eyes to locate a gesture monitoring area, and then when waving the hand, only a little swing is needed, and it is more conducive to improving the operation accuracy.

[0054] Please refer to Figure 3 , the steps of determining the gesture monitoring area according to the fixation point position and the user position information include:

[0055] Step S201: Obtain the position information of the eye tracking device as the user position information. The position information of the eye tracking device is the position information relative to the computer screen.

[0056] Step S202: Obtain the mapped position of the user's location information on the computer screen (the user's location information is in the form of coordinates in three-dimensional space, and the mapped position is in the form of two-dimensional coordinates on the screen, specifically referring to the corresponding point where the three-dimensional coordinates of the user relative to the computer screen are converted or mapped into the two-dimensional coordinate system of the screen);

[0057] Step S203: Calculate the distance between the mapped position and the fixation point position;

[0058] Step S204: Determine whether the distance is less than a first preset threshold. When the distance is less than the first preset threshold, construct a gesture monitoring area with a preset size centered on the fixation point position. The gesture monitoring area is generally square or circular; the preset size is set according to actual applications;

[0059] Step S205: When the distance is not less than the first preset threshold, construct a gesture monitoring area with a preset size centered on the mapped position.

[0060] In this embodiment, when the distance between the mapped position and the fixation point position is less than the first preset threshold, a gesture monitoring area with a preset size is constructed centered on the fixation point position to improve the operation accuracy. When the distance between the mapped position and the fixation point position is not less than the first preset threshold, it means that the user is far from the fixation point position. At this time, a gesture monitoring area convenient for operation can be generated without the user moving, which is convenient for the user. And in the present invention, by setting the gesture monitoring area, it is possible to avoid recognizing the gestures of others to a certain extent, which is beneficial to reducing the recognition error.

[0061] In this embodiment, the steps of receiving the user's gesture within the gesture monitoring area and further determining the mouse operation instruction according to the gesture include:

[0062] Capture the user's gesture within the gesture monitoring area through a camera, or use other sensors (such as an accelerometer, gyroscope) to detect the movement and posture of the hand;

[0063] Classify the gesture to determine the gesture type. The gesture types include single-point gestures and multi-point gestures. Single-point gestures include click, long press, and swipe. Multi-point gestures include pinch, spread, rotate, and zoom;

[0064] Determine the mouse operation instruction according to the gesture type.

[0065] Among them, click: The user touches the screen lightly with a finger or makes a clicking motion in the air to simulate a single click operation of a mouse. This is usually used to select an object, open a link, or perform other basic interactions. Long press: The user continuously presses the screen or holds a certain gesture for a period of time to simulate a long press operation of a mouse. This can be used to activate a specific function or start a dragging operation. Slide: The user slides a finger on the screen or makes a sliding motion in the air to simulate the movement or scrolling operation of a mouse. The sliding speed and direction can affect the movement speed or scrolling direction of the mouse pointer. Pinch: The user brings two fingers closer together and pinches them to simulate a double click operation of a mouse. This is usually used to quickly open or close a certain function, zoom in or out on a view, etc. Expand: Opposite to pinching, the user separates two fingers from a close state, which may be used to restore the view to its original size or perform other operations opposite to zooming in / zooming out. Rotate: The user makes a rotating motion with two fingers on the screen to simulate certain specific rotation operations, such as rotating a picture or adjusting the perspective. Zoom: The user simulates a zoom operation by bringing two fingers closer or farther apart, such as zooming in or out on a picture, map, etc.

[0066] Please refer to Figure 4 , another object of the present invention is to provide a control system for an intelligent mouse, and the system includes:

[0067] The preliminary mouse position movement module 100 is configured to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine the fixation point position of the user on the computer screen according to eye tracking technology, and initially automatically move the mouse pointer position according to the fixation point position;

[0068] The gesture monitoring area determination module 200 is configured to determine a gesture monitoring area according to the fixation point position and user position information;

[0069] The mouse operation instruction generation module 300 is configured to receive a user's gesture within the gesture monitoring area and further determine a mouse operation instruction according to the gesture.

[0070] In this embodiment, the preliminary mouse position movement module 100 includes:

[0071] The monitoring model wake-up unit is configured to receive an eye detection start signal input by a user (which can be a user's gesture action or a preset button operation), wake up an eye monitoring model according to the eye detection start signal, and the eye monitoring model includes an eye tracking device, and the eye tracking device is generally worn on the user's face;

[0072] A fixation position determination unit, which is configured to collect the user's eye movement data based on an eye tracking device, calculate the user's line of sight direction using a line of sight estimation algorithm, combine the line of sight direction with the coordinate system of the computer screen, and calculate the fixation point position of the user on the screen, where the fixation point position is in the form of two-dimensional coordinates;

[0073] A mouse preliminary movement unit, which is configured to preliminarily and automatically move the mouse pointer position according to the fixation point position, so that the mouse pointer is at the fixation point position, and record the residence time of the mouse pointer at the fixation point position in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, and the eye monitoring model is put into a sleep state.

[0074] In this embodiment, the gesture monitoring area determination module 200 includes:

[0075] A position information acquisition unit, which is configured to acquire the position information of the eye tracking device as the user position information, where the position information of the eye tracking device is the position information relative to the computer screen;

[0076] A position mapping unit, which is configured to acquire the mapped position of the user position information on the computer screen (the user position information is in the form of coordinates in a three-dimensional space, and the mapped position is in the form of two-dimensional coordinates of the screen, specifically referring to the corresponding point where the three-dimensional coordinates of the user relative to the computer screen are converted or mapped to the two-dimensional coordinate system of the screen);

[0077] A distance calculation unit, which is configured to calculate the distance between the mapped position and the fixation point position;

[0078] A distance first determination unit, which is configured to determine whether the distance is less than a first preset threshold. When the distance is less than the first preset threshold, a gesture monitoring area with a preset size is constructed with the fixation point position as the center point, and the gesture monitoring area is generally square or circular;

[0079] A distance second determination unit, which is configured to, when the distance is not less than the first preset threshold, construct a gesture monitoring area with a preset size with the mapped position as the center point.

[0080] It should be noted that in this article, the term "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0081] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for controlling an intelligent mouse, characterized in that: The method comprises: Receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position; Determine a gesture monitoring area according to the gaze point position and user position information; receiving a user's gesture in the gesture monitoring area, and further determining a mouse operation instruction according to the gesture; The step of determining the gesture monitoring area according to the gaze point position and the user position information comprises: Obtaining location information of the eye tracking device as user location information; Get the mapping position of the user's location information on the computer screen; Calculating the distance between the mapping position and the gaze point position; Determine whether the distance is less than a first preset threshold, and when the distance is less than the first preset threshold, construct a gesture monitoring area of ​​a preset size with the gaze point position as the center point; When the distance is not less than a first preset threshold, a gesture monitoring area of ​​a preset size is constructed with the mapping position as a center point.

2. The control method of an intelligent mouse according to claim 1, characterized in that: The steps of receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position include: receiving an eye detection start signal input by a user, and waking up an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; The user's eye movement data is collected based on the eye tracking device, and the user's gaze direction is calculated using the gaze estimation algorithm. The gaze direction is combined with the coordinate system of the computer screen to calculate the user's gaze point position on the screen; The mouse pointer position is automatically moved according to the gaze point position so that the mouse pointer is at the gaze point position, and the residence time of the mouse pointer at the gaze point position is recorded in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

3. The control method of an intelligent mouse according to claim 1, characterized in that: The step of receiving a user's gesture within the gesture monitoring area and further determining a mouse operation instruction according to the gesture comprises: Capturing the user's gestures in the gesture monitoring area through a camera; Classify gestures to determine gesture types, which include single-point gestures and multi-point gestures. Single-point gestures include click, long press, and slide, and multi-point gestures include pinch, expand, rotate, and zoom. Determine the mouse operation instruction according to the gesture type.

4. A control system for an intelligent mouse, characterized in that: The system comprises: A preliminary mouse position movement module is used to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine the user's gaze point position on the computer screen according to the eye tracking technology, and preliminarily automatically move the mouse pointer position according to the gaze point position; A gesture monitoring area determination module, used to determine the gesture monitoring area according to the gaze point position and user position information; A mouse operation instruction generating module, used for receiving a user's gesture in the gesture monitoring area, and further determining a mouse operation instruction according to the gesture; The gesture monitoring area determination module includes: A position information acquisition unit, used to acquire position information of the eye tracking device as user position information; A location mapping unit, used to obtain a mapping position of the user's location information on the computer screen; A distance calculation unit, used to calculate the distance between the mapping position and the gaze point position; a first distance determination unit, configured to determine whether the distance is less than a first preset threshold, and when the distance is less than the first preset threshold, construct a gesture monitoring area of ​​a preset size with the gaze point position as the center point; The second distance determination unit is used to construct a gesture monitoring area of ​​a preset size with the mapping position as the center point when the distance is not less than a first preset threshold.

5. The control system of an intelligent mouse according to claim 4, characterized in that: The mouse position preliminary movement module includes: A monitoring model awakening unit, configured to receive an eye detection start signal input by a user, and awaken an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; A gaze position determination unit, for collecting eye movement data of the user based on an eye tracking device, calculating the gaze direction of the user using a gaze estimation algorithm, combining the gaze direction with a coordinate system of a computer screen, and calculating the gaze point position of the user on the screen; The mouse preliminary movement unit is used to automatically move the mouse pointer position according to the gaze point position, so that the mouse pointer is at the gaze point position, and record the residence time of the mouse pointer at the gaze point position in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

Images