Control method and system of intelligent mouse

Abstract

The invention relates to a control method and system for an intelligent mouse, and the method comprises the steps: awakening an eyeball monitoring model according to an eyeball detection starting signal, determining the position of a fixation point of a user on a computer screen according to an eyeball tracking technology, and preliminarily and automatically moving the position of a mouse pointer according to the position of the fixation point; determining a gesture monitoring area according to the fixation point position and the user position information; and receiving a gesture of a user in the gesture monitoring area, and further determining a mouse operation instruction according to the gesture. The position of the fixation point and the position of the mouse pointer are preliminarily determined by starting the eyeball tracking technology, and the gesture monitoring area is flexibly and accurately determined according to the position of the fixation point and the position of the user, so that the gesture monitoring area is beneficial to improving the operation precision and effectively reducing the recognition error on the basis of facilitating user operation, and the user experience is improved. The gesture of the user is received in the gesture monitoring area, the mouse operation instruction is further determined according to the gesture, and the operation amount can be effectively reduced.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent mouse, in particular to a control method and system of an intelligent mouse. Background Art

[0002] With the continuous development of wireless technology, smart mice also generally adopt wireless connection technology, such as Bluetooth or wireless radio frequency technology. These technologies allow users to get rid of the constraints of cables and move and use the mouse more freely. At the same time, wireless connection technology also improves the flexibility and portability of the device, making the smart mouse more suitable for scenarios such as mobile office and remote work.

[0003] The existing intelligent mouse uses gesture recognition to control the mouse, which does bring unprecedented convenience and innovation. However, it also has the problem of misrecognition, resulting in low control accuracy. Summary of the invention

[0004] The present invention provides a control method and system for an intelligent mouse, which solve the technical problems mentioned in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for controlling an intelligent mouse, the method comprising: Receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position; Determine a gesture monitoring area according to the gaze point position and user position information; The user's gesture is received in the gesture monitoring area, and the mouse operation instruction is further determined according to the gesture.

[0006] As a further technical solution of the present invention, the steps of receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position include: receiving an eye detection start signal input by a user, and waking up an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; The user's eye movement data is collected based on the eye tracking device, and the user's gaze direction is calculated using the gaze estimation algorithm. The gaze direction is combined with the coordinate system of the computer screen to calculate the user's gaze point position on the screen; The mouse pointer position is automatically moved according to the gaze point position so that the mouse pointer is at the gaze point position, and the residence time of the mouse pointer at the gaze point position is recorded in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

[0007] As a further technical solution of the present invention, the step of determining the gesture monitoring area according to the gaze point position and the user position information includes: Acquire position information of an eye tracking device as user position information, wherein the position information of the eye tracking device is position information relative to a computer screen; Get the mapping position of the user's location information on the computer screen (the user's location information is in the form of coordinates in three-dimensional space, and the mapping position is in the form of two-dimensional coordinates of the screen, specifically, the three-dimensional coordinates of the user relative to the computer screen are converted or mapped to the corresponding point in the two-dimensional coordinate system of the screen); Calculating the distance between the mapping position and the gaze point position; Determine whether the distance is less than a first preset threshold, and when the distance is less than the first preset threshold, construct a gesture monitoring area of ​​a preset size with the gaze point position as the center point; When the distance is not less than a first preset threshold, a gesture monitoring area of ​​a preset size is constructed with the mapping position as a center point.

[0008] As a further technical solution of the present invention, the step of receiving a user's gesture in the gesture monitoring area and further determining a mouse operation instruction according to the gesture includes: Capturing the user's gestures in the gesture monitoring area through a camera, or using other sensors (such as accelerometers and gyroscopes) to detect hand movements and postures; Classify gestures to determine gesture types, which include single-point gestures and multi-point gestures. Single-point gestures include click, long press, and slide, and multi-point gestures include pinch, expand, rotate, and zoom. Determine the mouse operation instruction according to the gesture type.

[0009] Another object of the present invention is to provide a control system for an intelligent mouse, the system comprising: A preliminary mouse position movement module is used to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine the user's gaze point position on the computer screen according to the eye tracking technology, and preliminarily automatically move the mouse pointer position according to the gaze point position; A gesture monitoring area determination module, used to determine the gesture monitoring area according to the gaze point position and user position information; The mouse operation instruction generating module is used to receive the user's gesture in the gesture monitoring area and further determine the mouse operation instruction according to the gesture.

[0010] As a further technical solution of the present invention, the mouse position preliminary movement module includes: A monitoring model awakening unit, configured to receive an eye detection start signal input by a user, and awaken an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; A gaze position determination unit, for collecting eye movement data of the user based on an eye tracking device, calculating the gaze direction of the user using a gaze estimation algorithm, combining the gaze direction with a coordinate system of a computer screen, and calculating the gaze point position of the user on the screen; The mouse preliminary movement unit is used to automatically move the mouse pointer position according to the gaze point position, so that the mouse pointer is at the gaze point position, and record the residence time of the mouse pointer at the gaze point position in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

[0011] As a further technical solution of the present invention, the gesture monitoring area determination module includes: A position information acquisition unit, used to acquire position information of the eye tracking device as user position information; A location mapping unit, used to obtain a mapping position of the user's location information on the computer screen; A distance calculation unit, used to calculate the distance between the mapping position and the gaze point position; a first distance determination unit, configured to determine whether the distance is less than a first preset threshold value, and when the distance is less than the first preset threshold value, constructing a gesture monitoring area of ​​a preset size with the gaze point position as the center point, wherein the gesture monitoring area is generally square or circular; The second distance determination unit is used to construct a gesture monitoring area of ​​a preset size with the mapping position as the center point when the distance is not less than a first preset threshold.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a control method and system for an intelligent mouse, in which the gaze point position and the mouse pointer position are preliminarily determined by starting the eye tracking technology, and on this basis, the gesture monitoring area is flexibly and accurately determined according to the gaze point position and the user position. The gesture monitoring area of ​​the present invention is conducive to improving the operation accuracy and effectively reducing the recognition error on the basis of facilitating the user operation, receiving the user's gesture in the gesture monitoring area, and further determining the mouse operation instruction according to the gesture, which can effectively reduce the operation amount. In the present invention, the eye tracking technology is combined with the gesture recognition technology to realize the precise control of the intelligent mouse, and the recognition error is effectively reduced on the basis of reducing the operation amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flowchart of a control method of an intelligent mouse.

[0014] Figure 2 The present invention is a flowchart of initially moving the position of the mouse pointer in a method for controlling an intelligent mouse.

[0015] Figure 3 The present invention is a flowchart of determining a gesture monitoring area in a control method of an intelligent mouse.

[0016] Figure 4 The structure block diagram of a control system of an intelligent mouse. DETAILED DESCRIPTION

[0017] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] like Figure 1 As shown, an embodiment of the present invention provides a method for controlling an intelligent mouse, the method comprising: Step S100, receiving an eye detection start signal input by a user (which may be a user's gesture action or a preset button operation), waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to an eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position, wherein the eye monitoring model includes an eye tracking device, and the eye monitoring model is used to determine the user's gaze point position on the computer screen; Step S200: determining a gesture monitoring area according to the gaze point position and user position information; Step S300: receiving a user's gesture in the gesture monitoring area, and further determining a mouse operation instruction according to the gesture.

[0019] In this embodiment, the gaze point position and the mouse pointer position are preliminarily determined by starting the eye tracking technology. On this basis, the gesture monitoring area is flexibly and accurately determined according to the gaze point position and the user position. The gesture monitoring area of ​​the present invention is conducive to improving the operation accuracy and effectively reducing the recognition error on the basis of facilitating the user operation. The user's gesture is received in the gesture monitoring area, and the mouse operation instruction is further determined according to the gesture, which can effectively reduce the operation amount. In the present invention, the eye tracking technology is combined with the gesture recognition technology to achieve precise control of the intelligent mouse, and the recognition error is effectively reduced on the basis of reducing the operation amount.

[0020] See also Figure 2 The steps of receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position include: Step S101, receiving an eye detection start signal input by a user (which may be a user's gesture action or a preset button operation), and waking up an eye monitoring model according to the eye detection start signal, the eye monitoring model including an eye tracking device, which is generally worn on the user's face; Step S102: Collect the user's eye movement data based on the eye tracking device, calculate the user's line of sight direction using the line of sight estimation algorithm, combine the line of sight direction with the coordinate system of the computer screen, and calculate the user's gaze point position on the screen, which is in the form of two-dimensional coordinates; wherein, the line of sight estimation algorithm is the core of the eye tracking technology. It uses the eye movement data collected by the eye tracking device and calculates the user's line of sight direction through complex algorithm processing. This process usually involves multiple steps such as eyeball structure model, geometric feature extraction, feature learning to line of sight direction mapping; eyeball structure model: in some cases, the algorithm will use parameter information such as the geometric shape of the eyeball and the direction of the visual axis to assist in calculating the line of sight direction. Geometric feature extraction: the algorithm will extract eye appearance information or geometric features from the human eye image, such as pupil position, corneal reflection, etc. Feature learning to line of sight direction mapping: based on technologies such as deep learning, the algorithm will learn the mapping relationship between these features and the line of sight direction of the human eye, thereby completing the estimation of the line of sight direction. After obtaining the user's line of sight direction, the next step is to combine this direction with the coordinate system of the computer screen to determine the user's gaze point position on the screen. This process usually involves the following steps: Screen calibration: First, the screen needs to be calibrated to determine the position and direction of the screen in three-dimensional space. Calculation of the intersection of the line of sight and the screen: In the three-dimensional coordinate system, the intersection of the line of sight and the screen is calculated based on the user's line of sight and the position information of the screen. This intersection is the position of the user's gaze point on the screen.

[0021] Step S103, preliminarily and automatically moving the mouse pointer position according to the gaze point position, so that the mouse pointer is at the gaze point position, and recording the residence time of the mouse pointer at the gaze point position in real time. When the residence time exceeds a preset time threshold, turning off the eye tracking device, so that the eye monitoring model is in a sleep state.

[0022] In this embodiment, eye recognition is used to reduce the amount of user gestures. For example, when moving from the right side of the screen to the left side, the hand needs to swing greatly at the beginning. First, the eyes are used to locate a gesture monitoring area, and then when the hand is waved again, only a little bit needs to be swung, which is more conducive to improving the operation accuracy.

[0023] See also Figure 3 The step of determining the gesture monitoring area according to the gaze point position and the user position information includes: Step S201, obtaining position information of an eye tracking device as user position information, wherein the position information of the eye tracking device is position information relative to a computer screen; Step S202, obtaining the mapping position of the user position information on the computer screen (the user position information is in the form of coordinates in three-dimensional space, and the mapping position is in the form of two-dimensional coordinates of the screen, specifically referring to the corresponding point where the three-dimensional coordinates of the user relative to the computer screen are converted or mapped to the two-dimensional coordinate system of the screen); Step S203, calculating the distance between the mapping position and the gaze point position; Step S204, determining whether the distance is less than a first preset threshold value, when the distance is less than the first preset threshold value, constructing a gesture monitoring area of ​​a preset size with the gaze point position as the center point, wherein the gesture monitoring area is generally square or circular; the preset size is set according to actual application; Step S205: When the distance is not less than a first preset threshold, a gesture monitoring area of ​​a preset size is constructed with the mapping position as a center point.

[0024] In this embodiment, when the distance between the mapping position and the gaze point position is less than a first preset threshold, a gesture monitoring area of ​​a preset size is constructed with the gaze point position as the center point, so as to improve the operation accuracy. When the distance between the mapping position and the gaze point position is not less than the first preset threshold, it means that the user is far away from the gaze point position. At this time, a gesture monitoring area that is easy to operate can be generated without the user moving, which is convenient for the user. In the present invention, by setting the gesture monitoring area, it is possible to avoid recognizing other people's gestures to a certain extent, which is conducive to reducing recognition errors.

[0025] In this embodiment, the step of receiving a user's gesture within the gesture monitoring area and further determining a mouse operation instruction according to the gesture includes: Capturing the user's gestures in the gesture monitoring area through a camera, or using other sensors (such as accelerometers and gyroscopes) to detect hand movements and postures; Classify gestures to determine gesture types, which include single-point gestures and multi-point gestures. Single-point gestures include click, long press, and slide, and multi-point gestures include pinch, expand, rotate, and zoom. Determine the mouse operation instruction according to the gesture type.

[0026] Among them, click: the user touches the screen with a finger or makes a click action in the air to simulate a mouse click operation. This is usually used to select an object, open a link, or perform other basic interactions. Long press: the user presses the screen or maintains a gesture for a period of time to simulate a long press operation of the mouse. This can be used to activate a specific function or start a drag operation. Slide: the user slides the finger on the screen or makes a sliding action in the air to simulate the movement or scrolling of the mouse. The sliding speed and direction can affect the movement speed or scrolling direction of the mouse pointer. Pinch: the user brings two fingers close together and pinches them together to simulate a double-click operation of the mouse. This is usually used to quickly turn on or off a function, zoom in or out of the view, etc. Expand: contrary to pinching, the user separates the two fingers from the close state, which may be used to restore the view to the original size or perform other operations opposite to zooming in / out. Rotate: the user rotates the screen with two fingers to simulate certain specific rotation operations, such as rotating an image or adjusting the viewing angle. Zoom: the user simulates zooming operations by moving two fingers close together or apart, such as zooming in or out of an image or map, etc.

[0027] See also Figure 4 Another object of the present invention is to provide a control system for an intelligent mouse, the system comprising: The mouse position preliminary movement module 100 is used to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine the user's gaze point position on the computer screen according to the eye tracking technology, and preliminarily automatically move the mouse pointer position according to the gaze point position; A gesture monitoring area determination module 200, configured to determine a gesture monitoring area according to the gaze point position and user position information; The mouse operation instruction generating module 300 is used to receive a user's gesture in the gesture monitoring area, and further determine a mouse operation instruction according to the gesture.

[0028] In this embodiment, the mouse position preliminary moving module 100 includes: A monitoring model awakening unit is used to receive an eye detection start signal input by a user (which may be a user's gesture action or a preset button operation), and wake up the eye monitoring model according to the eye detection start signal. The eye monitoring model includes an eye tracking device, which is generally worn on the user's face. A gaze position determination unit, for collecting eye movement data of the user based on an eye tracking device, calculating the gaze direction of the user using a gaze estimation algorithm, combining the gaze direction with a coordinate system of a computer screen, and calculating the gaze point position of the user on the screen, wherein the gaze point position is in a two-dimensional coordinate form; The mouse preliminary movement unit is used to automatically move the mouse pointer position according to the gaze point position, so that the mouse pointer is at the gaze point position, and record the residence time of the mouse pointer at the gaze point position in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

[0029] In this embodiment, the gesture monitoring area determination module 200 includes: A position information acquisition unit, used to acquire position information of an eye tracking device as user position information, wherein the position information of the eye tracking device is position information relative to a computer screen; A position mapping unit, used to obtain the mapping position of the user position information on the computer screen (the user position information is in the form of coordinates in three-dimensional space, and the mapping position is in the form of two-dimensional coordinates of the screen, specifically, the three-dimensional coordinates of the user relative to the computer screen are converted or mapped to the corresponding points in the two-dimensional coordinate system of the screen); A distance calculation unit, used to calculate the distance between the mapping position and the gaze point position; a first distance determination unit, configured to determine whether the distance is less than a first preset threshold value, and when the distance is less than the first preset threshold value, constructing a gesture monitoring area of ​​a preset size with the gaze point position as the center point, wherein the gesture monitoring area is generally square or circular; The second distance determination unit is used to construct a gesture monitoring area of ​​a preset size with the mapping position as the center point when the distance is not less than a first preset threshold.

[0030] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0031] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for controlling an intelligent mouse, characterized in that: The method comprises: Receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position; Determine a gesture monitoring area according to the gaze point position and user position information; The user's gesture is received in the gesture monitoring area, and the mouse operation instruction is further determined according to the gesture.

2. The control method of an intelligent mouse according to claim 1, characterized in that: The steps of receiving an eye detection start signal input by a user, waking up an eye monitoring model according to the eye detection start signal, determining the user's gaze point position on the computer screen according to the eye tracking technology, and automatically moving the mouse pointer position according to the gaze point position include: receiving an eye detection start signal input by a user, and waking up an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; The user's eye movement data is collected based on the eye tracking device, and the user's gaze direction is calculated using the gaze estimation algorithm. The gaze direction is combined with the coordinate system of the computer screen to calculate the user's gaze point position on the screen; The mouse pointer position is automatically moved according to the gaze point position so that the mouse pointer is at the gaze point position, and the residence time of the mouse pointer at the gaze point position is recorded in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

3. The control method of an intelligent mouse according to claim 2, characterized in that: The step of determining the gesture monitoring area according to the gaze point position and the user position information comprises: Obtaining location information of the eye tracking device as user location information; Get the mapping position of the user's location information on the computer screen; Calculating the distance between the mapping position and the gaze point position; Determine whether the distance is less than a first preset threshold, and when the distance is less than the first preset threshold, construct a gesture monitoring area of ​​a preset size with the gaze point position as the center point; When the distance is not less than a first preset threshold, a gesture monitoring area of ​​a preset size is constructed with the mapping position as a center point.

4. The control method of an intelligent mouse according to claim 1, characterized in that: The step of receiving a user's gesture within the gesture monitoring area and further determining a mouse operation instruction according to the gesture comprises: Capturing the user's gestures in the gesture monitoring area through a camera; Classify gestures to determine gesture types, which include single-point gestures and multi-point gestures. Single-point gestures include click, long press, and slide, and multi-point gestures include pinch, expand, rotate, and zoom. Determine the mouse operation instruction according to the gesture type.

5. A control system for an intelligent mouse, characterized in that: The system comprises: A preliminary mouse position movement module is used to receive an eye detection start signal input by a user, wake up an eye monitoring model according to the eye detection start signal, determine the user's gaze point position on the computer screen according to the eye tracking technology, and preliminarily automatically move the mouse pointer position according to the gaze point position; A gesture monitoring area determination module, used to determine the gesture monitoring area according to the gaze point position and user position information; The mouse operation instruction generating module is used to receive the user's gesture in the gesture monitoring area and further determine the mouse operation instruction according to the gesture.

6. The control system of an intelligent mouse according to claim 5, characterized in that: The mouse position preliminary movement module includes: A monitoring model awakening unit, configured to receive an eye detection start signal input by a user, and awaken an eye monitoring model according to the eye detection start signal, wherein the eye monitoring model includes an eye tracking device; A gaze position determination unit, for collecting eye movement data of the user based on an eye tracking device, calculating the gaze direction of the user using a gaze estimation algorithm, combining the gaze direction with a coordinate system of a computer screen, and calculating the gaze point position of the user on the screen; The mouse preliminary movement unit is used to automatically move the mouse pointer position according to the gaze point position, so that the mouse pointer is at the gaze point position, and record the residence time of the mouse pointer at the gaze point position in real time. When the residence time exceeds a preset time threshold, the eye tracking device is turned off, so that the eye monitoring model is in a sleep state.

7. The control system of an intelligent mouse according to claim 6, characterized in that: The gesture monitoring area determination module includes: A position information acquisition unit, used to acquire position information of the eye tracking device as user position information; A location mapping unit, used to obtain a mapping position of the user's location information on the computer screen; A distance calculation unit, used to calculate the distance between the mapping position and the gaze point position; a first distance determination unit, configured to determine whether the distance is less than a first preset threshold, and when the distance is less than the first preset threshold, construct a gesture monitoring area of ​​a preset size with the gaze point position as the center point; The second distance determination unit is used to construct a gesture monitoring area of ​​a preset size with the mapping position as the center point when the distance is not less than a first preset threshold.

Images