Mouse control method and device, mouse and mouse system

Abstract

The invention relates to the technical field of man-machine interaction, and discloses a mouse control method and device, a mouse and a mouse system. The control method comprises the steps that data information corresponding to hand movement is obtained, and the data information comprises image information; performing preset processing on the image information to obtain a gesture recognition result; mapping the gesture recognition result into a corresponding mouse event, and sending a control instruction corresponding to the mouse event to the controlled object. According to the mouse control method, the problems of inaccurate gesture recognition and slow response in the prior art are solved through recognition from the original image to the gesture and mapping from the recognition to the control instruction, so that the mouse control method can respond to the hand action of the user more accurately and sensitively.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to mouse control methods, devices, mice, and mouse systems. Background Technology

[0002] Traditional computer mice primarily operate through physical buttons and scroll wheels. Prolonged use can easily lead to finger and wrist fatigue, and even occupational diseases such as carpal tunnel syndrome. Furthermore, the mechanical structure of a mouse is subject to wear and tear and has a limited lifespan. Summary of the Invention

[0003] In view of this, the present invention provides a mouse control method, device, mouse and mouse system to solve the problems of user hand fatigue and poor mouse response speed.

[0004] In a first aspect, the present invention provides a mouse control method, comprising: acquiring data information corresponding to hand movements, wherein the data information includes image information; performing preset processing on the image information to obtain a gesture recognition result; mapping the gesture recognition result to a corresponding mouse event and sending the control command corresponding to the mouse event to the controlled object.

[0005] Beneficial effects: By acquiring image information and recognizing gestures without contact, the problems of user fatigue, mechanical wear and tear and dependence on support surface caused by long-term use of traditional mice are solved.

[0006] In an optional embodiment, the present invention further proposes that the image information includes an image sequence, and that the image information is subjected to preset processing to obtain a gesture recognition result, including: performing image segmentation and feature extraction on each image in the image sequence to obtain the corresponding position information of a preset part of the hand in three-dimensional space; calculating the movement trajectory and acceleration information of the preset part based on the sequence relationship of the image information and the corresponding position information of the preset part in three-dimensional space; and determining the gesture recognition result based on the movement trajectory and acceleration information.

[0007] Beneficial effects: It realizes the conversion from raw images to precise gestures, and solves the defects of inaccurate recognition and slow response in the existing technology, so that the mouse control method can respond to the user's hand movements more accurately and sensitively.

[0008] In one optional embodiment, the present invention further proposes determining the gesture recognition result based on the movement trajectory and acceleration information, including: determining a first gesture result based on the movement of a preset point of the palm outline; determining a second gesture result based on the index fingertip moving towards the mouse and the acceleration exceeding a first threshold; determining a third gesture result based on the index fingertip repeatedly moving towards the mouse twice and the acceleration exceeding a second threshold; determining a fourth gesture result based on the middle fingertip moving towards the mouse and the acceleration exceeding a first threshold; determining a fifth gesture result based on the index finger outline sliding upward; and determining a sixth gesture result based on the index finger outline sliding downward.

[0009] Beneficial effects: The introduction of gesture recognition rules enables the system to efficiently distinguish between various gesture types, improving the accuracy and reliability of gesture recognition and providing users with a more precise, smooth, and low-latency non-press-based mouse operation experience.

[0010] In one optional implementation, the present invention also proposes mapping the gesture recognition result to a corresponding mouse event, including: mapping a first gesture result to mouse movement; mapping a second gesture result to clicking the left mouse button; mapping a third gesture result to double-clicking the left mouse button; mapping a fourth gesture result to clicking the right mouse button; mapping a fifth gesture result to scrolling the mouse wheel upwards; and mapping a sixth gesture result to scrolling the mouse wheel downwards.

[0011] Beneficial effects: The mapping mechanism is based on the precise calculation of the three-dimensional spatial position, movement trajectory and acceleration information of the preset part of the hand, which ensures that each gesture intention can be accurately captured and converted into the expected mouse operation, thus improving the smoothness and accuracy of the non-press mouse.

[0012] In one optional embodiment, the present invention further proposes that acquiring data information corresponding to hand movements includes: acquiring acceleration measurement information measured by an accelerometer; capturing image information of the hand, the image information including a positioning component worn on the user's hand; and receiving rotation measurement information fed back by a gyroscope, the gyroscope being mounted on the positioning component.

[0013] Beneficial effects: The synergistic effect of multi-source data can more comprehensively and accurately calculate the movement trajectory and acceleration information of preset parts of the hand, thereby improving the reliability, response speed and overall accuracy of gesture recognition, and providing users with a smoother and more accurate non-press-based mouse operation experience.

[0014] In an optional embodiment, the present invention further proposes that, after calculating the movement trajectory and acceleration information of the preset part based on the sequence relationship of image information and the corresponding position information of the preset part in three-dimensional space, the invention further includes: performing data calibration on the movement trajectory and acceleration information of the preset part based on acceleration measurement information and rotation measurement information, respectively.

[0015] Beneficial effects: By calibrating the movement trajectory and acceleration information separately, motion parameters can be optimized in a targeted manner, enabling effective fusion of multi-source sensor data, improving the accuracy and stability of gesture recognition, and reducing the false trigger rate.

[0016] Secondly, the present invention also provides a mouse control device, comprising: an information acquisition unit for acquiring data information corresponding to hand movements, wherein the data information includes image information; a gesture recognition unit for performing preset processing on the image information to obtain a gesture recognition result; and a control unit for mapping the gesture recognition result to a corresponding mouse event and sending the control command corresponding to the mouse event to the controlled object.

[0017] Beneficial effects: By acquiring image information and recognizing gestures without contact, the problems of user fatigue, mechanical wear and tear and dependence on support surface caused by long-term use of traditional mice are solved.

[0018] Thirdly, the present invention also provides a mouse, comprising: a data acquisition module, a data processing module, a wireless communication module, a power management module, and a housing assembly. The data acquisition module is used to acquire data information corresponding to hand movements; the data processing module is used to run the method provided above; the wireless communication module is used to send control commands corresponding to mouse events to the controlled object; the power management module is used to provide power consumption management for the mouse; and the housing assembly is used to fix the data acquisition module, the data processing module, the wireless communication module, and the power management module.

[0019] Beneficial effects: Effectively solves technical problems such as user fatigue, mechanical wear, high recognition delay, high false trigger rate, and poor environmental adaptability.

[0020] In one optional embodiment, the present invention also proposes that the data acquisition module includes a camera, an infrared fill light, and an accelerometer. The infrared fill light is used to provide supplementary lighting for the camera's shooting environment; the camera is used to acquire image information corresponding to hand movements; and the accelerometer is used to acquire acceleration measurement information of the hand.

[0021] Beneficial effects: The fusion of multi-source data enables more accurate calculation of the position, movement trajectory, and acceleration information of the preset hand parts in three-dimensional space, significantly improving the accuracy and robustness of gesture recognition and reducing the false recognition rate and latency.

[0022] Fourthly, the present invention also provides a mouse system, including the above-described mouse and a positioning component, wherein the positioning component is worn on the user's hand and includes a gyroscope.

[0023] Beneficial effects: By fusing rotational information with positional data, dynamic correction of movement trajectories is achieved, improving the accuracy and response speed of gesture recognition. This avoids the reliance of traditional mice on physical buttons and desktop support surfaces, reducing the mechanical load on fingers and wrists. It provides users with a more comfortable, precise, and adaptable operating experience, effectively overcoming the limitations of existing non-press-based mice in displacement recognition, click mapping, and environmental adaptability. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the first process of a mouse control method according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the second process of a mouse control method according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of a mouse control device according to an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the structure of an electronic mouse according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Computer mice in related technologies can easily cause finger and wrist fatigue with prolonged use, potentially leading to occupational diseases such as carpal tunnel syndrome. Furthermore, their mechanical structures are subject to wear and tear, have limited lifespan, and require regular cleaning. Operation also relies on a support surface like a desktop, making them inconvenient to use in environments without a flat surface. Some non-press-based mouse solutions in related technologies commonly suffer from limitations such as supporting only basic displacement recognition, lacking precise mapping of click and scroll wheel equivalents, high gesture recognition latency, high false trigger rates, and susceptibility to ambient light. These limitations make it difficult to provide a non-press-based mouse solution that offers precise operation, low latency, and a smooth user experience.

[0031] To address this issue, this invention proposes a mouse control method. This method acquires data information corresponding to hand movements (including image information), performs pre-processing on the image information to obtain gesture recognition results, maps the gesture recognition results to corresponding mouse events, and then sends the control commands corresponding to the mouse events to the controlled object. This method overcomes the physical limitations of traditional mice and the shortcomings of existing non-press-based solutions in recognition, providing a non-contact, high-precision, and low-latency mouse operation experience. For ease of understanding, some key terms in this embodiment are explained below: Hand movement data refers to various data that reflect the dynamic characteristics of a user's hand in space, such as posture, position, and movement. This data can be collected by cameras and sensors and serves as the basic input for subsequent gesture recognition.

[0032] Image information: Specifically refers to two-dimensional or three-dimensional image data that contains visual features of a user's hand, acquired through optical imaging devices (such as cameras). Image information is the primary data source for gesture recognition.

[0033] Pre-defined processing refers to a series of predefined algorithms and operations performed on the raw image information. These processes are used to extract effective features related to gesture recognition from complex image data, such as hand contours and key point locations.

[0034] Gesture recognition result: This refers to the specific gesture type determined by the system based on the user's hand movements after pre-processing. For example, the recognition result could be "palm open" or "index finger click," and these results will be directly associated with specific mouse functions.

[0035] Mouse events: These refer to standard mouse actions defined in a computer operating system or application. Examples of mouse events include "mouse movement," "left click," "right click," and "scroll wheel movement."

[0036] Control commands: These refer to the electronic signals or data packets that map gesture recognition results to mouse events and send them to the controlled object to execute the corresponding mouse events. These commands are the final output for implementing contactless mouse control.

[0037] Controlled object: refers to the electronic device or system that receives and responds to control commands. Typically, the controlled object is a computer, smart TV, or other terminal device that requires mouse operation.

[0038] This embodiment provides a mouse control method, such as... Figure 1 As shown, the specific implementation method may include the following steps: Step S101: Obtain data information corresponding to hand movements, wherein the data information includes image information.

[0039] In one example, a regular visible light camera can continuously capture a video stream of the user's hand, using each frame as image information. This camera can be fixed inside the mouse. Through continuous image frames, dynamic changes in the hand can be captured. In another example, a basic infrared camera can be used to acquire images of the hand under specific lighting conditions to reduce the impact of ambient light variations on image quality.

[0040] Step S102: Perform preset processing on the image information to obtain the gesture recognition result.

[0041] Specifically, basic image processing operations can be performed on the acquired image information. For example, the hand can be separated from the background using a background subtraction algorithm to obtain a binarized image of the hand. Subsequently, contour extraction can be performed on this binarized image to obtain the two-dimensional contour information of the hand. By analyzing the shape changes of this two-dimensional contour, such as the overall size of the palm or the extension state of the fingers, a preset gesture can be recognized.

[0042] Step S103: Map the gesture recognition result to the corresponding mouse event and send the control command corresponding to the mouse event to the controlled object.

[0043] In one example, when an open hand gesture is detected, it can be mapped to a mouse movement event. The movement of the mouse cursor is controlled by the change in the two-dimensional position of the hand in the image. Subsequently, these mouse events are converted into corresponding mouse operations and sent to the controlled object via a wired connection or a basic wireless communication module, thereby realizing the control of the mouse function of the controlled object.

[0044] It is understandable that the mouse control method in this implementation solves the problems of user fatigue, mechanical wear, and dependence on a support surface caused by long-term use of traditional mice through non-contact image information acquisition and gesture recognition. At the same time, this method avoids the limitations of existing non-press-type solutions, such as misoperation caused by physical contact and susceptibility to ambient light, providing a convenient, hygienic, and spatially flexible mouse control method.

[0045] In some embodiments, the present invention further proposes a mouse control method, such as... Figure 2 As shown, it includes: Step S201: Obtain data information corresponding to hand movements, including image information. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0046] Step S202: The image information includes an image sequence. Pre-processing is performed on the image information to obtain the gesture recognition result, specifically including: Step S2011: Perform image segmentation and feature extraction on each image in the image sequence to obtain the corresponding position information of the preset part of the hand in three-dimensional space.

[0047] Specifically, image information includes image sequences, which refer to a series of image frames arranged in chronological order, acquired through continuous shooting or acquisition. Image sequences can completely record the continuous movement of the hand over a period of time, providing the necessary temporal information for the recognition of dynamic gestures. For example, a high-speed camera can continuously capture hand images at a fixed frame rate (such as 30 or 60 frames per second) to form an image stream; or, multiple synchronized cameras can be used to simultaneously acquire images from different perspectives to construct an image sequence containing depth information.

[0048] Image segmentation and feature extraction are performed on each image in the image sequence to accurately separate the hand region from a complex background and identify key points or regions on the hand. Image segmentation can be implemented by using models, such as semantic segmentation networks based on convolutional neural networks, to perform pixel-level segmentation of the hand region, accurately separating the hand from the background; or, when the image contains depth information, the hand region can be identified and extracted based on depth thresholding or color space analysis (such as skin color detection). Feature extraction can be implemented by using deep learning-based hand keypoint detection models to identify specific key points on the hand, such as fingertips, knuckles, and the center of the palm, where these key points are preset locations; or, geometric features of the hand contour (such as shape descriptors) or texture features of the region can be extracted to describe the shape of the hand.

[0049] Step S2022: Calculate the movement trajectory and acceleration information of the preset part based on the sequence relationship of the image information and the corresponding position information of the preset part in three-dimensional space.

[0050] Specifically, obtaining the corresponding position information of a preset part of the hand in three-dimensional space refers to acquiring the precise position data (e.g., X, Y, Z coordinates) of key points or regions of the hand in a three-dimensional coordinate system. This can be achieved by: using a binocular or multi-view vision system and leveraging the principle of triangulation to calculate the three-dimensional spatial coordinates of the preset part based on the two-dimensional image coordinates of the same preset part from different viewpoints; or by directly acquiring a depth map of the scene using a depth camera, and combining this with image segmentation results to directly extract the 3D coordinates of the preset part of the hand.

[0051] Based on the sequential relationship of image information and the corresponding position information of the preset part in three-dimensional space, the movement trajectory and acceleration information of the preset part are calculated to quantify the dynamic characteristics of hand movement. The movement trajectory is calculated by connecting the three-dimensional position points of the same preset part in consecutive frames in temporal order to form its movement path in three-dimensional space. The acceleration information is calculated by performing differential calculation on the velocity information of the preset part in consecutive frames, where the velocity information can be obtained from the temporal difference of consecutive position points; or by curve fitting the movement trajectory of the preset part and then obtaining smooth acceleration information by calculating the second derivative of the fitted curve.

[0052] Step S2023, and determine the gesture recognition result based on the movement trajectory and acceleration information.

[0053] Specifically, determining the gesture recognition result based on movement trajectory and acceleration information means matching quantized hand dynamic parameters with predefined gesture patterns to identify the specific gesture. Implementation methods may include pre-setting a series of gesture rules, such as "when the index fingertip moves more than a preset distance in a specific direction and the average acceleration is greater than a certain threshold, it is recognized as gesture A," and then matching the calculated movement trajectory and acceleration information with these rules to determine the gesture recognition result.

[0054] Step S203: Map the gesture recognition result to the corresponding mouse event and send the control command corresponding to the mouse event to the controlled object. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0055] It is understood that the technical solution in this embodiment effectively solves the accuracy and latency problems in gesture recognition by refining the image processing flow. Specifically, image information is acquired as an image sequence, which provides a basis for capturing continuous hand movements, avoiding the static limitations of a single image and ensuring that dynamic gestures can be completely recorded. Image segmentation and feature extraction are performed on each image in the image sequence, which can separate the hand region from complex backgrounds and identify key parts such as fingers or palms, thereby obtaining the corresponding position information of the preset part of the hand in three-dimensional space, eliminating the influence of ambient light or interference, and improving the accuracy of positioning. Based on the sequence relationship of image information and the corresponding position information of the preset part in three-dimensional space, the movement trajectory and acceleration information of the preset part are calculated. Time series analysis is used to analyze the continuous changes of hand movements, ensuring that the calculation of trajectory and acceleration can reflect the real motion state and reduce recognition latency. The gesture recognition result is determined based on the movement trajectory and acceleration information, which is directly based on physical motion parameters, improving the reliability of mapping. Overall, it achieves the transformation from raw images to precise gestures, solving the defects of inaccurate recognition and slow response in existing technologies, thus enabling the mouse control method to respond to the user's hand movements more accurately and sensitively.

[0056] In some embodiments, the present invention further proposes determining the gesture recognition result based on movement trajectory and acceleration information, including: Step a1: Determine the first gesture result based on the movement of preset points on the palm outline.

[0057] Specifically, gestures are identified by monitoring the continuous displacement of pre-defined key points on the palm (such as the center of the palm, wrist joints, or specific finger joints) in three-dimensional space. These pre-defined points can be obtained from image information using image processing techniques (such as skeleton extraction and key point detection algorithms). When the movement trajectory of these pre-defined points conforms to a pre-defined translation pattern, it is determined as the first gesture result, typically used to simulate mouse movement. In one example, by analyzing the positional changes of the center of gravity or geometric center of the overall palm contour within consecutive time frames, when the displacement and direction meet specific conditions, it is determined as the first gesture result.

[0058] Step a2: Determine the second gesture result based on the index fingertip moving towards the mouse and the acceleration exceeding the first threshold.

[0059] Specifically, the system first identifies the position of the index fingertip in three-dimensional space and determines whether it is moving towards a preset virtual mouse position or area. Index fingertip recognition can be achieved through image segmentation and fingertip detection algorithms. The system calculates the acceleration information of the index fingertip during its movement. When the index fingertip moves towards the mouse and its acceleration value exceeds a preset first threshold, it is determined as a second gesture result, which typically maps to a mouse click operation. The system is configured to determine this gesture when the index fingertip moves rapidly towards the target area within a short period of time and its instantaneous acceleration reaches a certain value or higher.

[0060] Step a3: Determine the third gesture result based on the fact that the index fingertip moves towards the mouse twice repeatedly and the acceleration exceeds the second threshold.

[0061] Specifically, based on the second gesture recognition, the system further identifies two rapid movements of the index fingertip towards the mouse within a short period. The acceleration of each movement must exceed a preset second threshold, and the time interval between the two movements must be within a preset range. The second threshold is typically set higher than the first threshold to ensure the force and intent of the double-click action. For example, the system detects two independent, rapid index fingertip movements pointing towards the mouse and determines whether the interval between the two movements matches the characteristics of a double-click, thus identifying it as the third gesture result.

[0062] Step a4: Determine the fourth gesture result based on the middle fingertip moving towards the mouse and the acceleration exceeding the first threshold.

[0063] Specifically, the recognition method for the second gesture is similar, but the recognition object changes to the fingertip of the middle finger. The system identifies the position of the middle fingertip in three-dimensional space and determines whether it is moving towards a preset virtual mouse position or area. At the same time, it calculates the acceleration information of the middle fingertip during the movement. When its movement direction is towards the mouse and the acceleration value exceeds a preset first threshold, it is determined to be the result of the fourth gesture.

[0064] Step a5: Slide the index finger upwards to determine the result of the fifth gesture.

[0065] Specifically, the system analyzes the vertical displacement of the index finger's outline or key points. It identifies the overall outline of the index finger or its major joints (such as the base, middle, and tip) in three-dimensional space. When the index finger's outline or key points maintain a relatively stable horizontal position while undergoing upward displacement, this is identified as the fifth gesture result, and this gesture is mapped to scrolling the mouse wheel upwards. Alternatively, the system determines this sliding operation by detecting the continuous, smooth upward trajectory of the index fingertip in the vertical direction and combining this with its speed.

[0066] Step a6: Determine the result of the sixth gesture by sliding down according to the outline of the index finger.

[0067] Specifically, similar to the recognition method for the fifth gesture, but in the opposite direction, the system identifies the overall outline of the index finger or the positional changes of its major joints in three-dimensional space. When the outline of the index finger or its key points maintains a relatively stable horizontal position while undergoing downward displacement, it is determined to be the result of the sixth gesture, which is typically mapped to scrolling the mouse wheel downwards. Alternatively, this swipe operation can be determined by detecting the continuous, smooth downward trajectory of the index fingertip in the vertical direction, combined with its movement speed.

[0068] It is understood that this invention solves the problems of insufficient gesture recognition accuracy and high false trigger rate in existing technologies by defining a series of specific and quantifiable gesture recognition rules for different mouse operations. Specifically, by recognizing the movement of preset points on the palm outline, it can accurately capture the user's intended mouse movement, avoiding misjudgments caused by ambient light or slight jitter. Single-click and double-click operations are distinguished by the direction of movement of the index fingertip towards the mouse and the acceleration threshold, allowing users to achieve precise clicks with quick and forceful movements. Different acceleration thresholds effectively differentiate between single-click and double-click operations, significantly reducing the false trigger rate. Right-click is recognized by the movement of the middle fingertip, and scrolling of the scroll wheel is intuitively realized by recognizing the upward or downward swiping of the index finger outline, allowing users to browse pages in a natural and smooth manner. The introduction of these rules enables the system to efficiently distinguish multiple gesture types, improving the accuracy and reliability of gesture recognition and providing users with a more precise, smooth, and low-latency non-press-based mouse operation experience.

[0069] In some embodiments, the present invention further proposes mapping the above gesture recognition results to corresponding mouse events, specifically including: Step b1: Map the first gesture result to mouse movement.

[0070] Specifically, the first gesture result is mapped to mouse movement to associate the system-recognized first gesture result with the mouse movement function. The first gesture result is determined based on the movement of preset points on the palm outline, representing the user's intention to move the mouse cursor. By directly converting the two-dimensional or three-dimensional movement vector of the preset points on the palm in the first gesture result into the corresponding movement vector of the mouse cursor on the screen, or by setting a virtual mouse movement area, when the first gesture result instructs the palm to move within this area, the relative displacement of the gesture is converted into the relative displacement of the mouse cursor, thereby achieving precise cursor control.

[0071] Step b2: Map the second gesture result to a left mouse button click.

[0072] Specifically, the second gesture result is mapped to a left mouse button click, converting the system-recognized second gesture result into a left mouse button click operation. The second gesture result is determined based on the characteristic of the index fingertip moving towards the mouse with acceleration exceeding a first threshold, simulating the user's left-click action. When the system detects that the condition of the second gesture result is met, it immediately sends a simulated left mouse button press and release event to the operating system, thus completing a click operation. Alternatively, after detecting the second gesture result, a brief key press signal is simulated by software. The duration of this signal can be adjusted according to user habits or system settings to ensure the validity of the click operation.

[0073] Step b3: Map the third gesture result to a double-click of the left mouse button.

[0074] Specifically, the third gesture result is mapped to a double-click of the left mouse button, converting the system-recognized third gesture result into a double-click operation of the left mouse button. The third gesture result is determined based on the characteristic of the index fingertip repeatedly moving towards the mouse twice with acceleration exceeding a second threshold, simulating the user's left-click action. When the system detects that the conditions of the third gesture result are met, it can send two simulated left-click events to the operating system consecutively, with the time interval between the two clicks conforming to the operating system's double-click recognition criteria. Alternatively, a specific double-click event signal can be sent to the operating system, which directly triggers the double-click function without simulating two independent click events, improving response speed and accuracy.

[0075] Step b4: Map the fourth gesture result to a right-click.

[0076] Specifically, the fourth gesture result is mapped to a right mouse button click, converting the system-recognized fourth gesture result into a right mouse button click operation. The fourth gesture result is determined based on the characteristic of the middle fingertip moving towards the mouse with acceleration exceeding a first threshold, simulating the user's right-click action. When the system detects that the conditions of the fourth gesture result are met, it can send a simulated right-click press and release event to the operating system, thus completing a right-click operation. Alternatively, after detecting the fourth gesture result, a right-click button signal can be simulated by software.

[0077] Step b5: Map the result of the fifth gesture to the mouse wheel scrolling upwards.

[0078] Specifically, the fifth gesture result is mapped to mouse wheel scrolling upwards, converting the system-recognized fifth gesture result into a mouse wheel scrolling upwards operation. The fifth gesture result is determined based on the characteristics of the index finger's upward swiping outline, simulating the user's action of scrolling the mouse wheel upwards. When the system detects that the conditions of the fifth gesture result are met, it sends a simulated mouse wheel scrolling upwards event to the operating system, determining the scrolling amplitude based on the speed or distance of the gesture swiping. Alternatively, it continuously detects the upward swiping of the index finger's outline and sends mouse wheel scrolling upwards events to the operating system at a certain frequency and step size based on the duration or cumulative distance of the swiping, until the gesture stops.

[0079] Step b6, and mapping the result of the sixth gesture to scroll the mouse wheel down.

[0080] Specifically, the sixth gesture result is mapped to mouse wheel scrolling downwards, converting the system-recognized sixth gesture result into a mouse wheel scrolling operation. The sixth gesture result is determined based on the characteristics of the index finger's downward sliding outline, simulating the user's action of scrolling the mouse wheel downwards. When the system detects that the conditions of the sixth gesture result are met, it sends a simulated mouse wheel scrolling downwards event to the operating system. The scrolling amplitude can be adjusted according to the speed or distance of the gesture. Alternatively, by continuously detecting the downward sliding of the index finger's outline, and based on the duration or cumulative distance of the sliding, the system sends mouse wheel scrolling downwards events to the operating system at a certain frequency and step size until the gesture stops.

[0081] It is understood that the technical solution in this embodiment, by clearly defining specific mapping rules, accurately converts different gesture recognition results into corresponding mouse events, solving the problems of accidental triggering and inaccurate operation that may occur in gesture control. Mapping the first gesture result to mouse movement, based on the movement characteristics of preset points on the palm contour, achieves natural control of the cursor position, avoiding accidental actions caused by ambiguous gestures. Mapping the second gesture result to clicking the left mouse button, utilizing the characteristic of the index fingertip moving towards the mouse with acceleration exceeding a first threshold, ensures reliable recognition of the click action and reduces misoperation. Mapping the third gesture result to double-clicking the left mouse button, based on the characteristic of the index fingertip moving twice repeatedly with acceleration exceeding a second threshold, simulates double-click behavior, improving operational efficiency. Mapping the fourth gesture result to clicking the right mouse button, based on the characteristic of the middle fingertip moving with acceleration exceeding a first threshold, distinguishes between left and right button functions, enhancing control diversity. Mapping the fifth gesture result to scrolling the mouse wheel upwards, based on the characteristic of the index finger contour sliding upwards, achieves accurate response to scrolling operations. Mapping the sixth gesture result to scrolling the mouse wheel downwards, based on the characteristic of the index finger contour sliding downwards. The mapping mechanism is based on the precise calculation of the three-dimensional spatial position, movement trajectory and acceleration information of the preset part of the hand, which ensures that every gesture intention can be accurately captured and converted into the expected mouse operation. This improves the smoothness and accuracy of the non-press mouse, significantly improves the user experience, and avoids misoperation and high latency issues.

[0082] In some embodiments, the present invention further proposes that acquiring data information corresponding to hand movements includes: Step c1: Obtain acceleration measurement information from the accelerometer.

[0083] Specifically, the acceleration of the hand in three-dimensional space is directly measured using a sensor mounted on the mouse. This accelerometer can be either a sensor-based accelerometer or a piezoelectric accelerometer. By directly acquiring acceleration data, raw, real-time physical quantities can be provided for calculating the hand's trajectory and velocity, which can be used to calibrate for potential error accumulation and latency caused by indirect derivations from image sequences.

[0084] Step c2: Capture image information of the hand, including the positioning device worn on the user's hand.

[0085] Specifically, visual data of a user's hand is acquired using image acquisition devices, and these images show the user wearing specific positioning components on their hand. The image acquisition devices can employ visible light cameras to capture images of the hand and the positioning components with specific colors, shapes, or patterns, facilitating subsequent image segmentation and feature extraction; or they can use infrared cameras, in conjunction with infrared illumination, to capture images of the hand in the infrared spectrum. In this case, the positioning components are designed as materials or structures capable of emitting or reflecting infrared light, ensuring stable identification under various ambient lighting conditions. These positioning components serve as visual reference points, improving the accuracy and stability of image processing algorithms in identifying key parts of the hand.

[0086] Step c3: Receive rotation measurement information from the gyroscope, which is mounted on the positioning component.

[0087] Specifically, a gyroscope integrated into the positioning component is used to acquire hand rotation data. Placing the gyroscope on the positioning component ensures a close correlation between rotation measurement information and hand movements, accurately reflecting changes in hand posture and rotation speed, and providing important supplementary data for gesture recognition.

[0088] It is understood that the technical solution in this embodiment addresses the problem of insufficient gesture recognition accuracy when relying solely on image information by introducing multi-source sensor data. The accelerometer directly provides acceleration measurement information of hand movements, calibrating against potential errors and delays inferring acceleration from image sequences, thus improving the real-time performance and accuracy of data acquisition. By capturing image information including the positioning component worn on the user's hand, and using the positioning component as a stable visual reference point, the reliability of image segmentation and feature extraction is enhanced, ensuring the stability of the positional information of the preset hand parts. Receiving rotation measurement information from the gyroscope mounted on the positioning component allows for precise capture of hand posture changes and rotation speed, supplementing the shortcomings of image information in processing complex dynamic gestures. The synergistic effect of multi-source data enables more comprehensive and accurate calculation of the movement trajectory and acceleration information of the preset hand parts, thereby improving the reliability, response speed, and overall accuracy of gesture recognition, providing users with a smoother and more precise non-press-based mouse operation experience.

[0089] In some embodiments, the present invention further proposes that, after calculating the movement trajectory and acceleration information of the preset part based on the sequence relationship of image information and the corresponding position information of the preset part in three-dimensional space, the method further includes: Step d1: Based on the acceleration measurement information and the rotation measurement information, data calibration is performed on the movement trajectory and acceleration information of the preset part.

[0090] Specifically, data calibration is used to correct or optimize the movement trajectory and acceleration information of preset parts calculated from image information, thereby improving the accuracy and stability of gesture recognition. When calibrating acceleration information based on acceleration measurement information, the acceleration measurement information is the object's motion acceleration data directly measured by the accelerometer, which has high real-time performance and accuracy. For example, when the acceleration value calculated from the image deviates from the sensor measurement value by more than a preset threshold, the sensor measurement value can be used preferentially or a weighted average can be applied.

[0091] The movement trajectory is calibrated based on rotation measurement information, which is the angular velocity or attitude data of an object in three-dimensional space measured by a gyroscope. For example, the rotation data provided by the gyroscope is compared and fused with the hand posture changes identified in the image sequence. When the image recognition algorithm fails to accurately capture the subtle rotation of the hand in some frames, the rotation data from the gyroscope is used to correct the relative position changes of a preset part of the hand in three-dimensional space, thereby correcting the movement trajectory.

[0092] It is understood that the technical solution in this embodiment introduces a data calibration step, utilizing acceleration measurement information provided by an accelerometer and rotation measurement information provided by a gyroscope to correct the image calculation results. Acceleration measurement information directly reflects the linear movement of the hand, effectively correcting acceleration calculation errors that may be caused by insufficient light, blurring, or occlusion in image recognition, ensuring the accuracy of acceleration data. Rotation measurement information accurately captures the rotational movement of the hand, correcting potentially missed or inaccurately recognized posture changes in the image sequence, and correcting the movement trajectory of preset parts. By calibrating the movement trajectory and acceleration information separately, motion parameters can be optimized in a targeted manner, achieving effective fusion of multi-source sensor data, improving the accuracy and stability of gesture recognition, and reducing the false trigger rate. Especially in complex or dynamic usage environments, the calibration mechanism ensures the robustness of gesture recognition, enabling non-press-based mouse solutions to better replace traditional mice and improve user satisfaction.

[0093] This embodiment also provides a mouse control device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] This embodiment provides a mouse control device, such as... Figure 3 As shown, it includes: The information acquisition unit 301 is used to acquire data information corresponding to hand movements, including image information.

[0095] The gesture recognition unit 302 is used to perform preset processing on image information to obtain gesture recognition results.

[0096] The control unit 303 is used to map the gesture recognition result into the corresponding mouse event and send the control command corresponding to the mouse event to the controlled object.

[0097] The mouse control device provided in this embodiment of the invention can execute the mouse control method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0098] This embodiment also provides a mouse, including a data acquisition module, a data processing module, a wireless communication module, a power management module, and a housing assembly. The data acquisition module collects data corresponding to hand movements, avoiding physical contact between the user's fingers and the device, thus eliminating fatigue caused by prolonged pressing and reducing reliance on a support surface. The data processing module executes the above method, generating accurate mouse events through real-time analysis of the collected data, effectively reducing recognition latency and false triggering rate. The wireless communication module sends control commands corresponding to mouse events to the controlled object, allowing users to operate within a larger spatial range and increasing freedom of use. The power management module provides energy management for the mouse, optimizing power distribution, extending device battery life, and addressing portability issues. The housing assembly secures the data acquisition module, data processing module, wireless communication module, and power management module, ensuring stable operation of each module and reliable support for complex gesture operations.

[0099] In some embodiments, the data acquisition module includes a camera, an infrared fill light, and an accelerometer. The infrared fill light is used to provide supplementary lighting for the camera's shooting environment; the camera is used to acquire image information corresponding to hand movements; and the accelerometer is used to acquire acceleration measurement information of the hand.

[0100] Specifically, a camera is an optical imaging device used to capture image information corresponding to hand movements. Cameras can be of various types; for example, visible light cameras are used to acquire hand images under normal lighting conditions, while infrared cameras are used to acquire images by capturing reflected infrared light in low-light or no-light environments. An infrared fill light is an illumination device that emits infrared light to supplement the lighting of the camera's shooting environment. Its function is to provide an additional infrared light source when ambient light is insufficient, ensuring that the camera can acquire clear, high-quality hand images. An accelerometer is a device that measures the acceleration of an object, used to acquire acceleration measurement information of the hand. An accelerometer can provide dynamic change data of the hand in three-dimensional space, supplementing the lack of motion details in the image information.

[0101] It's understandable that the data acquisition module, by integrating a camera, infrared fill light, and accelerometer, solves the problems of unclear image acquisition and lack of accurate motion data in low-light environments. The fusion of multi-source data allows for more precise calculation of the position, trajectory, and acceleration information of the preset hand parts in three-dimensional space, significantly improving the accuracy and robustness of gesture recognition and reducing false recognition rates and latency. Especially in scenarios requiring precise operations (such as single clicks, double clicks, and scrolling), combining acceleration data can more accurately determine the user's intent, thus providing a smoother and more precise mouse control experience.

[0102] In some embodiments, the present invention provides a mouse system including the mouse described above and a positioning component, the positioning component being worn on the user's hand, the positioning component including a gyroscope.

[0103] Specifically, the positioning component is configured to fit closely to the user's hand, ensuring close-range and high-precision motion data acquisition and effectively reducing the interference of ambient light changes on the recognition process. The gyroscope is set to monitor the hand's rotation angle and angular velocity in real time, fusing rotation information with position data to dynamically correct the movement trajectory, improving the accuracy and response speed of gesture recognition. This avoids the dependence of traditional mice on physical buttons and desktop support surfaces, reducing the mechanical load on fingers and wrists. At the same time, non-contact operation eliminates the risk of accidental triggering due to hand sweat or oil, enabling the system to achieve stable and low-latency mouse function control even in scenarios without a flat support surface.

[0104] In one example, to improve the gesture recognition performance when medical staff wear surgical masks, this invention optimizes the data processing module using a glove pattern adaptive recognition algorithm.

[0105] A new glove pattern recognition submodule and dynamic parameter adjustment unit have been added to the existing gesture recognition algorithm module. Users activate glove mode via gestures or software settings. Once activated, the module preprocesses the incoming hand image, including: reflection suppression filtering to suppress glove surface reflections and highlight the hand contours; texture and edge enhancement to improve the clarity of fingertips and hand edges; and grayscale adaptation to adjust contrast based on glove color or infrared characteristics.

[0106] When extracting key points from preprocessed images, the mouse integrates training models with bare hands and various gloves, dynamically selecting or weighting the models based on image features to improve the accuracy of key point recognition under glove occlusion.

[0107] Through the technical solution in this embodiment, the mouse can maintain high-precision, low-latency gesture recognition when wearing surgical gloves, reduce optical interference, and ensure smooth and reliable operation in special scenarios.

[0108] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0109] The following is a detailed reference. Figure 4 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0110] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0111] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a memory 408, or installed from a ROM 402. When the computer program is executed by the processor 401, it performs the functions defined in the mouse control method of the embodiments of the present invention.

[0112] Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0113] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the mouse control method shown in the above embodiments is implemented.

[0114] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0115] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for controlling a mouse, characterized in that, include: Acquire data information corresponding to hand movements, wherein the data information includes image information; The image information is processed using a preset method to obtain the gesture recognition result; The gesture recognition result is mapped to the corresponding mouse event, and the control command corresponding to the mouse event is sent to the controlled object.

2. The method according to claim 1, characterized in that, The image information includes an image sequence, and the preset processing of the image information to obtain the gesture recognition result includes: Image segmentation and feature extraction are performed on each image in the image sequence to obtain the corresponding position information of the preset part of the hand in three-dimensional space; Based on the sequence relationship of the image information and the corresponding position information of the preset part in three-dimensional space, the movement trajectory and acceleration information of the preset part are calculated; The gesture recognition result is determined based on the movement trajectory and the acceleration information.

3. The method according to claim 2, characterized in that, Determining the gesture recognition result based on the movement trajectory and the acceleration information includes: The first gesture result is determined based on the movement of preset points on the palm outline; The second gesture result is determined based on the index fingertip moving towards the mouse and the acceleration exceeding the first threshold. The third gesture result is determined based on the index fingertip repeatedly moving towards the mouse twice with acceleration exceeding the second threshold. The fourth gesture result is determined based on the middle fingertip moving towards the mouse and the acceleration exceeding the first threshold. The result of the fifth gesture is determined by sliding the index finger upwards based on its outline. The sixth gesture is determined by sliding the index finger downwards based on its outline.

4. The method according to claim 3, characterized in that, The step of mapping the gesture recognition result to the corresponding mouse event includes: Map the first gesture result to mouse movement; Map the second gesture result to a left mouse button click; Map the third gesture result to a double-click of the left mouse button; Map the fourth gesture result to a right-click mouse button; The result of the fifth gesture is mapped to the mouse wheel scrolling upwards; The result of the sixth gesture is mapped to the mouse wheel scrolling down.

5. The method according to claim 2, characterized in that, The data information obtained corresponding to hand movements includes: Acquire acceleration measurement information from the accelerometer; Capture image information of the hand, the image information including a positioning component worn on the user's hand; The system receives rotation measurement information from a gyroscope mounted on the positioning component.

6. The method according to claim 5, characterized in that, After calculating the movement trajectory and acceleration information of the preset part based on the sequence relationship of the image information and the corresponding position information of the preset part in three-dimensional space, the method further includes: The movement trajectory and acceleration information of the preset part are calibrated based on the acceleration measurement information and the rotation measurement information, respectively.

7. A control device for a mouse, characterized in that, include: An information acquisition unit is used to acquire data information corresponding to hand movements, wherein the data information includes image information; A gesture recognition unit is used to perform preset processing on the image information to obtain a gesture recognition result; The control unit is used to map the gesture recognition result into a corresponding mouse event and send the control command corresponding to the mouse event to the controlled object.

8. A mouse, characterized in that, The mouse includes: The system includes a data acquisition module, a data processing module, a wireless communication module, a power management module, and a housing assembly. The data acquisition module is used to collect data information corresponding to hand movements; The data processing module is used to execute the method provided by any one of claims 1-6; The wireless communication module is used to send the control command corresponding to the mouse event to the controlled object; The power management module is used to provide power consumption management for the mouse; The housing assembly is used to secure the data acquisition module, the data processing module, the wireless communication module, and the power management module.

9. The mouse according to claim 8, characterized in that, The data acquisition module includes a camera, an infrared fill light, and an accelerometer. The infrared fill light is used to provide supplemental lighting for the shooting environment of the camera; The camera is used to acquire image information corresponding to hand movements; The accelerometer is used to acquire acceleration measurement information of the hand.

10. A mouse system, characterized in that, Includes a mouse and a positioning component as described in claim 8 or 9, wherein the positioning component is worn on a user's hand and includes a gyroscope.

Images