Intelligent mouse auxiliary control method and device, electronic equipment and storage medium

Abstract

The invention discloses an intelligent mouse auxiliary control method and device, electronic equipment and a storage medium, and belongs to the technical field of man-machine interaction. According to the method, by means of the computer vision technology, the action of the face of a user is captured in real time, the position of a mouse pointer is accurately calculated, and the user can achieve efficient and visual mouse control through natural movement of the face; meanwhile, due to the introduction of the Bluetooth control ring, the user can send a control signal through simple touch operation, specific operations such as clicking and dragging of the mouse are realized, and the convenience and efficiency of the operation are remarkably improved. The method is equivalent to a traditional mouse in operation precision and intuition, and the learning cost of a user is remarkably reduced. In the aspect of privacy protection, the method only depends on face key points and Bluetooth to control signals of the ring, voice information of a user is not involved, and privacy leakage is effectively prevented. According to the invention, brand new interaction experience can be brought to the user, and a new thought is provided for the development of a future man-machine interaction technology.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and more specifically, to an intelligent mouse auxiliary control method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of computer technology, human-computer interaction methods are also constantly innovating. Although traditional mouse control methods have been widely used, their limitations are gradually revealed in certain specific scenarios, such as limited space, hand fatigue, or when more efficient operations are required. For example, in an environment with limited desk space, a traditional mouse may take up too much space, be inconvenient to carry, and be difficult to move. For users who need to work for long periods of time, such as programmers and designers, frequently switching between keyboard and mouse not only interrupts the workflow, but also reduces work efficiency. In this context, it is particularly important to explore a new mouse control method based on natural user behavior (such as facial movement) and assisted by intelligent hardware.

[0003] Currently, there are many solutions to replace the traditional mouse, such as touchpads, styluses and voice recognition technology. Although these innovations have enriched the user's interactive experience to a certain extent, each solution has its inherent limitations. Although the touchpad is portable and simple in design, it is difficult to achieve the same precision as the traditional mouse. The size limitation makes its operation accuracy seriously insufficient, especially in situations where fine text editing and image processing are required. A study shows that 70% of designers have difficulties when using touchpads for image processing, which shows that touchpads perform poorly in scenarios with high precision requirements. Although the stylus performs well in the fields of painting and writing, its scope of application is relatively narrow and requires additional hardware support, which increases the user's cost of use and limits its popularity. According to the survey, only less than 30% of creative workers frequently use styluses, and most users believe that it is not suitable for daily office operations. In addition, although voice recognition technology has greatly improved convenience, its accuracy rate drops significantly in noisy environments, and there is a risk of privacy leakage. Studies on office environments have shown that due to the influence of background noise, the misrecognition rate of voice recognition systems can reach 15%, which seriously affects its reliability in actual work.

[0004] In addition to these technical limitations, existing alternatives generally lack intuitiveness and ease of use, requiring users to adapt over a long period of time, increasing learning costs. Studies have found that the learning curve for new programmers using new operating tools is nearly 30% higher than that for traditional mice, which not only affects work efficiency but also reduces user acceptance. More importantly, these solutions often ignore ergonomic design, and prolonged use may cause hand fatigue or even injury, especially in professions such as programmers and designers who need to work long hours.

[0005] In summary, the current alternatives on the market have many deficiencies in terms of accuracy, ease of use, learning cost, and ergonomics. In order to meet user needs, it is urgent to develop a technical solution that better meets user needs to overcome the limitations of existing solutions, improve operating accuracy, reduce learning costs, and focus on ergonomic design to reduce hand fatigue and damage caused by long-term use. Summary of the invention

[0006] In order to overcome the technical problems existing in the above prior art, the present invention provides an intelligent mouse auxiliary control method, device, electronic device and storage medium.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows: An intelligent mouse auxiliary control method comprises the following steps: Receive a facial video stream collected in real time by a user, perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image, and obtain a cropped image containing the user's face area; perform grayscale processing on each cropped image to obtain multiple grayscale images; Performing facial region detection on each of the preprocessed grayscale images, and obtaining coordinates of a plurality of facial key points in the facial region; adding and averaging the coordinate values ​​of the plurality of facial key points to obtain the facial center coordinates of each grayscale image; Acquire a facial video stream of the user when facing the screen, perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user when facing the screen, perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user when facing the screen, and calculate the relative coordinates of the center of the user's face; Obtaining the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculating a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; Creating a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moves with the movement of the user's face; multiplying the movement amount of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; determining a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displaying the mouse simulation pointer at the display position; Receive a user's mouse operation signal, and perform corresponding mouse operations based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation.

[0008] Furthermore, a facial video stream of the user is collected in real time through a camera facing the user's face; The video stream is preprocessed frame by frame through the OpenCV machine vision library, and the preprocessing includes cropping and uniformly extracting each frame of the image, and performing grayscale processing; Perform face detection on the preprocessed frames using the Dlib face detection model, which can locate 68 key points in the face image; Obtain the coordinates of three facial key points in the face area detected by the Dlib face detection model; Among them, the three facial key points are the 54th key points located by the Dlib face detection model 、Key Point No. 56 and Key Point No. 58 ; The key point No. 54 corresponds to the position of the user's nose tip, the key point No. 56 corresponds to the position below the user's left nostril, and the key point No. 58 corresponds to the position below the user's right nostril.

[0009] Further, a facial video stream of the user facing the screen in a working posture is obtained, and frame-by-frame preprocessing is performed to obtain a plurality of cropped grayscale images of the user facing the screen, a face region is detected for each of the preprocessed grayscale images, and the coordinates of three facial key points in the face region are obtained; the coordinate values ​​of the three facial key points are added and averaged to obtain the facial center coordinates of each grayscale image; Face center coordinates The calculation formula is as follows: ; in, , , They are the horizontal coordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; , , They are the ordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; The facial center coordinates of each grayscale image are calculated by the above formula, and the average of the obtained multiple facial center coordinates is calculated to obtain the relative coordinates of the center of the user's face. .

[0010] Further, using a pre-trained face detection model to perform face detection on the pre-processed picture frame, obtaining a detected face rectangular frame and a pixel width and height of the face rectangular frame; When the same picture frame contains multiple faces, the areas of all detected facial rectangular frames in each frame are calculated and compared one by one, and only the facial rectangular frame with the largest area is retained as the face region.

[0011] Further, the left boundary value of the facial activity rectangle is obtained by the user's left head turning action, and the right boundary value of the facial activity rectangle is obtained by the user's right head turning action; the width w of the facial activity rectangle is obtained by calculating the difference between the left boundary value and the right boundary value; The upper boundary value of the facial activity rectangle is obtained by the user's upward lifting action, and the lower boundary value of the facial activity rectangle is obtained by the user's downward lowering action; the height h of the facial activity rectangle is obtained by calculating the difference between the upper boundary value and the lower boundary value; Based on the width w of the user's facial activity rectangle and the width W of the screen used by the user, the mapping compensation ratio in the horizontal direction is calculated. , the formula is as follows: ; Based on the height h of the user's facial activity rectangle and the height H of the screen used by the user, calculate the mapping compensation ratio in the vertical axis direction , the formula is as follows: .

[0012] Furthermore, the center relative coordinates are , the facial center coordinates are , the mapping compensation ratio in the horizontal direction is , the mapping compensation ratio in the longitudinal direction is , the width of the screen is W, and the height of the screen is H; Mouse pointer offset in horizontal direction The calculation formula is: ; Mouse pointer offset in the vertical direction The calculation formula is: ; The center coordinates of the screen are obtained by the width and height of the screen: ; Add the mouse pointer offset to the center coordinate of the screen to obtain the coordinates of the mouse simulation pointer on the screen used by the user .

[0013] Further, receiving a mouse operation signal sent by a user through a Bluetooth controlled ring, wherein the Bluetooth controlled ring includes two touch buttons supporting a long press recognition function, wherein the two touch buttons are a click button C and a slide button S; Send a left-click mouse operation signal by clicking the click button C; Sending a left-button long-press mouse operation signal by long-pressing the click button C; Sending a right-click mouse operation signal by clicking the click button C and the slide button S simultaneously; By long pressing the sliding button S, a mouse operation signal of the scroll wheel sliding is sent.

[0014] An intelligent mouse auxiliary control device, comprising: A preprocessing unit, the preprocessing unit is used to receive a facial video stream collected in real time by a user, and perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image to obtain a cropped image containing the user's face area, and perform grayscale processing on each cropped image to obtain multiple grayscale images; A face detection unit, the face detection unit is used to detect the face area of ​​each of the preprocessed grayscale images and obtain the coordinates of multiple facial key points in the face area; A facial center coordinate calculation unit, the facial center coordinate calculation unit is used to add the coordinate values ​​of the plurality of facial key points and take an average value to obtain the facial center coordinates of each grayscale image; A data calibration unit, the data calibration unit is used to obtain a facial video stream when the user is looking directly at the screen, and perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user looking directly at the screen, and perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user looking directly at the screen to calculate the relative coordinates of the center of the user's face; the data calibration unit is also used to obtain the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculate a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; a pointer moving unit, the pointer moving unit being used to create a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moving as the user's face moves; the pointer moving unit multiplies the amount of movement of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; the pointer moving unit determines a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displays the mouse simulation pointer at the display position; A mouse operation execution unit, wherein the mouse operation execution unit is used to receive a user's mouse operation signal and execute a corresponding mouse operation based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the aforementioned intelligent mouse auxiliary control method when executing the computer program.

[0016] A computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the aforementioned intelligent mouse auxiliary control method.

[0017] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention significantly improves the accuracy and intuitiveness of operation by adopting advanced computer vision technology. By accurately capturing and locating the key points of the user's face, the present invention can accurately calculate the position of the mouse pointer. This technological innovation allows users to intuitively see the results of facial movement through the simulated mouse pointer when performing delicate operations, such as text editing and image processing, and enjoy the same accuracy as a traditional mouse. The direct association between facial movement and the simulated mouse pointer makes the operation process more intuitive and easy to understand, reducing the user's learning cost. The control of the actual mouse pointer relies on the signal of the Bluetooth control ring, ensuring the accuracy and efficiency of the operation.

[0018] In addition, the present invention combines a Bluetooth control ring, which greatly enhances ease of use and convenience. Users can send control signals through simple touch operations to achieve wireless control of the mouse pointer. This design eliminates unnecessary actions when users use traditional mice, such as looking for the mouse, adjusting posture, etc., thereby improving the convenience of operation. Especially for professionals who need to frequently switch between the keyboard and the mouse, the technical solution of the present invention significantly improves work efficiency.

[0019] The present invention also effectively reduces the learning cost and adaptation time. Since the operation mode is based on the natural facial movement of the human body and simple finger touch, users do not need to spend a lot of time to adapt to the new operation mode. Compared with other alternatives, the technical solution of the present invention performs well in the learning curve, and users can quickly master and use it proficiently, thereby reducing the learning cost.

[0020] The present invention has also achieved significant improvements in ergonomics. By reducing direct contact between the hand and the mouse, the present invention effectively reduces the risk of hand fatigue and injury caused by long-term use of the mouse. The combination of facial movement and ring touch allows users to maintain a more natural posture during operation, further improving the comfort of use.

[0021] The present invention also performs well in terms of privacy protection and security. Compared with technical solutions such as voice recognition that may leak personal privacy, the present invention only relies on the key points of the user's face and the signal of the Bluetooth control ring to operate, and does not involve the user's voice information, thereby effectively improving the level of privacy protection. In addition, the communication between the Bluetooth control ring and the computer uses encryption technology to ensure the security of data transmission.

[0022] Finally, the present invention has wide applicability and compatibility. The technical solution does not depend on a specific hardware platform or operating system, and can be used as long as the user has a device with a camera and Bluetooth function. This feature enables the present invention to meet the needs of different users in different scenarios, such as tablet computers, smart TVs, etc.

[0023] In summary, the mouse-assisted control method and device based on facial key point positioning proposed in the present invention have achieved remarkable technical effects in improving operation accuracy and intuitiveness, enhancing ease of use and convenience, reducing learning costs and adaptation time, improving ergonomic experience, improving privacy protection and security, and wide applicability and compatibility. This technical solution not only provides users with a more efficient and intuitive mouse control method, but also provides new ideas and directions for the development of future human-computer interaction technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of an intelligent mouse auxiliary control method provided in one embodiment of the present application; Figure 2 is a schematic diagram of the center relative coordinates and the facial activity rectangle in the intelligent mouse auxiliary control method provided in one embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a Bluetooth controlled ring in an intelligent mouse auxiliary control method provided in one embodiment of the present application; Figure 4 It is a schematic diagram of the working principle of the intelligent mouse auxiliary control method provided in one embodiment of the present application; Figure 5 For this application Figure 4 A schematic diagram of the working principle of the context data processing module in FIG. Figure 6 For this application Figure 4 Schematic diagram of the working principle of the mouse pointer control module in; Explanation of the markings in the figure: 1. Click button C; 2. Slide button S. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0027] Embodiment 1:

[0028] like Figure 1 As shown, the present invention provides a technical solution: An intelligent mouse auxiliary control method comprises the following steps: Receive a facial video stream collected in real time by a user, perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image, and obtain a cropped image containing the user's face area; perform grayscale processing on each cropped image to obtain multiple grayscale images; Performing facial region detection on each of the preprocessed grayscale images, and obtaining coordinates of a plurality of facial key points in the facial region; adding and averaging the coordinate values ​​of the plurality of facial key points to obtain the facial center coordinates of each grayscale image; Acquire a facial video stream of the user when facing the screen, perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user when facing the screen, perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user when facing the screen, and calculate the relative coordinates of the center of the user's face; Obtaining the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculating a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; Creating a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moves with the movement of the user's face; multiplying the movement amount of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; determining a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displaying the mouse simulation pointer at the display position; Receive a user's mouse operation signal, and perform corresponding mouse operations based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation.

[0029] Embodiment 2:

[0030] Based on Example 1, a facial video stream of the user is collected in real time by a camera facing the user's face; The video stream is preprocessed frame by frame through the OpenCV machine vision library, and the preprocessing includes cropping and uniformly extracting each frame of the image, and performing grayscale processing; Perform face detection on the preprocessed frames using the Dlib face detection model, which can locate 68 key points in the face image; Obtain the coordinates of three facial key points in the face area detected by the Dlib face detection model; Among them, the three facial key points are the 54th key points located by the Dlib face detection model 、Key Point No. 56 and Key Point No. 58 ; The key point No. 54 corresponds to the position of the user's nose tip, the key point No. 56 corresponds to the position below the user's left nostril, and the key point No. 58 corresponds to the position below the user's right nostril.

[0031] Specifically, computer vision technology, especially facial key point detection, has been widely used in the fields of face recognition, head position recognition, etc. The OpenCV machine vision processing library and the Dlib face detection model are used to capture and analyze the position information and key point coordinates of the user's face. Research in the field of human-computer interaction points out that intuitive feedback and learning costs are important factors affecting users' acceptance of new technologies. The present invention collects contextual data (including real-time position information of the user's face, key point coordinates, screen size and other necessary parameters) to enable the system to calculate the position of the user's face relative to the screen and simulate the movement of the mouse pointer.

[0032] Further, a facial video stream of the user facing the screen in a working posture is obtained, and frame-by-frame preprocessing is performed to obtain a plurality of cropped grayscale images of the user facing the screen, a face region is detected for each of the preprocessed grayscale images, and the coordinates of three facial key points in the face region are obtained; the coordinate values ​​of the three facial key points are added and averaged to obtain the facial center coordinates of each grayscale image; Face center coordinates The calculation formula is as follows:

[0033] in, , , They are the horizontal coordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; , , They are the ordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; The facial center coordinates of each grayscale image are calculated by the above formula, and the average of the obtained multiple facial center coordinates is calculated to obtain the relative coordinates of the center of the user's face. .

[0034] Further, using a pre-trained face detection model to perform face detection on the pre-processed picture frame, obtaining a detected face rectangular frame and a pixel width and height of the face rectangular frame; When the same picture frame contains multiple faces, the areas of all detected facial rectangular frames in each frame are calculated and compared one by one, and only the facial rectangular frame with the largest area is retained as the face region.

[0035] Further, if Figure 2 As shown, the left boundary value of the facial activity rectangle is obtained by the user's left head turning action, and the right boundary value of the facial activity rectangle is obtained by the user's right head turning action; the width w of the facial activity rectangle is obtained by calculating the difference between the left boundary value and the right boundary value; The upper boundary value of the facial activity rectangle is obtained by the user's upward lifting action, and the lower boundary value of the facial activity rectangle is obtained by the user's downward lowering action; the height h of the facial activity rectangle is obtained by calculating the difference between the upper boundary value and the lower boundary value; Based on the width w of the user's facial activity rectangle and the width W of the screen used by the user, the mapping compensation ratio in the horizontal direction is calculated. , the formula is as follows: ;

[0036] Based on the height h of the user's facial activity rectangle and the height H of the screen used by the user, calculate the mapping compensation ratio in the vertical axis direction , the formula is as follows: .

[0037] Furthermore, the center relative coordinates are , the facial center coordinates are , the mapping compensation ratio in the horizontal direction is , the mapping compensation ratio in the longitudinal direction is , the width of the screen is W, and the height of the screen is H; Mouse pointer offset in horizontal direction The calculation formula is: ; Mouse pointer offset in the vertical direction The calculation formula is: ; The center coordinates of the screen are obtained by the width and height of the screen: ; Add the mouse pointer offset to the center coordinate of the screen to obtain the coordinates of the mouse simulation pointer on the screen used by the user .

[0038] Further, if Figure 3As shown, a mouse operation signal sent by a user through a Bluetooth control ring is received, and the Bluetooth control ring includes two touch buttons that support a long press recognition function, and the two touch buttons are a click button C and a slide button S; Send a left-click mouse operation signal by clicking the click button C; Sending a left-button long-press mouse operation signal by long-pressing the click button C; Sending a right-click mouse operation signal by clicking the click button C and the slide button S simultaneously; By long pressing the sliding button S, a mouse operation signal of the scroll wheel sliding is sent.

[0039] Preferably, the user wears the Bluetooth control ring on the right index finger and sends multiple control signals to the receiving end program by touching the designated button on the ring. Combined with the processing logic of the receiving end program, four common functions can be realized, including left-click, left-click long press (including long press and drag), right-click and scroll wheel. Please note that the right index finger in the above description should be adjusted to a suitable wearing position according to actual conditions; however, in this design, wearing the ring on the right index finger is the most recommended choice.

[0040] This application designs a method for generating mouse control signals using a Bluetooth ring, which greatly reduces the extra actions required for users to perform common mouse operations; users can conveniently send corresponding control signals to the computer through simple operations on the Bluetooth ring, thereby covering the needs of ordinary users for basic mouse functions.

[0041] Embodiment 3:

[0042] The present invention provides a technical solution: An intelligent mouse auxiliary control device, comprising: A preprocessing unit, the preprocessing unit is used to receive a facial video stream collected in real time by a user, and perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image to obtain a cropped image containing the user's face area, and perform grayscale processing on each cropped image to obtain multiple grayscale images; A face detection unit, the face detection unit is used to detect the face area of ​​each of the preprocessed grayscale images and obtain the coordinates of multiple facial key points in the face area; A facial center coordinate calculation unit, the facial center coordinate calculation unit is used to add the coordinate values ​​of the plurality of facial key points and take an average value to obtain the facial center coordinates of each grayscale image; A data calibration unit, the data calibration unit is used to obtain a facial video stream when the user is looking directly at the screen, and perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user looking directly at the screen, and perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user looking directly at the screen to calculate the relative coordinates of the center of the user's face; the data calibration unit is also used to obtain the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculate a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; a pointer moving unit, the pointer moving unit being used to create a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moving as the user's face moves; the pointer moving unit multiplies the amount of movement of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; the pointer moving unit determines a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displays the mouse simulation pointer at the display position; A mouse operation execution unit, wherein the mouse operation execution unit is used to receive a user's mouse operation signal and execute a corresponding mouse operation based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation.

[0043] like Figure 4 As shown, the intelligent mouse auxiliary control device can be summarized as a context data processing module and a mouse pointer control module. The module runs on the graphical operating system interface used by the user, captures the user's facial position information in real time, and calculates it in combination with the screen size and system API, maps the facial movement into the continuous movement of the mouse simulation pointer (the actual mouse pointer is still valid), and adjusts the position of the actual mouse pointer according to the signal sent by the Bluetooth control ring and performs corresponding mouse operations.

[0044] Among them, Figure 5 As shown, the context data processing module receives the video stream image captured by the camera facing the user's face (such as the built-in camera of a laptop computer or a USB or network camera installed and properly placed by the user); the video stream image is preprocessed frame by frame through the OpenCV machine vision library, including cropping and uniform extraction of each frame image, and grayscale processing is performed to improve the recognition performance of the Dlib key point model; the preprocessed image frame is subjected to Dlib face detection to obtain the relative coordinates of the face in the frame relative to the pixel origin of the image frame and the pixel width and height of its rectangular frame.

[0045] In actual applications, the same frame may contain multiple faces. Therefore, in addition to adjusting the necessary weight parameters of the face detection model, it is also necessary to calculate the area of ​​all detected face rectangles in each frame and compare them one by one. Since the user's face is usually facing the camera and working close to the screen, the area of ​​the extracted face rectangle is relatively large. The system ultimately retains only the coordinate information of the face rectangle with the largest area, thereby completing the location information of the user's face.

[0046] Next, the Dlib facial feature point extraction technology is used to obtain the position information of 68 key points of the face. In this application, only the coordinates of the three key points 54 (nose tip), 56 (below the left nostril) and 58 (below the right nostril) are concerned. Through the above steps, the width and height of the picture frame, the relative position and width and height of the user's facial rectangular frame, and the relative coordinates of the three facial center key points are obtained.

[0047] In addition, before the program can work properly, it is necessary to calibrate the key point data to obtain the center relative coordinates. And the mapping compensation ratio R. The main steps are as follows: Step 1: The user faces the screen in a working posture and performs the following actions as required by the program: (1) Keep looking at the screen for a period of time t: Take No. 54 、No.56 and No. 58 The coordinate values ​​of the three key points are added and averaged; ;

[0048] The facial center coordinates are obtained , calculate the mean of multiple facial center coordinates obtained within a period of time t to obtain the center relative coordinate , this coordinate points to the center of the screen used by the user; (2) Turn head left: obtain the left boundary value (mean); (3) Turn right: get the right boundary value (mean); (4) Look up: get the upper boundary value (mean); (5) Lower the head: obtain the lower boundary value (mean); Step 2: Calculate the size of the facial activity rectangle based on the surrounding boundary values: Width of facial activity rectangle: w; Face activity rectangle height: h; Step 3: Calculate the mapping compensation ratio ;

[0049] First, call the system API to obtain the width and height of the user screen, and get the screen rectangle size: user screen rectangle width: W; user screen rectangle height: H; Then calculate the mapping compensation ratios in the horizontal (X) axis direction and the vertical (Y) axis direction respectively: ; ;

[0050] At this point, the context data processing module can provide the center relative coordinates for the next level (mouse pointer control module) , Current face center coordinates and mapping compensation ratio and It should be noted that the calculation of the center relative coordinates is only required when verifying the key point data, and the current face center coordinates The acquisition of is necessary throughout the program. In other words, the center relative coordinates are used to provide the center position information required by the system and only need to be calculated once; while the current face center coordinates reflect the user's facial movement data in real time, and the program relies on this data to achieve real-time adjustment of the simulated mouse pointer position.

[0051] Among them, Figure 6 As shown, after the upper module (context data processing module) completes data calibration and starts to provide data, the mouse pointer control module starts to simulate the mouse pointer first. The specific steps are as follows: Step 1: Calculate the offset and : According to the center relative coordinates provided by the upper module , Current face center coordinates and mapping compensation ratio and , we can get the offset formula of the horizontal (X) axis and the vertical (Y) axis: ;

[0052] ;

[0053] From the above, we can know that the width and height of the user screen rectangle are: W, H respectively, and the coordinates of the center of the screen can be obtained: ;

[0054] Finally, by adding the above offset to the screen center coordinates, the actual drawing coordinates of the mouse pointer can be obtained: ;

[0055] It is not difficult to see from the formula that when When it is a positive value, the analog pointer coordinate moves to the right, and when it is a negative value, it moves to the left. When it is a positive value, the simulated pointer coordinates move downward (the computer image coordinate system takes the upper left corner as the origin, the X-axis to the right is the positive direction, and the Y-axis downward is the positive direction), and when it is a negative value, it moves upward. In this way, the mouse pointer moves along with the changes in the user's facial position.

[0056] It should be noted that the coordinates here are only used to simulate the display of the mouse pointer, while the actual mouse operations (such as clicking, dragging, etc.) are still performed by the actual mouse pointer according to the signal of the Bluetooth control ring.

[0057] Step 2: Draw a simulated mouse pointer: After the above steps, the exact position of the simulated mouse pointer has been obtained. Then, a simulated mouse pointer similar to the actual mouse pointer but with slightly lower transparency is simulated through the program to indicate the current position of the simulated pointer to the user. At this time, if the user tries to move his face, the simulated mouse pointer will move accordingly.

[0058] Step 3: Process the command and generate mouse actions: When the mouse pointer control module monitors the signal (four types in total) generated by the Bluetooth control ring, it will generate and execute different mouse control instructions according to the type of signal, as follows: (1) Left click: When a click signal is detected, the program will call the system API to quickly position the actual mouse pointer to the position of the simulated mouse pointer, and perform a click operation after the positioning is completed.

[0059] (2) Long press the left button: When a long press signal is detected and the finger is not released, the program will continue to keep the real mouse pointer at the location of the simulated mouse pointer through the system API, and perform a long press action after the first positioning. If the user moves his face during this process, the position of the simulated pointer will change accordingly, causing the real mouse pointer to perform a long press and drag operation along the new trajectory.

[0060] (3) Right click: When a right-click signal is detected, the program will also use the system API to quickly position the actual mouse pointer to the simulated mouse pointer, and perform a right-click after positioning.

[0061] (4) Roller sliding: When the scroll wheel sliding signal is detected and the finger is not released, the program will continuously call the system API to simulate the scroll wheel. At this time, the program only reads the vertical movement changes of the user's face and generates the corresponding scrolling effect accordingly.

[0062] Among them, the methods of using the Bluetooth control ring to complete four basic mouse operations are as follows: Wear the Bluetooth control ring on the user's right middle finger, right index finger or other fingers. The Bluetooth control ring contains two touch buttons that support long press recognition function, marked as click button C (click) and slide button S (slide), which can realize the following four common mouse operations: Left click (click the C button); Long press the left button (long press the C button); Right-click (press the C and S buttons at the same time); Scroll wheel (press and hold the S button).

[0063] Preferably, when the Bluetooth control ring is worn on the index finger of the user's right hand, the pulp surface of the index finger is preferably located between the click button C and the slide button S, so that the thumb and middle finger can press the click button C and the slide button S on the index finger.

[0064] Specifically, the mouse pointers involved in this application are divided into two types: 1. Mouse simulation pointer: This pointer is generated based on the data provided by the context module. After the data verification is completed, it will be drawn on the user's screen and displayed in real time. When the user's face moves (i.e. the coordinates of the center of the face change), the program will map this change into the movement of the simulated mouse pointer through a specific mathematical algorithm.

[0065] In short, the current facial center coordinates reflect the user's facial movement data in real time, and the program relies on this data to implement real-time adjustment of the mouse simulation pointer position. The main function of the mouse simulation pointer is to intuitively feedback the results of the user's facial movement on the screen.

[0066] 2. Actual mouse pointer: This refers to the mouse pointer that comes with the user's operating system (usually represented by a white arrow). In the present invention, single click, right click and other operations are ultimately completed by the actual mouse pointer. The difference is that these operations are actively triggered according to the control signal sent by the ring.

[0067] For example, when you need to click the "×" icon in the upper right corner of the screen to close the current page, the user only needs to move his face in the same direction. At this time, the mouse simulation pointer will move in that direction accordingly, while the actual mouse pointer remains unchanged. When the mouse simulation pointer reaches the position of the "×" icon through the user's facial movement, the user only needs to click the left button C on the ring, and the program will immediately position the actual mouse pointer to the same position as the mouse simulation pointer and perform a left click.

[0068] Embodiment 4:

[0069] The present invention provides a technical solution: An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the aforementioned intelligent mouse auxiliary control method when executing the computer program.

[0070] In this embodiment, the computer system suitable for implementing the electronic device of the embodiment of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM), such as executing the method described in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other through a bus. Input / output (I / O) interface is also connected to the bus.

[0071] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage part as needed.

[0072] Embodiment 5:

[0073] The present invention provides a technical solution: A computer-readable storage medium stores a computer program executable by an electronic device, and when the computer program is executed on the electronic device, the electronic device executes the steps of the aforementioned intelligent mouse auxiliary control method. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0074] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0077] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0078] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the application point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0079] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0080] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be construed as a limitation to the claims. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. An intelligent mouse auxiliary control method, characterized in that: The following steps are involved: Receive a facial video stream collected in real time by a user, perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image, and obtain a cropped image containing the user's face area; perform grayscale processing on each cropped image to obtain multiple grayscale images; Performing facial region detection on each of the preprocessed grayscale images, and obtaining coordinates of a plurality of facial key points in the facial region; adding and averaging the coordinate values ​​of the plurality of facial key points to obtain the facial center coordinates of each grayscale image; Acquire a facial video stream of the user when facing the screen, perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user when facing the screen, perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user when facing the screen, and calculate the relative coordinates of the center of the user's face; Obtaining the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculating a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; Creating a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moves with the movement of the user's face; multiplying the movement amount of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; determining a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displaying the mouse simulation pointer at the display position; Receive a mouse operation signal from a user, and perform a corresponding mouse operation based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation; The steps of obtaining the width and height of the user's facial activity rectangle are as follows: Obtaining a left boundary value of a facial activity rectangle by a user's left-turning head action, and obtaining a right boundary value of the facial activity rectangle by a user's right-turning head action; and obtaining a width w of the facial activity rectangle by calculating a difference between the left boundary value and the right boundary value; The upper boundary value of the facial activity rectangle is obtained by the user's upward lifting action, and the lower boundary value of the facial activity rectangle is obtained by the user's downward lowering action; the height h of the facial activity rectangle is obtained by calculating the difference between the upper boundary value and the lower boundary value; The steps for calculating the mapping compensation ratio are as follows: Based on the width w of the user's facial activity rectangle and the width W of the screen used by the user, the mapping compensation ratio in the horizontal direction is calculated. , the formula is as follows: ; Based on the height h of the user's facial activity rectangle and the height H of the screen used by the user, calculate the mapping compensation ratio in the vertical axis direction , the formula is as follows: .

2. The intelligent mouse auxiliary control method according to claim 1, characterized in that: The user's facial video stream is collected in real time through a camera facing the user's face; The video stream is preprocessed frame by frame through the OpenCV machine vision library, and the preprocessing includes cropping and uniformly extracting each frame of the image, and performing grayscale processing; Perform face detection on the preprocessed frames using the Dlib face detection model, which can locate 68 key points in the face image; Obtain the coordinates of three facial key points in the face area detected by the Dlib face detection model; Among them, the three facial key points are the 54th key points located by the Dlib face detection model 、Key Point No. 56 and Key Point No. 58 ; The key point No. 54 corresponds to the position of the user's nose tip, the key point No. 56 corresponds to the position below the user's left nostril, and the key point No. 58 corresponds to the position below the user's right nostril.

3. The intelligent mouse auxiliary control method according to claim 2, characterized in that: Acquire a facial video stream of a user facing the screen in a working posture, and perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user facing the screen, perform face region detection on each of the preprocessed grayscale images, and obtain the coordinates of three facial key points in the face region; respectively add up the coordinate values ​​of the three facial key points and obtain the average value to obtain the facial center coordinates of each grayscale image; Face center coordinates The calculation formula is as follows: ; in, , , They are the horizontal coordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; , , They are the ordinate values ​​of key point No. 54, key point No. 56 and key point No. 58 respectively; The facial center coordinates of each grayscale image are calculated by the above formula, and the average of the obtained multiple facial center coordinates is calculated to obtain the relative coordinates of the center of the user's face. .

4. The intelligent mouse auxiliary control method according to claim 1, characterized in that: Use a pre-trained face detection model to perform face detection on the pre-processed picture frame, and obtain the detected face rectangular frame and the pixel width and height of the face rectangular frame; When the same picture frame contains multiple faces, the areas of all detected facial rectangular frames in each frame are calculated and compared one by one, and only the facial rectangular frame with the largest area is retained as the face region.

5. The intelligent mouse auxiliary control method according to claim 1, characterized in that: The relative coordinates of the center are , the facial center coordinates are , the mapping compensation ratio in the horizontal direction is , the mapping compensation ratio in the vertical direction is , the width of the screen is W, and the height of the screen is H; Mouse pointer offset in horizontal direction The calculation formula is: ; Mouse pointer offset in the vertical direction The calculation formula is: ; The center coordinates of the screen are obtained by the width and height of the screen: ; Add the mouse pointer offset to the center coordinate of the screen to obtain the coordinates of the mouse simulation pointer on the screen used by the user .

6. The intelligent mouse auxiliary control method according to claim 1, characterized in that: Receiving a mouse operation signal sent by a user through a Bluetooth controlled ring, wherein the Bluetooth controlled ring includes two touch buttons supporting a long press recognition function, wherein the two touch buttons are a click button C and a slide button S; Send a left-click mouse operation signal by clicking the click button C; Sending a left-button long-press mouse operation signal by long-pressing the click button C; Sending a right-click mouse operation signal by clicking the click button C and the slide button S simultaneously; By long pressing the sliding button S, a mouse operation signal of the scroll wheel sliding is sent.

7. An intelligent mouse auxiliary control device, used to implement the intelligent mouse auxiliary control method according to any one of claims 1 to 6, characterized in that: include: A preprocessing unit, the preprocessing unit is used to receive a facial video stream collected in real time by a user, and perform frame-by-frame preprocessing on the facial video stream, crop each frame of the image to obtain a cropped image containing the user's face area, and perform grayscale processing on each cropped image to obtain multiple grayscale images; A face detection unit, the face detection unit is used to detect the face area of ​​each of the preprocessed grayscale images and obtain the coordinates of multiple facial key points in the face area; A facial center coordinate calculation unit, the facial center coordinate calculation unit is used to add the coordinate values ​​of the plurality of facial key points and take an average value to obtain the facial center coordinates of each grayscale image; A data calibration unit, the data calibration unit is used to obtain a facial video stream when the user is looking directly at the screen, and perform frame-by-frame preprocessing to obtain a plurality of cropped grayscale images of the user looking directly at the screen, and perform mean calculation on a plurality of facial center coordinates obtained through the plurality of cropped grayscale images of the user looking directly at the screen to calculate the relative coordinates of the center of the user's face; the data calibration unit is also used to obtain the width and height of the user's facial activity rectangle and the width and height of the screen used by the user, and calculate a mapping compensation ratio based on the width and height of the user's facial activity rectangle and the width and height of the screen used by the user; a pointer moving unit, the pointer moving unit being used to create a mouse simulation pointer on the screen used by the user, the mouse simulation pointer moving as the user's face moves; the pointer moving unit multiplies the amount of movement of the face center coordinates compared to the center relative coordinates after the user moves the face by the mapping compensation ratio to obtain a mouse pointer offset; the pointer moving unit determines a display position of the mouse simulation pointer on the screen used by the user based on the mouse pointer offset, and displays the mouse simulation pointer at the display position; A mouse operation execution unit, wherein the mouse operation execution unit is used to receive a user's mouse operation signal and execute a corresponding mouse operation based on the mouse operation signal, wherein the mouse operation signal includes a left-click operation, a left-button long-press operation, a right-click operation, and a scroll wheel sliding operation.

8. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the intelligent mouse auxiliary control method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the intelligent mouse auxiliary control method according to any one of claims 1 to 6.

Images