A mouse control method, medium, and device based on head pose recognition

Abstract

This invention discloses a mouse control method, medium, and device based on head posture recognition. It acquires a facial image and collects key feature points of the zygomatic arch, eyebrows, nose tip, and mouth. Based on the geometric relationship of these key feature points, it determines the head posture movement data in the horizontal and vertical directions. Based on the vector relationship of this movement data in a pre-established Cartesian coordinate system of the facial image, it calculates the mouse offset and controls the cursor movement accordingly. Based on the key feature point of the mouth, within a continuous fixed frame cycle, a mouse click event is triggered when the distance difference between the two corners of the mouth remains consistently greater than a set distance threshold. Advantages: Applicable to a contactless scenario, especially in the field of accessibility applications, it strengthens the solution to the problem of inconvenience for disabled users and has a certain degree of fun, with a broad market potential.

Description

Technical Field

[0001] This invention relates to a mouse control method, medium, and device based on head posture recognition, belonging to the technical field of biometrics and human-computer interaction technology. Background Technology

[0002] With the rapid development of biometric and human-computer interaction technologies, facial recognition technology has matured, and human-computer interaction methods are becoming increasingly user-friendly and intelligent. Traditional human-computer interaction methods typically rely on hands to control mouse movement and clicking. While this method is widely used, it significantly impacts the human-computer interaction experience for users with arm disabilities who cannot operate the mouse with their hands. Therefore, a mouse control method based on head posture recognition would be more convenient and faster. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a mouse control method, medium and device based on head posture recognition.

[0004] To address the aforementioned technical problems, this invention provides a mouse control method based on head posture recognition, comprising:

[0005] Acquire facial images and collect key feature points such as the cheekbone arch, eyebrows, tip of the nose, and mouth;

[0006] Based on the geometric relationship of key feature points of the zygomatic arch, eyebrows and nose tip, the movement data of the head posture in the horizontal and vertical directions are determined. Based on the vector relationship of the movement data in the pre-established Cartesian coordinate system of the face image, the mouse offset is calculated, and the movement of the cursor is controlled according to the mouse offset.

[0007] Based on the key feature points of the mouth, a mouse click event is triggered when the distance difference between the two corners of the mouth remains greater than a set distance threshold within a cycle of continuous fixed frames.

[0008] Furthermore, the acquisition of facial images, including the collection of key feature points such as the zygomatic arch, eyebrows, tip of the nose, and mouth, includes:

[0009] Key facial feature points are obtained using image recognition technology and a pre-constructed Cartesian coordinate system for facial images;

[0010] The rectangular coordinate system is a coordinate system with the top left corner of the screen as the origin, the horizontal x-axis of the screen from left to right, and the vertical y-axis of the screen from top to bottom;

[0011] The key facial feature points include: the most prominent point of the left zygomatic arch F1(x,y), the most prominent point of the right zygomatic arch F2(x,y), the outermost point of the left eyebrow F3(x,y), the outermost point of the right eyebrow F4(x,y), the outermost point of the left corner of the mouth F5(x,y), the outermost point of the right corner of the mouth F6(x,y), the tip of the nose F7(x,y), and 12 key feature points of the eyes.

[0012] Furthermore, the calculation of the mouse offset includes:

[0013] By utilizing the positional relationship between the most prominent point F1(x,y) of the left zygomatic arch, the most prominent point F2(x,y) of the right zygomatic arch, and the tip of the nose F7(x,y), the rotation of the head in the horizontal direction is determined, and the offset F of the mouse in the horizontal x-axis direction is determined. x , represented as:

[0014] F x =K0·(F 1x +F 2x -2F 7x ) / (F 1x -F 2x )

[0015] By using the positional relationship between the outermost points F3(x,y) and F4(x,y) of the left and right eyebrows and the outermost points F5(x,y) and F6(x,y) of the left and right corners of the mouth, the vertical rotation of the head is determined, and the offset F of the mouse in the vertical y-axis direction is determined. y :

[0016] F y =K1·(F 5y +F 6y -F 3y -F 4y )

[0017] F 1x F represents the x-axis coordinate of the most prominent point of the left zygomatic arch. 2x F represents the x-axis coordinate of the most prominent point of the right zygomatic arch. 7x F represents the x-axis coordinate of the tip of the nose. 3y F represents the y-axis coordinate of the outermost point of the left eyebrow. 4y F represents the y-axis coordinate of the outermost point of the right eyebrow. 5y F represents the y-axis coordinate of the outermost point of the left corner of the mouth. 6y K0 represents the y-axis coordinate of the outermost point of the right corner of the mouth, and K1 represents the mouse movement sensitivity in the horizontal x-axis and vertical y-axis directions of the screen, respectively.

[0018] Furthermore, the key facial feature points also include key eye feature points, including:

[0019] P, outer corner of right eye 11 (n,y), P, slightly to the right of the center of the upper eyelid 12 (x,y), P, slightly to the left of the center of the upper eyelid 13 (x,y), inner corner of eye P 14 (x,y), P, slightly to the left of the center of the lower eyelid 15 (x,y), P, slightly to the right of the center of the lower eyelid 16 (x,y); P, the outer corner of the left eye 21 (x,y), P, slightly to the left of the center of the upper eyelid 22 (x,y), P, slightly to the right of the center of the upper eyelid 23 (x,y), inner corner of eye P 24 (x,y), P, slightly to the right of the center of the lower eyelid 25 (x,y), P, slightly to the left of the center of the lower eyelid 26 (x,y).

[0020] Furthermore, it also includes:

[0021] Calculate the aspect ratio E of the eye based on key feature points of the eye, and determine whether the eye is open based on the aspect ratio;

[0022] Within a preset cycle, the number of blinks is detected based on whether both eyes are open. If the number of blinks reaches a preset threshold, the cursor coordinates are initialized; otherwise, the cursor remains in its current position. The initial position of the cursor coordinates is the center of the screen.

[0023] Furthermore, the formula for calculating the aspect ratio E is as follows:

[0024]

[0025] Pn1 y P represents the y-axis coordinate value of the outer corner of the eye. n2y P represents the y-axis coordinate value slightly to the right of the center of the upper eyelid. n3y P represents the y-axis coordinate value of the upper eyelid center slightly to the left. n4y P represents the y-axis coordinate value of the inner corner of the eye. n5y P represents the y-axis coordinate value of the lower eyelid center slightly to the left. n6y This represents the y-axis coordinate value slightly to the right of the center of the lower eyelid, where n = 1 or 2, used to represent either the right or left eye.

[0026] Furthermore, the calculation process for the distance difference between the two corners of the mouth includes:

[0027] The distance between the outermost point F5(x,y) on the left corner of the mouth and the outermost point F6(x,y) on the right corner of the mouth is denoted as L1, and is expressed as:

[0028] L1 = F 5x -F 6x

[0029] When the corners of the mouth turn upwards, the distance between the two points is denoted as L2, and expressed as:

[0030] L2=F′ 5x -F′ 6x

[0031] The relative change between L2 and L1 is denoted as the distance difference L3 between the two corners of the mouth, and is expressed as:

[0032] L3 = (L2 - L1) / L2 + L1).

[0033] Furthermore, a function selection page for left-click, double-click, and right-click is provided, which is selected by triggering a mouse click event. The default initial click state is left-click; when double-click is selected, the double-click command is executed in the first cycle, and the initial click state is restored in the second cycle; when right-click is selected, the right-click command is executed in the first cycle, and the initial click state is restored in the second cycle.

[0034] A computer-readable storage medium for storing one or more programs, characterized in that the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described.

[0035] A computing device, comprising,

[0036] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described.

[0037] The beneficial effects achieved by this invention are as follows:

[0038] The present invention discloses a mouse control method based on head posture recognition, which is mainly applied to a contactless scenario, especially in the field of barrier-free application. It strengthens the solution to the problem of inconvenience for disabled people, and is also interesting. It is also convenient for normal people to use for entertainment and games. Moreover, this system has greatly improved its performance in terms of speed and convenience compared with traditional mouse operation, and has a broad market for development. Attached Figure Description

[0039] Figure 1 This is a flowchart of a mouse control method based on head posture recognition provided by the present invention;

[0040] Figure 2 This is a schematic diagram of key facial feature points excluding the eyes in this invention;

[0041] Figure 3 This is a schematic diagram illustrating the determination of the corner of the mouth in this invention;

[0042] Figure 4 This is a schematic diagram of the judgment of the eye part in this invention (taking the right eye as an example). Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0044] A mouse control method based on head posture recognition uses image recognition technology to obtain key feature points such as the zygomatic arch, nose tip, eyebrows, and corners of the mouth in a face image. By judging the changes in their geometric relationships and combining the additional offset calculated by the relationship between the head posture movement information and the changes in the coordinates of the eyeball center point, the mouse movement is realized and the mouse click function is triggered.

[0045] Figure 1 A flowchart of a mouse control method based on head pose recognition provided by the present invention is shown. It includes the following steps:

[0046] Step S1: Use image recognition technology to obtain key feature points of the face image. The key points used include the most prominent point of the left zygomatic arch F1(x,y), the most prominent point of the right zygomatic arch F2(x,y), the outermost point of the left eyebrow F39(x,y), the outermost point of the right eyebrow F4(x,y), the outermost point of the left corner of the mouth F5(x,y), the outermost point of the right corner of the mouth F6(x,y), the tip of the nose F7(x,y), and 12 key feature points of the eyes.

[0047] The 12 key feature points of the eye area include: corresponding to the outer corner of the right eye P 11 (x,y), P, slightly to the right of the center of the upper eyelid 12 (x, y0, P, center of upper eyelid slightly to the left) 13 (x,y), inner corner of eye P 14 (x,y), P, slightly to the left of the center of the lower eyelid 15 (x,y), P, slightly to the right of the center of the lower eyelid 16 (x, y); corresponding to the outer corner of the left eye P respectively. 21 (x,y), P, slightly to the left of the center of the upper eyelid 22 (x,y), P, slightly to the right of the center of the upper eyelid 23 (x, y0, inner corner of eye P) 24 (x, y0, P, center of lower eyelid slightly to the right) 25 (x,y), P, slightly to the left of the center of the lower eyelid 26 (x, y). Specific location as follows: Figure 2 , Figure 3 , Figure 4 As shown.

[0048] Step S2: Based on the geometric relationship between the most prominent points of the left and right cheekbone arches, the outermost points of the left and right eyebrows and the tip of the nose, determine the movement of the head in the horizontal and vertical directions, and calculate the offset of the mouse in the x and y axes by combining the positioning of the center point of the eyeball.

[0049] Mouse offset F in the horizontal x-axis direction x , represented as:

[0050] F x =K0·(F 1x +F 2x -2F 7x ) / (F 1x -F 2x )

[0051] Mouse offset F along the vertical y-axis y , represented as:

[0052] F y =K1·(F 5y +F 6y -F 3y -F 4y )

[0053] F 1x F represents the x-axis coordinate of the most prominent point of the left zygomatic arch. 2x F represents the x-axis coordinate of the most prominent point of the right zygomatic arch. 7x F represents the x-axis coordinate of the tip of the nose. 3y F represents the y-axis coordinate of the outermost point of the left eyebrow. 4y F represents the y-axis coordinate of the outermost point of the right eyebrow. 5y F represents the y-axis coordinate of the outermost point of the left corner of the mouth. 6y This represents the y-axis coordinate of the outermost point of the right corner of the mouth. K0 and K1 represent the mouse movement sensitivity along the horizontal x-axis and vertical y-axis of the screen, respectively.

[0054] Step S3: Record the relative change relationship of the corner of the mouth as L3. If L3 is greater than the threshold L0, it means that there is a corner of the mouth raising action, otherwise there is no such action; acquire the face image of each frame, and in a cycle of 5 frames, if the action is captured in all 5 frames, a mouse click event is triggered, otherwise it is not triggered. The threshold L0 can be taken as 0.05cm.

[0055] The distance between the outermost point F5(x,y) on the left corner of the mouth and the outermost point F6(x,y) on the right corner of the mouth is denoted as L1, and is expressed as:

[0056] L1 = F 5x -F 6x

[0057] When the corners of the mouth turn upwards, the distance between the two points is denoted as L2, and expressed as:

[0058] L2=F′ 5x -F′ 6x

[0059] The relative change between L2 and L1 is denoted as the distance difference L3 between the two corners of the mouth, and is expressed as:

[0060] L3 = (L2 - L1) / L2 + L1).

[0061] Step S4: Use QT components to set up multiple click function page options. Initially, keep the "left click" function, and you can choose from three states: "left click", "double click" and "right click". After making a selection, repeat step S3.

[0062] Step S5: If the "left click" function is selected, this state will be maintained in each cycle; if the "double click" or "right click" function is selected, after being triggered in the first cycle, it will return to the "left click" state starting from the second cycle.

[0063] Specifically, in combination Figure 2 The schematic diagram of the present invention does not include key facial feature points of the eye area.

[0064] Points F1, F2, and F7 form a line, with F7 as the midpoint. The distance between F1 and F7 is approximately the same as the distance between F2 and F7, so the offset range in the x-axis direction is the same. The total length of the line formed by these three points is fixed. When moving left, the distance on the left side decreases significantly, and the distance on the right side increases significantly; similarly, when moving right again, the distance on the left side increases significantly, and the distance on the right side decreases significantly, thus achieving mouse movement in the x-axis direction.

[0065] Points F3, F4, F5, and F6 form a quadrilateral. The distance between points F3 and F4 and between points F5 and F6 are generally constant. Therefore, by combining the changes in the distance between points F3 and F5 and between points F4 and F6, the mouse can be moved along the y-axis.

[0066] Specifically, in combination Figure 3The schematic diagram of the judgment regarding the corner of the mouth in this invention shows that when the mouth is kept in a normal closed state, the distance between points F5 and F6 at the corner of the mouth is generally maintained at 6cm. If the user consciously raises the corner of the mouth, the distance between points F′5 and F′6 at the corner of the mouth is generally maintained at 7cm. However, considering the different degrees of mouth corner raising between individuals, the trigger threshold can be set to 0.05cm based on the relative change of the mouth before and after. In addition, in order to exclude such brief unconscious mouth corner raising behavior, a method is adopted to determine whether the mouth corner is continuously raised in a cycle of five image frames, so as to accurately trigger the mouse click event.

[0067] Specifically, in combination Figure 4 The schematic diagram of the eye part judgment in this invention (taking the right eye as an example) uses the aspect ratio of the eye to judge blinking. An individual normally blinks 10-15 times per minute, and each blink takes about 0.3 seconds. 24 frames are captured in 1 second. Therefore, if every 24 frames is a cycle, an individual can consciously blink 3 times or more to initialize the cursor position. This judgment method can minimize the occurrence of unconscious blinking.

[0068] The formula for calculating the aspect ratio E is as follows (taking the right eye as an example, and the same applies to the left eye):

[0069]

[0070] P 11y P represents the y-axis coordinate of the outer corner of the right eye. 12y P represents the y-axis coordinate value of the right upper eyelid center, slightly to the right. 13y P represents the y-axis coordinate value of the right upper eyelid center, slightly to the left. 14y P represents the y-axis coordinate of the inner corner of the right eye. 15y P represents the y-axis coordinate value of the lower right eyelid center, slightly to the left. 16y This represents the y-axis coordinate value of the lower eyelid of the right eye, slightly to the right of its center.

[0071] Since mouse movement affects head movement, a blinking initialization function is implemented to prevent excessive head movement. This function allows the user to return to the center of the screen at any time, based on their control needs.

[0072] Accordingly, the present invention also provides a computer-readable storage medium for storing one or more programs, characterized in that the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described.

[0073] Accordingly, the present invention also provides a computing device, comprising,

[0074] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described.

[0075] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A mouse control method based on head pose recognition, characterized by, The method comprises the following steps: acquiring a face image, and collecting cheekbone, eyebrow, nose tip and mouth key feature points; judging head posture movement data in horizontal and vertical directions according to geometric relations of the cheekbone, eyebrow and nose tip key feature points, calculating mouse offset according to vector relations of the movement data on a pre-established face image rectangular coordinate system, and controlling cursor movement according to the mouse offset; when a distance difference between two sides of the mouth is greater than a set distance threshold, triggering a mouse click event once in a cycle with consecutive fixed frames as a loop; the acquiring of the face image and the collecting of the cheekbone, eyebrow, nose tip and mouth key feature points comprise the following steps: acquiring face key feature points by using image recognition technology and a pre-established face image rectangular coordinate system; the rectangular coordinate system is a coordinate system with a left top corner of a screen as an origin, a screen from left to right as a horizontal x-axis, and a screen from top to bottom as a vertical y-axis; The human face key feature points include: left side zygomatic arch most protruding point , right side zygomatic arch most protruding point , left eyebrow outermost point , right eyebrow outermost point , left corner of the mouth outermost point , right corner of the mouth outermost point , nose tip point and 12 key feature points of eye part; the calculating of the mouse offset comprises the following steps: Utilizing the most prominent point of the left zygomatic arch The most prominent point of the right zygomatic arch With the tip of the nose Based on the positional relationship, determine the head's horizontal rotation and the mouse's horizontal position. Offset F in the axial direction x , is represented as: F x = K0·(F 1x + F 2x - 2F 7x ) / ( F 1x - F 2x ) Use the outermost points of both eyebrows and and the outer corners of the left and right mouth and Based on the positional relationship, determine the vertical rotation of the head and the vertical movement of the mouse. Offset F in the axial direction y : F y = K1 · (F 5y + F 6y - F 3y - F 4y ) F 1x x-axis coordinate value of the most protruding point of the left zygomatic arch, F 2x x-axis coordinate value of the most protruding point of the right zygomatic arch, F 7x x-axis coordinate value of the tip of the nose, F 3y y-axis coordinate value of the most outer point of the left eyebrow, F 4y y-axis coordinate value of the most outer point of the right eyebrow, F 5y y-axis coordinate value of the most outer point of the left corner of the mouth, F 6y y-axis coordinate value of the most outer point of the right corner of the mouth, and respectively represent the mouse movement sensitivity in the horizontal x-axis and vertical y-axis directions of the screen.

2. The head-pose-recognition-based mouse control method according to claim 1, wherein, the face key feature points further comprise eye key feature points, and the method further comprises the following steps: outer corner of the right eye , upper eyelid center is right biased , upper eyelid center is left biased , inner corner of the eye , lower eyelid center is left biased , lower eyelid center is right biased ; outer corner of the left eye , upper eyelid center is left biased , upper eyelid center is right biased , inner corner of the eye , lower eyelid center is right biased , lower eyelid center is left biased . 3.The head-pose-recognition-based mouse control method according to claim 2, wherein, the method further comprises the following steps: calculating an aspect ratio E of the eye part according to the eye key feature points, and judging whether the eyes are open according to the aspect ratio; in a preset cycle, detecting whether a blink count reaches a preset count threshold according to whether the eyes are open, and if yes, initializing cursor coordinates; if not, the cursor remains at a current position; the initialized position of the cursor coordinates is a screen center.

4. The head-pose-identification-based mouse control method according to claim 3, wherein, a calculation formula of the aspect ratio E is as follows: ； P n1y y-coordinate value of the outer corner of the eye, P n2y y-coordinate value of the center of the upper eyelid shifted to the right, P n3y y-coordinate value of the center of the upper eyelid shifted to the left, P n4y y-coordinate value of the inner corner of the eye, P n5y y-coordinate value of the center of the lower eyelid shifted to the left, P n6y y-coordinate value of the center of the lower eyelid shifted to the right, n = 1 or 2, indicating the right eye or the left eye. 5.The head-pose-recognition-based mouse control method according to claim 1, wherein, the calculating of the distance difference between the two sides of the mouth comprises the following steps: the distance between the left corner of the mouth lateral point and the right corner of the mouth lateral point is denoted as , and is expressed as: ； When the mouth corner up action, the distance between the two points at this time is recorded as , expressed as: ； The relative change between and is noted as the difference in distance of the corners of the mouth , expressed as: 。 6. The head posture recognition-based mouse control method according to claim 1, wherein a function selection page with left click, double click and right click is provided, and selection is performed by triggering a mouse click event; a default initial click state is left click; when double click is selected, a double click command is executed in a first cycle, and the initial click state is restored in a second cycle; when right click is selected, a right click command is executed in a first cycle, and the initial click state is restored in a second cycle.

7. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method of any of claims 1-6. The one or more programs include instructions that when executed by a computing device, cause the computing device to perform any of the methods of claims 1-6.

8. A computing device, comprising: The method comprises the following steps: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any of the methods of claims 1-6.

Images