Gesture cursor mapping method based on machine vision, network device and storage medium

Abstract

The application provides a gesture cursor mapping method based on machine vision, a network device and a storage medium, wherein the gesture cursor mapping method based on machine vision comprises the following steps: acquiring a key point set of a body; the key point set comprises two shoulder end point coordinates of shoulders, and the two shoulder end point coordinates are a and b respectively; calculating a shoulder width D; acquiring a hand key point coordinate m; calculating a hand center point coordinate c according to the hand key point coordinate; calculating two diagonal point coordinates of a virtual frame according to the shoulder width D; obtaining a cursor mapping coordinate M of the hand key point on a projection screen based on the two diagonal point coordinates of the virtual frame, the hand key point coordinate m, and the length width and the high height of the projection screen. The size and the position of the virtual frame are added and generated by an algorithm, so that the cursor moving speed is self-adaptive at different distances, the operation range is smaller, the user fatigue is reduced, and the cursor moving speed is adjustable.

Description

Technical Field

[0001] This application relates to the field of communication equipment technology, and in particular to a machine vision-based gesture cursor mapping method, network equipment, and storage medium. Background Technology

[0002] With advancements in computer performance and AI, more interaction methods have emerged in the field of human-computer interaction. Compared to traditional methods such as mice, keyboards, touchscreens, and remote controls, gesture control based on machine vision will bring a completely new operating experience. In the field of large-screen terminals, gesture control has become an indispensable feature of high-end devices from various manufacturers.

[0003] In vision-based gesture interaction, simulating user gestures as mouse operations not only reduces the learning curve for users but also better integrates with the current software ecosystem, eliminating the need for third-party applications to adapt their gesture interaction methods. The most crucial aspect of simulating mouse operations with gestures is cursor mapping; its accuracy and stability are key to user experience. Currently, cursor mapping often uses direct mapping, which has the major problem of inconsistent cursor movement speed at different distances—the farther the distance, the slower the cursor moves. Furthermore, direct mapping requires larger user gestures, increasing user fatigue. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this application provides a machine vision-based gesture cursor mapping method, network device, and storage medium.

[0005] According to the first aspect of the embodiments of this application, a gesture cursor mapping method based on machine vision is provided.

[0006] Obtain a set of key points about the body;

[0007] The set of key points includes the coordinates of the two endpoints of the shoulder, which are a(x) and a(x) respectively. a ,y a ), b(x b ,y b );

[0008] Through the coordinates of the two endpoints of the shoulder a(x) a ,y a ), b(x b ,y b ), calculate shoulder width D;

[0009] Get the coordinates of the key points of the hand m(x) m ,y m );

[0010] Calculate the coordinates c(x) of the center point of the hand based on the coordinates of the key points of the hand. c ,y c );

[0011] Calculate the coordinates p(x) of the two diagonal points of the virtual frame based on the shoulder width D. p ,y p )=(x c -D / 2,y c -D / 2)q(x q ,y q )=(x c +D / 2,y c +D / 2);

[0012] Based on the virtual bounding box coordinates p and q, the obtained hand keypoint coordinates m, and the projection screen's width and height, the cursor mapping coordinates M(x, q) of the hand keypoints on the projection screen are obtained. M ,y M ), (x M ,y M )=(width*(x m -x p ) / (x q -x p ),height*(y m -y p ) / (y q -y p )).

[0013] Preferably, it also includes obtaining the cursor velocity gain, including

[0014] Calculate the current speed level to obtain the speed V = V min +(K-1) / (N-1)*(V max -V min );

[0015] gain = V / V max ;

[0016] Among them, V min V represents the minimum cursor movement speed supported by the system. max This represents the maximum cursor movement speed supported by the system, where N is the number of speed levels (N≥1), and the current speed level K∈[1,N];

[0017] The coordinates of the two diagonal points p(x) of the virtual box p ,y p )=(x t -gain*D / 2,y t -gain*D / 2)q(x q ,yq )=(x t +gain*D / 2,y t +gain*D / 2).

[0018] Preferably, it also includes

[0019] Determine the coordinates m(x) of key points on the hand m ,y m Is it within the virtual frame?

[0020] If it is not within the virtual frame, update the virtual frame;

[0021] If the cursor coordinates are calculated within a virtual bounding box.

[0022] Preferably, updating the virtual frame includes:

[0023] Scenario 1:

[0024] x m <x p When, offset = x p -x m ;x p =x m ;x q =x q -offset;

[0025] Scenario 2:

[0026] x m >x q When, offset = x m -x q ;x q =x m ;x p =x p +offset;

[0027] Scenario 3:

[0028] y m <y t When, offset = y p -y m ;y p =y m ;y q =y q -offset;

[0029] Scenario 4:

[0030] y m >y q When, offset = y m -y q ;yq =y m ;y p =y p +offset.

[0031] Preferably, before calculating the shoulder width, the coordinates a(x) of the two endpoints of the shoulder are... a ,y a ), b(x b ,y b Perform filtering.

[0032] The network device provided in the second aspect of this application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it performs the aforementioned machine vision-based gesture cursor mapping method.

[0033] The third aspect of this application provides a storage medium storing computer program instructions thereon, which, when executed by a processor, are used to implement the above-described machine vision-based gesture cursor mapping method.

[0034] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0035] In this embodiment, a virtual frame is added to perform scale conversion on the traditional direct mapping method. Unlike the direct mapping method, where the screen size and position are fixed, the virtual frame is generated by an algorithm. Therefore, it can achieve adaptive cursor movement speed at different distances, reduce user fatigue with smaller operation ranges, and allow for adjustable cursor movement speed.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the application, serve to explain the principles of this application.

[0038] Figure 1 This is a diagram illustrating the gesture interaction process;

[0039] Figure 2 This is a schematic diagram illustrating the acquisition of full-body keypoint coordinates using the MediaPipe Pose model in an embodiment of this application.

[0040] Figure 3 This is a schematic diagram illustrating the acquisition of key hand coordinates using the MediaPipe Hands model in an embodiment of this application.

[0041] Figure 4 This is a schematic flowchart of the cursor mapping method according to an embodiment of this application;

[0042] Figure 5 This is a schematic diagram illustrating the transformation relationship between the virtual frame and the projection screen in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the hardware framework of the network device in an embodiment of this application. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] First, let's introduce a relatively complete process of gesture interaction. This includes data acquisition, hand key point recognition, data preprocessing and control determination, key point to gesture conversion, command matching, smoothing filtering, cursor mapping, and system response, such as... Figure 1 As shown.

[0046] Specifically, data acquisition involves capturing images using a monocular camera, with the sampling rate determined by the camera's frame rate. Keypoint acquisition can be implemented using the Mediapipe framework, an open-source multimedia machine learning model application framework developed by Google. Hand keypoint recognition feeds the captured images into a hand keypoint detection model for detection; if a hand is detected, it outputs the keypoint coordinates. These keypoint coordinates are typically a set of coordinates, commonly 21, but the amount can be adjusted based on processing capabilities and specific needs. Data preprocessing and control authority determination preprocess the identified keypoint coordinates and determine control authority, outputting the set of keypoint coordinates for the hand with control. These coordinates are then input into the "keypoint-to-gesture" model. The keypoint-to-gesture model takes the output from the previous step and feeds it into a gesture classification model for inference. It outputs the name corresponding to the current gesture, such as FIVE, FIRST, or POINTER. Command matching, based on a command matching algorithm, outputs the matching operation command for the current gesture, such as MOVE, CLICK_DOWN / UP. Cursor smoothing (cursor debouncing) refers to filtering the coordinate information of key points. Cursor mapping (cursor positioning) refers to converting the coordinates of key points into the coordinates of the cursor on the projection screen. System response refers to processing instructions and positioning the cursor.

[0047] To address the problems existing in the background technology, embodiments of this application provide a gesture cursor mapping method based on machine vision, such as... Figure 4 As shown, it includes:

[0048] Obtain the set of keypoint coordinates for the body. In this embodiment, this is achieved using the MediaPipe Pose model within the MediaPipe framework described above. The MediaPipe Pose model is a high-fidelity body pose tracking model. It can deduce the coordinates of all 33 keypoints from a single frame. This includes the coordinates of the two shoulder endpoints, a(x...). a ,y a ), b(x b ,y b ), calculate shoulder width like Figure 2 As shown.

[0049] like Figure 3 As shown, obtain the coordinates m(x) of the key points of the hand. m ,y m The MediaPipe Hands model, part of the MediaPipe architecture, is used to obtain a set of hand keypoint coordinates. The MediaPipe Hands model is a high-fidelity hand and finger tracking model. Machine learning is used to infer the hand keypoint coordinates from a single frame; in this case, there are 21 keypoints. Each keypoint output by the MediaPipeHands model consists of x, y, and z coordinates. x and y are normalized to [0.0, 1.0] using the image width and height, respectively. z represents the coordinate depth with the wrist as the origin; the smaller the value, the closer the coordinate is to the camera. The size of z uses approximately the same scale as x.

[0050] Calculate the coordinates c(x) of the center point of the hand based on the coordinates of the key points of the hand. c ,y c It is calculated based on pre-determined algorithms, such as weighted or linear summation of hand key point coordinates.

[0051] Based on shoulder width D and the coordinates of the center point of the hand c(x) c ,y c Calculate the coordinates p(x) of the two diagonal points of the virtual bounding box. p ,y p )=(x c -D / 2,y c -D / 2), q(x) q ,y q )=(x c +D / 2,y c +D / 2), such as Figure 5 As shown.

[0052] Based on the virtual bounding box coordinates p and q, the obtained hand keypoint coordinates m, and the projection screen's width and height, the cursor mapping coordinates M(x, q) of the hand keypoints on the projection screen are obtained. M ,y M ), (x M ,y M )=(width*(x m -x p ) / (x q -x p ),height*(y m -y p ) / (y q -y p )),like Figure 5 As shown. This differs from the direct mapping method, i.e. (x... M ,y M ) = (width*x m height*y m This paper introduces a virtual frame cursor mapping method. First, the position of key points relative to the virtual frame is calculated, and then multiplied by the screen width and height to obtain the final coordinates on the screen. Clearly, when the size of the hand key points remains constant, a larger virtual frame results in a slower cursor movement speed, while a smaller virtual frame results in a faster cursor movement speed. The virtual frame is generated based on the user's body and hand key points. When the user is close to the camera, the user occupies a larger proportion of the camera's view, resulting in a larger virtual frame; when the user is far from the camera, the user occupies a smaller proportion of the camera's view, resulting in a smaller virtual frame. Therefore, cursor mapping based on the virtual frame can achieve a consistent cursor movement speed at different distances. Furthermore, the generation of the virtual frame depends on shoulder width and the center position of the hand, allowing users to control the cursor in any corner of the screen with sufficiently small movements during operation, thereby reducing fatigue from gesture interaction.

[0053] This application embodiment also includes obtaining the cursor speed gain after calculating the shoulder width D, as follows: including calculating the current speed level to obtain the speed V = V min +(K-1) / (N-1)*(V max -V min gain = V / V max Among them, V min V represents the minimum cursor movement speed supported by the system. max This represents the maximum cursor movement speed supported by the system, where N is the number of speed levels (N≥1), and the current speed level K∈[1,N]; the coordinates of the two diagonal points of the virtual bounding box are p(x). p ,y p )=(x t -gain*D / 2,yt -gain*D / 2)q(x q ,y q )=(x t +gain*D / 2,y t +gain*D / 2). The cursor movement speed is adjustable through the speed gain (gain).

[0054] This application embodiment also includes determining the coordinates m(x) of key hand points. m ,y m Check if the cursor is within the virtual bounding box; if not, update the virtual bounding box; if within the virtual bounding box, calculate the cursor mapping coordinates. Methods for updating the virtual bounding box include:

[0055] Case 1: x m <x p When, offset = x p -x m ;x p =x m ;x q =x q -offset;

[0056] Case 2: x m >x q When, offset = x m -x q ;x q =x m ;x p =x p +offset;

[0057] Case 3: y m <y t When, offset = y p -y m ;y p =y m ;y q =y q -offset;

[0058] Case 4: y m >y q When, offset = y m -y q ;y q =y m ;y p =y p +offset.

[0059] When the cursor moves to the screen boundary, the virtual bounding box will automatically adjust, eliminating the error of the cursor moving out of the virtual bounding box. This adaptive adjustment, compared to simply setting the cursor directly to the screen boundary after determining whether it has moved outside the virtual bounding box, effectively solves the damping problem when returning from the boundary.

[0060] Before calculating the shoulder width D in this embodiment, the coordinates a(x) of the two endpoints of the shoulder are... a ,y a ), b(x b ,y b Perform filtering processing. This can be data filtering methods such as smoothing filtering or wavelet filtering.

[0061] The second aspect of this application also provides a network device, such as... Figure 6 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it performs the aforementioned machine vision-based gesture cursor mapping method. It also acquires images captured by a camera, such as those from a computer or iPad.

[0062] The third aspect of this application also provides a storage medium storing computer program instructions thereon, which, when executed by a processor, are used to implement the above-described machine vision-based gesture cursor mapping method.

[0063] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0064] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A machine vision-based gesture cursor mapping method, characterized in that, Obtain the set of key points of the body; The set of key points includes the coordinates of the two endpoints of the shoulder, which are a(x) and a(x) respectively. a ,y a ), b(x b ,y b ); Through the coordinates of the two endpoints of the shoulder a(x) a ,y a ), b(x b ,y b ), calculate shoulder width D; Get the coordinates of the key points of the hand m(x) m ,y m ); Calculate the coordinates c(x) of the center point of the hand based on the coordinates of the key points of the hand. c ,y c ); Calculate the coordinates p(x) of the two diagonal points of the virtual frame based on the shoulder width D. p ,y p )=(x c -D / 2,y c -D / 2)q(x q ,y q )=(x c +D / 2,y c +D / 2); Based on the virtual bounding box coordinates p and q, the obtained hand keypoint coordinates m, and the projection screen's width and height, the cursor mapping coordinates M(x, q) of the hand keypoints on the projection screen are obtained. M ,y M ), (x M ,y M )=(width*(x m -x p ) / (x q -x p ),height*(y m -y p ) / (y q -y p )).

2. The machine vision-based gesture cursor mapping method according to claim 1, characterized in that, It also includes obtaining the cursor velocity gain, including Calculate the current speed level to obtain the speed V = V min +(K-1) / (N-1)*(V max -V min ); gain=V / V max ; Among them, V min V represents the minimum cursor movement speed supported by the system. max This represents the maximum cursor movement speed supported by the system, where N is the number of speed levels (N≥1), and the current speed level K∈[1,N]; The coordinates of the two diagonal points p(x) of the virtual frame p ,y p )=(x t -gain*D / 2,y t -gain*D / 2)q(x q ,y q )=(x t +gain*D / 2,y t +gain*D / 2).

3. The machine vision-based gesture cursor mapping method according to claim 1 or 2, characterized in that, Also includes Determine the coordinates m(x) of key points on the hand m ,y m Is it within the virtual frame? If it is not within the virtual frame, update the virtual frame; If the cursor coordinates are calculated within a virtual bounding box.

4. The machine vision-based gesture cursor mapping method according to claim 3, characterized in that, The updated virtual frame includes: Scenario 1: x m <x p When, offset = x p -x m ;x p =x m ;x q =x q -offset; Scenario 2: x m >x q When, offset = x m -x q ; x q = x m ; x p = x p + offset; Scenario 3: y m <y t When, offset = y p -y m ; y p = y m ; y q = y q -offset; Scenario 4: y m >y q When, offset = y m -y q ; y q = y m ; y p = y p + offset.

5. The machine vision-based gesture cursor mapping method according to claim 1, characterized in that, Before calculating shoulder width, the coordinates of the two endpoints of the shoulder, a(x) a ,y a ), b(x b ,y b Perform filtering.

6. A network device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program by performing a machine vision-based gesture cursor mapping method as described in any one of claims 1-5.

7. A storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the machine vision-based gesture cursor mapping method according to any one of claims 1-5.

Images