Gesture cursor mapping method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a gesture cursor mapping method and device, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: obtaining a space coordinate of each key point in a plurality of key points of a hand in a current frame image in a space rectangular coordinate system, the plurality of key points including a wrist key point and at least two target key points, each target key point is a root key point of a finger except a thumb, and the original point of the space rectangular coordinate system is a wrist key point; based on the space coordinate of each key point, determining a target yaw angle and a target pitch angle of the gesture posture in the current frame image; determining coordinates of a cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relation; the target plane coordinate is determined according to the plane coordinate of the at least one target key point. The recognition efficiency of determining cursor movement according to the change of the gesture posture can be improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a gesture cursor mapping method, device, electronic device, storage medium and vehicle. Background Art

[0002] At present, there are more and more applications for controlling the movement of the cursor of electronic devices based on user gestures. In the existing technology, user gesture recognition is based on a two-dimensional plane to construct the coordinates of the key points of the user's hand, and then convert them based on the coordinates of the key points of the hand to obtain the coordinates of the cursor of the electronic device screen. However, due to the two-dimensional plane, when the user's gesture posture has a small change, it is impossible to sensitively capture the change of the coordinates of the key points of the hand, and then it is impossible to respond to the change of the gesture posture to obtain the change of the cursor coordinates. Therefore, the recognition efficiency of determining the cursor movement based on the change of gesture posture is very low, and the user experience is poor. Summary of the invention

[0003] The present application provides a gesture cursor mapping method, device, electronic device, storage medium and vehicle, which can improve the recognition efficiency of determining cursor movement based on changes in gesture posture.

[0004] In a first aspect, an embodiment of the present application provides a gesture cursor mapping method, comprising: obtaining the spatial coordinates of each of a plurality of key points of a hand in a current frame image in a spatial rectangular coordinate system, the plurality of key points comprising a wrist key point and at least two target key points, each target key point being a root key point of a finger other than a thumb, and the origin of the spatial rectangular coordinate system being the wrist key point; determining a target yaw angle and a target pitch angle of a gesture posture in the current frame image based on the spatial coordinates of each key point; determining the coordinates of a cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; the target plane coordinates are determined based on the plane coordinates of at least one target key point, the plane coordinates of each target key point being the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship being a mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

[0005] In some embodiments of the present application, the multiple key points include a wrist key point, a first target key point, and a second target key point; based on the spatial coordinates of each key point, determining the target yaw angle and the target pitch angle of the gesture posture in the current frame image, including: determining the initial yaw angle of the gesture posture based on a first angle between a projection of a first vector on a horizontal plane of the spatial rectangular coordinate system and the target vector, the first vector being a vector determined by the first target key point and the second target key point, and the direction of the first vector is from the first target key point to the second target key point; the target direction The quantity is a vector on the common coordinate axis of the first vertical plane and the horizontal plane of the spatial rectangular coordinate system and pointing to the positive direction of the common coordinate axis, and the first vertical plane is parallel to the horizontal axis of the screen; based on the second angle between the projection of the second vector on the first vertical plane and the target vector, the initial pitch angle of the gesture posture is determined, the second vector is a vector determined by the wrist key point and the first target key point, and the direction of the second vector is from the wrist key point to the first target key point; based on the initial yaw angle and the initial pitch angle, the target yaw angle and the target pitch angle are determined.

[0006] In some embodiments of the present application, the multiple key points include wrist key points and four target key points; the initial yaw angle of the gesture posture is determined based on the first angle between the projection of the first vector on the horizontal plane of the spatial rectangular coordinate system and the target vector, including: taking two key points in each key point combination in the multiple key point combinations as the first target key point and the second target key point, respectively, and determining the first angle between the projection of the first vector on the horizontal plane and the target vector to determine multiple first angles, and the multiple key point combinations are obtained by combining different target key points in the four target key points; the average value of the multiple first angles is determined as the initial yaw angle; the initial pitch angle of the gesture posture is determined based on the projection of the second vector on the first vertical plane and the second angle of the target vector, including: taking multiple key points in the four target key points as the first target key points, respectively, and determining the second angle between the projection of the second vector on the first vertical plane and the target vector to determine multiple second angles; the average value of the multiple second angles is determined as the initial pitch angle.

[0007] In some embodiments of the present application, the target yaw angle and the target pitch angle are determined based on the initial yaw angle and the initial pitch angle, including: filtering the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle.

[0008] In some embodiments of the present application, filtering the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle includes: determining a state matrix and a state transfer matrix in a Kalman filter formula based on a first angle formula, a first angular velocity formula, a second angle formula, and a second angular velocity formula;

[0009] Based on the state matrix, the state transfer matrix, the preset observation matrix, the preset state transfer process noise matrix and the preset system observation noise matrix, and the Kalman filter iteration principle formula, the initial yaw angle and the initial pitch angle are subjected to Kalman filter iteration processing to obtain the target yaw angle and the target pitch angle;

[0010] Among them, the first angle formula is:

[0011] The second angle formula is:

[0012] The first angular velocity formula is: ω 1t =ω 1(t-1) +a 1(t-1) Δt;

[0013] The first angular velocity formula is: ω 2t =ω 2(t-1) +a 2(t-1) Δt;

[0014] The Kalman filter formula is:

[0015] The state matrix is:

[0016] The state transfer matrix is:

[0017] The default observation matrix is:

[0018] The noise matrix of the preset state transfer process is:

[0019] The preset system observation noise matrix is:

[0020] The Kalman filter iteration principle formula includes:

[0021] Among them, θ 1t is the initial yaw angle at time t, θ 1(t-1) is the target yaw angle at time t-1, w 1t is the angular velocity in the yaw direction at time t, w 1(t-1)is the angular velocity in the yaw direction at time t-1, Δt is the time change between time t and time t-1, a 1(t-1) is the acceleration in the yaw angle direction at time t-1; θ 2t is the initial pitch angle at time t, θ 2(t-1) is the target pitch angle at time t-1, w 2t is the angular velocity in the pitch direction at time t, w 2(t-1) is the angular velocity in the pitch direction at time t-1, a 2(t-1) is the acceleration in the pitch angle direction at time t-1; X t is the observation state matrix at time t, is the predicted state matrix at time t, is the optimized state matrix at time t-1, B t-1 is the control matrix, is the optimal state covariance at time t-1, is the predicted state covariance at time t, K t is the Kalman fusion coefficient at time t, and I is the unit matrix with a diagonal of 1.

[0022] In some embodiments of the present application, the preset mapping relationship is:

[0023] x=θ1·S1·λ1+x0·(1-λ1), y=θ2·S2·λ2+y0·(1-λ2)

[0024] Among them, (x0, y0) is the plane coordinate of the target, (x, y) is the coordinate of the cursor, θ1 is the target yaw angle, θ2 is the target pitch angle, S1 is the preset mapping sensitivity corresponding to the target yaw angle, S2 is the preset mapping sensitivity corresponding to the target pitch angle, λ1 and λ2 are cursor adjustment parameters, and 0<λ1≤1, 0<λ2≤1.

[0025] In some embodiments of the present application, before determining the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and the preset mapping relationship, the method further includes: when entering the drag gesture control state is detected, adjusting λ1 from the first adjustment parameter to the second adjustment parameter, and adjusting λ2 from the third adjustment parameter to the fourth adjustment parameter;

[0026] When exiting the drag gesture control state is detected, λ1 is adjusted from the second adjustment parameter to the first adjustment parameter, and λ2 is adjusted from the fourth adjustment parameter to the third adjustment parameter; wherein the first adjustment parameter is smaller than the second adjustment parameter, and the third adjustment parameter is smaller than the fourth adjustment parameter.

[0027] In a second aspect, an embodiment of the present application provides a gesture cursor mapping device, comprising: an acquisition module, used to acquire the spatial coordinates of each of a plurality of key points of a hand in a current frame image in a spatial rectangular coordinate system, the plurality of key points comprising a wrist key point and at least two target key points, each target key point being a root key point of a finger other than a thumb, and the origin of the spatial rectangular coordinate system being the wrist key point; a determination module, used to determine a target yaw angle and a target pitch angle of a gesture posture in the current frame image based on the spatial coordinates of each key point; the determination module is further used to determine the coordinates of a cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; the target plane coordinates are determined according to the plane coordinates of at least one target key point, the plane coordinates of each target key point being the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship being a mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

[0028] In some embodiments of the present application, the multiple key points include a wrist key point, a first target key point and a second target key point; the determination module is specifically used to determine the initial yaw angle of the gesture posture based on a first angle between a projection of a first vector on a horizontal plane of the spatial rectangular coordinate system and a target vector, the first vector being a vector determined by the first target key point and the second target key point, and the direction of the first vector is from the first target key point to the second target key point; the target vector is a vector on a common coordinate axis of a first vertical plane and the horizontal plane of the spatial rectangular coordinate system, and pointing to a positive direction of the common coordinate axis, and the first vertical plane is parallel to the horizontal axis of the screen; the initial pitch angle of the gesture posture is determined based on a second angle between a projection of the second vector on the first vertical plane and the target vector, the second vector is a vector determined by the wrist key point and the first target key point, and the direction of the second vector is from the wrist key point to the first target key point; based on the initial yaw angle and the initial pitch angle, the target yaw angle and the target pitch angle are determined.

[0029] In some embodiments of the present application, the multiple key points include a wrist key point and four target key points; the determination module is specifically used to use two key points in each key point combination in the multiple key point combinations as the first target key point and the second target key point, respectively, to determine the first angle between the projection of the first vector on the horizontal plane and the target vector to determine multiple first angles, and the multiple key point combinations are obtained by combining different target key points among the four target key points; the average value of the multiple first angles is determined as the initial yaw angle; the determination module is specifically used to use multiple key points among the four target key points as the first target key points, respectively, to determine the second angle between the projection of the second vector on the first vertical plane and the target vector to determine multiple second angles; the average value of the multiple second angles is determined as the initial pitch angle.

[0030] In some embodiments of the present application, the determination module is specifically used to filter the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle.

[0031] In some embodiments of the present application, the filtering processing module is specifically used to determine the state matrix and the state transfer matrix in the Kalman filter formula based on the first angle formula, the first angular velocity formula, the second angle formula, and the second angular velocity formula;

[0032] Based on the state matrix, the state transfer matrix, the preset observation matrix, the preset state transfer process noise matrix and the preset system observation noise matrix, and the Kalman filter iteration principle formula, the initial yaw angle and the initial pitch angle are subjected to Kalman filter iteration processing to obtain the target yaw angle and the target pitch angle;

[0033] Among them, the first angle formula is:

[0034] The second angle formula is:

[0035] The first angular velocity formula is: ω 1t =ω 1(t-1) +a 1(t-1) Δt;

[0036] The first angular velocity formula is: ω 2t =ω 2(t-1) +a 2(t-1) Δt;

[0037] The Kalman filter formula is:

[0038] The state matrix is:

[0039] The state transfer matrix is:

[0040] The default observation matrix is:

[0041] The noise matrix of the preset state transfer process is:

[0042] The preset system observation noise matrix is:

[0043] The Kalman filter iteration principle formula includes:

[0044] Among them, θ 1t is the initial yaw angle at time t, θ 1(t-1) is the target yaw angle at time t-1, w 1t is the angular velocity in the yaw direction at time t, w 1(t-1) is the angular velocity in the yaw direction at time t-1, Δt is the time change between time t and time t-1, a 1(t-1) is the acceleration in the yaw angle direction at time t-1; θ 2t is the initial pitch angle at time t, θ 2(t-1) is the target pitch angle at time t-1, w 2t is the angular velocity in the pitch direction at time t, w 2(t-1) is the angular velocity in the pitch direction at time t-1, a 2(t-1) is the acceleration in the pitch angle direction at time t-1; X t is the observation state matrix at time t, is the predicted state matrix at time t, is the optimized state matrix at time t-1, B t-1 is the control matrix, is the optimal state covariance at time t-1, is the predicted state covariance at time t, K t is the Kalman fusion coefficient at time t, and I is the unit matrix with a diagonal of 1.

[0045] In some embodiments of the present application, the preset mapping relationship is:

[0046] x=θ1·S1·λ1+x0·(1-λ1), y=θ2·S2·λ2+y0·(1-λ2)

[0047] Among them, (x0, y0) is the plane coordinate of the target, (x, y) is the coordinate of the cursor, θ1 is the target yaw angle, θ2 is the target pitch angle, S1 is the preset mapping sensitivity corresponding to the target yaw angle, S2 is the preset mapping sensitivity corresponding to the target pitch angle, λ1 and λ2 are cursor adjustment parameters, and 0<λ1≤1, 0<λ2≤1.

[0048] In some embodiments of the present application, the device also includes: an adjustment module, which is used to adjust λ1 from the first adjustment parameter to the second adjustment parameter, and adjust λ2 from the third adjustment parameter to the fourth adjustment parameter when entering into a drag gesture control state is detected before determining the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; when exiting from the drag gesture control state is detected, adjust λ1 from the second adjustment parameter to the first adjustment parameter, and adjust λ2 from the fourth adjustment parameter to the third adjustment parameter; wherein the first adjustment parameter is smaller than the second adjustment parameter, and the third adjustment parameter is smaller than the fourth adjustment parameter.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, and when the computer program is executed by the processor, the steps of any one of the gesture cursor mapping methods provided in the first aspect are implemented.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the gesture cursor mapping methods provided in the first aspect are implemented.

[0051] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: a gesture cursor mapping device as described in the second aspect, or an electronic device as described in the third aspect, or a computer-readable storage medium as described in the fourth aspect.

[0052] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the gesture cursor mapping method as described in the first aspect.

[0053] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor, a memory and a communication interface, the communication interface is coupled to the processor, the memory is used to store programs or instructions that can be run on the processor, and the processor is used to execute the program or instructions to implement the steps of the gesture cursor mapping method described in the first aspect.

[0054] The technical solution provided by the embodiment of the present application has the following advantages over the prior art: In the embodiment of the present application, the spatial coordinates of each key point in a spatial rectangular coordinate system (the origin of the spatial rectangular coordinate system is the wrist key point) of a plurality of key points of the hand in the current frame image (the plurality of key points include a wrist key point and at least two target key points, each target key point is a root key point of a finger other than the thumb) are obtained; based on the spatial coordinates of each key point, the target yaw angle and target pitch angle of the gesture posture in the current frame image are determined; based on the target yaw angle, the target pitch angle, the target plane coordinates (the target plane coordinates are determined based on the plane coordinates of at least one target key point, and the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image) and a preset mapping relationship (the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor), the coordinates of the cursor are determined. In this way, compared with the prior art that only maps the coordinates of the cursor on the screen based on the coordinates of a target key point in the current frame image and the corresponding mapping relationship, the coordinates of the cursor on the screen are mapped based on the yaw angle, the pitch angle, the coordinates of at least one target key point in the current frame image and a preset mapping relationship. When mapping the coordinates of the cursor, not only the influence of the change of the coordinates of at least one target key point in the current frame image on the coordinates of the cursor is considered, but also the influence of the change of gesture posture (yaw angle and pitch angle) on the coordinates of the cursor is considered. Furthermore, when the user's gesture posture has a small change, although the change of the coordinates of the hand key point cannot be sensitively captured, the change of the yaw angle and pitch angle of the gesture posture can be sensitively captured, and then the change of the cursor coordinates can be obtained in response to the change of the yaw angle and pitch angle of the gesture posture. Therefore, the recognition efficiency of determining the cursor movement according to the change of gesture posture can be improved, and the user experience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 A flowchart of a gesture cursor mapping method provided in this application;

[0058] Figure 2A schematic diagram of key points of a hand in a gesture image provided by the present application;

[0059] Figure 3 A schematic diagram of a spatial rectangular coordinate system corresponding to a gesture image provided in this application;

[0060] Figure 4 A schematic diagram of a yaw angle of a gesture image provided in this application;

[0061] Figure 5 A schematic diagram of the pitch angle of a gesture image provided by the present application;

[0062] Figure 6 A gesture cursor mapping schematic diagram provided for this application;

[0063] Figure 7 A schematic diagram of the structure of a gesture cursor mapping device provided by the present application;

[0064] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0067] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0068] At present, there are three methods for controlling the movement of the cursor on the screen according to the user gesture. One is to control the movement of the cursor by detecting the change in the distance between the target fingers in the user gesture in real time through the camera module; the other is to directly map the coordinates of the cursor on the screen based on the coordinates of the hand key points; and another is to calculate the shoulder width D according to the coordinates a and b of the two end points of the shoulder; calculate the coordinates c of the center point of the hand according to the coordinates m of the hand key points; calculate the coordinates of the two diagonal points of the virtual frame according to the shoulder width D; based on the coordinates p and q of the virtual frame mark point, the acquired coordinates m of the hand key point, and the length width and height height of the projection screen, obtain the cursor mapping coordinates M of the hand key point on the projection screen.

[0069] However, no matter which of the above methods is used, the coordinates of the cursor are mapped based on the coordinates of the hand key points, or combined with the coordinates of the hand key points and the shoulder key points, and are all constructed on a two-dimensional plane. Therefore, if the user needs to move a large range on the two-dimensional plane, the changes in the coordinates of the hand key points or the changes in the coordinates of the hand key points and the changes in the coordinates of the shoulder key points can be detected. Therefore, the detection sensitivity of the key point coordinate changes is low, and the recognition efficiency of determining the cursor movement based on the change of the key point coordinates is low.

[0070] In order to solve the above technical problems, a gesture cursor mapping method is provided in an embodiment of the present application. Compared with the prior art, the coordinates of the cursor on the screen are mapped only based on the coordinates of a target key point in the current frame image and the corresponding mapping relationship. In this scheme, the coordinates of the cursor on the screen are mapped based on the yaw angle, the pitch angle, the coordinates of a target key point in the current frame image and the preset mapping relationship. When mapping the coordinates of the cursor, not only the influence of the change of the coordinates of a target key point in the current frame image on the coordinates of the cursor is considered, but also the influence of the change of gesture posture (yaw angle and pitch angle) on the coordinates of the cursor is considered. Furthermore, when the user's gesture posture has a small change, although the change of the coordinates of the hand key points cannot be sensitively captured, the change of the gesture posture can be sensitively captured, and then the change of the cursor coordinates is obtained in response to the change of the gesture posture. Therefore, the recognition efficiency of determining the cursor movement according to the change of gesture posture can be improved, and the user experience can be improved.

[0071] The present application is applied to gesture cursor mapping scenarios. The electronic device in the embodiments of the present application may be a vehicle-mounted terminal, or a mobile phone, notebook, computer, etc. that is connected to the vehicle for communication. The specific details may be determined based on actual conditions and are not limited here.

[0072] The technical solution of the present application is explained in detail below through several specific embodiments.

[0073] Figure 1 A flowchart of a gesture cursor mapping method provided in this application is shown in FIG. Figure 1 As shown, the gesture cursor mapping method may include the following steps 101 to 103.

[0074] 101. Obtain the spatial coordinates of each key point of the hand in the current frame image in a spatial rectangular coordinate system.

[0075] The multiple key points include a wrist key point and at least two target key points, each target key point is a root key point of a finger other than a thumb, and the origin of the spatial rectangular coordinate system is the wrist key point.

[0076] In some embodiments of the present application, the electronic device captures a user's gesture image through a camera, applies a pre-trained neural network, and extracts the coordinates of each of the multiple key points of the hand in the currently captured gesture image (current frame image) in a 3D space coordinate system (space rectangular coordinate system), and the coordinates of each key point in a 2D plane coordinate system, wherein the origin of the 3D space coordinate system is the wrist joint point, and the origin of the 2D plane coordinate system is the fixed point in the upper left corner of the gesture image (i.e., the current frame image).

[0077] The gesture image is a monocular RGB camera image, and the extraction of hand key points based on the monocular RGB camera image is a general technology and is not limited here.

[0078] Exemplarily, pre-training a neural network model includes: a model training phase and a model deployment inference phase.

[0079] The model training phase includes:

[0080] 1.1 Data annotation: RGB cameras collect hand photos, or use motion capture technology to output hand model photos through the rendering engine, classify the images into left and right hands according to the actual size, and annotate the plane coordinates of 21 hand key points on the image and the x, y, and z coordinate axes with the wrist joint as the 3D coordinate origin. Accurate data annotation in the depth direction is usually obtained through motion capture;

[0081] 1.2 Model design: Use CNN convolutional neural network model, including multiple convolutional layers, pooling layers, and residual connections. Use fully connected layers to regress 3D coordinates, or use Gaussian heatmaps to regress 2D and 3D coordinates;

[0082] 1.3 Model training: Divide the data into training set, test set, and validation set, use the model to infer the training set, define the mean square error between the inference value and the label of the 2D and 3D coordinates as the loss, use gradient descent for training, update the network model weights, and continue to iterate until the loss reaches the optimal value. Obtain a pre-trained model.

[0083] The model deployment and inference phase includes:

[0084] 2.1 Image acquisition: Deploy cameras to collect RGB images in real time, standardize the image data and wait for the pre-trained neural network inference;

[0085] 2.2 Model reasoning: The model reads standardized image data, and obtains a set of feature maps through multiple convolutions, pooling layers, and residual connections. Through the full connection of the neural network or Gaussian heat map regression, it outputs the left and right hand categories, the plane coordinates of 21 hand key points, and the x, y, and z coordinate axes of the 3D space coordinate system with the wrist joint as the origin.

[0086] 102. Determine a target yaw angle and a target pitch angle of the gesture posture in the current frame image based on the spatial coordinates of each key point.

[0087] For example, Figure 2 (a) in the figure is a gesture image. Figure 2 (b) in the figure shows the key points of each hand in the gesture image. Figure 2 (c) in the figure shows that the key points of the hand in the gesture image are numbered from 0 to 10. The index relationship of the coordinates of the key points of the hand in the spatial rectangular coordinate system is as follows: Figure 3 As shown, the first vertical plane of the spatial rectangular coordinate system is the YOZ plane, and the horizontal plane is the XOZ plane. According to the hand joint activity characteristics, the wrist joint (i.e., key point 0) and the joints connecting the fingers and the palm (i.e., key points 2, 3, 4, and 5) are relatively stable. Therefore, the multiple key points include the wrist key point (i.e., key point 0) and four target key points (i.e., at least two of the key points 2, 3, 4, and 5).

[0088] Combination Figure 3 As shown, in the embodiment of the present application, the angle between the projection of a vector formed by any two key points among the multiple key points on the XOZ plane and the Z axis can be determined as the target yaw angle (yaw); the average value of the angle between the projection of a vector formed by each key point combination in the multiple key point combinations (each key point combination includes two key points) on the XOZ plane and the Z axis can also be determined as the yaw angle; the specific angle can be determined based on actual conditions and is not limited here.

[0089] Combination Figure 3 As shown, in the embodiment of the present application, the angle between the projection of the vector formed by any two key points among the multiple key points on the YOZ plane and the Z axis can be determined as the pitch angle (pitch); the average value of the angle between the projection of the vector formed by each key point combination in the multiple key point combinations (each key point combination includes two key points) on the YOZ plane and the Z axis can also be determined as the target pitch angle; the specific angle can be determined based on actual conditions and is not limited here.

[0090] 103. Determine the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates, and a preset mapping relationship.

[0091] Among them, the target plane coordinates are determined based on the plane coordinates of at least one target key point, the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

[0092] The preset mapping relationship may be specifically determined according to actual conditions and is not limited here.

[0093] A gesture cursor mapping method is provided in an embodiment of the present application. Compared with the prior art, the coordinates of the cursor on the screen are mapped based only on the coordinates of a target key point in the current frame image and the corresponding mapping relationship. In this scheme, the coordinates of the cursor on the screen are mapped based on the yaw angle, the pitch angle, the coordinates of at least one target key point in the current frame image and a preset mapping relationship. When mapping the coordinates of the cursor, not only the influence of the change of the coordinates of at least one target key point in the current frame image on the coordinates of the cursor is considered, but also the influence of the change of gesture posture (yaw angle and pitch angle) on the coordinates of the cursor is considered. Furthermore, when the user's gesture posture has a small change, although the change of the coordinates of the hand key point cannot be sensitively captured, the change of the yaw angle and pitch angle of the gesture posture can be sensitively captured, and then the change of the cursor coordinates can be obtained in response to the change of the yaw angle and pitch angle of the gesture posture. Therefore, the recognition efficiency of determining the cursor movement according to the change of gesture posture can be improved, and the user experience can be improved.

[0094] In some embodiments of the present application, the multiple key points include a wrist key point, a first target key point, and a second target key point; the above step 102 can be specifically implemented through the following steps 102a to 102c.

[0095] 102a. Determine an initial yaw angle of the gesture based on a first angle between a projection of the first vector on a horizontal plane of the spatial rectangular coordinate system and the target vector.

[0096] Among them, the first vector is a vector determined by the first target key point and the second target key point, and the direction of the first vector is from the first target key point to the second target key point; the target vector is a vector on the common coordinate axis of the first vertical plane and the horizontal plane of the spatial rectangular coordinate system, and points to the positive direction of the common coordinate axis, and the first vertical plane is parallel to the horizontal axis of the screen.

[0097] The target vector may be a unit vector pointing in the positive direction of the common coordinate axis.

[0098] For example, Figure 4 As shown, the first target key point is key point 2, the second target key point is key point 5, and the calculation process of the initial yaw angle (θ1) is:

[0099] Assume that the coordinates of joint point i are (x i ,y i ,z i ), the coordinates of the first target key point are (x2, y2, z2), and the coordinates of the second target key point are (x5, y5, z5). The first vector is V 25 =[x5-x2,y5-y2,z5-z2]; V 25 Project V on the XOZ plane 25xz =[x5-x2,0,z5-z2]; the target vector is the Z-axis unit vector V z =[0,0,1]; According to the inner product formula of vector V 25xz ·V z =|V 25xz |*|V z |*arccosθ1, we can get θ1=arccos((V 25xz ·V z ) / (|V 25xz |*|V z |)).

[0100] In the embodiment of the present application, according to the characteristics of hand joint movement, it can be known that key point No. 2 and key point No. 5 are relatively stable. Since key point No. 2 and key point No. 5 are far apart, key point No. 2 is used as the first target key point and key point No. 5 is used as the second target key point. When the gesture has a slight change, the sensitivity of the first vector is greater, thereby improving the recognition efficiency of determining the cursor movement according to the change of gesture posture, and improving the user experience.

[0101] 102b. Determine an initial pitch angle of the gesture posture based on a second angle between a projection of the second vector on the first vertical plane and the target vector.

[0102] The second vector is a vector determined by the wrist key point and the first target key point, and the direction of the second vector is from the wrist key point to the first target key point.

[0103] For example, Figure 5 As shown in the figure, the first target key point is key point 2, the wrist key point is key point 0, and the calculation process of the initial pitch angle (θ2) is:

[0104] Assume that the coordinates of joint point i are (x i ,y i ,z i), then the coordinates of the first target key point are (x2, y2, z2), and the coordinates of the wrist key point are (x0, y0, z0) = (0, 0, 0). Second vector: V 02 =[x2-x0,y2-y0,z2-z0]; V 02 Project V on the YOZ plane 02yz =[0,y2-y0,z2-z0]; the target vector is the Z-axis unit vector V z =[0,0,1]; According to the inner product formula of vector V 02yz ·V z =|V 02yz |*|V z |*arccosθ2, we can get θ2=arccos((V 02yz ·V z ) / (|V 02yz |*|V z |)).

[0105] In the embodiment of the present application, according to the characteristics of hand joint movement, it can be known that key point No. 2 and the wrist key point are relatively stable. Since key point No. 2 and the wrist key point are far apart, key point No. 2 is used as the first target key point. When the gesture has a slight change, the sensitivity of the second vector is greater, thereby improving the recognition efficiency of determining the cursor movement according to the change in gesture posture, thereby improving the user experience.

[0106] In some embodiments of the present application, the multiple key points include a wrist key point and four target key points; the above step 102a can be specifically implemented through the following steps 102a1 and 102a2.

[0107] 102a1. Use two key points in each key point combination in the multiple key point combinations as the first target key point and the second target key point, respectively, and determine the first angle between the projection of the first vector on the horizontal plane and the target vector to determine multiple first angles.

[0108] The multiple key point combinations are obtained by combining different target key points among the four target key points.

[0109] 102a2. Determine an average value of the multiple first angles as the initial yaw angle.

[0110] For example, the mean value of the angles between the projections of the line vectors of the key points 2 to 5, 2 to 3, 3 to 4, and 4 to 5 on the XOZ plane and the Z axis (θ 1avg ) as the initial yaw angle, the value is more stable, which can better improve the recognition efficiency of determining the cursor movement according to the change of gesture posture and improve the user experience.

[0111] θ1avg The specific calculation process is:

[0112] The coordinates of key point i are (x i ,y i ,z i ), based on the calculation method of calculating the angle between the projection of the line vector connecting key points 2 to 5 on the XOZ plane and the Z axis described in the above step 102a, it can be obtained:

[0113] The angle θ between the projection of the line vector connecting key points 2 to 5 on the XOZ plane and the Z axis 25 =arccos((V 25xz ·V z ) / (|V 25xz |*|V z |)); The angle between the projection of the line vector connecting key points 2 to 3 and the Z axis is θ 23 =arccos((V 23xz ·V z ) / (|V 23xz |*|V z |)); The angle between the projection of the line vector connecting key points 3 to 4 and the Z axis is θ 34 =arccos((V 34xz ·V z ) / (|V 34xz |*|V z |)); The angle between the projection of the line vector connecting key points 4 to 5 and the Z axis is θ 45 =arccos((V 45xz ·V z ) / (|V 45xz |*|V z |)); mean θ 1avg =(θ 25 +θ 33 +θ 34 +θ 45 ) / 4.

[0114] In some embodiments of the present application, the multiple key points include a wrist key point and four target key points; the above step 102b can be specifically implemented through the following steps 102b1 and 102b2.

[0115] 102b1. Use two key points in each key point combination in the multiple key point combinations as the first target key point and the second target key point, respectively, and determine the second angle between the projection of the second vector on the first vertical plane and the target vector to determine multiple second angles.

[0116] 102b2. Determine an average value of the multiple second angles as the initial pitch angle.

[0117] For example, the mean value of the angle between the projection of the line vectors of the key points 0 to 2, 0 to 3, 0 to 4, and 0 to 5 on the YOZ plane and the Z axis (θ 2avg ) as the initial pitch angle, the value is more stable, which can better improve the recognition efficiency of determining the cursor movement according to the change of gesture posture and improve the user experience.

[0118] θ 2avg The specific calculation process is:

[0119] The coordinates of key point i are (x i ,y i ,z i ), based on the calculation method of calculating the angle between the projection of the line vector connecting key points 0 to 2 on the YOZ plane and the Z axis described in the above step 102b, it can be obtained:

[0120] The angle θ between the projection of the line vector connecting key points 0 to 2 on the YOZ plane and the Z axis 02 =arccos((V 02yz ·V z ) / (|V 02yz |*|V z |)); The angle between the projection of the line vector connecting key points 0 to 3 and the Z axis is θ 03 =arccos((V 03yz ·V z ) / (|V 03yz |*|V z |)); The angle between the projection of the line vector connecting key points 0 to 4 and the Z axis is θ 04 =arccos((V 04yz ·V z ) / (|V 04yz |*|V z |)); The angle between the projection of the line vector connecting key points 0 to 5 and the Z axis is θ 05 =arccos((V 05yz ·V z ) / (|V 05yz |*|V z |)); mean θ 2avg =(θ 02 +θ 03 +θ 04 +θ 05 ) / 4.

[0121] 102c. Determine the target yaw angle and the target pitch angle based on the initial yaw angle and the initial pitch angle.

[0122] In some embodiments of the present application, the initial yaw angle can be determined as the target yaw angle, and the initial pitch angle can be determined as the target pitch angle. The initial yaw angle and the initial pitch angle can also be filtered to obtain the target yaw angle and the target pitch angle. The specific angles can be determined based on actual conditions and are not limited here.

[0123] In some embodiments of the present application, the above step 102c can be specifically implemented through the following step 102c1.

[0124] 102c1. Filter the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle.

[0125] In some embodiments of the present application, the filtering processing algorithm can be Kalman filtering, mean filtering, median filtering, exponential filtering, Holt double exponential filtering, ARMA autoregressive sliding average filtering and other filtering smoothing methods, which can be determined based on actual conditions and are not limited here.

[0126] In the embodiment of the present application, since the coordinates of the key points contain a lot of noise, the initial yaw angle and the initial pitch angle obtained by the above steps 102a and 102b based on the coordinates of multiple key points, if directly applied to cursor mapping, may cause cursor jitter. Therefore, before the initial yaw angle and the initial pitch angle obtained by the above steps 102a and 102b based on the coordinates of multiple key points are applied to cursor mapping, filtering and smoothing processing must be performed first to avoid the occurrence of cursor jitter.

[0127] In some embodiments of the present application, the above step 102c1 can be specifically implemented through the following steps 102c1a and 102c1b.

[0128] 102c1a. Determine a state matrix and a state transfer matrix in a Kalman filter formula based on the first angle formula, the first angular velocity formula, the second angle formula, and the second angular velocity formula.

[0129] 102c1b. Based on the state matrix, the state transfer matrix, the preset observation matrix, the preset state transfer process noise matrix and the preset system observation noise matrix, and the Kalman filter iteration principle formula, the initial yaw angle and the initial pitch angle are subjected to Kalman filter iteration processing to obtain the target yaw angle and the target pitch angle.

[0130] Among them, the first angle formula is:

[0131] The second angle formula is:

[0132] The first angular velocity formula is: ω 1t =ω 1(t-1) +a1(t-1) Δt;

[0133] The first angular velocity formula is: ω 2t =ω 2(t-1) +a 2(t-1) Δt;

[0134] The Kalman filter formula is:

[0135] The state matrix is:

[0136] The state transfer matrix is:

[0137] The default observation matrix is:

[0138] The noise matrix of the preset state transfer process is:

[0139] The preset system observation noise matrix is:

[0140] The Kalman filter iteration principle formula includes:

[0141] Among them, θ 1t is the initial yaw angle at time t, θ 1(t-1) is the target yaw angle at time t-1, w 1t is the angular velocity in the yaw direction at time t, w 1(t-1) is the angular velocity in the yaw direction at time t-1, Δt is the time change between time t and time t-1, a 1(t-1) is the acceleration in the yaw angle direction at time t-1; θ 2t is the initial pitch angle at time t, θ 2(t-1) is the target pitch angle at time t-1, w 2t is the angular velocity in the pitch direction at time t, w 2(t-1) is the angular velocity in the pitch direction at time t-1, a 2(t-1) is the acceleration in the pitch angle direction at time t-1; X t is the observation state matrix at time t, is the predicted state matrix at time t, is the optimized state matrix at time t-1, B t-1 is the control matrix, is the optimal state covariance at time t-1; is the predicted state covariance at time t; K t is the Kalman fusion coefficient at time t.

[0142] It can be understood that the above steps 102c1a and 102c1b can be specifically implemented through the following steps 201 to 204.

[0143] 201. Define the kinematic model:

[0144] θ at time t 1t and θ 2t and θ at time t-1 1(t-1) and θ 2(t-1) The relationship is the following first angle formula and second angle formula: w 1t with a 1(t-1) , w 2t with a 2(t-1) The relationship between is the following first angular velocity formula and second angular velocity formula: ω 1t =ω 1(t-1) +a 1(t-1) Δt,ω 2t =ω 2(t-1) +a 2(t-1) Δt; combined with Kalman filter formula The following matrix representation can be obtained:

[0145] Then we can get the state matrix as: The state transfer matrix is: Because there are 4 observations involved, the preset observation matrix is ​​set as:

[0146] 202. Define the preset state transfer process noise matrix (Q) and the preset system observation noise matrix (R). Since the kinematic model in step 201 is reasonable, the Q coefficient is 0.1. Since the inference noise is actually large, the R coefficient is 2. Therefore,

[0147] and

[0148] 203. Perform Kalman filtering on the initial yaw angle and the initial pitch angle.

[0149] Step 1: Initialization.

[0150] Assume the optimal state at time t = 0 Optimal state covariance Right now

[0151] Step 2: Calculate the predicted state and predicted state covariance at time t based on the optimal state and optimal state covariance at time t-1.

[0152] According to the Kalman filter formula The control matrix B is omitted in the actual calculation. t-1 , attribute it to the preset state transfer process noise matrix Q, and calculate the predicted state at time t = 1 According to the optimal state covariance at time t-1 Get the predicted state covariance at time t Formula Calculate the predicted state covariance of the state at time t=1

[0153] Step 3: Calculate the Kalman fusion coefficient from the predicted state covariance and the system observation covariance.

[0154] According to the observation noise R and the predicted state covariance Calculate the Kalman fusion coefficient (K t ) Calculate the Kalman fusion coefficient at time t = 1

[0155] Step 4: Iteratively calculate the optimal state matrix at time t=1 based on the Kalman fusion coefficient at time t=1, the observed state matrix at time t=1, and the predicted state matrix at time t=1.

[0156] According to the observed state at time t (X t ), prediction status and fusion coefficient (K t ), calculate the optimal state Formula Calculate the optimal state matrix at time t = 1 Optimal state matrix It is the stable system state after filtering and smoothing optimization at time t=1 where θ 1t＝1 ,θ 2t＝1 is the parameter involved in cursor mapping at time t = 1. and fusion coefficient (K t ), calculate the optimal state covariance Formula Calculate the optimal state covariance at time t = 1 Where I is the identity matrix with diagonal 1,

[0157] 204. Iteratively repeat the above step 203 for the initial yaw angle and the initial pitch angle until the end.

[0158] In an embodiment of the present application, a uniform acceleration motion model is defined in the Kalman filter processing, and a preset observation matrix, a preset state transfer process noise matrix and a preset system observation noise matrix corresponding to the Kalman filter processing are predefined in combination with the uniform acceleration motion model, thereby realizing Kalman filter processing of the initial yaw angle and the initial pitch angle in combination with the uniform acceleration motion model. After experimental analysis, the initial yaw angle and the initial pitch angle are subjected to Kalman filter processing in combination with the uniform acceleration motion model, and the noise smoothing processing effect is better, which can well avoid the occurrence of cursor jitter.

[0159] In some embodiments of the present application, the preset mapping relationship is:

[0160] x=θ1·S1·λ1+x0·(1-λ1), y=θ2·S2·λ2+y0·(1-λ2)

[0161] Among them, (x0, y0) is the plane coordinate of the target, (x, y) is the coordinate of the cursor, θ1 is the target yaw angle, θ2 is the target pitch angle, S1 is the preset mapping sensitivity corresponding to the target yaw angle, S2 is the preset mapping sensitivity corresponding to the target pitch angle, λ1 and λ2 are cursor adjustment parameters, and 0<λ1≤1, 0<λ2≤1.

[0162] Among them, S1 and S2 are related to at least one of the values ​​of λ1 and λ2, the screen size, the values ​​of θ1 and θ2, etc., and can be determined according to actual conditions, and are not limited here.

[0163] Exemplarily, when λ1=1, S1≥screen width / maximum value of θ1; when λ2=1, S2≥screen height / maximum value of θ2; the specific value can be determined according to actual conditions and is not limited here.

[0164] In the embodiment of the present application, the above-mentioned preset mapping relationship is a mapping relationship of a linear function, and a linear function has a characteristic of linear change. Therefore, the coordinates of the cursor mapped using the above-mentioned preset mapping relationship are relatively accurate.

[0165] In some embodiments of the present application, the preset mapping relationship may also be a quadratic function mapping relationship or a cubic function mapping relationship, etc., which is not limited here.

[0166] In some embodiments of the present application, the values ​​of λ1 and λ2 may be fixed, or the values ​​of λ1 and λ2 may be adjusted based on an event (such as whether the drag gesture control state is entered), which is not limited here.

[0167] like Figure 6As shown, in an embodiment of the present application, the cursor is moved by using the changes in the target yaw angle and target pitch angle of the gesture posture, so as to achieve the purpose of the cursor parameters responding to the changes in the target yaw angle and target pitch angle of the gesture posture. The cursor movement can respond to both the change in the gesture posture (finger pointing) and the change in the position of the gesture (hand displacement). Therefore, the user can move the cursor by finger pointing or by hand displacement, without having to move the arm over a large range on a two-dimensional plane. The recognition efficiency of determining the cursor movement based on the change in gesture posture can be better improved, thereby improving the user experience.

[0168] In some embodiments of the present application, before the above step 103, the gesture cursor mapping method provided by the embodiment of the present application may further include the following step 105 or step 106.

[0169] 105. When it is detected that the drag gesture control state has been entered, adjust λ1 from the first adjustment parameter to the second adjustment parameter, and adjust λ2 from the third adjustment parameter to the fourth adjustment parameter.

[0170] 106. When exiting the drag gesture control state is detected, adjust λ1 from the second adjustment parameter to the first adjustment parameter, and adjust λ2 from the fourth adjustment parameter to the third adjustment parameter.

[0171] Among them, the first adjustment parameter is smaller than the second adjustment parameter, and the third adjustment parameter is smaller than the fourth adjustment parameter.

[0172] It can be understood that it is first detected whether the drag gesture control state is entered. If the drag gesture control state is entered, the above step 105 is executed, and if the drag gesture control state is not entered, the above step 106 is executed. The drag gesture control state refers to the drag gesture control state of the control or slider. When the drag gesture control state is entered, the values ​​of λ1 and λ2 are increased, so that the gesture control can be more sensitive and comfortable when the user drags the control or slider.

[0173] The present application also provides a gesture cursor mapping device, Figure 7 A schematic diagram of the structure of a gesture cursor mapping device provided in this application, such as Figure 7 As shown, the gesture cursor mapping device includes:

[0174] The acquisition module 701 is used to obtain the spatial coordinates of each of the multiple key points of the hand in the current frame image in the spatial rectangular coordinate system, the multiple key points include a wrist key point and at least two target key points, each target key point is a root key point of a finger other than the thumb, and the origin of the spatial rectangular coordinate system is the wrist key point; the determination module 702 is used to determine the target yaw angle and target pitch angle of the gesture posture in the current frame image based on the spatial coordinates of each key point; the determination module 702 is also used to determine the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; the target plane coordinates are determined according to the plane coordinates of at least one target key point, the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

[0175] In some embodiments of the present application, the multiple key points include a wrist key point, a first target key point, and a second target key point; the determination module 702 is specifically used to determine the initial yaw angle of the gesture posture based on a first angle between a projection of a first vector on a horizontal plane of the spatial rectangular coordinate system and a target vector, the first vector being a vector determined by the first target key point and the second target key point, and the direction of the first vector is from the first target key point to the second target key point; the target vector is a vector on a common coordinate axis of a first vertical plane and the horizontal plane of the spatial rectangular coordinate system, and pointing to a positive direction of the common coordinate axis, and the first vertical plane is parallel to the horizontal axis of the screen; the initial pitch angle of the gesture posture is determined based on a second angle between a projection of the second vector on the first vertical plane and the target vector, the second vector being a vector determined by the wrist key point and the first target key point, and the direction of the second vector is from the wrist key point to the first target key point; based on the initial yaw angle and the initial pitch angle, the target yaw angle and the target pitch angle are determined.

[0176] In some embodiments of the present application, the multiple key points include a wrist key point and four target key points; the determination module 702 is specifically used to use two key points in each key point combination in the multiple key point combinations as the first target key point and the second target key point, respectively, to determine the first angle between the projection of the first vector on the horizontal plane and the target vector to determine multiple first angles, and the multiple key point combinations are obtained by combining different target key points in the four target key points; the average value of the multiple first angles is determined as the initial yaw angle; the determination module 702 is specifically used to use multiple key points in the four target key points as the first target key points, respectively, to determine the second angle between the projection of the second vector on the first vertical plane and the target vector to determine multiple second angles; the average value of the multiple second angles is determined as the initial pitch angle.

[0177] In some embodiments of the present application, the determination module 702 is specifically configured to filter the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle.

[0178] In some embodiments of the present application, the filtering processing module is specifically used to determine the state matrix and the state transfer matrix in the Kalman filter formula based on the first angle formula, the first angular velocity formula, the second angle formula, and the second angular velocity formula;

[0179] Based on the state matrix, the state transfer matrix, the preset observation matrix, the preset state transfer process noise matrix and the preset system observation noise matrix, and the Kalman filter iteration principle formula, the initial yaw angle and the initial pitch angle are subjected to Kalman filter iteration processing to obtain the target yaw angle and the target pitch angle;

[0180] Among them, the first angle formula is:

[0181] The second angle formula is:

[0182] The first angular velocity formula is: ω 1t =ω 1(t-1) +a 1(t-1) Δt;

[0183] The first angular velocity formula is: ω 2t =ω 2(t-1) +a 2(t-1) Δt;

[0184] The Kalman filter formula is:

[0185] The state matrix is:

[0186] The state transfer matrix is:

[0187] The default observation matrix is:

[0188] The noise matrix of the preset state transfer process is:

[0189] The preset system observation noise matrix is:

[0190] The Kalman filter iteration principle formula includes:

[0191] Among them, θ 1t is the initial yaw angle at time t, θ 1(t-1) is the target yaw angle at time t-1, w 1t is the angular velocity in the yaw direction at time t, w 1(t-1) is the angular velocity in the yaw direction at time t-1, Δt is the time change between time t and time t-1, a 1(t-1) is the acceleration in the yaw angle direction at time t-1; θ 2t is the initial pitch angle at time t, θ 2(t-1) is the target pitch angle at time t-1, w 2t is the angular velocity in the pitch direction at time t, w 2(t-1) is the angular velocity in the pitch direction at time t-1, a 2(t-1) is the acceleration in the pitch angle direction at time t-1; X t is the observation state matrix at time t, is the predicted state matrix at time t, is the optimized state matrix at time t-1, B t-1 is the control matrix, is the optimal state covariance at time t-1, is the predicted state covariance at time t, K t is the Kalman fusion coefficient at time t, and I is the unit matrix with a diagonal of 1.

[0192] In some embodiments of the present application, the preset mapping relationship is:

[0193] x=θ1·S1·λ1+x0·(1-λ1), y=θ2·S2·λ2+y0·(1-λ2)

[0194] Among them, (x0, y0) is the plane coordinate of the target, (x, y) is the coordinate of the cursor, θ1 is the target yaw angle, θ2 is the target pitch angle, S1 is the preset mapping sensitivity corresponding to the target yaw angle, S2 is the preset mapping sensitivity corresponding to the target pitch angle, λ1 and λ2 are cursor adjustment parameters, and 0<λ1≤1, 0<λ2≤1.

[0195] In some embodiments of the present application, the device also includes: an adjustment module, which is used to adjust λ1 from the first adjustment parameter to the second adjustment parameter, and adjust λ2 from the third adjustment parameter to the fourth adjustment parameter when entering into a drag gesture control state is detected before determining the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; when exiting from the drag gesture control state is detected, adjust λ1 from the second adjustment parameter to the first adjustment parameter, and adjust λ2 from the fourth adjustment parameter to the third adjustment parameter; wherein the first adjustment parameter is smaller than the second adjustment parameter, and the third adjustment parameter is smaller than the fourth adjustment parameter.

[0196] It should be noted that the gesture cursor mapping device may be the electronic device in the method embodiment of the present application, or may be a functional module and / or functional entity in the electronic device that can implement the functions of the device embodiment, and the present application embodiment does not limit this.

[0197] In the embodiment of the present application, each module can implement the gesture cursor mapping method provided in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0198] Figure 8 The structural diagram of an electronic device provided in the embodiment of the present application is used to exemplarily illustrate the electronic device that implements any gesture cursor mapping method in the embodiment of the present application, and should not be understood as a specific limitation on the embodiment of the present application.

[0199] like Figure 8 As shown, the electronic device 800 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0200] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device 800 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0201] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processor 801, the functions defined in any gesture cursor mapping method provided in the embodiment of the present application can be executed.

[0202] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0203] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0204] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0205] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the spatial coordinates of each of the multiple key points of the hand in the current frame image in a spatial rectangular coordinate system, the multiple key points include a wrist key point and at least two target key points, each target key point is a root key point of a finger other than the thumb, and the origin of the spatial rectangular coordinate system is the wrist key point; based on the spatial coordinates of each key point, determines the target yaw angle and target pitch angle of the gesture posture in the current frame image; based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship, determines the coordinates of the cursor; the target plane coordinates are determined according to the plane coordinates of at least one target key point, the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

[0206] In an embodiment of the present application, a computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0207] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0208] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0209] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0210] In the context of the present application, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0211] An embodiment of the present application also provides a vehicle, comprising: the above-mentioned gesture cursor mapping device, or the above-mentioned electronic device, or the above-mentioned computer-readable storage medium.

[0212] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0213] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0214] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A gesture cursor mapping method, characterized in that: The method comprises: Acquire the spatial coordinates of each of the multiple key points of the hand in the current frame image in a spatial rectangular coordinate system, wherein the multiple key points include a wrist key point and at least two target key points, each target key point is a root key point of any finger except a thumb, and the origin of the spatial rectangular coordinate system is the wrist key point; Determine a target yaw angle and a target pitch angle of the gesture posture in the current frame image based on the spatial coordinates of each key point; The coordinates of the cursor are determined based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; the target plane coordinates are determined based on the plane coordinates of at least one target key point, the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

2. The method according to claim 1, characterized in that The plurality of key points include the wrist key point, a first target key point, and a second target key point; The determining, based on the spatial coordinates of each key point, a target yaw angle and a target pitch angle of the gesture posture in the current frame image includes: Determine the initial yaw angle of the gesture posture based on a first angle between a projection of a first vector on a horizontal plane of the spatial rectangular coordinate system and a target vector, wherein the first vector is a vector determined by the first target key point and the second target key point, and the direction of the first vector is from the first target key point to the second target key point; the target vector is a vector on a common coordinate axis of a first vertical plane of the spatial rectangular coordinate system and the horizontal plane, and points to a positive direction of the common coordinate axis, and the first vertical plane is parallel to the horizontal axis of the screen; Determining an initial pitch angle of the gesture posture based on a second angle between a projection of a second vector on the first vertical plane and the target vector, wherein the second vector is a vector determined by the wrist key point and the first target key point, and a direction of the second vector is from the wrist key point to the first target key point; The target yaw angle and the target pitch angle are determined based on the initial yaw angle and the initial pitch angle.

3. The method according to claim 2, characterized in that The multiple key points include the wrist key points and four target key points; the determining the initial yaw angle of the gesture posture based on the first angle between the projection of the first vector on the horizontal plane of the spatial rectangular coordinate system and the target vector includes: Using two key points in each key point combination in a plurality of key point combinations as the first target key point and the second target key point, respectively, and determining a first angle between the projection of the first vector on the horizontal plane and the target vector to determine a plurality of first angles, wherein the plurality of key point combinations are obtained by combining different target key points in the four target key points; Determine an average value of the plurality of first angles as the initial yaw angle; The determining the initial pitch angle of the gesture posture based on a second angle between a projection of the second vector on the first vertical plane and the target vector comprises: Using multiple key points among the four target key points as the first target key points, respectively, and determining a second angle between the projection of the second vector on the first vertical plane and the target vector to determine multiple second angles; An average value of the plurality of second angles is determined as the initial pitch angle.

4. The method according to claim 2, characterized in that: The determining the target yaw angle and the target pitch angle based on the initial yaw angle and the initial pitch angle comprises: The initial yaw angle and the initial pitch angle are filtered to obtain the target yaw angle and the target pitch angle.

5. The method according to claim 4, characterized in that The filtering process on the initial yaw angle and the initial pitch angle to obtain the target yaw angle and the target pitch angle includes: Based on the first angle formula, the first angular velocity formula, the second angle formula, and the second angular velocity formula, a state matrix and a state transfer matrix in the Kalman filter formula are determined; Based on the state matrix, the state transfer matrix, the preset observation matrix, the preset state transfer process noise matrix and the preset system observation noise matrix, and the Kalman filter iteration principle formula, the initial yaw angle and the initial pitch angle are subjected to Kalman filter iteration processing to obtain the target yaw angle and the target pitch angle; Wherein, the first angle formula is: The second angle formula is: The first angular velocity formula is: ω 1t =ω 1(t-1) +a 1(t-1) Δt; The first angular velocity formula is: ω 2t =ω 2(t-1) +a 2(t-1) Δt; The Kalman filter formula is: The state matrix is: The state transfer matrix is: The preset observation matrix is: The preset state transfer process noise matrix is: The preset system observation noise matrix is: The Kalman filter iteration principle formula includes: Among them, the θ 1t is the initial yaw angle at time t, the θ 1(t-1) is the target yaw angle at time t-1, and w 1t is the angular velocity in the yaw direction at time t, and w 1(t-1) is the angular velocity in the yaw angle direction at time t-1, Δt is the time change between time t and time t-1, and a 1(t-1) is the acceleration in the yaw angle direction at time t-1; the θ 2t is the initial pitch angle at time t, the θ 2(t-1) is the target pitch angle at time t-1, and w 2t is the angular velocity in the pitch direction at time t, and w 2(t-1) is the angular velocity in the pitch angle direction at time t-1, and a 2(t-1) is the acceleration in the pitch angle direction at time t-1; t is the observation state matrix at time t, is the predicted state matrix at time t, is the optimized state matrix at time t-1, the B t-1 is the control matrix, is the optimal state covariance at time t-1, is the predicted state covariance at time t, the K t is the Kalman fusion coefficient at time t, and I is a unit matrix with a diagonal of 1.

6. The method according to any one of claims 1 to 5, characterized in that The preset mapping relationship is: x=θ1·S1·λ1+x0·(1-λ1), y=θ2·S2·λ2+y0·(1-λ2) Among them, (x0, y0) is the target plane coordinate, (x, y) is the coordinate of the cursor, θ1 is the target yaw angle, θ2 is the target pitch angle, S1 is the preset mapping sensitivity corresponding to the target yaw angle, S2 is the preset mapping sensitivity corresponding to the target pitch angle, λ1 and λ2 are cursor adjustment parameters, and 0<λ1≤1, 0<λ2≤1.

7. The method according to claim 6, characterized in that Before determining the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and the preset mapping relationship, the method further includes: When entering the drag gesture control state is detected, λ1 is adjusted from the first adjustment parameter to the second adjustment parameter, and λ2 is adjusted from the third adjustment parameter to the fourth adjustment parameter; When exiting the drag gesture control state is detected, λ1 is adjusted from the second adjustment parameter to the first adjustment parameter, and λ2 is adjusted from the fourth adjustment parameter to the third adjustment parameter; The first adjustment parameter is smaller than the second adjustment parameter, and the third adjustment parameter is smaller than the fourth adjustment parameter.

8. A gesture cursor mapping device, characterized in that: The device comprises: An acquisition module, used for acquiring the spatial coordinates of each of the multiple key points of the hand in the current frame image in a spatial rectangular coordinate system, wherein the multiple key points include a wrist key point and at least two target key points, each target key point is a root key point of a finger other than a thumb, and the origin of the spatial rectangular coordinate system is the wrist key point; A determination module, configured to determine a target yaw angle and a target pitch angle of the gesture posture in the current frame image based on the spatial coordinates of each key point; The determination module is also used to determine the coordinates of the cursor based on the target yaw angle, the target pitch angle, the target plane coordinates and a preset mapping relationship; the target plane coordinates are determined according to the plane coordinates of at least one target key point, the plane coordinates of each target key point are the coordinates of the corresponding target key point in the current frame image, and the preset mapping relationship is the mapping relationship between the target yaw angle, the target pitch angle, the target plane coordinates and the coordinates of the cursor.

9. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the gesture cursor mapping method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the gesture cursor mapping method according to any one of claims 1 to 7 are implemented.

11. A vehicle, characterized in that: include: The gesture cursor mapping device as claimed in claim 8, or the electronic device as claimed in claim 9, or comprising the computer-readable storage medium as claimed in claim 10.

Images