A three-dimensional target selection method based on finger motion as ray mapping

Abstract

The application discloses a three-dimensional target selection method based on finger motion as a ray mapping basis, and belongs to the technical field of human-computer interaction. The method comprises the following steps: (1) capturing hand state information, switching the mapping subject of ray tracking according to the stretching and bending degree of fingers and the motion speed of fingers, and determining the mapping subject; the mapping subject comprises the whole hand and each finger; (2) determining the information type number of mapping, the mapping scale, the projection mode of mapping according to the current mapping subject, and selecting the target in the three-dimensional space by using the mapped ray. The application switches the current mapping subject, improves the selection efficiency of the target in the three-dimensional scene, and reduces the fatigue of hands and arms; different mapping information type numbers, projection modes and projection scales are flexibly used, so that the limitations of different finger types are compensated, and the mapped ray can have a larger selection range in the virtual three-dimensional scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of human-computer interaction, including three-dimensional interaction, virtual reality, augmented reality, mixed reality field, and particularly relates to a three-dimensional target selection method based on finger movement as a ray mapping basis. BACKGROUND

[0002] With the continuous development of virtual reality and augmented reality technology, how to efficiently and naturally select virtual three-dimensional targets has always been a research hotspot. The common three-dimensional selection methods can be divided into two types, virtual hand technology and ray technology. Virtual hand is usually aimed at objects that can be directly touched within the user's arm range, similar to the user's grabbing behavior in real life; the ray is usually aimed at the acquisition of distant target objects, because the ray starts from a certain vertex and is infinite in length, so there is no depth limitation. Generally, users will select distant objects by shooting a ray through a handle or a bare hand gesture. Although the handle is less tiring to use, it is less portable in some daily life scenarios, especially in travel scenarios. The starting point of the ray of bare hand interaction is usually the entire hand, and the muscle groups on the arm are frequently used during the interaction process, causing muscle fatigue. SUMMARY

[0003] In order to reduce the fatigue of bare hand ray interaction and meet the user's selection of virtual objects in different distribution ranges, the present application provides a finger gesture ray interaction method based on finger gesture, which improves the comfort of users in the three-dimensional interaction process while ensuring selection efficiency.

[0004] The technical solutions adopted by the present application are as follows:

[0005] A three-dimensional target selection method based on finger movement as a ray mapping basis, comprising:

[0006] T1, capturing hand state information, switching the mapping subject of ray tracking according to the stretching and bending degree of the fingers and the finger movement speed, and determining the mapping subject; the mapping subject includes the whole hand and each finger;

[0007] T2, determining the number of information types, mapping scale, and projection mode of mapping according to the current mapping subject, and selecting targets in a three-dimensional space using the mapped ray; at least one of the number of information types, mapping scale, and projection mode of mapping corresponding to different mapping subjects is different.

[0008] As a preferred embodiment of the present application, the hand state information includes finger position information and angle information.

[0009] As a preferred embodiment of the present application, switching the mapping subject according to the stretching and bending degree of the fingers and the finger movement speed to determine the mapping subject comprises:

[0010] T1.1: reconstructing hand features according to the position information and angle information of the captured fingers in the initial state;

[0011] T1.2: dividing the finger flexion degree into three levels according to the hand features, namely a first flexion level, a second flexion level and a third flexion level;

[0012] T1.3: when the flexion degree of all fingers is the third level, or the flexion degree of more than or equal to two fingers is the first flexion level, the mapping subject of ray tracing is the whole hand;

[0013] When it is detected that the flexion degree of only one finger is the first flexion level, the mapping subject of ray tracing is the corresponding finger.

[0014] As a preferred embodiment of the present application, when the current mapping subject is a finger, if the motion speed of the finger is greater than that of other fingers, the mapping subject is ensured to be the finger, otherwise, the finger with the greatest motion speed is taken as the mapping subject.

[0015] As a preferred embodiment of the present application, the method for dividing the flexion level according to the finger flexion degree in step T1.2 is as follows:

[0016] When the finger R>R1 and P>P1, the finger is in the first flexion level, R represents the angle between the finger and the palm, P is the average of the angles between different knuckles of the finger, R1 is the first threshold value, and P1 is the second threshold value;

[0017] When the finger 0<R<R2 and 0<P<P2, the finger is in the third flexion level, R2 is the third threshold value; wherein R1>R2 and P1>P2;

[0018] When the first flexion level condition or the third flexion level condition is not met, the finger is in the second flexion level.

[0019] As a preferred embodiment of the present application, the mapping information type is one or both of the position information and the angle information of the finger.

[0020] As a preferred embodiment of the present application, the mapping ratio changes according to the level change of the motion speed or angular speed of the finger or the whole hand;

[0021] When the motion speed or angular speed of the finger or the whole hand is in the lowest level, the ratio of the movement distance of the ray starting point to the movement distance of the finger / whole hand is a constant value D1, and the ratio of the direction angle change of the ray to the direction angle change of the finger / whole hand is a constant value Q1;

[0022] When the movement speed or angular velocity of the finger or whole hand is at the highest level, the ratio of the moving distance of the ray starting point to the moving distance of the finger / whole hand is a constant value D2, and the ratio of the direction angle change of the ray to the direction angle change of the finger / whole hand is a constant value Q2; where D2>D1, Q2>Q1;

[0023] When the movement speed or angular velocity of the finger or the whole hand is between the lowest level and the highest level, the ratio of the moving distance of the starting point of the ray to the moving distance of the finger / whole hand is the variable value D, and the ratio of the change in the direction angle of the ray to the change in the direction angle of the finger / whole hand is the variable value Q; and the value of D is positively correlated with the movement speed of the finger or the whole hand, and the value of Q is positively correlated with the angular velocity of the finger or the whole hand, D1<D<D2, Q1<Q<Q2.

[0024] As a preference of the present invention, at the same movement speed or angular velocity, the mapping ratio of the whole hand is smaller than that of the fingers.

[0025] As a preferred embodiment of the present invention, the projection method of the mapping refers to a method of defining a starting point and an end point in a finger or a whole hand, and the direction of the line connecting the starting point and the end point is used as a reference direction of the ray.

[0026] Compared with the prior art, the advantages of the present invention are:

[0027] 1. Predict the user's interaction intention by the extension and flexion level of the fingers, switch the current mapping subject, improve the efficiency of selecting targets in the 3D scene, and reduce hand and arm fatigue

[0028] 2. According to the current mapping subject, different types of mapping information, projection methods, and projection ratios are flexibly used to compensate for the limitations of different finger types, so that the mapped rays can have a wider range of choices in the virtual three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A method for determining the finger extension and flexion level provided in an embodiment of the present disclosure;

[0030] Figure 2 A schematic diagram of switching a mapping subject provided by an embodiment of the present disclosure;

[0031] Figure 3 A schematic diagram of mapping finger motion to ray motion provided by an embodiment of the present disclosure;

[0032] Figure 4 A coordinate relationship diagram of a mapping ratio of finger motion to ray motion provided by an embodiment of the present disclosure;

[0033] Figure 5A schematic diagram of a three-dimensional target selection method based on finger motion as ray mapping provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is further explained in detail through specific implementation methods.

[0035] The above and other technical contents, features and principles of the present invention are clearly presented in the detailed description of the specific embodiments with accompanying drawings below. Through the detailed description of this embodiment, you can have a deeper understanding of the various technical means adopted by the present invention to achieve the above objectives.

[0036] like Figure 5 As shown, the present invention provides a three-dimensional target selection method based on finger motion, comprising:

[0037] S1, capturing basic information of the hand through a sensor, which can restore status information of each finger of the hand, wherein the status information of the finger includes position information and angle information of the finger;

[0038] In this step, the captured information includes but is not limited to the image information, electromyographic information, posture information, etc. of the fingers, which can reflect the state information of the fingers. This can be achieved through sensors such as cameras, electromyographic sensors, and gyroscopes; the initial state of the hand is captured according to the sensor, and the hand features are reconstructed in the computer.

[0039] S2, judging the current extension and flexion level of each finger based on the initial state of each finger in the reconstructed hand features;

[0040] In this step, the finger flexion degree is divided into three levels, namely the first flexion level, the second flexion level and the third flexion level. Figure 1 The following describes a method for judging the finger extension and flexion level:

[0041] According to the position and angle information of the fingers, the angle between the fingers and the palm is set as R, and the angle between different joints in the fingers is set as P. If there are three joints, then P is the average of the angles between the first and second joints, and the angles between the second and third joints.

[0042] When the finger R>R1 and P>P1, the finger is at the first extension and flexion level, R1 is the first threshold, and P1 is the second threshold;

[0043] When finger 0 <R<R2且0<P<P2时,手指为第三伸曲等级,R2为第三阈值,P2为第四阈值;其中R1> R2, P1>P2;

[0044] When the above-mentioned first flexion level condition or third flexion level condition is not met, the finger is at the second flexion level.

[0045] S3, determining a current mapping subject based on the current extension and flexion level of each finger, wherein the mapping subject includes each finger and the entire hand;

[0046] In this step, if Figure 2 The figure shows a finger mapping subject switching method, which includes the following steps:

[0047] When the extension and flexion degree of all fingers is the third level, or when the extension and flexion state of two or more fingers is the first level, the mapping subject of ray tracing is the whole hand;

[0048] When it is detected that the extension and bending state of one and only one finger is the first extension and bending level, the mapping subject of the ray tracing is the finger;

[0049] In a specific implementation of the present invention, when the current mapping subject is a finger type, if the movement speed of the finger is greater than that of other fingers, it is ensured to remain the mapping subject; otherwise, the finger with the fastest movement speed is used as the mapping subject.

[0050] S4, according to the current mapping subject, confirming information such as the number of types of information to be mapped, the mapping ratio, and the mapping projection method, and using the mapped ray to select a target in the three-dimensional space.

[0051] In this step, the mapping method includes, but is not limited to, the number of types of mapping information. The number of types of mapping information refers to the number of types of information related to the finger used for ray mapping. For example, the starting point and direction of the ray can be obtained by mapping the finger's position information, or by mapping the finger's position information and angle information. The number of types of mapping information for the former is 1, while the number of types of mapping information for the latter is 2.

[0052] Mapping scales include dynamic and static mapping scales. For example, the dynamic mapping scale changes based on the speed of the finger's displacement, while the static mapping scale uses a constant scale to map the finger's position or angle to the starting position and direction of the ray.

[0053] The projection method of the mapping refers to how the starting and ending points of the finger direction are defined. The direction of the line connecting the starting and ending points serves as a reference for the ray direction. For example, the starting and ending points can be the ends of the distal phalanx of the finger, or the base of the proximal phalanx can be used as the starting point and the tip of the distal phalanx can be used as the ending point.

[0054] like Figure 3 The figure shows a schematic diagram of mapping finger motion to ray motion.

[0055] When the left index finger is used as the mapping subject, the finger's position and angle information are mapped to the motion information of a ray in three-dimensional space, and the target in the direction of the ray mapped in three-dimensional space is selected. As the finger's movement speed and angular velocity change, the ratio of the distance traveled by the ray's starting point, as mapped from the finger's position coordinates, to the finger's actual movement distance changes, and the ratio of the angular change in the ray's direction to the angular change in the finger's actual direction also changes.

[0056] In this embodiment, Figure 4 As shown in (a) in FIG, the movement speed of the finger in the air is set to three levels, namely the first speed level, the second speed level and the third speed level.

[0057] When the finger's movement speed in the air is the first speed level, the ratio of the moving distance of the ray starting point to the moving distance of the finger vertex is D1, where 0 <D1<1;

[0058] When the finger moves in the air at the third speed level, the ratio of the moving distance of the ray starting point to the moving distance of the finger vertex is D2, where D2>1;

[0059] When the finger moves in the air at the second speed level, the ratio of the moving distance of the ray starting point to the moving distance of the finger vertex is D, where D1 <D<D2,D随着移动速度的增大而增大。

[0060] In this embodiment, Figure 4 As shown in (b) in FIG, the angular velocity of the finger moving in the air is set to three levels, namely the first angular velocity level, the second angular velocity level and the third angular velocity level.

[0061] When the angular velocity of the finger in the air is the first angular velocity level, the ratio of the ray direction angle change to the direction angle change of the distal phalanx of the finger is Q1, where 0 <Q1<1;

[0062] When the angular velocity of the finger in the air is the third angular velocity level, the ratio of the ray direction angle change to the direction angle change of the distal phalanx of the finger is Q2, where Q2>1;

[0063] When the angular velocity of the finger in the air is the second angular velocity level, the ratio of the ray direction angle change to the direction angle change of the distal phalanx of the finger is Q, where Q1 <Q<Q2,Q随着角速度的增大而增大。

[0064] In a specific implementation of the present invention, the mapping ratios at the same speed level will vary depending on the type of finger, including the ratio of the moving distance of the ray starting point to the moving distance of the finger vertex, and the ratio of the ray direction angle change to the direction angle change of the distal phalanx of the finger. For example, when the mapping subject is composed of Figure 2 The index finger shown switches to Figure 2 When the thumb is shown, the values ​​of D1, D2, D, Q1, Q2, and Q change. Those skilled in the art can define the ratios corresponding to different finger types according to actual needs, so that when the ray is actually manipulated by the finger, the selectable range is not reduced due to the limited actual motion space, ensuring that the mapped ray has a larger selection range in the virtual three-dimensional scene.

[0065] Similarly, those skilled in the art can define the number of types of information to be mapped and the mapping projection method according to different finger types according to actual needs. For example, when the mapping subject is Figure 2 When the index finger is shown, the number of types of mapping information is 1, that is, the starting point information and direction information of the ray are obtained by mapping the position information of the index finger, and the direction of the line connecting the base of the proximal phalanx and the top of the distal phalanx of the index finger is used as the reference basis for the ray direction; when the mapping subject is as follows Figure 2 For the thumb shown, the number of types of mapping information is 2, that is, the starting point information and direction information of the ray are obtained by mapping the position information and angle information of the thumb, and the direction of the line connecting the two ends of the distal phalanx of the thumb is used as a reference for the direction of the ray.

[0066] In a specific implementation of the present invention, when the entire hand is the mapping subject, the number of types of mapping information projected can be 1 or 2. The mapping projection method includes starting from the center of the wrist and ending at the tip of the index finger, or starting from the center of the palm and ending at the base of the proximal phalanx of the index finger. The mapping ratio can be a static mapping ratio or a Figure 4 Dynamic mapping scale shown.

[0067] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or imagined by a person skilled in the art from the disclosure of the present invention should be considered to be within the scope of protection of the present invention.

Claims

1. A three-dimensional target selection method based on finger motion as ray mapping, characterized in that: include: T1, capturing hand state information, switching the mapping subject of ray tracing according to the extension and flexion degree of the fingers and the finger movement speed, and determining the mapping subject; the mapping subject includes the whole hand and each finger; T2, based on the current mapping subject, determining the number of mapped information types, mapping ratio, and mapping projection method, and using the mapped rays to select a target in the three-dimensional space; different mapping subjects may correspond to different numbers of mapped information types, mapping ratios, and mapping projection methods; The mapping ratio changes according to the level of the movement speed or angular velocity of the finger or the whole hand; When the movement speed or angular velocity of the finger or whole hand is at the lowest level, the ratio of the moving distance of the ray starting point to the moving distance of the finger / whole hand is a constant value D1, and the ratio of the direction angle change of the ray to the direction angle change of the finger / whole hand is a constant value Q1; When the movement speed or angular velocity of the finger or whole hand is at the highest level, the ratio of the moving distance of the ray starting point to the moving distance of the finger / whole hand is a constant value D2, and the ratio of the direction angle change of the ray to the direction angle change of the finger / whole hand is a constant value Q2; where D2>D1, Q2>Q1; When the movement speed or angular velocity of the finger or the whole hand is between the lowest level and the highest level, the ratio of the moving distance of the starting point of the ray to the moving distance of the finger / whole hand is the variable value D, and the ratio of the change in the direction angle of the ray to the change in the direction angle of the finger / whole hand is the variable value Q; and the value of D is positively correlated with the movement speed of the finger or the whole hand, and the value of Q is positively correlated with the angular velocity of the finger or the whole hand, D1<D<D2, Q1<Q<Q2.

2. The three-dimensional target selection method based on finger motion as ray mapping according to claim 1, characterized in that: The hand state information includes finger position information and angle information.

3. The three-dimensional target selection method based on finger motion as ray mapping according to claim 2, characterized in that: The switching of the mapping subject according to the extension and flexion degree of the finger and the finger movement speed to determine the mapping subject includes: T1.1: Reconstruct hand features based on the captured initial finger position and angle information; T1.2: Based on hand characteristics, the finger flexion degree is divided into three levels: first flexion level, second flexion level, and third flexion level; T1.3: When all fingers are at the third level of flexion, or when two or more fingers are at the first level of flexion, the ray tracing mapping subject is the entire hand. When it is detected that the extension degree of one and only one finger is the first extension level, the mapping subject of the ray tracing is the corresponding finger.

4. The three-dimensional target selection method based on finger motion as ray mapping according to claim 3, characterized in that: When the current mapping subject is a finger, if the movement speed of the finger is greater than that of other fingers, the mapping subject is guaranteed to remain the finger, otherwise, the finger with the fastest movement speed is used as the mapping subject.

5. The three-dimensional target selection method based on finger motion as ray mapping according to claim 3, characterized in that: The method for classifying the finger flexion levels in step T1.2 is as follows: When R>R1 and P>P1 of the finger, the finger is in the first extension and flexion level, R represents the angle between the finger and the palm, P is the average of the angles between different knuckles in the finger, R1 is the first threshold, and P1 is the second threshold; When the finger satisfies 0 < R < R2 and 0 < P < P2, the finger is at the third flexion and extension level, where R2 is the third threshold value and P2 is the fourth threshold value; here, R1 > R2 and P1 > P2. When the conditions of the first flexion and extension level or the third flexion and extension level are not met, the finger is at the second flexion and extension level.

6. The three-dimensional target selection method based on finger motion as ray mapping according to claim 1, characterized in that: The types of information to be mapped are one or both of the position information and the angle information of the finger.

7. The three-dimensional target selection method based on finger motion as ray mapping according to claim 1, characterized in that: At the same movement speed or angular velocity, the mapping ratio of the whole hand is smaller than that of the finger.

8. The three-dimensional target selection method based on finger motion as ray mapping according to claim 1, characterized in that: The projection method of the mapping described refers to the definition method of the starting point and the ending point of the finger or the whole hand, and the direction of the line connecting the starting point and the ending point is used as the reference direction of the ray.

Images