A 3D sketching method in virtual reality based on a gesture-drawing surface

Abstract

The present invention discloses a method for three-dimensional sketching in virtual reality based on a gesture-drawing surface, including: making a static gesture with the non-dominant hand to create a drawing surface, and making a dynamic gesture with the non-dominant hand to manipulate the drawing surface and perform drawing on the drawing surface. In this application, the drawing surface is presented in the form of gesture interaction. Through the interaction method of Leapmotion gesture somatosensory recognition, using the two-handed interaction mechanism, it provides sketching assisted by the drawing surface, improving the aesthetics and interest of sketching in the VR environment.

Description

Technical Field

[0001] The present invention relates to the field of VR drawing, and particularly to a method for three-dimensional sketch drawing in virtual reality based on a gesture drawing surface. Background Art

[0002] Sketches are an important part of the design process. They have a positive impact on stimulating the creative thinking of artists and improving the quality of design works. Due to their autonomy and expressiveness, sketches have become an intuitive method for various visual design tasks such as painting, 3D modeling, and product design.

[0003] In particular, air sketches allow artists to directly draw 3D strokes in the air, but usually require extensive training. With the increasing popularity of consumer-grade augmented reality (AR) and virtual reality (VR) technologies such as Microsoft HoloLens and HTC Vive, many commercial applications have been developed to make it easier for artists to create air 3D sketches. Most of these applications, including Tilt Brush, Gravity Sketch, and Quill, provide an effective way to conceptualize ideas while also supporting creativity.

[0004] However, existing VR sketch software has the defect that the generated 3D sketches have lower accuracy compared to 2D sketches. This is manifested as a lack of depth perception ability. Research also shows that visual guides such as grids, scaffolding curves, and drawing surfaces can not only enhance the user's depth perception in a virtual reality environment but also improve the accuracy of strokes. However, the current method for creating drawing surfaces is both cumbersome and time-consuming.

[0005] To solve this problem, we introduce gestures to define the visual drawing surface. As a direct and powerful communication tool, gestures can conveniently describe the spatial characteristics and position information of these drawing surfaces. If designed properly, they are easy to remember because they are very similar to real-world references. Therefore, we believe that air gestures can allow users to naturally create drawing surfaces in VR that are similar to real-world objects. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for three-dimensional sketch drawing in virtual reality based on a gesture drawing surface to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above purpose, the present invention provides the following solution:

[0008] The present invention provides a method for three-dimensional sketch drawing in virtual reality based on a gesture drawing surface, including:

[0009] Make a static gesture with the non-dominant hand to create a drawing surface, and make a dynamic gesture with the non-dominant hand to manipulate the drawing surface and perform drawing on the drawing surface.

[0010] Optionally, the non-dominant hand is the non-preferred hand of the user.

[0011] Optionally, the drawing surface includes: a plane, a curved surface, a cylinder, a cone, and a sphere, and the dynamic gestures include: moving, rotating, and scaling.

[0012] Optionally, when making a static gesture with the non-dominant hand to create a drawing surface, it includes:

[0013] Obtain the gesture information of the user through a gesture tracker, and the gesture information includes: the type and parameters for creating the drawing surface and the parameters for operation;

[0014] Obtain the coordinates of the fingers based on the gesture information of the user;

[0015] Calculate the gesture type through a decision tree algorithm;

[0016] When the gesture type is determined to be a static gesture, generate a curved surface through a curved surface definition algorithm to create a drawing surface.

[0017] Optionally, when obtaining the gesture information of the user through a gesture tracker, it includes:

[0018] Divide the five fingers into three groups, the first group is the index finger, the second group is the thumb, and the third group is the remaining fingers;

[0019] Obtain the thumb direction, index finger direction, and the average direction of the remaining fingers, and the connection lines of each group of finger joints through the three groups of fingers, where the connection line of the third group of fingers is obtained by connecting the average coordinates of the same joint of the three fingers.

[0020] Optionally, when calculating the gesture type through a decision tree algorithm, it includes:

[0021] Obtain the normal direction of the palm;

[0022] Calculate all the included angles between the thumb direction, index finger direction, the average direction of the remaining fingers, and the normal direction of the palm;

[0023] And based on all the values of the included angles calculated by the decision tree algorithm, used to distinguish the gesture type.

[0024] Optionally, when generating a curved surface through a curved surface definition algorithm based on the gesture type, it includes:

[0025] When a planar or curved gesture is detected, a smooth curve of the index finger is obtained based on the connection line of the finger joints of the index finger as the baseline, a vertical line passing through the first joint of the third group of fingers is obtained for the baseline, and the baseline is moved along the vertical line to generate a surface;

[0026] When a cone or cylinder gesture is detected, an axis is obtained based on the connection line of the thumb joint and the connection line of the index finger joint, a generatrix is calculated based on the first joint, the second joint of the third group of fingers and the axis, and a surface is generated by rotating based on the generatrix;

[0027] When a sphere gesture is detected, a surface is generated using the default sphere.

[0028] Optionally, the process of making a dynamic gesture with the non-dominant hand to control the drawing surface includes:

[0029] When the non-dominant hand is less than 5 cm away from the drawing surface, the gesture is determined to be a dynamic gesture;

[0030] Calculate the distance between the index finger and the thumb. If it is less than 2 cm, the gesture is determined to be a pinch. Calculate the average distance between the five fingers as the grasping force. If the force is less than 5 cm, the gesture is determined to be a grasp;

[0031] Obtain the parameters required for moving, rotating and scaling and perform corresponding operations.

[0032] Optionally, the process of drawing on the drawing surface includes:

[0033] When the distance of the user's dominant hand from the drawing surface is less than 5 cm, drawing starts in two-dimensional mode;

[0034] The drawn strokes are bound within the drawing surface and automatically extend when drawn beyond the plane or surface range.

[0035] The present invention discloses the following technical effects:

[0036] The present invention presets a basic geometric body as a drawing surface; presets a drawing surface auxiliary drawing process, including surface generation, surface operation and stroke drawing; the drawing surface includes a simple drawing surface and a complex drawing surface assembled by geometric bodies; obtains user gesture data through the Leapmotion plug-in introduced in Unity; controls the generation of the basic geometric body as the drawing surface according to the user's static gesture data; modifies the position, angle and size parameters of the drawing surface according to the user's dynamic gesture data; controls the drawing of strokes through the user's controller movement and key operation; generates a corresponding type of drawing surface by identifying gesture categories through a decision tree algorithm; can solve the problem of inaccurate accuracy of existing 3D sketch drawing; can enhance the expression form of 3D sketches by combining multiple basic geometric bodies to form a drawing surface of a complex geometric body, and provide a high-fidelity sketch drawing effect. In this application, the drawing surface is presented in the form of gesture interaction. Through the interactive mode of Leapmotion gesture somatosensory recognition, a two-handed interaction mechanism is used to provide sketch drawing assisted by the drawing surface, thereby improving the aesthetics and fun of sketch drawing in a VR environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 Schematic diagram of gestures in the embodiments of the present application, where (a) is a grabbing gesture, (b) is a half-grabbing gesture, and (c) is an OK gesture and the cones generated by them;

[0039] Figure 2 The gestures included in the consensus set in the embodiment of the present application, wherein (a) is a plane gesture, (b) is a curved gesture, (c) is a cone gesture, (d) is a sphere gesture, (e) is a cylinder gesture, (f) is a move gesture, (g) is a rotation gesture, and (h) is a zoom gesture;

[0040] Figure 3 is the agreement score of each task in the embodiment of the present application;

[0041] Figure 4 Schematic diagram of gesture information in an embodiment of the present application, wherein (a) is a vector diagram and (b) is a joint line diagram;

[0042] Figure 5 A flowchart of a decision tree in an embodiment of the present application;

[0043] Figure 6Schematic diagrams of curved surface generation and sphere generation in the embodiments of this application, where (a) is a plane, (b) is a curved surface, and (c) is a sphere;

[0044] Figure 7 Schematic diagrams of the generation of a cylinder and a cone in the embodiments of this application, where (a) and (b) are cylinders, and (c) is a cone;

[0045] Figure 8 Flowchart in the embodiments of this application. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0047] This invention is carried out in a virtual reality environment. The participants wear HTC Vive head-mounted displays (HMDs) equipped with Leap Motion hand trackers. Leap Motion tracks the participants' hands and displays them in VR. These gestures are recorded as data records with millisecond timestamps for later analysis. The system was implemented using Unity (version 2020.1.3f1) in C# on a 3.60 GHz PC with Windows 10 and Nvidia GTX 1080 installed. In each task, a video recorder records the screen on the PC, and an RGB camera records the participants' gestures from the front. In addition, the participants are required to follow the think-aloud protocol. The entire process is recorded for later analysis.

[0048] Steps: First, the researchers briefly introduced VR and the Leap Motion hand tracker to the participants and described the experimental tasks in detail. Each participant was allocated 10 minutes to familiarize themselves with their virtual hand models in VR. Then, the software presented the tasks to each participant in a Latin square balanced order. For each geometric shape, the participants were required to depict it using static gestures made with their non-dominant hand, which best expressed its spatial characteristics while thinking aloud. In addition, how the participants relied on their non-dominant hands to manipulate these drawing surfaces (i.e., move, rotate, scale) was also studied. For example, after showing an animation of a drawing surface moving from left to right, the software restored the surface to its original position and waited for the participants' gesture input for movement.

[0049] Each task was repeated three times at different positions and perspectives. A total of 12 participants × 8 tasks × 3 times = 288 gestures were obtained. Among them, 16 were discarded due to user confusion and gesture overlap. After each task, participants were asked to rate the goodness and ease of use of the gestures on a 5-point Likert scale. Finally, semi-structured interviews were conducted to further understand the subjects' attitudes towards the gesture settings observed in the study. Each session lasted approximately 45 minutes to 1 hour.

[0050] The most commonly used gesture sets for creating and manipulating the drawing surface were analyzed. The observed gestures were listed according to different task categories. Further discussions on the observed gestures were made from the perspective of the participants and previous studies.

[0051] Plane: All participants reported a preference for using a flat hand with straight fingers. Regarding this gesture, 70% of the participants spread their fingers, and 30% kept their fingers together. This indicates that the user's gesture intention is not affected by whether the fingers are together or not, but by user fatigue, as users prefer relaxed gestures.

[0052] Curved surface: All participants made a curved hand to create a curved surface. Similarly, when making this gesture, participants focused more on the remaining four fingers rather than the thumb (50% put the thumb and index finger together, and 50% separated the thumb and index finger).

[0053] Cylinder: 90% of the participants made a C-shaped gesture with the thumb pointing at the other curved fingers. Among the remaining participants, the most common gesture was the grasping gesture. It can actually be regarded as a variant of the C-shaped gesture because it describes a cylinder with a radius smaller than that of the C-shaped gesture.

[0054] Cone: 50% of the participants made an OK gesture, which pinched the thumb and index finger and extended the other three fingers. 30% made a grasping gesture. Other participants reported a semi-grasping gesture in which the five fingers were spread out to imitate a cone, with the palm representing the vertex and the fingers representing the edges. These gestures are as Figure 1 shown, which can be explained by the fact that it is difficult to imagine creating a cone with a static one-handed gesture. But in reality, this posture would be very strange and difficult to associate with a cone. This was also illustrated in the later semi-interview. When using the OK gesture to make a cone, 90% of the participants agreed. Therefore, the OK gesture was designated for the cone.

[0055] Sphere: 70% of the participants preferred to use a fist gesture to create a sphere. Another 30% made a semi-grasping gesture to describe a sphere as if they had caught it in their imagination.

[0056] Movement: 23% of the participants reported making point gestures with the index finger, 24% made pinch gestures, and 53% preferred to move the surface with the palm rather than the fingers. Compared with previous studies, the results differed from those of dragging while shaking hands with the dominant hand, which might be due to the different proficiencies of the two hands. Since it is difficult to recognize meaningful pushing actions of the palm in a VR environment and it is also difficult to precisely control the position of the target, a pinch gesture was selected to pinch and move the drawing surface. However, this is still a valuable finding that can serve as guidance for gesture-based interfaces.

[0057] Rotation: All participants directly applied the relative rotation of the wrist to the object. After pinching the object, the user could directly rotate the drawing surface in 6DoF.

[0058] Scaling: In terms of the stretching task, 50% of the participants opened their fingers after grasping, and 40% of the participants opened their thumb and index finger after pinching. Therefore, an open grasp gesture was adopted to scale the drawing surface after grasping.

[0059] Two Likert scales were analyzed to determine the goodness and usability of the gestures. By comparing the scores of the consensus set (193 gestures) and the discarded set (79 gestures), the average scores for matching gestures and functions were 4.01 (SD = 0.715) and 3.53 (SD = 1.08), respectively. The average scores for usability were 4.41 (SD = 0.663) and 3.85 (SD = 0.891), respectively. In terms of goodness (F 1,270 = 14.93, P <.0001) and usability (F 1,270 = 32.464, p <.0001), the scores of the consensus set were significantly higher than those of the discarded set. Therefore, the gestures in the user-defined set were on average better than the discarded gestures. Figure 2 The gestures included in the consensus set are shown.

[0060] After that, the agreement scores were calculated according to the process adopted in previous work to measure the degree of consensus among the participants. Figure 3 The protocol scores for each task are shown. It can be found that the participants had the highest degree of consensus on planar gestures and curved gestures. The pose of the cone was difficult to imagine, and it was not easy for the participants to reach an agreement.

[0061] The drawing surface, as a complex three-dimensional geometric shape, can be composed of basic geometric shapes. Therefore, five basic geometric shapes (i.e., plane, curved surface, cylinder, cone, sphere) were defined for the study. For each of these geometric figures, multiple manipulation animations (i.e., movement, rotation, scaling) were generated. The participants were instructed to depict the shapes with gestures or use hand movements to perform the operations shown in the animations.

[0062] This application proposes a 3D sketching workflow that integrates gesture-based drawing surface generation and 2D sketching. First, the user can directly draw 3D strokes in mid-air or 2D strokes on a drawing surface according to their intention. For surface-based sketching, the user needs to make a static gesture with their non-dominant hand to indicate the intended drawing surface. Then, these gestures are recognized by an algorithm, and a corresponding feedback surface is presented, which can be regarded as a preview of the drawing surface. Once the trigger of the controller is triggered near the feedback surface, the same drawing surface is confirmed and created. If the generated surface does not meet the user's expectations (e.g., inappropriate size or position), the user can modify the surface by dynamically manipulating gestures (i.e., scaling, translating, and rotating). The incremental and iterative workflow includes a series of steps such as creating a drawing surface, manipulating the drawing surface, and drawing strokes. It allows the user to add more drawing surfaces and 3D strokes at any stage.

[0063] Leap Motion is used to track the user's gestures. The gesture information contains not only the type and parameters of the drawing surface to be created but also the parameters for operation.

[0064] Hand configuration: The five fingers of one hand are divided into three groups. These three groups of phalanges are used to define the surface. As Figure 1 shown, one is the index finger, which is the most commonly used finger to describe the surface, one is the thumb, which is used to distinguish expandable surfaces (such as cones, cylinders, and spheres) from other surfaces (such as planes and curved surfaces). The last group is composed of the other three fingers, which are not often used to depict surface features and are not used proficiently. Here, tJoints, iJoints, and oJoints represent the lines of the three groups of finger joints respectively. oJoints are connected by the average coordinates of the same joints of the three fingers in the last group. om is the second finger joint of oJoints, which will be used in the calculation later. represent the thumb direction, index finger direction, and the average direction of the other fingers respectively. O is the center of the palm, is the normal vector of the palm, as Figure 4 shown.

[0065] Gesture recognition: The type of gesture is distinguished by calculating the angles between vectors such as . Since the gesture changes continuously during the interaction, the decision tree (DT) algorithm is used to recognize the gesture and update the feedback drawing surface in real time. The gesture is judged based on the angles between four vectors . Specifically, there are six variables as the input of the model, which correspond to the angles between two of the four vectors, including the normal vector of the palm and the three vectors The predicted output of the model corresponds to five surface categories. After multiple actual tests, the optimal parameters of the decision tree model for gesture classification are selected. The test results show that the prediction accuracy of the optimal model is 97.6%. The algorithm process and required parameters are as Figure 5 shown. The distance between fingers is used to adjust the drawing surface parameters, so these parameters are not used in the recognition algorithm.

[0066] The rules for classifying operating gestures are much simpler. When a non-dominant hand is detected near the existing drawing surface (distance less than 5 cm), the drawing surface will be highlighted. At this time, the gesture is classified as a dynamic gesture for executing an operation command, so the rendering of the feedback surface is stopped. By calculating the distance between the index finger and the thumb and the average distance between the five fingers, it is easy to identify the pinching and grasping gestures. The determination method for pinching is: the distance between the index finger and the thumb is less than the distance when the index finger and the thumb are pinched together. In this embodiment, the distance between the index finger and the thumb is calculated. If it is less than 2 cm, the gesture is determined to be a pinch; the determination method for grasping is: the average distance between the five fingers is less than the distance when the five fingers grasp. In this embodiment, the average distance between the five fingers is calculated as the grasping force. If the force is less than 5 cm, the gesture is determined to be a grasp; and the Leap Motion API is used to obtain the parameters required for rotation and translation. Figure 5 In, tpAngle is and the angle between. toAngle is and the angle between. opAngle is and the angle between. ioAngle is and the angle between. ipAngle and tiAngle are discarded as they are useless. Here, the leaf node represents the gesture classification result after executing the decision tree.

[0067] Surface definition: Based on the hand contour defined above, a mapping from the gesture to the 3D primitive shape (i.e., the drawing surface) can be constructed. Two surface definition methods are introduced. One is the surface definition method based on translation, and the other is the surface definition method based on rotation. Most existing research relies on projecting the drawn strokes or creating the drawing surface. In the present invention, the lines of finger joints are used to replace the strokes.

[0068] Both the plane and the surface are based on the translation of the finger joint lines. If a planar gesture or a curved gesture is captured, the algorithm uses the smooth curve of iJoints as the baseline ( Figure 6 the baseline in), and the vertical direction of om and the baseline ( Figure 6The l) in it is used as the translation direction. In practice, the system expands the default surface size to a comfortable size (12 cm in our implementation). It can be seen that the curvature of the curved surface here is determined by the curvature of the baseline. In other words, the more the hand bends, the more curved the surface becomes.

[0069] Regarding the sphere, when a fist gesture is captured, we will create a default sphere at the position of the non-dominant hand ( Figure 6 (C)). In fact, its center O is located at the center of the fist, and the diameter is approximately the width of the palm (8 cm). All spheres created by default have the same size and can be scaled later using manipulation gestures. Figure 6 In (A) the plane and (B) the curved surface in it are calculated and generated by translating the lines of the finger joints, and (C) the spheres created by default have the same size.

[0070] If the Leap Motion Controller captures a C-shaped gesture or an OK gesture, the algorithm first calculates an axis perpendicular to the approximate circle formed by the tJoints and the iJoints ( Figure 7 the axis in it). This axis intersects the circle at its center point O, which determines the position and orientation of the curved surface. Then, we can calculate the generatrix l of the fictional shape based on om, the second joint of the oJoints, and the axis (see Figure 7 ). A cone or a cylinder can be formed by rotating l around the axis. In fact, the default axis length is set to a suitable size (12 cm in our implementation) to create a curved surface slightly longer than the width of the palm.

[0071] Regarding the cylinder, its generatrix l is a straight line passing through the point om and parallel to the axis ( Figure 4 (A)). For the cone, l is the hypotenuse of the triangle formed by om and the axis ( Figure 7 (C)). The projected length of the two generatrix curves on the axis is equal to the length of the axis. The distance d between om and the axis determines the scale of the fictional shape (i.e., the radius of the cylinder and the radius of the base of the cone). It can only be modified by adjusting the bending angle of the fingers. For example, when Figure 7 (A) the hand is held more tightly, the distance d becomes smaller, and the radius of the cylinder also becomes smaller ( Figure 7 (B)). Figure 7 In (A)(B) the cylinder and (C) the cone in it are calculated and generated by rotating the lines of the finger joints. The distance d between om and the axis in (A) is greater than that in (B), so the radius of the cylinder is larger.

[0072] The surface definition algorithm is based on finger joint vectors, so it is not limited to these five gestures. It has the potential to be extended to more gestures and surfaces. For shapes other than planes and spheres, users can create shapes with different proportions by changing the positions of specific fingers to meet the requirements of the sketch. The finger joints required by the algorithm are as Figure 6 and Figure 7 shown.

[0073] The process of creating a shape can be summarized as follows, as Figure 8 shown:[[]]

[0074] 1. Use Leap Motion to extract the information of the key joints in Figure 1 and calculate the relevant vectors.

[0075] 2. Use the decision tree shown in Figure 5 to identify gestures.

[0076] 3. Select the surface definition algorithm according to the gesture type:[[]]

[0077] a) When a plane or curve gesture is detected, calculate the baseline and l, and then use the translation method to generate the surface.

[0078] b) When a cone or cylinder gesture is detected, calculate the axis and l, and then use the rotation method to generate the surface.

[0079] c) When a sphere gesture is detected, use the default sphere to generate the surface.

[0080] The most popular method for creating 3D graphics using a computer is to use a NURBS-based modeler such as Maya or 3D Studio Max. This method is used in the system to generate the surface. In fact, due to the limitations of NURBS efficiency, multiple shapes are generated in advance following the surface definition algorithm instead of in real time. And according to the gesture types defined above and the relationships between the vectors, surfaces with corresponding effects are displayed.

[0081] Creating and manipulating the drawing surface using gestures:[[]]

[0082] When the user moves the non-dominant hand within the monitoring range of Leap Motion, the drawing surface is rendered as a semi-transparent feedback surface that follows the non-dominant hand to find a suitable position. Only when the user pulls the trigger of the controller or draws a stroke on it will the feedback surface "freeze" into the drawing surface. The presence of the drawing surface can reduce accidental surface creation.

[0083] Once the user's non-dominant hand approaches the drawing surface (distance less than 5 cm), the system will highlight the drawing surface. Instead, the drawing surface remains semi-transparent. This mechanism avoids having too many surfaces in the space to interfere with the user. When the drawing surface is highlighted by the non-primary hand, the system then interprets the next gesture as manipulation of the existing surface. Additionally, changing the distance between the controller and the drawing surface can also make the surface highlighted or semi-transparent.

[0084] The user can make all surfaces transparent by clicking the menu button on the controller to display the content consisting only of strokes, which helps to view the details of the sketch. The user can modify the strokes by undoing and redoing, or remove the surface by throwing it out of sight. To reduce accidental operations, a pinning function is also designed to prohibit any further modification of the surface.

[0085] Drawing on the surface

[0086] The system incorporates two-dimensional constrained sketching. The user needs to move the controller held by the dominant hand to the drawing surface (distance less than 5 cm). In this way, the strokes drawn by the user are bound to the highlighted surface and cannot penetrate or break away from it. The drawing surface reduces the inaccuracies caused by the user's arm movement. Additionally, when the user draws strokes beyond the default range of the plane or surface, the surface automatically extends without the need to create another surface again. This helps the user reduce the number of interaction steps involved in drawing larger objects and reduces fatigue.

[0087] In addition, the system also allows the user to draw freely in mid-air, which provides more freedom than a single sketch mode.

[0088] The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional sketching method in virtual reality based on a gesture-drawing surface, characterized in that, Including the following steps: Making a static gesture with the non-dominant hand to create a drawing surface, and making a dynamic gesture with the non-dominant hand to manipulate the drawing surface and perform drawing on the drawing surface; During the process of obtaining the user's gesture information through a gesture tracker, it includes: Dividing five fingers into three groups, the first group is the index finger, the second group is the thumb, and the third group is the remaining fingers; Obtaining the thumb direction, index finger direction, and average direction of the remaining fingers, as well as the connection lines of each group of finger joints through the three groups of fingers, where the connection line of the third group of fingers is obtained by connecting the average coordinates of the same joints of the three fingers; During the process of calculating the gesture type through a decision tree algorithm, it includes: Obtaining the normal direction of the palm; Calculating all the included angles between the thumb direction, index finger direction, average direction of the remaining fingers, and the normal direction of the palm; And using all the obtained included angle values calculated by the decision tree algorithm to distinguish the gesture type; During the process of making a static gesture with the non-dominant hand to create a drawing surface, it includes: Obtaining the user's gesture information through a gesture tracker, and the gesture information includes: the type, parameters for creating the drawing surface, and parameters for operation; Obtaining the coordinates of the fingers based on the user's gesture information; Calculating the gesture type through a decision tree algorithm; When the gesture type is determined to be a static gesture, generating a surface through a surface definition algorithm to create a drawing surface; During the process of generating a surface through a surface definition algorithm based on the gesture type, it includes: When detecting a plane or curve gesture, obtaining the smooth curve of the index finger based on the connection line of the index finger finger joints as the baseline, obtaining the vertical line of the baseline passing through the first joint of the third group of fingers, and moving the baseline along the vertical line to generate a surface; When detecting a cone or cylinder gesture, obtaining the axis based on the connection line of the thumb joint and the index finger joint, calculating the generatrix based on the first joint, second joint of the third group of fingers, and the axis, and generating a surface by rotating based on the generatrix; When detecting a sphere gesture, using the default sphere to generate a surface.

2. The method according to claim 1, wherein The non-dominant hand is the non-preferred hand of the user.

3. The method according to claim 1, wherein The drawing surface includes: plane, surface, cylinder, cone, and sphere, and the dynamic gestures include: moving, rotating, and scaling.

4. The method according to claim 1, wherein During the process of making a dynamic gesture with the non-dominant hand to manipulate the drawing surface, it includes: When the distance between the non-dominant hand and the drawing surface is less than 5 cm, determining the gesture as a dynamic gesture; Calculating the distance between the index finger and the thumb, if it is less than 2 cm, determining the gesture as a pinch, calculating the average distance between the five fingers as the grasping force, and if the force is less than 5 cm, determining the gesture as a grasp; Obtaining the parameters required for moving, rotating, and scaling and performing corresponding operations.

5. The method according to claim 1, wherein During the process of performing drawing on the drawing surface, it includes: When the distance between the user's dominant hand and the drawing surface is less than 5 cm, starting to draw in two-dimensional mode; The drawn strokes are bound within the drawing surface and automatically extend when drawn beyond the plane or surface range.

Images