Utilizing immersive pose selection surfaces in 3D modeling

By utilizing gestures and palm/back-of-hand orientation planes to identify the faces of 3D objects in a 3D immersive environment, the problems of selection ambiguity and cumbersome operation in existing technologies are solved, achieving more efficient and accurate face selection.

CN113126749BActive Publication Date: 2026-11-10DASSAULT SYSTEMES SA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202011592238.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-30
Filing Date
2020-12-29
Publication Date
2026-11-10
Estimated Expiration
2040-12-29

AI Technical Summary

Technical Problem

Existing CAD applications struggle to accurately select the faces of 3D objects in immersive 3D environments through natural hand interaction, resulting in selection ambiguity and cumbersome operations.

Method used

By displaying 3D objects in a 3D immersive environment, detecting gestures, and using the orientation plane formed by the palm and/or back of the hand, the face closest to the orientation plane is identified, and face selection is achieved by combining position and orientation information.

Benefits of technology

It resolves user selection ambiguity, improves selection accuracy and operational efficiency, provides a direct and natural interaction method, and reduces the difficulty of selecting faces of 3D objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113126749B_ABST
    Figure CN113126749B_ABST
Patent Text Reader

Abstract

This invention particularly relates to a computer-implemented method for selecting faces among the faces of a 3D object in a 3D immersive environment. Each face is oriented within the 3D immersive environment. The method includes displaying the 3D object in the 3D immersive environment. The method also includes detecting a gesture (S20). The gesture includes adduction of all fingers except the thumb. The method further includes determining an orientation plane formed by the palm and / or back of the hand in the 3D immersive environment. The method also includes identifying the face of the 3D object having the orientation closest to the orientation plane. This constitutes an improved method for selecting faces among the faces of a 3D object in a 3D immersive environment through hand interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly, to a method for selecting faces among the faces of a 3D object in a 3D immersive environment. Background Technology

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD is an acronym for Computer-Aided Design, which, for example, refers to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, which, for example, refers to software solutions for simulating the physical behavior of future products. CAM is an acronym for Computer-Aided Manufacturing, which, for example, refers to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play a crucial role in technical efficiency. These technologies may be embedded in Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage corporate knowledge to extend product development across the entire product lifecycle, from concept to the end of the product lifecycle. Dassault Systèmes (traded as CATIA, ENOVIA, and DELMIA) offers PLM solutions that provide an engineering center to organize product engineering knowledge, a manufacturing center to manage manufacturing engineering knowledge, and an enterprise center that enables the company to integrate and connect both the engineering and manufacturing centers. The entire system provides an open object model to link products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, which drives optimized product definition, manufacturing preparation, production, and service.

[0003] CAD applications allow for the interactive creation of accurate 3D objects. A 3D object is considered accurate when it faithfully replicates the corresponding real-world object. One standard is to minimize the difference between the envelope of the 3D object and the envelope of the actual object. Therefore, 3D objects traditionally created with CAD applications contain many small topological elements for the rough parts of the 3D object. Several methods have been implemented for selecting topological elements of interest for design. Traditionally, mouse-based methods are used.

[0004] Here, a small topological element can be defined as an element with a small visible surface. In a mouse-based approach, a small topological element can be defined as an element with a small clickable surface. In other words, because users need to define the mouse position very precisely, it is difficult to select small topological elements in a mouse-based approach. Topological elements may also frequently overlap and / or be partially or completely occluded. Therefore, even if topological elements are not small, their clickable surface may be very small or even zero in the case of a mouse-based approach.

[0005] In 3D applications, a well-known problem is the ambiguity surrounding which elements a user should select. In CAD applications, mouse-based methods tend to select the smallest element the cursor is pointing to, but this doesn't completely resolve ambiguity. A well-known solution to reduce this ambiguity is to allow users to specify which type of topological element they want to select. Using this information, elements can be filtered by type during selection. This solution is time-consuming and requires, for example, displaying various available filters in a 2D panel. Another issue during element selection is reducing the number of forced clicks required to select an element. For example, it is often necessary to zoom in on the area of ​​interest to select elements of interest within a 3D object. Changing the viewpoint of the 3D scene is also necessary when the element of interest is occluded. Both of these situations require several steps from the user and are time-consuming.

[0006] Recently, hand interaction has become increasingly important in 3D design applications. In fact, recent advancements in immersive technology have enabled the widespread application of full hand tracking in Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Virtual Reality is a 3D immersive environment that can be defined as an artificial environment created by software and presented to the user in a way that leads the user to believe and accept it as a real environment. In Virtual Reality (VR), the user's perception of reality is entirely based on virtual information. Augmented Reality (AR) and Mixed Reality (MR) differ from Virtual Reality (VR) in that a 3D immersive environment consists of an actual, real surrounding environment and virtual objects added as layers to that real environment. In AR and MR, additional computer-generated information is provided to the user, enhancing their perception of reality. On the other hand, in Virtual Reality (VR), the surrounding environment is entirely virtual. The difference between AR and MR is that users cannot directly interact with virtual objects in AR. In Mixed Reality, additional computer-generated information is "blended" together to create a realistic environment. Users can navigate the environment and interact with objects that are both real and virtual.

[0007] For example, augmented reality (AR) can display a virtual 3D box on a physical table. In the case of mixed reality (MR), the user might be able to pick up and open the box.

[0008] In the definitions of virtual, augmented, and mixed reality, a real object is an object that physically exists in the environment.

[0009] Existing CAD applications do not allow for the creation of accurate 3D objects in immersive environments using natural hand interactions.

[0010] Within this context, an improved approach is still needed for selecting topological elements such as faces of 3D objects using gestures in a 3D immersive environment within a CAD system. Summary of the Invention

[0011] Therefore, a method is provided for selecting faces among the faces of a 3D object in a 3D immersive environment. Each face is oriented in the 3D immersive environment. The method includes displaying the 3D object in the 3D immersive environment. The method also includes detecting gestures. Gestures include adduction of all fingers except the thumb. The method also includes determining an orientation plane formed by the palm and / or back of the hand in the 3D immersive environment. The method also includes identifying the face of the 3D object having an orientation closest to the orientation plane.

[0012] This approach constitutes an improved method for creating accurate 3D objects using hand interaction in immersive environments.

[0013] The method may include one or more of the following: - Each face of the 3D object and the orientation plane also have a normal, and wherein the identification (S40) further includes: identifying the face of the 3D object that has the closest orientation between the normal of the face and the normal of the orientation plane. - Each face of the 3D object and the orientation plane also have a position in the 3D immersive environment, and wherein the recognition (S40) further includes: ∙ Minimize function f : in: o It's the side that interests me; o and and ; o It is in a 3D immersive environment The position relative to the orientation plane of the hand H The Euclidean distance between them; and o It is a surface in a 3D immersive environment The angle between the normal to the direction plane of the hand and the normal to the direction plane of the hand. - Function item Replace with ,in, Calculated in the following way: in: o H’ It is the position of the orientation plane of the virtual hand; o E It is the position of the user's viewpoint; o H It is the position of the user's hand in the orientation plane; o EH It is a vector from E to H; o Omax It is the position of the face of the 3D object that is farthest from E; o It is the position of the user's hand center that is furthest from E on the horizontal plane of the user's body; o ; o From E arrive Omax The norm of the vector; and o From E arrive The norm of the vector; - Function item or It also includes the position of the face in the 3D immersive environment and the Euclidean distance between at least one intersection of the 3D model and a ray, the ray being projected from the center of the head or a point between the dominant eye or eyes to the center of the 3D model. - Identification (S40) includes: • Identify (S400) the face of the 3D object that has the closest orientation to the orientation plane; • Calculate one or more n neighboring faces of the face determined by (S410), one or more neighboring faces, and the determined face to form a subset of selectable faces; and • Identify (S420) the face of the 3D object that has the closest orientation to the orientation plane from a subset of selectable faces. - Modify the first rendering of faces in a subset of selectable faces. - Modify the second rendering of the identified faces. - The faces of the 3D object can be triangles and / or quadrilaterals and / or topological surfaces and / or parametric surfaces and / or process surfaces. - The gesture (S20) detected includes the adduction of all fingers except the thumb and the abduction of the thumb. - Selecting identified faces of 3D objects by detecting gestures, including thumb tucking. - The detected gesture also includes the abduction of all fingers except the thumb and deselecting the selected face. - Detected gestures also include: • Extension of the proximal and distal interphalangeal joints of all fingers except the thumb was detected; and / or • Extension of the metacarpophalangeal joints of all fingers except the thumb was detected; and / or • Gesture detection also includes detecting actual contact between all adjacent fingers except the thumb.

[0014] A computer program is also provided, which includes instructions for performing the method.

[0015] A system is also provided, which includes a display and processing circuitry communicatively coupled to a memory storing the computer program. Attached Figure Description

[0016] Embodiments of the invention will now be described by way of non-limiting examples and with reference to the accompanying drawings, in which:

[0017] - Figure 1 An example flowchart of the method is shown;

[0018] - Figure 2 An example flowchart of step S40 of the method is shown;

[0019] - Figure 3 An example of the system is shown;

[0020] - Figure 4 An example of a rear-front view of the skeleton of the right hand in the default pose is shown;

[0021] - Figure 5 This is an example of the finger abduction and adduction directions in the front-back view of the virtual skeleton of the right hand in the default pose.

[0022] - Figure 6 This is an example showing the directions of finger flexion, extension, and hyperextension in a side view of the virtual skeleton of the right hand in the default posture;

[0023] - Figure 7 An example of a rear-front view of the virtual skeleton of the right hand in the default pose is shown;

[0024] - Figure 8 An example side view of the virtual skeleton of the right hand in the default pose is shown;

[0025] - Figure 9 An example of a rear-front view of the virtual skeleton of the right hand in the detected pose is shown;

[0026] - Figure 10 An example side view of the virtual skeleton of the right hand in the detected pose is shown;

[0027] - Figures 11 to 21 The method is shown. Detailed Implementation

[0028] refer to Figure 1 The flowchart presents a computer-implemented method for selecting faces from the faces of a 3D object in a 3D immersive environment. Each face is oriented within the 3D immersive environment. The method includes displaying the 3D object S10 in the 3D immersive environment. The method also includes detecting a gesture S20. The gesture includes adduction of all fingers except the thumb. The method further includes determining an orientation plane S30 formed by the palm and / or back of the hand in the 3D immersive environment. The method also includes identifying the face of the 3D object with the closest orientation to the orientation plane S40.

[0029] This constitutes an improved method for selecting a face of a 3D object using hand interaction within a 3D immersive environment of a CAD system. Notably, this method resolves the ambiguity surrounding which elements a user wants to select. As mentioned above, 3D objects in CAD applications contain many small elements. Selecting elements using mouse-based methods can be difficult due to the element's size or the size of its visible surface from the user's perspective. In fact, if the size is small, using mouse-based methods requires the user to move the mouse very precisely and / or zoom in and / or change the viewpoint of the 3D scene. The method presented here addresses this deficiency. Essentially, this method uses the orientation of the planes of the back and / or palm of the hand during the gesture to determine the element the user wants to select. This method can also combine the orientation of the planes of the back and / or palm of the hand with the hand's position during the user's gesture to determine which element the user wants to select. By using this combination of position and orientation, all elements of the 3D object can be distinguished and thus selected by the user using this method.

[0030] The method is computer-implemented. This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the execution of the method's steps by a computer may be fully automatic or semi-automatic. In the example, at least some steps of the method may be triggered through user-computer interaction. The required level of user-computer interaction can depend on the anticipated level of automation and be balanced with the need to fulfill the user's intentions. In the example, this level may be user-defined and / or predefined.

[0031] For example, detection S20 and determination S30 depend in part on the user's action. Detection S20 is performed as a result of a user action that includes a user gesture in which all fingers except the thumb are adducted. Determination S30 is performed as a result of a user action that includes a user gesture (wherein an orientation plane is formed using the user's palm and / or the back of the hand).

[0032] A typical example of a computer implementation of the method is to execute it using a system suitable for this purpose. This system may include a processor coupled to memory and a graphical user interface (GUI) on which a computer program is recorded, containing instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for this type of storage and may comprise several physically distinct parts (e.g., one for the program and possibly another for the database).

[0033] Figure 3 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0034] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and random access memory (RAM) 1070 also connected to the bus. The client computer is also provided with a graphics processing unit (GPU) 1110 associated with video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device (e.g., a hard disk drive 1030). Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the above may be supplemented by or included in a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used on the client computer to allow the user to selectively position the cursor at any desired location on the monitor 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0035] The computer program may include computer-executable instructions, which include units for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, as digital electronic circuitry or computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps can be performed by the programmable processor executing the instruction program to perform the function of the method by manipulating input data and generating output. Therefore, the processor may be programmable and coupled to receive and send data and instructions from the data storage system, at least one input device, and at least one output device. If desired, the application program may be implemented in a high-level procedural or object-oriented programming language, or assembly or machine language. In any case, the language may be a compiled language or an interpreted language. The program may be a complete installation or update program. In any case, the application of the program on the system receives instructions to perform the method.

[0036] This method manipulates 3D objects. Any subsequent references to a 3D object will refer to the digitally modeled 3D object, not the physical object. A digitally modeled 3D object can represent the geometry of a product to be manufactured in the real world after its virtual design is completed using, for example, CAD software solutions or CAD systems. Examples include (e.g., machinery) parts or assemblies of parts (or equivalently, assemblies of parts, since from a methodological perspective, an assembly of parts can be viewed as the parts themselves, or the method can be applied independently to each part of the assembly) or more generally, any rigid body assembly (e.g., a moving mechanism). CAD software solutions allow for product design in a wide range of unrestricted industrial sectors, including: aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine and / or offshore oil / gas production or transportation. Therefore, the 3D modeling object here can represent an industrial product, which can be any mechanical part, such as: parts of land vehicles (including, for example, automobile and light truck equipment, racing cars, motorcycles, truck and motor vehicle equipment, trucks and buses, trains), parts of aircraft (including, for example, fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), parts of marine vehicles (including, for example, naval equipment, commercial ships, marine equipment, yachts and workboats, marine equipment), general mechanical parts (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment products, metal products, tire products), electromechanical or electronic parts (including, for example, consumer electronics products, safety and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical devices and equipment), consumer goods (including, for example, furniture, home and garden products, leisure products, fashion products, products of hard goods retailers, products of soft goods retailers), and packaging (including, for example, food and beverage and tobacco, beauty and personal care, and household product packaging).

[0037] 3D objects consist of at least three topological entities: faces, edges, and vertices. By definition, a face is a bounded portion of a surface. An edge is a bounded portion of a curve. A vertex is a point in 3D space. Their relationships are as follows: A bounded portion of a curve is defined by two points (vertices) located on the curve. A bounded portion of a surface is defined by its boundary, which is a set of edges located on the surface. The edges of a face's boundary are connected by sharing vertices. Faces are connected by sharing edges. If two faces share an edge, they are adjacent. Similarly, if two edges share a vertex, they are adjacent. All edges of a face may or may not lie in the same plane. The normal of a face can be calculated by taking the cross product of the vectors of two edges of the face. If not all edges of a face lie in the same plane, it is worth noting that various normals can be calculated based on the edges chosen for calculating the cross product. Normals can also be edited. For example, if all edges of a face are not in the same plane, the normals are typically edited. Normals give the orientation of a face. In other words, normals determine the front and back faces of a face. Traditionally, for 3D objects, the normals of their faces point outwards from the 3D object. For simplicity, we will consider this situation from now on, and it can be understood that the method of the present invention can be performed using the normals of faces of a 3D object that point inwards from the 3D object. The orientation of a face can be derived from its direction. The orientation of a face is specified by a line parallel to its normal. The orientation of this line of the face can also be edited. Faces also have positions. The position of a face can be given by its centroid. The position of a face can also be edited.

[0038] From now on, we assume that all orientations, directions, and positions are calculated within the reference frame of the 3D immersive environment.

[0039] 3D objects can also be defined using edges or lines (and in some cases, faces or surfaces). Lines, edges, or surfaces can be represented in various ways, such as non-uniform rational B-splines (NURBS). These edges, lines, and / or surfaces can be used as input, from which the geometry of a 3D object can be generated, which in turn allows for the generation of a representation of the 3D object. The methods described herein can be used for 3D objects based on non-uniform rational B-spline (NURBS) lines, edges, and surfaces. More generally, surfaces can also be parametric surfaces and / or procedural surfaces. Parametric surfaces are Euclidean surfaces. The surface in, which consists of two parameters The parametric equations are defined. A process surface is a surface defined as a process. A subdivision surface can be considered as a process surface. A subdivision surface is a method of representing a smooth surface by specifying a coarser piecewise linear polygonal mesh.

[0040] Now for reference Figure 1The method displays at least one 3D object in a 3D immersive environment (S10). If more than one 3D object is displayed in the 3D immersive environment, one of the 3D objects is identified as the 3D object of interest. From now on, the 3D object of interest will be referred to as the 3D object.

[0041] As described above, 3D immersive environments can be used in virtual, augmented, or mixed reality. A 3D immersive environment has a reference frame. Therefore, in this 3D immersive environment, all objects and all their topological elements can be positioned relative to this reference frame. This method can then be used in virtual, augmented, and mixed reality to select the faces of any 3D object in the 3D immersive environment. This method can also be used in any other reality that includes environments with a reference frame that allows the positioning of at least one 3D object.

[0042] Then, a gesture is detected (S20). The gesture includes the adduction of all fingers except the thumb.

[0043] The interaction between a user and a 3D immersive environment can be accomplished by tracking the position and orientation of a part of the user's body, or by tracking the corresponding position and orientation of several parts of the user's body, or even by tracking the position and orientation of the entire user's body.

[0044] In the example, the part of the user's body is the user's hand.

[0045] Hand tracking provides direct and natural interaction and improves the immersive experience. In fact, users don't need to use specific virtual, mixed, or augmented reality hand controllers. Traditionally, to interact with 3D objects in a 3D immersive environment, one approach involves pointing a virtual 3D line at the 3D object and then clicking a button on a specific controller.

[0046] Hand tracking allows the detection of specific hand poses or gestures. For example, a video recorder or glove can be used to track a hand in a specific pose, while an accelerometer can be used to track a hand in a specific gesture. This method can be used with either the user's right or left hand. A pose is the position of a part of the body, and a gesture can be defined as a movement that allows a part to change its pose. In other words, a second pose can be seen as the result of a gesture that begins with a first pose. Therefore, a gesture can be detected by detecting a second pose and comparing it to the first pose. If the detected second pose is the same as the first pose, no gesture has occurred. If the detected pose is different from the first pose, the gesture can be defined as the difference between the second and first poses. More generally, a gesture can be defined as a change in pose triggered by a change in the position and / or orientation of a part of the user's body (e.g., the user's hand). For example, in this method, gesture detection can consist of calculating the difference between the current pose and a default pose. In this example, the default pose corresponds to the first pose, and the current gesture corresponds to the second pose. The difference can be inferred using changes in the position and / or orientation of the user's hand. It is worth noting that hand tracking for a specific pose is performed by a device that captures the pose at a high frame rate (e.g., at least 10 frames per second), so the risk of not detecting such motion is quite low: it involves motion from the first pose to the first pose with a second intermediate pose. The default pose can be any gesture that allows for the detection of adduction of all fingers except the thumb. An example of the default pose for the right hand is... Figure 7 and Figure 8 The default posture is shown in the diagram. This default posture is an example of a common posture for a resting right hand. In other words, the default posture is an example of a right-hand posture in which all muscles are relaxed. The default posture can also be defined as any posture that will not be detected as a specific posture used in any step of the method. From now on, for illustrative purposes only, all gestures will be defined as... Figure 7 and Figure 8 The difference in the default pose shown.

[0047] Gesture detection can be accomplished in various ways. In this example, gestures can be detected by mathematically comparing the relative positions and / or orientations of different body parts, with a tolerance to account for morphological variations. Note that the system may need to calibrate line segments to fit the user's morphology. In another example, gesture detection can also be achieved using machine learning techniques by training a neural network to distinguish between different possible gestures.

[0048] In the example, this can be achieved through a virtual skeleton 150 (e.g., as...). Figure 5(As shown) to provide kinematic parameters for hand gestures. A virtual skeleton comprises one or more joints and one or more bones. This virtual skeleton does not need to replicate the real skeleton. In other words, there may not be a direct correspondence between the joints and bones of the virtual skeleton and those of the real skeleton. For example, a virtual skeleton of the hand may have fewer or more joints than the real skeleton of the hand. Furthermore, even if each joint of the virtual skeleton corresponds to a joint of the real skeleton, their position and orientation may vary. The gestures or postures of the real skeleton can be detected using the linear and angular velocities, position, and orientation of the joints of the virtual skeleton. For simplicity, we will consider replicating the real skeleton of the hand later.

[0049] It is worth noting that the method of the present invention is independent of the detection methods used to detect the various postures described. Therefore, any technology capable of detecting posture changes (e.g., changes in the position and / or orientation of a part of the user's body) based on or not based on a virtual skeleton can be used.

[0050] exist Figure 4 The image shows an example of the actual skeleton of a hand. A hand contains five fingers, commonly referred to as: thumb, index finger, middle finger, ring finger, and little finger. The thumb is composed of the metacarpophalangeal joint 10 and the interphalangeal joint 60. The index finger is composed of the metacarpophalangeal joint 20, the proximal interphalangeal joint 70, and the distal interphalangeal joint 110. The middle finger is composed of the metacarpophalangeal joint 30, the proximal interphalangeal joint 80, and the distal interphalangeal joint 120. The ring finger is composed of the metacarpophalangeal joint 40, the proximal interphalangeal joint 90, and the distal interphalangeal joint 130. The little finger is composed of the metacarpophalangeal joint 50, the proximal interphalangeal joint 100, and the distal interphalangeal joint 140. The rest of the hand can be divided into three regions. The first region is the palm, which is the central area of ​​the front part of the hand. The second region is the back of the hand, which is the corresponding area of ​​the palm at the back of the hand. The third region is the heel of the hand, located in the proximal part of the palm.

[0051] Adduction is a posture in which body parts align with the body's midline. For example, in... Figure 5 As shown in the example, for the fingers other than the thumb, adduction is a gesture that moves each finger toward line 190. Figure 5 In the example shown, line 190 can be defined by the center of gravity of the hand and the longitudinal axis of the hand. Adduction 240 of all fingers except the thumb is the adduction of the metacarpophalangeal joints 20, 30, 40, and 50 of the index, middle, ring, and little fingers, respectively. This gesture is the result of contraction of muscles including the intercarpal muscles. Adduction 240 of the thumb moves the thumb away from line 190 by moving it into the plane of the palm. Adduction 240 of the thumb is the adduction of the metacarpophalangeal joint 10. Adduction of the thumb is the result of contraction of muscles including the adductor pollicis muscle.

[0052] As in Figure 6As shown in the examples, flexion 260 is a gesture of moving the index, middle, ring, and little fingers from line 190 toward the palm of the hand. Extension 270 is a gesture of moving the index, middle, ring, and little fingers toward line 190 and away from the palm of the hand. Hyperextension 280 is a gesture of moving the index, middle, ring, and little fingers away from line 190 and toward the back of the hand. These three gestures involve the metacarpophalangeal joints 20, 30, 40, and 50 of the index, middle, ring, and little fingers, the proximal interphalangeal joints 70, 80, 90, and 100, and the distal interphalangeal joints 110, 120, 130, and 140, respectively. Extension is the result of the contraction of muscles including the extensor digitorum muscles, the interosseous muscles of the back of the hand and the palmar muscles. Thumb adduction is a gesture of bringing the thumb toward the palm of the hand. Thumb adduction is primarily produced by the adductor muscles. Thumb adduction may also bring the thumb toward the index finger and the side of the palm. Thumb abduction is the gesture of moving the thumb away from the palm of the hand. Thumb abduction can also move the thumb away from the index finger and the side of the palm. Adduction of the thumb is caused by the abductor pollicis longus muscle. Adduction and abduction of the thumb involve the metacarpophalangeal joint 10 of the thumb. Thumb flexion moves the thumb toward line 190. Thumb flexion is mainly produced by the flexor pollicis brevis muscle. Thumb extension moves the thumb away from line 190. Thumb extension is mainly caused by the extensor pectineus longus muscle. Adduction, abduction, flexion, and extension of the thumb involve the metacarpophalangeal joint 10 of the thumb.

[0053] Back Figure 1 After detecting a gesture S20 involving the adduction of all fingers except the thumb, the method includes determining an orientation plane S30 formed by the palm and / or back of the hand. Determining the orientation plane S30 is based on evaluating the orientation of the palm and / or back of the hand. When all fingers except the thumb are adducted, the palm and / or back of the hand can be considered a flat surface. Therefore, when all fingers except the thumb are adducted, the back of the hand can be considered a plane. To approximate the orientation of the palm and / or back of the hand when all fingers except the thumb are adducted, the normal to the plane is calculated using the cross product by using the normal to any point on the plane or by using the positions of three points on the plane.

[0054] Now let's discuss Figure 18 . Figure 18 An example is shown where S30 is determined by using the orientation of the wrist joint 160 of the virtual skeleton 150. In fact, the wrist joint 160 of the virtual skeleton 150 is a point representing the orientation plane of the palm and / or back of the hand, so the orientation of the wrist joint 160 is the orientation of the orientation plane.

[0055] Figure 19Another example is shown where S30 is determined based on the positions of the wrist joint 160, the index finger's metacarpophalangeal joint 170, and the little finger's metacarpophalangeal joint 180. These three positions form a triangle. The normal to the triangle can be calculated by taking the cross product of the vectors of two sides of the triangle. The normal gives the direction of the palm and / or back of the hand, depending on the sides chosen for calculating the cross product. The normal also depends on the order in which these sides are used to calculate the cross product. The orientation of the triangle can be specified by a line perpendicular to the plane on which all the angles of the triangle lie. The orientation of the triangle can also be edited and therefore not perpendicular to the plane.

[0056] Determining S30 can also be done without any virtual skeleton. As an example, determining S30 can be based on a neural network trained to determine S30 from a video recorder.

[0057] Using the palm and / or back of the hand allows the user to provide orientation to the system, making the method more reliable. In fact, the relative flatness of these areas is slightly determined by the hand's gesture. Furthermore, these areas represent the largest surface area of ​​the hand, which increases the reliability of determining its orientation. The position of the palm and / or back of the hand can, for example, be located at the wrist joint 160 or at the centroid of the triangle formed by the wrist joint 160, the metacarpophalangeal joint 170 of the index finger, and the metacarpophalangeal joint 180 of the little finger.

[0058] The method then further includes identifying, in S40, the faces of the 3D object that have the orientation closest to the orientation plane. It should be understood that each face of the 3D object defines a plane, and the orientation closest to that orientation plane is determined by comparing the orientation plane with each (or at least a portion of) the planes formed by these faces. An example of determining the closest orientation will now be discussed.

[0059] As an example, the orientation of each face can be specified using lines. For each face, the angle between the line representing its orientation and the line representing the orientation of the plane formed by the palm and / or back of the hand is calculated. The face with the smallest angle value is the face of the 3D object with the orientation closest to the orientation plane. Here, we assume all angles are defined between [0; 2π].

[0060] In the examples disclosed herein, recognition S40 can be performed iteratively by considering each face of the 3D object. It should be understood that when determining the face of the 3D object having the orientation closest to the orientation plane, only a subset of the faces of the 3D object may be used. For example, only the face closest to the user's hand may be used (e.g., based on Euclidean distance).

[0061] Using orientation to select a face among the faces of a 3D object can speed up the selection of the face of interest. It also reduces ambiguity for the user regarding which face they want to select. In fact, an accurate 3D object is a digitally modeled 3D object that can replicate a physical object with very high precision. For example, an accurate 3D model typically has many small faces for replicating rough surfaces and large faces for replicating flat surfaces. In this case, adjacent faces of an accurate 3D object have significant orientation differences. Therefore, selecting a face among the faces of a 3D object by using orientation is particularly useful when the 3D object has many small faces, which is the case for accurate 3D objects.

[0062] In the example, identification S40 may include identifying the face of the 3D object that has the closest orientation between the normal vector of the face and the normal vector of the orientation plane. For example, identification S40 can be performed iteratively by considering each face or subset of faces of the 3D object. For each face considered, the angle between its normal vector and the normal vector of the orientation plane is calculated. The face with the smallest angle is the face of the 3D object that has the closest orientation to the orientation plane. Here, we assume all angles are defined between [0; 2π].

[0063] By using direction instead of orientation during S40 recognition, ambiguity regarding which face the user wants to select is further reduced. In fact, if two faces have opposite orientations, where the normal of one of these faces is parallel to the normal of the orientation plane, using direction instead of orientation during S40 recognition eliminates ambiguity about which face the user wants to select. Figure 15 As shown, using orientation to select a face among the faces of a 3D object is particularly useful for selecting the face 220 located on the back of the object from the user's perspective. As an example, here, the orientation plane of the back of the hand is used during recognition S40. Therefore, by turning the right palm towards the user's head, this method allows them to select the face located on the back of the object as viewed from the user's perspective. Figure 14 This shows a 3D object from the user's perspective. (Example) Figure 14 As shown, using orientation to select faces among the faces of a 3D object is also particularly useful for selecting faces 220 that have a small visible surface from the user's perspective.

[0064] In the example, identifying S40 can include identifying a surface within a surface, which is minimized by the following equation (1), denoted as . f Functions: (1) in: - It's the side that interests me; - and and ; - It is in a 3D immersive environment Yes, the position relative to the orientation plane of the hand. H The Euclidean distance between them; and - It is a surface in a 3D immersive environment The angle between the normal to the direction plane of the hand and the normal to the direction plane of the hand.

[0065] function It is a weighted sum of two terms. In fact, the first term... Multiply by a factor And the second item Multiply by a factor .

[0066] First weighting term It is in a 3D immersive environment The position relative to the orientation plane of the palm and / or back of the hand. H The Euclidean distance between them. For example, the surface The position can be located at the centroid of the plane. For example, the orientation plane of the palm and / or back of the hand can be located at the wrist joint 160 of the virtual skeleton.

[0067] Second weighting term It is noodles The angle between the normal vector of the hand and the normal vector of the orientation plane of the palm and / or back of the hand. The second weighting term can also be based on the surface. The orientation of the palm and / or back of the hand, and the orientation of the planes of the hands, rather than using their directions.

[0068] Each weight and The sum of the two weights can be equal to or greater than 0. The sum of the two weights must be strictly greater than 0. If one of the weights is equal to 0, it means the function only considers one of the two terms. For example, it can be... It can be set between 0.1 and 0.5, and can... Set between 0.5 and 0.9.

[0069] Weight and It can be pre-calculated. For example, it can be pre-calculated based on the number and size of the faces of a 3D object. Weights and For example, it can be pre-computed when loading a 3D immersive environment containing 3D objects. If the 3D object has many facets, the weight of the second term may be greater than the weight of the first term. The weights can also depend on the device used. For example, some devices may give limited accuracy in hand position but better orientation accuracy. The weights may also vary during the method. For example, the weights can vary depending on the number and size of the faces of the 3D object located on the surface of the 3D object in the user's field of view. The field of view can be defined as the extent of the observable environment at a given time. Therefore, since the field of view may change during the method, the weights... and It may also change.

[0070] In the example, recognition S40 can be completed iteratively by considering each face of the 3D object. In another example, recognition S40 can be completed iteratively by considering each face of a subset of the 3D object. For each face, a function is calculated. f The result. The surface with the minimum value is the surface that makes the function... f Minimize the faces of a 3D object.

[0071] In the example, the function f item Can be replaced with item Calculate using formula (2) in the following manner. : (2) in: - H’ It is the calculated position of the center of the virtual hand; - E It is the position of the user's viewpoint; - H It is the center position of the user's hand; - EH From E arrive H ; - Omax It is the position of the face of the 3D object that is farthest from E; - It is the position of the user's hand center that is furthest from E on the horizontal plane of the user's body; - ; - From E arrive Omax The norm of the vector; and - From E arrive The norm of the vector;

[0072] H’ This is the calculated position of the virtual hand's orientation plane. A virtual hand can be defined as a hand that, in a 3D immersive environment, is positioned from the user's viewpoint. E See, it's virtually positioned at least further than the user's hand. To ensure that from the user's perspective, the item... Since it's always related to the distance to the surface, using a virtual hand is useful. From the user's perspective... E From the user's perspective, a virtual hand might not be closer than the user's hand. E It may be located, for example, in the center of their head, or in the area of ​​their dominant eye, or in the middle of their eye. Omax From the user's perspective E Look at the position of the furthest face of the 3D object. For example, this can be determined from the user's viewpoint at the beginning of the method, such as at the start of detection S20. E Look at the face that is considered the furthest face of a 3D object. For example, from the user's viewpoint. E The face considered the furthest from the 3D object can also be updated throughout the method. This applies if the user's viewpoint changes during this method. E If the 3D object moves, the update is from the user's viewpoint. E Looking at the face that is considered the furthest face of a 3D object is particularly useful. In other words, as an example, it can be determined during detection S20. Omax And can be updated throughout the method. Omax . Hmax The distance between the user's hand orientation plane and the user's viewpoint in the horizontal plane of the user's body. E The furthest position. Therefore, Hmax Is it possible to consider the distance from the user's viewpoint when using this feature? E The distance method is used to select the surface index. Hmax For example, the position of the user's hand orientation plane when the user's arm is extended horizontally in front of them. It can be pre-calculated, for example, during the calibration of the line segment. It can also be set by the user. Used as Factors. For example, It is set by the user before using this method. If = 1 means that when in the horizontal plane of the user's body from the user's viewpoint E The distance between the furthest faces of a 3D object is greater than that from the user's viewpoint. E When viewing the farthest point of the user's hand in the directional plane, the virtual hand will appear farther away than the user's hand in a 3D immersive environment. a Using a value greater than 1 will allow the user to select from the user's viewpoint. ETo view the furthest face of a 3D object, you don't have to fully extend your arm. For example, a The value can be set between 1.5 and 2. For simplicity, we will consider the following... .

[0073] Using virtual hands can improve user interaction with 3D objects (and more generally, in 3D environments). Figure 20 and Figure 21 The example illustrates the improvements in ergonomics achieved by using a virtual hand.

[0074] exist Figure 20 In the example, using a virtual hand allows users to select any face of a 3D object without having to fully extend their arm. Keeping the arm extended can be tiring for users, so a virtual hand is used to address this issue.

[0075] exist Figure 21 In the example, using a virtual hand allows the user to select any face of a 3D object without moving the user's viewpoint E. In fact, for example, some faces of the 3D object might be unselectable without using a virtual hand when the distance between the farthest face of the 3D object on the horizontal plane of the user's body and the user's viewpoint E is greater than the distance between the center of the user's hand and the farthest point from E. In this case, the virtual hand is positioned further away from the user's viewpoint E than the user's hand E. Without using a virtual hand, the item... The choice of the farthest face of the 3D object viewed from the user's viewpoint E will be penalized. Therefore, the method will be biased by this factor. The virtual hand addresses this issue by virtually moving the user's hand within the 3D immersive environment to ensure that the virtual H' is always at least as far as the farthest face of the 3D object, where both positions are considered from the user's viewpoint E.

[0076] In the example, the use of a virtual hand can be combined with ray projection from the virtual hand. This combination can be activated, for example, if the user's hand is indeed close to the user's head. The direction of the ray can be orthogonal to the orientation plane of the virtual hand's palm / back or from the center of the virtual hand's palm / back towards the center of the 3D object. For example, in mixed reality or augmented reality, this combination can be very useful for selecting any face of a 3D object that may be surrounded by real obstacles.

[0077] In the example, the function item or (Calculated using equation (1) or equation (2) respectively) may also include the Euclidean distance between the positions of the following two in the 3D immersive environment: the face; and the 3D model and the position from the user's viewpoint. EAt least one intersection point between the projected rays. For example, rays can be projected from the center of the head, the dominant eye, or a point between the eyes to the center of the 3D model. The direction of projection can be defined, for example, by the user's gaze or the direction of the user's head. It can also be defined from the user's viewpoint. E The projection direction is calculated from the user's viewpoint E to the center of the 3D model. The center of the 3D model can be defined, for example, as the center of the bounding box of the 3D model. For instance, the projection direction can be calculated from the user's viewpoint E to the center of the 3D model.

[0078] Add a term based on the Euclidean distance in the 3D immersive environment (which is the Euclidean distance between the location of the face and the position of at least one intersection point of the 3D model and the ray projected from the user's viewpoint E) to the term. or This method allows the user's viewpoint E Position sensitivity. In other words, this allows for the selection of faces aligned with a line between the user's viewpoint E and at least one intersection point of the 3D model and the ray. This is particularly useful if the 3D model is partially rendered outside the viewport. In this case, the method will penalize the selection of faces outside the viewport.

[0079] To retrieve the position and / or orientation of a user's head, head tracking can be performed using any technology capable of detecting changes in posture or motion. For example, head-mounted receivers or glasses exist that provide this information in virtual, mixed, or augmented reality. To retrieve the direction and / or orientation of a user's gaze, eye tracking technology can be used.

[0080] In the example, refer to Figure 2 For example, identification S40 may further include: determining the face of the 3D object that has the closest orientation to the orientation plane in S400. Identification S40 may further include calculating one or more faces determined by S410. n Adjacent faces, one or more n Adjacent faces and the determined faces form a subset 230 of selectable faces. The determined faces... n Adjacent faces are those of the defined faces. n -1 Adjacent faces share an edge and are not connected to each other. n -2. Faces that share an edge with adjacent faces. Therefore, the first adjacent face of the determined face is a face that shares an edge with the determined face, and the second adjacent face of the determined face is a face that shares an edge with the first adjacent face of the determined face but does not share an edge with the determined face. Identification S40 also includes: from the selectable subset 230 of faces, identifying the face of the 3D object that has the closest orientation to the orientation plane in S420. This can be predetermined based on the size of the face of the 3D object. nIt can also be based on the size of the faces forming subset 230 or on the faces forming subset 230 from the user's viewpoint. E The size of the visible surface is updated based on the distance between the user's viewpoint and the determined surface. n It can also be predetermined based on device specifications or user preferences. n .

[0081] S400 and S420 can also be determined by determining the face of the 3D object with the closest orientation or by determining the function that makes equation (1) or (2) work. f This is performed by minimizing the faces. All combinations between S400 and S420 can be used. For example, determining S400 can be done by determining the faces of the 3D object with the closest orientation, and determining S420 can be done by determining the face of the function... f Complete it by minimizing the number of faces.

[0082] In another example, it can be determined by making the function f Minimize the surface to complete the determination of S400 and S420. In this case, during the determination of S400, the weights... Can be compared Four times larger, and during the determination of S420, the weights Can be compared Four times smaller.

[0083] In the example, the method may further include modifying the first rendering of a subset 230 of selectable faces. The subset 230 of selectable faces is calculated in step S410. As described above, for example, during determination S40, the subset 230 is calculated based on the determined faces. The size of the subset 230 depends on the faces used in the calculation S410. n The value of . In the example, it can be calculated. n The value ensures that the visible surface from the user's perspective is always large enough for the user to easily identify it. Modifying the rendering of subset 230 helps the user identify which faces will be considered selectable during step S420. Therefore, the user will easily identify that the face of interest is not in subset 230, and thus modify the position and / or orientation of their hand during step S410 to change subset 230.

[0084] like Figure 11 As shown in the example, when it is detected that S20 has not occurred, the rendering of the faces of the 3D object is not modified by this method.

[0085] In the example, after identifying a subset 230 of selectable faces, the user can modify the position of their hand to change the subset 230 of selectable faces and / or they can modify the orientation of their hand to change the identified face 210 within the same subset 230 of selectable faces. Figure 16 and Figure 17 As shown in the example, when the position and / or orientation of the hand is modified, a subset 230 of the selectable faces is updated.

[0086] Rendering modifications can be performed in various ways. Rendering modifications can be performed during computation S410 and / or recognition S420. As an example, rendering modifications can be performed by highlighting selectable faces. As another example, rendering modifications can be performed by applying a specific texture to selectable faces. Rendering can also modify only the edges of selectable faces. For example, edges can be highlighted or thickened. In another example, rendering modifications can also include zooming in on selectable faces and disconnecting selectable faces from the 3D object. Rendering modifications can be performed through any other feedback from the user (e.g., automatically capturing graphical user information on the recognized faces). Rendering modifications can also appear on the user's hand by changing the representation of the hand. For example, the color of the hand might change, or lines might be overlaid on the fingers.

[0087] In the example, the method may further include modifying the second rendering of the identified face 210. The identified face 210 may be, for example, the face identified during step S40. Thus, modification of the rendering of the identified face 210 begins and helps the user select the face of interest.

[0088] The identified face 210 can also be, for example, the face identified during step S420. The identified face 210 can also be, for example, the face determined during step S400. In these cases, for example, the second and third renderings may differ from the first rendering of the subset 230 of selectable faces. The second rendering may also differ from the third rendering. The first and / or second and / or third renderings improve the user's interaction with 3D objects in the 3D environment: in fact, if the user needs to change the position of their hand to change the subset 230 of selectable faces, the user can more easily grasp the position during step S420, and / or they can modify the orientation of their hand to change the identified face 210 within the same subset 230 of selectable faces. Less user interaction is required to select the desired face.

[0089] Figure 11 and Figure 12 An example is shown in which the rendering of 3D object 200 is modified only when a gesture including the adduction of all fingers except the thumb is detected during step S20.

[0090] As an example, the first and second renderings are different, so the user can easily identify the subset 230 of selectable faces and the identified face 210. Figure 16 and Figure 17 An example is shown in the image.

[0091] Rendering modifications can be performed in various ways. In one example, rendering can be performed by highlighting selectable faces. In another example, rendering can be performed by applying a specific texture to selectable faces. Rendering can also modify only the edges of selectable faces; for example, edges can be highlighted or thickened. In another example, rendering modifications could also include enlarging selectable faces and disconnecting selectable faces from the 3D object. Rendering modifications can be made based on any other feedback from the user, such as automatically capturing graphical user information on the identified faces. Modifications to the rendering can also appear on the user's hand by changing its representation; for example, the color of the hand might change, or lines might be overlaid on the fingers.

[0092] A good balance can be found between providing continuous user feedback and avoiding excessive pop-ups. Providing continuous user feedback can be defined as having a method that responds to user input. In other words, for example, once the user provides input, the method should update the result of identifying S40 and / or determining the result of S400 and / or calculating the result of S410 and / or identifying the result of S420 without a visible delay. With a method that responds to user input, there is a risk that the method will provide excessive pop-ups. The term defines the tendency of the user interface to switch quickly and continuously between various states. In this method, for example, it could be a rapid and fast switch between two or more faces considered as being identified.

[0093] As an example, identified faces with modified renderings can be retained until a better candidate of the same dimension has been confidently identified. In other words, identifying S40 may involve minimizing a function. f Once the function on the other side f The value is less than the function of the currently identified face. f If the value is not specified, a switch from the currently identified face to another face will not occur. To avoid excessive popping, a penalty threshold or penalty factor can be used during recognition S40 to reduce the value of the currently identified face as a function. f The result is compared to the other face. In other words, the identified face with the modified render can be preserved until a better candidate with the same dimensions is found. f Minimize it, which has an increasing penalty threshold and / or a function towards the candidate. fThe result is multiplied by a penalty factor. Alternatively, the system can switch from the currently identified face to another only after detecting that the user's hand has not moved for a specific period of time, thus avoiding excessive popping.

[0094] Continuous user feedback can be achieved, for example, by modifying the rendering of the 3D object 200 and / or by adding a virtual representation near the 3D object 200 and / or near the user's hand and / or by haptic feedback from the device.

[0095] In the example, the 200 faces of the 3D object in this method can be triangles and / or quadrilaterals and / or topological faces and / or parametric surfaces and / or process surfaces. A topological face is a set of adjacent triangles or a set of adjacent quadrilaterals that form a large “logical” continuous face while still potentially revealing a wide variety of different orientations between each primitive, especially when the topological face is large. In CAD programs, topological faces are mostly constrained by topological edges. The term “adjacent triangles and / or adjacent quadrilaterals” should be understood to mean that each triangle or quadrilateral in the group is one of the adjacent triangles or quadrilaterals of the group. For example, a CAD object representing a car can consist of thousands of triangles, which can be grouped into fewer than one hundred topological faces. For example, a topological face can represent the upper surface of a car seat.

[0096] For all types of faces, orientation is specified by its normal vector, and orientation is specified by a line parallel to that normal vector. For topological faces, the normal vector can be edited. In practice, topological faces can consist of triangles or quadrilaterals oriented in various orientations, making it potentially difficult to calculate meaningful normal vectors for this type of face. Displaying the normal vectors throughout all steps of the method can help improve user interaction in a 3D environment, for example, during the design of a 3D model. Essentially, it graphically assists users in identifying the orientation and location of topological faces.

[0097] In the example, the detected S20 gesture includes adduction of all fingers except the thumb and abduction of the thumb. Figure 5 As shown, abduction of 250° represents a posture where the body part is away from the body's midline. For all fingers, abduction moves the fingers 190° away from the line. Figure 9 and Figure 10 An example of the pose detected for the right hand during step S20 is shown.

[0098] Thumb abduction is the result of muscle contraction, including the abductor brevis muscle.

[0099] Detecting adduction and thumb abduction of all fingers except the thumb is advantageous for two reasons. First, it reduces the risk of confusion between the detected posture and the default posture. Second, as... Figure 10 As shown, the adduction of all fingers except the thumb and the abduction of the thumb tend to flatten the orientation plane formed by the palm and / or back of the hand, thus making the determination of S30 more reliable.

[0100] In the example, the method also includes selecting the identified face 210 of a 3D object by detecting a gesture, including thumb adduction. Figure 12 and Figure 13 The diagram illustrates an example of selection based on the detection of a gesture that also includes thumb adduction. In other words, when the gesture includes thumb adduction, the identified face 210 of the 3D object is added to the current selection of faces. Figures 13 to 15 As shown, the identified face 210 is therefore considered the selected face 220. As an example, these figures illustrate the application of a third rendering to the selected face. This detection can occur during steps S40, S400, or S420. Therefore, the user can iteratively use this method to select multiple faces of interest.

[0101] In the example, the method could also include detecting that the gesture includes the abduction of all fingers except the thumb and deselecting the selected face. This detection can occur during any step of the method. For example, the user could thus deselect the last selected face. As another example, the user could deselect all currently selected faces opposite them.

[0102] In the example, detecting a gesture may also include:

[0103] -Extension of the proximal and distal interphalangeal joints of all fingers except the thumb was detected; and / or

[0104] - Detected extension of the metacarpophalangeal joints of all fingers except the thumb; and / or

[0105] - It was detected that all fingers except the thumb were in substantial contact.

[0106] Any combination of these three detections is possible. These detections can occur during steps S20 and / or S40. These detections can also occur during S400 and / or S420.

[0107] The extension of the proximal and distal interphalangeal joints of all fingers except the thumb is the result of muscle contraction, including the dorsal and palmar interosseous muscles of these fingers and the humeral muscles of these fingers.

[0108] The extension of the metacarpophalangeal joints of all fingers except the thumb is the result of muscle contraction, including the extensor muscles of the fingers.

[0109] The side-by-side contact of all fingers except the thumb is a result of the abduction of these fingers. Contact may need to be substantial to be detected. In practice, for more robust detection of side-by-side finger contact, the detection only needs to consider substantial contact. In other words, due to the shape and consistency of the fingers, the detection may assume that the two side-by-side fingers may not be in contact even if there is one or more small surface contacts between them. Detecting one or more of these gestures reduces the risk of confusion between the detected posture and the default posture. If these three gestures are performed, all fingers except the thumb are aligned with the orientation plane of the palm and / or back of the hand, thus improving determination S30.

Claims

1. A computer-implemented method for selecting a face from faces of a 3D object in a 3D immersive environment of a CAD system, each face being oriented within a reference frame of the 3D immersive environment, the method comprising: - Display the 3D object in the 3D immersive environment (S10); - Detect (S20) a gesture of one of the user's hands, the gesture including adduction of all fingers except the thumb; - Determine (S30) the orientation plane formed by the palm and / or back of the hand within the reference frame of the 3D immersive environment; and - Identify (S40) a face of the 3D object in the reference frame of the 3D immersive environment, the orientation of which has a minimum angle value with the orientation of the orientation plane.

2. The computer-implemented method according to claim 1, wherein, Each face of the 3D object and the orientation plane also have a normal, and wherein the identification (S40) further includes: identifying the face of the 3D object that has the closest orientation between the normal of the face and the normal of the orientation plane.

3. The computer-implemented method according to claim 1 or 2, wherein, Each face of the 3D object and the orientation plane also have a position in the 3D immersive environment, and wherein the recognition (S40) further includes: - Make the function Minimize: in: - It's the side that interests me; - and and ; - It is in the 3D immersive environment The position relative to the orientation plane of the hand The Euclidean distance between them; and - The surface in the 3D immersive environment The angle between the normal of the hand and the normal of the orientation plane of the hand, wherein the angle is measured in radians and is within the interval [0, 2π].

4. The computer-implemented method according to claim 3, wherein, The function item Replace with ,in, Calculated in the following way: in: - H’ It is the position of the orientation plane of the virtual hand; - E It is the position of the user's viewpoint; - H It is the position of the orientation plane of the user's hand; - EH It is a vector from E to H; - Omax It is the position of the face of the 3D object that is farthest from E; - It is the position of the center of the user's hand that is furthest from E on the horizontal plane of the user's body; - ; - From E arrive Omax The norm of the vector; and - From E arrive The norm of a vector.

5. The computer-implemented method according to claim 3, wherein, The function item or It also includes the position of the face in the 3D immersive environment and the Euclidean distance between at least one intersection of the 3D model and a ray, the ray being projected from the center of the head or a point between the dominant eye or eyes to the center of the 3D model.

6. The computer-implemented method according to claim 1, wherein, The identification (S40) includes: - Determine (S400) the face of the 3D object that has the closest orientation to the orientation plane; - Calculate one or more n neighboring faces of the face determined by (S410), wherein the one or more n neighboring faces and the determined face form a subset of selectable faces; and - Identify (S420) the face of the 3D object that has the closest orientation to the orientation plane from the subset of the selectable faces.

7. The computer-implemented method according to claim 6, further comprising: Modify the first rendering of faces in the subset of selectable faces.

8. The computer-implemented method according to claim 1, further comprising: Modify the second rendering of the identified faces.

9. The computer-implemented method according to claim 1, wherein, The faces of the 3D object are selected from at least one of the following: triangles and / or quadrilaterals and / or topological surfaces and / or parametric surfaces and / or process surfaces.

10. The computer-implemented method according to claim 1, wherein, The detected gesture (S20) includes the adduction of all fingers except the thumb and the abduction of the thumb.

11. The computer-implemented method according to claim 1, further comprising: - The identified face of the 3D object is selected by detecting the gesture, which also includes the tucking of the thumb.

12. The computer-implemented method according to claim 11, further comprising: - The detected gesture also included the abduction of all fingers except the thumb; as well as - Deselect the selected face.

13. The computer-implemented method according to claim 1, wherein, Detecting the gesture also includes: - Extension of the proximal and distal interphalangeal joints of all fingers except the thumb was detected; and / or - Extension of the metacarpophalangeal joints of all fingers except the thumb was detected; and / or - Detecting the gesture also includes detecting that all fingers, except the thumb, are in substantial contact.

14. A computer program product comprising instructions that, when executed, cause a computer to perform the method according to any one of claims 1 to 13.

15. A system comprising a display and processing circuitry communicatively coupled to a memory storing instructions which, when executed, cause the processing circuitry to perform the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Method for extracting and identifying three-dimensional gesture and constructing three-dimensional gesture interaction system

    CN109753876A

  • Working with 3D Objects

    US20110164029A1