How to change the rendering of areas of a 3D scene in an immersive environment

Abstract

To provide a computer-implemented method of correcting rendering of a 3D scene area in an immersive environment.SOLUTION: An area is computed based on a 3D position of a head tracking device (HED) of a user and a 3D position of a hand tracking device(s) (HAD1, HAD2) of at least one user.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to the field of computer programs and systems, and in particular to the fields of product design and exploration in immersive environments, and to the fields of simulation and video games where exploration is useful. [Background technology]

[0002] Advances being made in rendering the environment are making the experience increasingly realistic. In particular, virtual reality headsets allow users to visualize the 3D space they are immersed in through a virtual camera. The virtual reality headset tracks the head position, so the orientation of the user's head corresponds to its orientation in the immersive environment. Using one or more handheld motion-tracked controllers, the user can interact with the environment.

[0003] However, in very crowded environments, the virtual camera's view can be partially blocked, making the virtual reality experience less enjoyable. For example, consider the scenario of exploring a wiring closet, where dozens of cables block the camera's view. R Congestion in wiring closets can make it difficult for users to find, author, and manipulate cables.

[0004] Furthermore, in immersive environments, especially in some video games, 3D scenes are often displayed in small rooms. be In that case, there may be many objects in front of the virtual camera.

[0005] The first way to deal with field of view occlusion is to define a frustum. This solution is called "user-centric" because the frustum is associated with the user. A frustum also speeds up the rendering of a 3D scene, since only a portion of it is rendered. In the following, the frustum refers to the region of space that can be rendered for the user. A frustum is a truncated pyramid. The planes that cut the frustum perpendicular to the virtual camera's line of sight are called the "near clipping plane" and the "far clipping plane". The frustum is bounded by the near clipping plane on the side of the virtual camera and the far clipping plane on the opposite side. The angle between the opposite sides of the frustum corresponds to the natural field of view of a human. Objects that are not in the frustum are not rendered. In particular, objects that are between the virtual camera and the near clipping plane are not rendered.

[0006] Therefore, it is the software developer's or user's responsibility to set the near clipping plane at an optimal distance relative to the virtual camera. If the near clipping plane is too close to the virtual camera, the user's view will be obstructed, and if the near clipping plane is too far from the virtual camera, too many objects will be removed. Therefore, setting the depth of the near clipping plane is inflexible and non-adaptive.

[0007] Furthermore, the axis of the virtual camera is perpendicular to the near clipping plane. As the user moves his head, objects that are outside the frustum on the axis of the virtual camera and close to the near clipping plane enter the frustum. Depending on the orientation of the head, the object or part of it may or may not be rendered, which can be confusing to the user.

[0008] A second way to deal with field of view obstructions is to manually select the parts of the object to explore in order to change the properties of the object. This solution is called "product-centric" because the properties of the object are changed.

[0009] Returning to the wiring closet example, the user ghosts the first layer of cables and repeats the ghosting operation for the next layer until the desired layer is reached. Hereinafter, ghosting refers to changing the rendering of an object in order to see through it. Ghosting also makes the object unselectable. Ghosting can be performed, for example, by changing the opacity of the object or by rendering only the edges of the object.

[0010] Ghosting is done manually: the user points the cursor over a portion of an object and presses and releases the "Tab" button (or another button depending on how the feature is implemented), ghosting that portion of the object. If the user wants to ghost a different portion of the object, the user presses and releases the "Tab" button when the cursor is over that portion, until the user has finished exploring the object. To un-ghost said portion (i.e., remove the ghosting effect), the user moves the cursor away from the ghosted portion. Note that the user may also choose to hide an object instead of ghosting it; hereafter, hidden means not to render it.

[0011] Manually ghosting objects has several drawbacks. First, it is inefficient when trying to visualize dense 3D scenes, since it requires ghosting all objects in front of the user. Second, exploring objects deeply in many layers requires multiple button presses, which is tedious. Third, when a user unghosts a part in front of them, they cannot unghost parts behind the unghosted part. Unghosting operations must be performed in the reverse order of the ghosting operations. Summary of the Invention [Problem to be solved by the invention]

[0012] Therefore, there is a need to provide a computer-implemented method for adaptively exploring regions of a 3D scene in an immersive environment while reducing search time. [Means for solving the problem]

[0013] The object of the invention is a computer-implemented method for modifying the rendering of an area of ​​a 3D scene in an immersive environment, characterized in that the area is calculated based on the 3D position of a user's head tracking device and the 3D position of at least one user's hand tracking device.

[0014] In a preferred embodiment, the region includes a convex volume having a planar portion that fits the near clipping plane of a virtual camera frustum rendering a portion of the 3D scene, said convex 3D volume extending along a segment defined by the 3D position of the head tracking device and by the 3D position of the hand tracking device.

[0015] In a preferred embodiment, the convex volume comprises a truncated cone or a cylinder of revolution.

[0016] In a preferred embodiment, the convex volume has a curved shape opposite the planar portion that partially surrounds the hand tracking device and is spaced from the hand tracking device by a non-null safety margin.

[0017] In a preferred embodiment, the method comprises: - receiving a user input that activates a hand tracking device; - modifying the rendering of the region as long as the hand tracking device is activated; Includes.

[0018] In a preferred embodiment, the method comprises: - receiving a user input that activates a second hand tracking device; - calculating the distance between a 3D position of a hand tracking device, called a first hand tracking device, and a 3D position of a second hand tracking device; - Enlarging or shrinking the convex volume over the segment by increasing or decreasing said distance, respectively; Includes.

[0019] In a preferred embodiment, receiving a user input to activate the second hand tracking device includes receiving a command to press and hold a button on the second hand tracking device, and expanding or contracting the convex volume includes receiving a command to press and hold a button on the second hand tracking device while the button on the second hand tracking device is being pressed. Medium , is executed.

[0020] In a preferred embodiment, the step of receiving a user input to activate the second hand tracking device includes receiving a command to press and release a button on the second hand tracking device, and the step of enlarging or shrinking the convex volume is performed after receiving another command to press and release the button.

[0021] In a preferred embodiment, if the distance between the 3D position of the first hand tracking device and the 3D position of the second hand tracking device exceeds a threshold distance, the convex 3D volume is transformed into a sphere having its center at the 3D position of the head tracking device and its radius equal to the distance between the 3D position of the head tracking device and the 3D position of the first hand tracking device.

[0022] In a preferred embodiment, the region includes the 3D object that has the smallest bounding box that intersects with the convex volume.

[0023] In a preferred embodiment, modifying the rendering of the region comprises ghosting said region.

[0024] Alternatively, modifying the rendering of the region comprises hiding said region.

[0025] The present invention also relates to a computer program stored on a computer readable data storage medium. To and computer executable instructions for causing a virtual reality system to perform the aforementioned method.

[0026] The present invention also relates to a computer-readable data storage medium containing computer-executable instructions for causing a virtual reality system to perform the aforementioned method.

[0027] The present invention also relates to a virtual reality system including a processor coupled to a memory, the memory storing computer-executable instructions for causing the virtual reality system to perform the aforementioned method, a head-mounted display device having a head tracking device, and at least one handheld controller having a hand tracking device. [Brief description of the drawings]

[0028] Additional features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 illustrates a top view of a user in an immersive environment, the user equipped with a head tracking device and a first hand tracking device. [Diagram 2] FIG. 1 illustrates a top view of a user in an immersive environment, the user equipped with a head tracking device and a first hand tracking device. [Diagram 3] A top view of a user in an immersive environment is shown, the user equipped with a head tracking device and two tracking devices. [Figure 4] 1 shows a top view of a user in an immersive environment where the area is a sphere. [Diagram 5] 1 shows a top view of a user in an immersive environment where the area is a sphere. [Figure 6]FIG. 2 is an illustration of a user's viewpoint when searching for an object according to the method according to the invention; [Figure 7] FIG. 2 is an illustration of a user's viewpoint when searching for an object according to the method according to the invention; [Figure 8] FIG. 1 illustrates a computing environment adapted to carry out the method according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] FIG. 1 shows an initial top view of a user in an immersive environment. The user includes a virtual reality headset and at least one wireless handheld controller. The virtual reality headset and the wireless handheld controller are part of a virtual reality system. One example of a virtual reality system is "HTCVive" (registered trademark).

[0030] The headset and at least one handheld controller are arranged in a coordinate system of the virtual environment. Head tracking of the headset is performed by the head tracking device HED in cooperation with a base station in various ways by surface detection, markers, environmental recognition, a gyroscope control unit, computer vision. Similarly, hand tracking of the at least one handheld controller is performed by the hand tracking device HAD1 in cooperation with a base station in various ways by surface detection, markers, environmental recognition, a gyroscope control unit, computer vision.

[0031] In the following it can be seen that the virtual reality headset includes a head tracking device HED and the handheld controller includes a hand tracking device HAD1.

[0032] Referring now to FIG. 1, the exact 3D position of the user's head corresponds to a point between the eyes (cyclops eyes) that is equidistant from both eyes. The point between the eyes is calculated by a software layer of the head tracking device HED. The exact position of the user's hands is calculated based on the 3D position of the hand tracking device HAD1. The hand positions are calculated by a software layer of the hand tracking device HAD1.

[0033] The rendering of the region is modified based on the 3D position of the user's head tracking device HED and the 3D position of at least one of the user's hand tracking devices, as shown in Figures 1 and 2. In Figure 1, the user holds the hand tracking device HAD1 in front of him / her on the right side. In Figure 2, the user holds the hand tracking device HAD1 in front of him / her on the left side.

[0034] Thus, as the user moves his / her hand, the regions where the rendering will be changed are calculated in real time. The user simply and adaptively sets the depth regions where the rendering will be changed.

[0035] Advantageously, modifying the rendering of the region includes ghosting it or hiding it. Thus, a user can view the 3D scene through the ghosted or hidden region while visualizing the context around the ghosted or hidden region. When a region is ghosted or hidden, ray tracing is performed without taking the region into account, thus proving the realistic colors of the scene.

[0036] The user can predetermine, for example by a menu setting, what modifications of the rendering will be performed (whether to hide or ghost areas).

[0037] In a preferred embodiment, the region comprises a convex volume VOL. In particular, the convex volume VOL has a planar portion PLP that fits the near clipping plane of the virtual camera frustum that renders the part of the 3D scene. As described above, the near clipping plane marks the virtual camera frustum on one side of the head tracking device HED, and the far clipping plane marks the virtual camera frustum on the opposite side. In contrast to known methods that modify the depth of the near clipping plane, the method of the invention does not modify the depth of the near clipping plane. More generally, the method of the invention is performed regardless of the size / shape / depth of the virtual camera frustum.

[0038] The convex volume VOL extends along a segment SEG defined by the 3D position of the head tracking device HED and the 3D position of the hand tracking device HAD1. By "extending along" we mean that the convex volume VOL can have any shape, provided that the segment between the 3D position of the head tracking device HED and the 3D position of the hand tracking device HAD1 is included in the convex volume VOL. Changing the rendering of a region is therefore intuitive and allows the user to explore by simply "digging" into the 3D object. There is no need to press a dedicated button on the keyboard for as many layers as you want to ghost. Vector

[0039]

number

[0040] corresponds to the vector between the 3D position of the head tracking device HED and the 3D position of the hand tracking device HAD1.

[0041] In Figures 1 and 2, the convex volume VOL comprises a cylinder of revolution with an axis substantially corresponding to the segment SEG. Alternatively, the convex volume VOL comprises a truncated cone, also called a frustum, although the latter should not be confused with the frustum of the virtual camera that renders the 3D scene. In the case of a truncated cone, the convex volume VOL has the shape of a "torch".

[0042] In a preferred embodiment, the convex volume VOL has a curved shape that partially surrounds the hand tracking device HAD1, opposite the planar portion PLP. In other words, at the tip of the cylinder or the tip of the truncated cone, opposite the planar portion PLP, the convex volume VOL is not planar, but rather curved, and surrounds the end of the hand tracking device HAD1.

[0043] The convex volume VOL also includes a safety margin SMA between the hand tracking device HAD1 and the end of the convex volume VOL that is opposite the planar part PLP in said curvature, so that the curved part and the hand tracking device HAD1 may be separated from each other. In an immersive environment, the hand tracking device, i.e. the controller, is often visualized by modeling it. Without a safety margin, the user's view may be blocked by the hand tracking device while exploring the object. A non-null safety margin SMA increases the visibility of the area where the rendering is changed.

[0044] The value of the safety margin SMA may be proportional to the length of the segment SEG. Alternatively, it may be exponential. According to another alternative, the value of the safety margin SMA is fixed as long as the arm is close to the user's head (arm folded). Then, as the arm extends from half extension to full extension, the value of the safety margin SMA increases linearly or exponentially. The same applies to the contraction movement.

[0045] The modification of the rendering of the region is controlled by the user. In an immersive environment, the hand tracking device receives a user input that activates it. The user input can be, for example, pressing and holding a button on the hand tracking device. The modification of the rendering of the region is then performed as long as the hand tracking device is activated, for example as long as the button is pressed. As a result, every time the user releases the button, the 3D scene is displayed without ghosting, hiding or without regions. The invented method is user-friendly, since the user needs to operate only one button to modify the rendering of the region.

[0046] Figure 3 shows another embodiment of the method of the invention, in which two hand tracking devices are required. The aforementioned hand tracking device HAD1 is referred to as the first hand tracking device HAD1. In addition, a second hand tracking device HAD2 is used. In the example of Figure 3, the user holds the first hand tracking device HAD1 on the left hand and the second hand tracking device HAD2 on the right hand. Of course, the first hand tracking device HAD1 can be mounted on the right hand and the second hand tracking device HAD2 on the left hand without affecting the method of the invention.

[0047] When a user input is received to activate the second hand tracking device HAD2, a distance DIS between the 3D position of the first hand tracking device HAD1 and the 3D position of the second hand tracking device HAD2 is calculated. Then, when the user increases or decreases the distance DIS, the convex volume VOL is enlarged or thinned, respectively, across the segment SEG. "Enlarging the convex volume VOL across the segment SEG" means that the cross-sectional area increases. "Shrinking the convex volume VOL across the segment SEG" means that the cross-sectional area decreases.

[0048] By adapting the distance DIS, the user can adapt the region to the context of the 3D scene and also to the object to be explored: for example, to explore the gears of a gearbox, a short distance DIS is sufficient, while to explore an axis that crosses the entire gearbox, a long distance DIS is more suitable.

[0049] If the convex volume VOL is a rotating cylinder, actuating the second hand tracking device HAD2 transforms the rotating cylinder into a frustum of a cone. Then, as the user continues to increase the distance DIS, the apex angle (i.e., the cone angle if the cone is not truncated) increases. If the convex volume VOL is initially a frustum of a cone, increasing the distance DIS also increases the apex angle.

[0050] Thus, the user can modify the region to be ghosted or hidden, while playing with the depth of the region starting from the near clipping plane, but also while playing with the opening angle of the region. Meanwhile, the user can change the orientation of the segment SEG using the first hand tracking device HAD1.

[0051] To enlarge or shrink the convex volume VOL, the user presses and holds the button on the second hand tracking device HAD2, and while the button on the second hand tracking device HAD2 is pressed, the enlargement or shrinking of the convex volume VO is performed. Thus, there is real-time visual feedback regarding the change in the distance DIS. When the user releases the button on the second hand tracking device HAD2, the distance DIS is considered fixed until the user presses the button again.

[0052] Alternatively, the enlargement or reduction of the convex volume VOL can be performed with an "on / off" command. The user presses and releases a button on the second hand tracking device HAD2, activating the functionality to change the width of the convex volume VOL. The enlargement or reduction of the convex volume VOL is performed after receiving another command of pressing and releasing the button. Thus, between the two switches of the button, the user can annotate objects that can be seen with ghosting or hiding of objects in the field of view of the virtual camera, which may be useful in the authoring process.

[0053] According to the preferred embodiment shown in Figures 4 and 5, the region where the rendering is changed may be a user-centered sphere with a radius equal to the distance between the 3D position of the head tracking device HED and the 3D position of the first hand tracking device HAD1. For example, all objects that are within the sphere are ghosted or hidden. As a result, the user does not need to move his / her hands to explore objects from one side to another because he / she only needs to move his / her head.

[0054] This configuration is also interesting when the user's real environment is crowded in front of him, for example due to a computer screen. If the user is aware that his lateral environment is free, he can adjust the radius of the sphere by extending or retracting his arms.

[0055] When the user holds the first hand tracking device HAD1 and the second hand tracking device HAD2 in his hands, he increases the distance DIS in one of the aforementioned ways. When the distance DIS exceeds a threshold distance, the convex 3D volume VOL, which is initially a truncated cone or a rotating cylinder, is transformed into a sphere SPH. The center of the sphere SPH is located at the 3D position of the head tracking device HED, and the radius Dhh is equal to the distance between the 3D position of the head tracking device HED and the 3D position of the first hand tracking device HAD1.

[0056] The user can switch back to a non-spherical shape by providing an input to the second hand tracking device HAD2, for example by pressing a button on the second hand tracking device HAD2.

[0057] Advantageously, the region whose rendering is changed also includes the 3D object with the smallest bounding box that intersects with the convex volume VOL. This is illustrated in figures 1 to 5. Around the convex volume VOL, a checkered box that intersects with the convex volume VOL is also ghosted / hidden. Thus, the object (or part of it) is not cut, but ghosted / hidden or rendered.

[0058] Figures 6 and 7 show the user's viewpoint when exploring an object with the method according to the invention. Figure 6 shows the viewpoint before exploring the object, in this case a cube. In Figure 7, the user extends his arm to "dig" into the cube, so that the excavated part of the cube is hidden.

[0059] The present invention has been shown to be particularly tailored for immersive environments, since the user only needs to interact with the controller of the virtual reality system without using other peripherals such as a keyboard, mouse, etc. Furthermore, the method of the present invention is user-friendly, since the user does not need to memorize specific buttons.

[0060] The methods of the present invention can be carried out by a suitably programmed general-purpose computer or virtual reality system, possibly including a computer network having a suitable program stored in non-volatile form on a computer readable medium such as a hard disk, solid state, disk or CD-ROM, and using its microprocessor and memory to execute said program.

[0061] A computer CPT suitable for carrying out the method according to an exemplary embodiment of the invention is described with reference to figure 8. In figure 8, the virtual reality system VRS comprises a central processing unit (CPU) P which carries out the above-mentioned method steps while executing an executable program, i.e. a set of computer readable instructions, stored in a memory device such as a RAM M1 or a ROM M2 or a hard disk drive (HDD) M3, a DVD / CD drive M4 or stored remotely.

[0062] The claimed invention is not limited by the form of the computer readable media on which the computer readable instructions and / or data structures of the processes of the invention are stored. For example, the instructions and files can be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or other information processing device with which the computer communicates, such as a server or computer. The programs and files can be stored on the same memory device or on different memory devices.

[0063] Furthermore, computer programs suitable for carrying out the methods of the present invention may be provided as a utility application, a background daemon, or a component of an operating system, or a combination thereof, and run in conjunction with a CPUP and an operating system such as Microsoft VISTA, Microsoft Windows 10, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art.

[0064] The CPU P may be a Xenon processor from Intel of America or an Opteron processor from AMD of America, or may be other processor types such as a Freescale ColdFire, IMX, or ARM processor from Freescale Corporation of America. Alternatively, the CPU may be a processor such as a Core2Duo from Intel Corporation of America, and may run on an FPGA, ASIC, PLD, or may use discrete logic circuitry, as will be appreciated by those skilled in the art. Additionally, the CPU may run as multiple processors operating in concert to execute the computer readable instructions of the process of the present invention described above.

[0065] The virtual reality system of FIG. 8 also includes a network interface NI, such as Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with a network, such as a local area network (LAN), a wide area network (WAN), or the Internet. The virtual reality system further includes a head mounted display device HMD having a head tracking device HED. A generic I / O interface IF interfaces with handheld controllers HHC1, HHC2 equipped with hand tracking devices HAD1, HAD2, respectively. The display, keyboard, and pointing device, together with the display controller and the I / O interface, form a graphical user interface. This interface is used by the user to provide input commands and by the computer to display 3D objects.

[0066] The disk controller DKC connects the HDDM3 and DVD / CD M4 to a communication bus CBS, such as ISA, EISA, VESA, PCI, etc., to interconnect all components of the computer.

[0067] Any method steps described herein should be understood as representing a module, segment, or portion of code that contains one or more executable instructions for performing a particular logical function or step in a process, and alternative implementations are included within the scope of the exemplary embodiments of the invention.

Claims

1. 1. A computer-implemented method for modifying a rendering of a region of a 3D scene in an immersive environment, comprising: the region is calculated based on a 3D position of a user's head tracking device (HED) and a 3D position of at least one of the user's hand tracking devices (HAD1, HAD2); the region includes a convex volume (VOL) having a planar portion (PLP) that fits a near clipping plane of a virtual camera frustum that renders a portion of the 3D scene; A computer-implemented method, characterized in that the convex volume (VOL) extends along a segment (SEG) defined by the 3D position of the head tracking device (HED) and the 3D position of the hand tracking device (HAD1).

2. 2. The computer-implemented method of claim 1, wherein the convex volume (VOL) comprises a truncated cone or a cylinder of revolution.

3. A computer-implemented method as described in any one of claims 1 or 2, characterized in that the convex volume has a curved shape that partially surrounds the hand tracking device (HAD1) as opposed to the planar portion (PLP) and is spaced from the hand tracking device (HAD1) by a non-null safety margin (SMA).

4. receiving a user input to actuate the hand tracking devices (HAD1, HAD2); - modifying the rendering of said region as long as said hand tracking devices (HAD1, HAD2) are activated; 4. A computer-implemented method according to claim 1, further comprising:

5. receiving a user input that activates a second hand tracking device (HAD2); - calculating the distance (DIS) between the 3D position of the first hand tracking device (HAD1), called the first hand tracking device, and the 3D position of the second hand tracking device (HAD2); - expanding or contracting the convex volume (VOL) over the segment (SEG) by respectively increasing or decreasing said distance (DIS); 5. A computer-implemented method according to claim 1, further comprising:

6. receiving a user input to actuate the second hand tracking device (HAD2) includes receiving a command to press and hold a button on the second hand tracking device (HAD2); 6. The computer-implemented method of claim 5, wherein the step of expanding or contracting the convex volume (VOL) is performed for as long as a button on the second hand tracking device (HAD2) is pressed.

7. receiving a user input to actuate the second hand tracking device (HAD2) includes receiving a command to press and release a button on the second hand tracking device (HAD2); 6. The computer-implemented method of claim 5, wherein the step of expanding or contracting the convex volume (VOL) is performed after receiving a separate command to press and release the button.

8. if the distance (DIS) between the 3D position of the first hand tracking device (HAD1) and the 3D position of the second hand tracking device (HAD2) exceeds a threshold distance, the convex volume (VOL) is transformed into a sphere (SPH); 8. A computer-implemented method according to any one of claims 5 to 7, characterized in that the sphere (SPH) has its center at the 3D position of the head tracking device (HED) and its radius is equal to the distance between the 3D position of the head tracking device (HED) and the 3D position of the first hand tracking device (HAD1).

9. 9. The computer-implemented method of claim 1, wherein the region comprises a 3D object having a smallest bounding box that intersects with a convex volume (VOL).

10. 10. The computer-implemented method of claim 1, wherein modifying the rendering of the region comprises ghosting the region.

11. 11. The computer-implemented method of claim 1, wherein modifying the rendering of the region comprises making the region invisible.

12. A computer program stored on a computer-readable data storage medium (M1, M2, M3, M4) and comprising computer executable instructions for causing a virtual reality system to carry out a method according to any one of claims 1 to 11.

13. A computer readable data storage medium (M1, M2, M3, M4) comprising computer executable instructions for causing a virtual reality system to carry out a method according to any of claims 1 to 11.

14. A processor (P) coupled to a memory (M1, M2, M3, M4), said memory storing computer executable instructions for causing a virtual reality system (VRS) to execute a method according to any one of claims 1 to 11; a head mounted display device (HMD) having a head tracking device (HED); At least one handheld controller (HHC) having a hand tracking device (HAD1, HAD2); A virtual reality system (VRS) comprising:

Images