Method for modifying rendering of a region of a 3D scene in an immersive environment

Abstract

The invention relates to a computer-implemented method for modifying the rendering of a region of a 3D scene in an immersive environment, characterized in that the region is computed based on the 3D position of a head tracking device (HED) of the user and the 3D position of at least one hand tracking device (HAD1, HAD2) of the user.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly to the field of product design and exploration in immersive environments, and also to the field of simulations and video games, in which exploration is useful. Background Technology

[0002] Advances in environment rendering have made experiences increasingly realistic. In particular, virtual reality headsets allow users to visualize the 3D space they are immersed in using a virtual camera. By tracking head position with the headset, the user's head orientation corresponds to the orientation within the immersive environment. Using one or more handheld motion-tracking controllers, users can interact with the environment. Summary of the Invention

[0003] However, in very crowded environments, the virtual camera's field of view can be partially obstructed, making the virtual reality experience less enjoyable. For example, consider a scenario of exploring a wiring room, where dozens of cables block the camera's view. The crampedness of the wiring room prevents users from discovering and creating with the cables, and also prevents them from manipulating them.

[0004] Furthermore, in immersive environments, especially in some video games, 3D scenes are small rooms. In such cases, there may be many objects in front of the virtual camera.

[0005] The first way to handle view occlusion is by defining the view frustum. Because the view frustum is associated with the user, this solution is called "user-centric." Since only a portion of the 3D scene is rendered, the view frustum also speeds up the rendering of the 3D scene. Hereafter, the view frustum refers to the area of ​​space that can be rendered for the user. The view frustum is a truncated pyramid. The planes perpendicular to the virtual camera's viewing direction that cut through the view frustum are called the "near clipping plane" and the "far clipping plane." The view frustum is defined by the near clipping plane on one side of the virtual camera and the far clipping plane on the opposite side. The angle between the opposite sides of the view frustum corresponds to the natural field of vision of humans. Objects not within the view frustum are not rendered. In particular, objects located between the virtual camera and the near clipping plane are not rendered.

[0006] Therefore, it is up to the developers or the software users to decide on the optimal distance for the near clipping plane relative to the virtual camera: if the near clipping plane is too close to the virtual camera, the user's view will be obstructed, and if the near clipping plane is too far from the virtual camera, too many objects will be removed. Thus, setting the depth of the near clipping plane is neither flexible nor adaptive.

[0007] Furthermore, the virtual camera's axis is perpendicular to the near clipping plane. As the user moves their head, objects located outside the view frustum, near the near clipping plane, and on the virtual camera's axis enter the view frustum. Depending on the head's orientation, objects or portions thereof may be rendered or not rendered, which can be confusing for the user.

[0008] The second way to handle field-of-view occlusion involves manually selecting portions of the objects to be explored in order to alter their properties. Because the object properties are modified, this solution is known as "product-centric."

[0009] Returning to the cabling room example, the user ghosts the first layer of the cable and repeats the ghosting operation for the next layer until the user has reached the layer of interest. From this point onward, ghosting refers to modifying the rendering of an object so that it can be viewed through it. Ghosting also makes the object unselectable. This can be done, for example, by modifying the object's opacity or by rendering only the object's edges.

[0010] The blurring operation is performed manually: the user-guided cursor hovers over a portion of the object, then they press and release the "tab" button (or another button, depending on how the functionality has been implemented), blurring that portion of the object. If the user wants to blur another portion of the object, they repeat the process of pressing and releasing the "tab" button while the cursor is hovering over that portion, and so on, until the user has stopped exploring the object. To unblur (unghost) that portion (i.e., remove the blur effect), the user moves the cursor away from the blurred portion. Note that the user can hide the object instead of blurring it. Hiding it afterwards means it is not rendered.

[0011] Manually blurring objects has several drawbacks. First, it's inefficient when trying to visualize crowded 3D scenes because it requires blurring every object in front of the user. Second, exploring objects with many layers involves pressing buttons multiple times, which is cumbersome. Third, if the user unblurs the portion they see in front of them, they cannot unblur the portion behind it: the unblurring operation must be performed in the reverse order of the blurring operations.

[0012] Therefore, there is a need to provide a computer-implemented method for adaptively exploring areas of a 3D scene in an immersive environment while reducing exploration time.

[0013] The object of the present invention is a computer-implemented method for modifying the rendering of a region of a 3D scene in an immersive environment, characterized in that the region is calculated based on the 3D position of a user’s head tracking device and the 3D position of at least one of the user’s hand tracking devices.

[0014] In a preferred embodiment, the region includes a convex volume having a planar portion that coincides with the near clipping plane of a virtual camera frustum rendering a portion of the 3D scene, the convex 3D volume extending along a segment defined by the 3D positions of the head tracking device and the hand tracking device.

[0015] In a preferred embodiment, the convex volume includes a truncated cone or a revolution cylinder.

[0016] In a preferred embodiment, the convex volume has a curved shape such that a portion of it surrounds the hand tracking device, is opposite to the planar portion, and is spaced apart from the hand tracking device by a non-empty safety margin.

[0017] In a preferred embodiment, the method includes:

[0018] - The steps for receiving user input to activate the hand tracking device, and

[0019] - Modify the rendering steps for the area as long as the hand tracking device is being activated.

[0020] In a preferred embodiment, the method includes:

[0021] - Steps for receiving user input to actuate the second hand tracking device.

[0022] - The step of calculating the distance between the 3D position of the hand tracking device, referred to as the first hand tracking device, and the 3D position of the second hand tracking device, and

[0023] - The steps of widening or thinning the convex volume of the span by increasing or decreasing the distance, respectively.

[0024] In a preferred embodiment, the step of receiving user input to actuate the second hand tracking device includes: receiving a command to press a button on the second hand tracking device, and performing a step of widening or thinning the convex volume while the button on the second hand tracking device is being pressed.

[0025] In a preferred embodiment, the step of receiving user input to actuate the second hand tracking device includes: receiving a command to press and then release a button on the second hand tracking device; and performing a step of widening or thinning the convex volume after receiving another command to press and then release the button.

[0026] In a preferred embodiment, if the distance between the 3D position of the first hand tracking device and the 3D position of the second hand tracking device exceeds a threshold distance, the convex 3D volume is transformed into a sphere, the center of which is located at the 3D position of the head tracking device, and the radius of the sphere is equal to the distance between the 3D position of the head tracking device and the 3D position of the first hand tracking device.

[0027] In a preferred embodiment, the region includes a 3D object having a minimum bounding box that intersects with the convex volume.

[0028] In a preferred embodiment, modifying the rendering of a region includes blurring the region.

[0029] Alternatively, modifying the rendering of a region may involve hiding that region.

[0030] The present invention also relates to a computer program product stored on a computer-readable data storage medium, the computer program product including computer-executable instructions for causing a virtual reality system to perform the aforementioned methods.

[0031] The present invention also relates to a computer-readable data storage medium comprising computer-executable instructions for causing a virtual reality system to perform the aforementioned methods.

[0032] The present invention also relates to a virtual reality system comprising: a processor coupled to a memory; a memory storing computer-executable instructions for causing the virtual reality system to perform the aforementioned methods; a head-mounted display device having a head-tracking device; and at least one handheld controller having a hand-tracking device. Attached Figure Description

[0033] Additional features and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings:

[0034] - Figure 1 and Figure 2 The image shows a top-down view of a user in an immersive environment; the user is equipped with a head-tracking device and a first-hand-tracking device.

[0035] - Figure 3 The image shows a top-down view of a user in an immersive environment; the user is equipped with head-tracking and hand-tracking devices.

[0036] - Figure 4 and Figure 5 A top-down view of the user in an immersive environment is shown, where the area is a sphere;

[0037] - Figure 6 and Figure 7 This illustrates the user's viewpoint while exploring an object, as per the method according to the present invention.

[0038] - Figure 8 A computer environment suitable for performing the method according to the invention is shown. Detailed Implementation

[0039] Figure 1 This image shows a first-person top-down view of a user in an immersive environment. The user is equipped with a virtual reality headset and at least one wireless handheld controller. The virtual reality headset and wireless handheld controller are part of a virtual reality system. An example of a virtual reality system is the HTC Vive. TM .

[0040] The headset and at least one handheld controller are located in the coordinate system of the virtual environment. Head tracking of the headset is performed by a head tracking device (HED) in various ways: through cooperation with a base station, via surface detection, marking, environment recognition, a gyroscope control unit, and computer vision. Similarly, hand tracking of at least one handheld controller is also performed by a hand tracking device (HAD1) in various ways: through cooperation with a base station, via surface detection, marking, environment recognition, a gyroscope control unit, and computer vision.

[0041] In the following text, virtual reality headsets including head-tracking devices (HED) and handheld controllers including hand-tracking devices (HAD1) will be considered.

[0042] Now for reference Figure 1 The precise 3D position of the user's head corresponds to the point between the eyes (the Cyclops's eyes), which is equidistant from both eyes. This point between the eyes is calculated by the software layer of the head tracking device HED. The precise position of the user's hands is calculated based on the 3D position of the hand tracking device HAD1. The hand position is calculated by the software layer of the head tracking device HAD1.

[0043] like Figure 1 and Figure 2 As shown, the rendering of the region was modified based on the 3D position of the user's head tracking device (HED) and the 3D position of at least one of the user's hand tracking devices. Figure 1 In the image, the user places the HAD1 hand tracking device on the right side in front of him. Figure 2 In the image, the user places the HAD1 hand tracking device on the left side in front of him.

[0044] Therefore, the area to be modified in the rendering is calculated in real time as the user moves their hand. The user can easily and adaptively set the depth area to be modified in the rendering.

[0045] Advantageously, modifying the rendering of a region involves blurring or hiding that region. Therefore, users can see the 3D scene through the blurred or hidden region while simultaneously visualizing the context surrounding it. When a region is blurred or hidden, ray tracing is performed without considering that region, thus revealing the true colors of the scene.

[0046] Users can predetermine what kind of modifications to the rendering to make: hide or blur the area.

[0047] In a preferred embodiment, the region includes a convex volume (VOL). Specifically, the VOL has a planar portion (PLP) that coincides with the near clipping plane of the virtual camera's view frustum, which is a portion of the 3D scene being rendered. As mentioned above, the near clipping plane marks the virtual camera's view frustum on one side of the head-tracking device (HED), while the far clipping plane marks the virtual camera's view frustum on the opposite side. In contrast to known methods that modify the depth of the near clipping plane, the method of the present invention does not change the depth of the near clipping plane. More generally, the method of the present invention is performed independently of the size / shape / depth of the virtual camera's view frustum.

[0048] The convex volume (VOL) extends along the segment SEG defined by the 3D positions of the head tracking device (HED) and the hand tracking device (HAD1). The phrase "extends along..." means that the convex volume (VOL) can have any shape, provided it includes the segment between the 3D positions of the head tracking device (HED) and the hand tracking device (HAD1). Therefore, rendering of the modified region is intuitive, and the user can simply "dig" within the 3D object to explore it. The user doesn't need to press so many dedicated keys on the keyboard or so many layers to blur. Vector The vector corresponding to the 3D position of the head tracking device HED and the 3D position of the hand tracking device HAD1.

[0049] exist Figure 1 and Figure 2 In this context, the convex volume VOL comprises a cylinder of revolution having an axis substantially corresponding to the segment SEG. Alternatively, the convex volume VOL comprises a truncated cone, also known as a view cone. However, the latter should not be confused with the view cone of a virtual camera rendering a 3D scene. In the case of the truncated cone, the convex volume VOL has a "flashlight" shape.

[0050] In a preferred embodiment, the convex volume VOL has a curved shape that partially surrounds the hand tracking device HAD1, the curved shape being opposite to the planar portion PLP. In other words, at the end of the cylinder or at the end of the truncated cone, opposite to the planar portion PLP, the convex volume VOL is not planar, but curved, so as to surround the end of the hand tracking device HAD1.

[0051] The convex volume VOL can also include a safety margin SMA with the aforementioned curvature between the hand-tracking device HAD1 and the end of the convex volume VOL opposite the planar portion PLP, allowing the curved portion and the hand-tracking device HAD1 to be spaced apart from each other. In immersive environments, the hand-tracking device (i.e., the controller) is often made visible through modeling. Without a safety margin, the user's field of vision may be obstructed by the hand-tracking device while exploring objects. A non-empty safety margin SMA enhances the visibility of the rendered modified areas.

[0052] The safety margin SMA value can be proportional to the length of the segment SEG. Alternatively, it can be exponential. According to another alternative, the safety margin SMA value is fixed as long as the arm is close to the user's head (arm folded). Then, as the arm extends from semi-extended to fully extended, the safety margin SMA value increases linearly or exponentially. The same applies to contraction movements.

[0053] Modifications to the rendering of a region are user-controlled. In an immersive environment, a hand-tracking device receives user input that actuates it. User input can be, for example, pressing a button on the hand-tracking device. Then, as long as the hand-tracking device is actuated (e.g., whenever the button is pressed), modifications to the rendering of the region are performed. Therefore, whenever the user releases the button, the 3D scene is displayed without any blurring or hiding of the region. Since the user only needs to actuate one button to modify the rendering of the region, the method of this invention is user-friendly.

[0054] Figure 3 Another embodiment of the method of the present invention is shown, wherein two hand tracking devices are required. The aforementioned hand tracking device HAD1 is referred to as the first hand tracking device HAD1. Additionally, a second hand tracking device HAD2 is used. Figure 3 In the example, the user carries the first hand tracking device HAD1 in their left hand and the second hand tracking device HAD2 in their right hand. Of course, the user can carry the first hand tracking device HAD1 in their right hand and the second hand tracking device HAD2 in their left hand without affecting the method of the present invention.

[0055] When user input is received to actuate the second hand tracking device HAD2, the distance DIS between the 3D positions of the first hand tracking device HAD1 and the second hand tracking device HAD2 is calculated. Then, as the user increases or decreases the distance DIS, the cross-segment SEG widens or thins the convex volume VOL, respectively. "Widening the convex volume VOL across the cross-segment SEG" means increasing the cross-section. "Thinning the convex volume VOL across the cross-segment SEG" means decreasing the cross-section.

[0056] By adjusting the Distance Indicator (DIS), users can adapt the area to the context of the 3D scene or to the object being explored. For example, a small distance DIS might be sufficient for exploring the gears in a gearbox, while a large distance DIS would be better for exploring the shaft that runs through the entire gearbox.

[0057] If the convex volume VOL is a cylinder of revolution, the actuated second hand tracking device HAD2 transforms the cylinder of revolution into a frustum. Then, if the user continues to increase the distance DIS, the vertex angle (i.e., the cone angle when the cone is not truncated) increases. If the convex volume VOL was initially a frustum, the vertex angle also increases as the distance DIS increases.

[0058] Therefore, users can not only process the depth of the region starting from the near-clipping plane, but also modify the region to be blurred or hidden while processing the angle of the region's aperture. Simultaneously, users can modify the direction of the segment SEG using the first-hand tracking device HAD1.

[0059] To widen or thin the convex volume VOL, the user presses and holds a button on the second hand tracking device HAD2, and the widening or thinning of the convex volume VOL is performed simultaneously with the button being pressed. Therefore, there is real-time visual feedback regarding the modification of the distance DIS. When the user releases the button on the second hand tracking device HAD2, the distance DIS is considered fixed until the user presses the button again.

[0060] Alternatively, the widening or thinning of the convex volume (VOL) can be achieved using an "on / off" command: the user presses and releases a button on the second hand-tracking device, HAD2, to activate the function of modifying the width of the convex volume (VOL). Upon receiving another press and release command, the widening or thinning of the convex volume (VOL) is performed. Therefore, between the two switches of the button, the user can annotate objects that are visible due to the blurring or hiding of objects in the virtual camera's field of view, which can be useful during the creative process.

[0061] according to Figure 4 and Figure 5 In the preferred embodiment shown, the rendered modified area can be a user-centric sphere with a radius equal to the distance between the 3D position of the head tracking device HED and the 3D position of the first hand tracking device HAD1. For example, all objects within the sphere are blurred or hidden. Therefore, since the user only needs to move his head, the user does not need to move his hand to explore objects from one side to the other.

[0062] This configuration can also be interesting when the user's real-world environment is crowded in front of them, for example, due to a computer screen. If the user is aware that their side environment is free, they can adjust the radius of the sphere by extending or contracting their arms.

[0063] When the user has both a first hand tracking device HAD1 and a second hand tracking device HAD2 in his hand, he uses one of the aforementioned methods to increase the distance DIS. When the distance DIS exceeds a threshold distance, the convex 3D volume VOL (which can initially be a truncated cone or a cylinder of revolution) is transformed into a sphere SPH. The center of the sphere SPH is located at the 3D position of the head tracking device HED, and the radius Dhh is equal to the distance between the 3D position of the head tracking device HED and the 3D position of the first hand tracking device HAD1.

[0064] Users can switch back to the non-spherical shape by providing input to the second hand tracking device HAD2 (for example, by pressing a button on the second hand tracking device HAD2).

[0065] Advantageously, rendering the modified region also includes 3D objects with a minimum bounding box intersecting the convex volume VOL. For example... Figures 1 to 5 As shown: The checkerboard pattern of squares surrounding and intersecting the convex volume VOL is also blurred / hidden. Therefore, the object (or a portion thereof) is not clipped: it is either blurred / hidden or rendered.

[0066] Figure 6 and Figure 7 The viewpoint of a user when exploring an object is shown due to the method according to the present invention. Figure 6 The viewpoint is shown before the object being explored (in this case, the cube). Figure 7 In the image, the user stretches out his arm and "digs" inside the cube, thus hiding the digging portion of the cube.

[0067] It has been shown that the present invention is particularly well-suited for immersive environments because users only need to interact with the controllers of the virtual reality system, without using other peripheral devices such as a keyboard or mouse. Furthermore, the method of the present invention is user-friendly because users do not need to memorize specific buttons.

[0068] The method of the present invention can be executed by a properly programmed general-purpose computer or virtual reality system (possibly including a computer network), which stores a suitable program in a non-volatile form on a computer-readable medium such as a hard disk, solid-state disk, or CD-ROM, and executes the program using its microprocessor and memory.

[0069] refer to Figure 8A computer CPT suitable for performing methods according to exemplary embodiments of the present invention is described. Figure 8 In this context, the Virtual Reality System (VRS) includes a Central Processing Unit (CPU) P, which executes the method steps described above while running an executable program (i.e., a set of computer-readable instructions). This executable program is stored in a memory device such as RAM M1 or ROM M2 or a hard disk drive (HDD) M3, DVD / CD drive M4, or is stored remotely.

[0070] The claimed invention is not limited to the form of a computer-readable medium on which computer-readable instructions and / or data structures of the invention process are stored. For example, instructions and files may be stored in a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other information processing device (e.g., a server or computer) in communication with the computer. Programs and files may be stored on the same memory device or on different memory devices.

[0071] Furthermore, computer programs suitable for performing the methods of the present invention can be provided as utility applications, background daemons, or components of operating systems, or combinations thereof, which are executed together with a CPU P and operating systems such as Microsoft Vista, Microsoft Windows 10, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art.

[0072] CPU P can be a Xenon processor from Intel or an Opteron processor from AMD, or it can be other processor types (e.g., Freescale ColdFire, IMX, or ARM processors from Freescale, Inc., USA). Alternatively, as those skilled in the art will recognize, the CPU can be a processor such as the Core2Duo from Intel, Inc., or it can be implemented on an FPGA, ASIC, PLD, or using discrete logic circuitry. Furthermore, the CPU can be implemented as multiple processors working together to execute computer-readable instructions of the inventive processes described above.

[0073] Figure 8The virtual reality system also includes a network interface (NI) (e.g., an Intel Ethernet PRO network interface card from Intel Corporation, USA) for interfacing with networks such as local area networks (LANs), wide area networks (WANs), and the Internet. The virtual reality system further includes a head-mounted display (HMD) with a head-tracking device (HED). A general-purpose I / O interface (IF) interfaces with hand controllers (HHC1 and HHC2) respectively equipped with hand-tracking devices (HAD1 and HAD2). The display, keyboard, and pointing devices, together with the display controllers and I / O interfaces, form a graphical user interface (GUI), which is used by the user to provide input commands and by the computer to display 3D objects.

[0074] The disk controller DKC connects HDD M3 and DVD / CD M4 to the communication bus CBS, which can be ISA, EISA, VESA, PCI, or the like, to interconnect all components of the computer.

[0075] Any method steps described herein should be understood as representing a module, segment, or part of code, including one or more executable instructions for implementing a particular logical function or step in the process, and alternative implementations are included within the scope of exemplary embodiments of the invention.

Claims

1. A computer-implemented method for modifying the rendering of regions of a 3D scene in an immersive environment, characterized in that, The region is calculated based on the 3D position of the user's head tracking device and the 3D position of at least one of the user's hand tracking devices; The region includes a convex volume having a planar portion that coincides with the near clipping plane of the virtual camera frustum rendering a portion of the 3D scene, and the convex volume extends along a segment defined by the 3D position of the head tracking device and the 3D position of the hand tracking device.

2. The computer-implemented method according to claim 1, wherein, The convex volume includes a truncated cone or a cylinder of revolution.

3. The computer-implemented method according to any one of claims 1 or 2, wherein, The convex volume has a curved shape such that a portion of it surrounds the at least one hand tracking device, is opposite to the planar portion, and is spaced apart from the at least one hand tracking device by a non-empty safety margin.

4. The computer-implemented method according to claim 1 or 2, comprising: The step of receiving user input to actuate the at least one hand tracking device, and The step of modifying the rendering of the region is performed as long as at least one hand-tracking device is activated.

5. The computer-implemented method according to any one of claims 1 to 2, comprising: The step of receiving user input to actuate the second hand tracking device in the at least one hand tracking device. The step of calculating the distance between the 3D position of the hand tracking device, referred to as the first hand tracking device in the at least one hand tracking device, and the 3D position of the second hand tracking device, and The steps of widening or thinning the convex volume across the segment by increasing or decreasing the distance, respectively.

6. The computer-implemented method according to claim 5, wherein: The steps of receiving user input to actuate the second hand tracking device include: receiving a command to press and hold a button on the second hand tracking device; While the button on the second hand tracking device is being pressed, the step of widening or thinning the convex volume is performed.

7. The computer-implemented method according to claim 5, wherein: The steps of receiving user input to actuate the second hand tracking device include: receiving a command to press and then release a button on the second hand tracking device; After receiving another command to press and then release the button, the step of widening or thinning the convex volume is performed.

8. The computer-implemented method according to claim 5, wherein, If the distance between the 3D position of the first hand tracking device and the 3D position of the second hand tracking device exceeds a threshold distance, the convex volume is transformed into a sphere, the center of which is located at the 3D position of the head tracking device, and the radius of which is equal to the distance between the 3D position of the head tracking device and the 3D position of the first hand tracking device.

9. The computer-implemented method according to any one of claims 1 to 2, wherein, The region includes 3D objects having a minimum bounding box that intersects with the convex volume.

10. The computer-implemented method according to claim 4, wherein, The rendering of the modified area includes blurring the area.

11. The computer-implemented method according to claim 4, wherein, Modifying the rendering of a region includes hiding that region.

12. A computer program product stored on a computer-readable data storage medium, the computer program product comprising computer-executable instructions for causing a virtual reality system to perform the method according to any one of claims 1-11.

13. A computer-readable data storage medium comprising computer-executable instructions for causing a virtual reality system to perform the method according to any one of claims 1 to 11.

14. A virtual reality system, comprising: A processor coupled to memory; A memory storing computer-executable instructions that cause the virtual reality system to perform the method according to any one of claims 1 to 11; Head-mounted display devices with head tracking capabilities; And at least one handheld controller having at least one hand tracking device.

Images