Viewing angle correction of user input objects
By obtaining the camera two-dimensional coordinates and depth information of the user input object in HMD, image transformation and depth map processing are performed, the distance perception and hand-eye coordination problems caused by the position differences of eyes, monitors and cameras in head-mounted devices are solved, and the problem of distance perception and hand-eye coordination caused by the difference in eyes, monitors and cameras in head-mounted devices is improved. The authenticity of the virtual reality experience.
Patent Information
- Application Number
- CN202380037050.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-29
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-06
AI Technical Summary
In head mounted devices, due to the different positioning of eyes, displays and cameras in space, the physical environment images seen by users in the HMD do not always reflect what the user actually sees, resulting in distance perception obstacles, directional problems and Bad hand-eye coordination.
By obtaining the camera two-dimensional coordinate set and depth information of the user input object in the physical environment, image transformation is performed based on the depth map, so that the image appears to be captured from the position of the user's eyes, and using a static depth map to reduce artifacts, determine the user Enter the correct display location for the object.
It improves users' distance perception and hand-eye coordination in HMD, reduces position errors when virtual objects interact with the physical environment, and enhances the authenticity of the virtual reality experience.
Smart Images

Figure CN119096537A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Patent No. 63 / 356,619 filed on June 29, 2022, which is hereby incorporated by reference in its entirety. Technical Field
[0003] The present disclosure generally relates to systems, methods, and devices for correcting a mismatch between a camera position of a user input object and a displayed position of the user input object. Background Art
[0004] In various embodiments, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures an image of the physical environment (e.g., a scene) in which a user is present and a display that displays the image to the user. In some cases, the image or a portion thereof may be combined with one or more virtual objects to present an XR experience to the user. In other cases, the HMD may operate in a pass-through mode in which an image or a portion thereof is presented to the user without adding virtual objects. Ideally, the image of the physical environment presented to the user is substantially similar to what the user would see if the HMD were not present. However, due to the different positioning of the eyes, display, and camera in space, this may not occur, resulting in impaired distance perception, disorientation, and poor hand-eye coordination. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] In order that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be made with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.
[0006] Figure 1 is a block diagram of an example operating environment according to some implementations.
[0007] Figure 2 Example scenarios related to capturing images of a physical environment and displaying the captured images are shown according to some implementations.
[0008] Figure 3 An image of the physical environment captured by an image sensor from a specific perspective.
[0009] Figure 4 yes Figure 3 A top-down perspective view of the physical environment.
[0010] Figure 5A shows what the user's left eye would see if the user was not wearing an HMD Figure 3 view of the physical environment.
[0011] Figure 5B shows the image captured by the left image sensor of the HMD Figure 3 An image of the physical environment.
[0012] Fig. 6A Shows Figure 5B The depth graph of the center row of the depth map of the image.
[0013] Figure 6B Shows Figure 5B A static depth graph of the center row of the depth map of the image.
[0014] 7A to 7C Shown based on Figure 5B The transformed image synthesized from the image .
[0015] Figure 8 is a flowchart representation of a method of determining a display position according to some specific implementations.
[0016] Fig. 9 is a block diagram of an example controller according to some specific implementations.
[0017] Fig.10 is a block diagram of an example electronic device according to some implementations.
[0018] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention
[0019] Various embodiments disclosed herein include devices, systems, and methods for determining a display position. In various embodiments, the method is performed by a device including one or more processors and a non-volatile memory. The method includes obtaining a camera two-dimensional coordinate set of a user input object in a physical environment. The method includes obtaining depth information of a physical environment that does not include the user input object. The method includes transforming the camera two-dimensional coordinate set into a display two-dimensional coordinate set based on the depth information of the physical environment that does not include the user input object.
[0020] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION
[0021] A physical environment refers to a physical place that people can sense and / or interact with without the help of an electronic device. A physical environment may include physical features, such as a physical surface or a physical object. For example, a physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic device. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For another example, an XR system may detect the movement of an electronic device (e.g., a mobile phone, a tablet computer, a laptop computer, a head-mounted device, etc.) that presents an XR environment, and in response, adjust the graphic content and sound field presented to a person by the electronic device in a manner similar to how such views and sounds will change in a physical environment. In some cases (eg, for accessibility reasons), the XR system may adjust characteristics of graphical content in the XR environment in response to representations of physical motion (eg, voice commands).
[0022] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.
[0023] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.
[0024] As described above, in an HMD having a display and a scene camera, if the HMD does not exist due to the different positioning of the eyes, display, and camera in space, the image of the physical environment presented to the user on the display may not always reflect what the user sees. In various cases, this leads to, for example, poor distance perception, disorientation of the user, and poor hand-eye coordination when interacting with the physical environment. Therefore, in various specific implementations, the images from the scene camera are transformed so that they appear to be captured at the location of the user's eyes using a depth map that represents, for each pixel of the image, the distance from the camera to the object represented by the pixel. In various specific implementations, the images from the scene camera are partially transformed so that they appear to be captured at a location closer to the location of the user's eyes than the location of the scene camera.
[0025] In various implementations, the depth map is changed to reduce artifacts. For example, in various implementations, the depth map is smoothed to avoid holes in the transformed image. In various implementations, the depth map is clamped to reduce large movements of pixels during the transformation. In various implementations, the depth map is made static so that dynamic objects do not contribute to the depth map. For example, in various implementations, the depth map value at the location of the dynamic object is determined by interpolating the depth map using positions around the location of the dynamic object. In various implementations, the depth map value at the location of the dynamic object is determined based on the depth map value determined when the dynamic object is not at the location. In various implementations, the depth map is determined using a three-dimensional model of the physical environment without dynamic objects. Using a static depth map can increase spatial artifacts, such as objects not being displayed at their true locations. However, using a static depth map can reduce temporal artifacts, such as flicker.
[0026] When using an altered depth map (or even an inaccurate depth map), a user input object (such as a user's hand) may be displayed at a different location than where the hand is detected in the physical environment. Thus, in various implementations, when a user attempts to interact with a virtual object at a location in the physical environment, it may be preferred to use the displayed location rather than the detected location.
[0027] Figure 1 1 is a block diagram of an example operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and in order not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.
[0028] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Fig. 9 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by electronic device 120 and / or is combined with the electronic device.
[0029] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. The following description is relative to Fig.10 The electronic device 120 is described in more detail.
[0030] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .
[0031] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.
[0032] Figure 2 An example scenario 200 is shown that involves capturing an image of an environment and displaying the captured image according to some implementations. A user wears a device (e.g., Figure 1 The image sensor 230 captures an image of the physical environment, and the display 210 displays the image of the physical environment to the user's eye 220. The image sensor 230 has a viewing angle that is vertically offset from the user's viewing angle (e.g., where the user's eye 220 is located) by a vertical offset 241. Additionally, the viewing angle of the image sensor 230 is longitudinally offset from the user's viewing angle by a longitudinal offset 242. Additionally, in various implementations, the viewing angle of the image sensor 230 is laterally offset from the user's viewing angle by a lateral offset (e.g., toward Figure 2 within or offset outside of the page).
[0033] Figure 3 is an image 300 of a physical environment 301 captured by an image sensor from a specific viewing angle. The physical environment 301 includes a structure 310 having a first surface 311 closer to the image sensor, a second surface 312 farther from the image sensor, and a third surface 313 connecting the first surface 311 and the second surface 312. The first surface 311 has letters A, B, and C painted thereon, the third surface 313 has letters D painted thereon, and the second surface 312 has letters E, F, and G painted thereon.
[0034] From a particular viewing angle, image 300 includes all of the letters painted on structure 310. However, from other viewing angles, as described below, the captured image may not include all of the letters painted on structure 310.
[0035] Figure 4 yes Figure 3 4 is a top-down perspective view of a physical environment 301 of FIG. 4. The physical environment 301 includes a structure 310 and a user 410 wearing an HMD 420. The user 410 has a left eye 411a at a left eye position, thereby providing a left eye perspective. The user 410 has a right eye 411b at a right eye position, thereby providing a right eye perspective. The HMD 420 includes a left image sensor 421a at a left image sensor position that provides a left image sensor perspective. The HMD 420 includes a right image sensor 421b at a right image sensor position that provides a right image sensor perspective. Since the left eye 411a of the user 410 and the left image sensor 421a of the HMD 420 are at different positions, they each provide a different perspective of the physical environment.
[0036] Figure 5A A view 501 of the physical environment 301 as would be seen by the left eye 411a of the user 410 if the user 410 was not wearing the HMD 420 is shown. In the view 501, the first surface 311 and the second surface 312 are present, but the third surface 313 is not present. On the first surface 311, the letters B and C are at least partially visible, while the letter A is not in the field of view of the left eye 411a. Similarly, on the second surface 312, the letters E, F, and G are visible. The view 501 also includes the user's left hand 599 in front of the first surface 311, which partially obscures the letter B.
[0037] Figure 5A An image 502 of the physical environment 301 captured by the left image sensor 421a is shown. In image 502, as in view 501, there is a first surface 311 of the structure 310 and a second surface 312 of the structure 310, but there is no third surface 313. On the first surface 311, the letters B and C can be seen, while the letter A is not in the field of view of the left image sensor 421a. Similarly, on the second surface 312, the letters F and G can be seen, while the letter E is not in the field of view of the left image sensor 421a. It is worth noting that in image 502, compared to view 502, the letter E is not present on the second surface 312. Therefore, the letter E is in the field of view of the left eye 411a, but not in the field of view of the left image sensor 421a. Image 502 also includes the user's left hand 599 in front of the first surface 311, roughly in the middle of the letters B and C and not blocking either of the two letters.
[0038] In various implementations, the HMD 420 transforms the image 502 so that it appears as if it is captured from a left eye perspective rather than from a left image sensor perspective, such as to look like view 501. In various implementations, the HMD 420 transforms the image 502 based on the image 502, the depth value associated with the image 502, and the difference between the left image sensor perspective and the left eye perspective. In various implementations, the difference between the left image sensor perspective and the left eye perspective is determined during a calibration process. In various implementations, the depth value of a pixel of the image represents the distance from the left image sensor 421a to an object represented by the pixel in the physical environment. In various implementations, the depth value is used to generate a depth map that includes a corresponding depth value for each pixel of the image 502. In various implementations, for each pixel position of the transformed image, the corresponding pixel position of the image 502 is determined based on the depth value associated with the pixel position.
[0039] In various implementations, the resulting transformed image includes holes, such as pixel positions of the transformed image where there are no corresponding pixel positions of image 502. Such holes may be filled via interpolation or using additional images, such as using another image from a different perspective (e.g., from the right image sensor 421b or from the left image sensor 421a at a different time).
[0040] In various implementations, the resulting transformed image includes blur, such as pixel locations of the transformed image that exist at multiple corresponding pixel locations of image 502. An averaging or consensus algorithm may be used to remove such blur.
[0041] In various implementations, the depth map does not include dynamic objects and / or only includes static objects. In various implementations, the depth map does not include movable objects and / or only includes fixed objects. In various implementations, the depth map does not include temporary objects and / or only includes permanent objects. Therefore, in various implementations, the depth map does not include the user's left hand 599. For each pixel position representing the user's left hand 599 in the image 502, a depth value is determined by ignoring the distance from the left image sensor 421a to the user's left hand 599. Instead, the depth value represents the distance from the left image sensor 421a to the static object behind the user's left hand 599. In various implementations, the depth value is determined by interpolating the depth values of the pixels around the pixel position representing the user's left hand 599. In various implementations, the depth value is determined based on the depth value of the pixel position when the user's left hand 599 is not at the pixel position. In various implementations, the depth value is determined using a three-dimensional model of the physical environment 301 that does not include the user's left hand 599.
[0042] Using a static depth map may increase spatial artifacts, such as the user's left hand 599 not being displayed at its true location in the physical environment 301. Therefore, the user senses (using proprioception or kinesthetic sense) that the user's left hand 599 is at a different location than the user sees the user's left hand 599. However, using a static depth map may reduce temporal artifacts, such as flickering.
[0043] Fig. 6A A depth graph 600 of a center row of a depth map of an image 502 is shown. The depth graph 600 includes a left first portion 601A corresponding to distances between the left image sensor 421A and various points on the first surface 311 of the structure 310 located to the left of the user's left hand 599, and a right first portion 601B corresponding to distances between the left image sensor 421A and various points on the first surface 311 of the structure 310 located to the right of the user's left hand 599. The depth graph 600 includes a second portion 602 corresponding to distances between the left image sensor 421A and various points located on the second surface 312 of the structure 310. The depth graph 600 includes a third portion 603 corresponding to distances between the left image sensor 421A and various points located on the user's left hand 599.
[0044] Figure 6B A static depth graph 610 of the center row of the static depth map of the image 502 is shown. The static depth graph 610 includes a left first portion 601A, a right first portion 601B, and a second portion 602. However, instead of including the third portion 603, the static depth graph 610 includes a static third portion 613 corresponding to the distances between the left scene camera 421A and various points on the first surface 311 of the structure 310 located behind the user's left hand 599. In various implementations, the static third portion 613 is generated via interpolation, earlier measurements when the left hand 599 is not present, or a three-dimensional model of the physical environment 301.
[0045] Fig. 7A A first synthesized transformed image 701 is shown. The first synthesized image 701 is based on the image 502 of the physical environment 301 captured by the left image sensor 421a and an image of virtual content including a virtual input point 751 and a virtual bubble 752. The virtual input point 751 is displayed at a location on the user's left hand 599 (at the tip of the index finger). The virtual bubble 752 is a three-dimensional virtual object that is virtually located in the physical environment 301 at a location between the user 410 and the structure 310, at approximately the same depth as the user's left hand 599.
[0046] In various implementations, the image 502 of the physical environment 301 is transformed before the transformed image is composited with the image of the virtual content. In various implementations, the image 502 of the physical environment 301 is composited with the image of the virtual content before the composite image is transformed.
[0047] exist Fig. 7A In, use Fig. 6A The depth graph 600 of the physical environment 301 is used to transform the image 502 (or the composite image). Therefore, as in the view 501, the user's left hand 599 partially blocks the letter B on the first surface 311.
[0048] Figure 7B A second synthesized transformed image 702 is shown. Figure 7B In, use Figure 6B The image 502 (or composite image) of the physical environment 301 is transformed using the depth graph 610 of the physical environment 301. Thus, as in the image 502, the user's left hand 599 is located between the letter B and the letter C on the first surface 311 and does not occlude either of the two letters.
[0049] In various implementations, the location of the user's left hand 599 in the physical environment 301 is determined in order to provide user input at that location in the physical environment 301. However, the location of the user's left hand 599 displayed in the second composite image 702 does not correspond to the location of the user's left hand 599 in the physical environment 301. If the user were to provide user input with the user's left hand 599 at the location of the virtual bubble 752 in the physical environment 301 to interact with the virtual bubble 752, the user would not see the user's left hand 599 at the location of the virtual bubble 752. Similarly, if the user were to provide user input with the user's left hand 599 at a location in the physical environment 301 that causes the user's left hand 599 to be displayed at the location of the virtual bubble 752, the user would not be able to interact with the virtual bubble 752.
[0050] Thus, in various implementations, the position of the user's left hand 599 in the physical environment 301 is transformed, and the transformed position is used to provide user input. In various implementations, the three-dimensional physical position in the physical environment 301 is projected into the two-dimensional left camera position in the image 502 (e.g., in Fig. 7A The left camera position is transformed into a two-dimensional left display position (e.g., in FIG. 1 ) using depth information that does not include the user's hand. Figure 7BThe right camera position is transformed into a two-dimensional right display position using depth information excluding the user's left hand 599. The left camera position and the right camera position are used to triangulate the three-dimensional input position in the physical environment 301.
[0051] Figure 7C A third synthesized transformed image 703 is shown. Figure 7C In, use Figure 6B The image 502 (or the composite image) of the physical environment 301 is transformed by using the depth curve graph 610 of the physical environment 301. Therefore, as in the image 502, the user's left hand 599 is located between the letter B and the letter C on the first surface 311 and does not block either of the two letters. In addition, in response to detecting the user's left hand 599 at the input location in the physical environment 301 (which is the same as the location of the virtual bubble 752 in the physical environment 301), the display of the virtual bubble 752 is changed. Figure 7C Change in (e.g., explosion).
[0052] Figure 8 8 is a flowchart representation of a method for determining a display position according to some specific implementations. In various specific implementations, method 800 is performed by a computer having one or more processors, non-volatile memory, an image sensor, and a display (e.g., Figure 1 In some implementations, the method 800 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).
[0053] Method 800 begins in block 810, where a device obtains a set of camera two-dimensional coordinates of a user input object in a physical environment. In various implementations, the user input object is a user's hand. In various implementations, the two-dimensional coordinates of the user's hand are positions of joints of the user's hand. Although method 800 is primarily described for a single joint, it should be understood that method 800 may be performed for multiple joints of a hand. In various implementations, the user input object is a handheld device, such as a stylus. In various implementations, the set of two-dimensional coordinates of the stylus is the position of the tip of the stylus.
[0054] In various implementations, obtaining a camera two-dimensional coordinate set of a user input object includes: obtaining a physical three-dimensional coordinate set in a physical environment of the user input object and projecting the physical three-dimensional coordinate set to a camera image plane. In various implementations, the physical three-dimensional coordinate set is obtained using a hand detection algorithm applied to an image of the physical environment. The image may be captured by one or more image sensors of the device or by other cameras in the physical environment. In various implementations, the physical three-dimensional coordinate set is obtained from the user input object (e.g., from a stylus) based on an inertial measurement unit (IMU) of the stylus.
[0055] In various specific implementations, obtaining a set of camera two-dimensional coordinates of a user input object includes detecting the user input object in an image of the physical environment.
[0056] The method 800 continues in block 820, where the device obtains depth information of the physical environment, where the depth information includes a depth value of the camera two-dimensional coordinate set that is different from the depth to the user input object. For example, in various specific implementations, the depth value of the camera two-dimensional coordinate set represents the depth to a static object behind the user input object.
[0057] In various implementations, the depth information includes a depth map for the physical environment. In various implementations, the depth map is based on an initial depth map in which the value of each pixel represents the depth from the object represented by the pixel. For example, the depth map can be a changed version of the initial depth map. In various implementations, the method 800 includes transforming the image of the physical environment based on the depth map. In various implementations, the transformation is based on a difference between the perspective of an image sensor that captured the image of the physical environment and the perspective of a user. In various implementations, the method 800 includes displaying the transformed image.
[0058] In various implementations, for each pixel position representing a user input object, a depth value of the depth map is determined while ignoring the depth from the user input object. Thus, in various implementations, the depth information of the physical environment is a static depth map. Thus, in various implementations, the depth information does not include dynamic objects and / or only includes static objects. In various implementations, the depth information does not include movable objects and / or only includes fixed objects. In various implementations, the depth information does not include temporary objects and / or only includes permanent objects.
[0059] In various implementations, the depth value is determined by interpolating the depth values of one or more pixels that are close to the pixel position but do not represent the user input object. Therefore, in various implementations, obtaining the depth information of the physical environment includes: determining the depth value of the camera two-dimensional coordinate set by interpolation using the depth values of the positions around the camera two-dimensional coordinate set.
[0060] In various implementations, the depth value is determined based on one or more depth values of the pixel location when the user input object is not at the pixel location. Thus, in various implementations, obtaining depth information of the physical environment includes: determining a depth value of the camera two-dimensional coordinate set when the user input object is not at the camera two-dimensional coordinate set.
[0061] In various implementations, a three-dimensional model of the physical environment is used to determine the depth value. For example, the depth value can be determined using ray tracing from the camera position through the image plane at the pixel position to the static object in the three-dimensional model. Therefore, in various implementations, obtaining the depth information of the physical environment includes: determining the depth value of the camera two-dimensional coordinate set based on the three-dimensional model of the physical environment.
[0062] In various implementations, the depth information of the physical environment is a smoothed depth map obtained by spatially filtering the initial depth map. In various implementations, the depth information of the physical environment is a clamped depth map, in which each pixel of the initial depth map having a value below a depth threshold is replaced by the depth threshold.
[0063] The method 800 continues at block 830 where the device transforms the camera 2D coordinate set into a display 2D coordinate set based on the depth information. In various implementations, the display 2D coordinate set is determined according to the following relationship, where x c and c is the camera's 2D coordinate set, x d and d is the set of two-dimensional coordinates of the display, P c is the 4x4 view projection matrix of the image sensor representing the viewing angle of the image sensor, P d is the user's 4x4 view projection matrix representing the user's point of view, and d is the depth map value at the camera's set of 2D coordinates:
[0064]
[0065] In various implementations, method 800 also includes determining an input three-dimensional coordinate set in a physical environment by triangulating the display two-dimensional coordinate set and the second display two-dimensional coordinate set. In various implementations, the second display two-dimensional coordinate set is obtained in a manner similar to the display two-dimensional coordinate set for a second camera plane or a second image sensor (e.g., for a second eye of a user), where the display two-dimensional coordinate set is determined for a first eye of the user. For example, in various implementations, the device projects the physical three-dimensional coordinate set to a second image plane to obtain a second camera two-dimensional coordinate set, and transforms it using the depth information to generate a second display two-dimensional coordinate set.
[0066] In various implementations, method 800 includes determining user input based on an input three-dimensional coordinate set. In various implementations, the user input is associated with the input three-dimensional coordinate set. In various implementations, the user input includes the presence of a user input object at the input three-dimensional coordinate set. In various implementations, the user input is associated with a position on a ray that passes through the input three-dimensional coordinate set (and in various implementations, a second input three-dimensional coordinate set that passes through the user input object). Thus, in various implementations, the user input includes a user input object pointing to a location from the input three-dimensional coordinate set. In various implementations, determining the user input includes: detecting a gesture. In various implementations, method 800 includes changing the display of virtual content in response to the user input. For example, in Figure 7C In response to detecting the input position of the user's left hand 599 at the position of the virtual bubble 752, the virtual bubble 752 pops.
[0067] In various implementations, method 800 includes displaying virtual content at a set of two-dimensional coordinates on a display. For example, Figure 7B Included is a virtual input point 751 displayed at a set of display two-dimensional coordinates.
[0068] Fig. 9 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 902 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, and / or the like), one or more input / output (I / O) devices 906, one or more communication interfaces 908 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, a memory 920, and one or more communication buses 204 for interconnecting these components and various other components.
[0069] In some implementations, the one or more communication buses 904 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 906 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.
[0070] The memory 920 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 920 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 920 or the non-transitory computer-readable storage medium of the memory 920 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 930 and an XR experience module 940.
[0071] The operating system 930 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 940 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 940 includes a data acquisition unit 942, a tracking unit 944, a coordination unit 946, and a data sending unit 948.
[0072] In some specific implementations, the data acquisition unit 942 is configured to at least Figure 1 The electronic device 120 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data acquisition unit 942 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0073] In some implementations, the tracking unit 944 is configured to map the physical environment 105 and track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various specific implementations, the tracking unit 944 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0074] In some implementations, the coordination unit 946 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 946 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0075] In some implementations, the data transmission unit 948 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transmission unit 948 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.
[0076] Although the data acquisition unit 942, the tracking unit 944, the coordination unit 946, and the data sending unit 948 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 942, the tracking unit 944, the coordination unit 946, and the data sending unit 948 may be located in separate computing devices.
[0077] also, Fig. 9 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig. 9 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.
[0078] Fig.10is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1006, one or more communication interfaces 1008 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1010, one or more XR displays 1012, one or more optional internal-facing and / or external-facing image sensors 1014, a memory 1020, and one or more communication buses 1004 for interconnecting these and various other components.
[0079] In some implementations, the one or more communication buses 1004 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1006 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.
[0080] In some implementations, one or more XR displays 1012 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 1012 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 1012 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 1012 can present MR and VR content.
[0081] In some implementations, the one or more image sensors 1014 are configured to obtain image data corresponding to at least a portion of the user's face (including the user's eyes) (and thus may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 1014 are configured to face forward so as to obtain image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 1014 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.
[0082] Memory 1020 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, memory 1020 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1020 optionally includes one or more storage devices remotely located from one or more processing units 1002. Memory 1020 includes non-transitory computer-readable storage media. In some specific implementations, memory 1020 or non-transitory computer-readable storage media of memory 1020 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1030 and an XR rendering module 1040.
[0083] The operating system 1030 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 1040 is configured to present XR content to a user via one or more XR displays 1012. To this end, in various implementations, the XR rendering module 1040 includes a data acquisition unit 1042, a perspective transformation unit 1044, an XR rendering unit 1046, and a data sending unit 1048.
[0084] In some specific implementations, the data acquisition unit 1042 is configured to at least Figure 1 The controller 110 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various specific implementations, the data acquisition unit 1042 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.
[0085] In some implementations, the perspective transformation unit 1044 is configured to transform the camera 2D coordinate set into the display 2D coordinate set. In various implementations, the perspective transformation unit 1044 is configured to transform the physical 3D coordinate set into the input 3D coordinate set. To this end, in various implementations, the perspective transformation unit 1044 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0086] In some implementations, the XR rendering unit 1046 is configured to display the transformed image via one or more XR displays 1012. To this end, in various implementations, the XR rendering unit 1046 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.
[0087] In some implementations, the data sending unit 1048 is configured to send data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data sending unit 1048 is configured to send authentication credentials to the electronic device. To this end, in various implementations, the data sending unit 1048 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.
[0088] Although the data acquisition unit 1042, the perspective transformation unit 1044, the XR rendering unit 1046, and the data sending unit 1048 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 1042, the perspective transformation unit 1044, the XR rendering unit 1046, and the data sending unit 1048 may be located in separate computing devices.
[0089] also, Fig.10 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig.10 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.
[0090] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.
[0091] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0092] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.
[0093] As used herein, the term “if” may be interpreted to mean “when the stated antecedent is true” or “when the stated antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the stated antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
Claims
1. A method, comprising: At a device having one or more processors and non-transitory memory: Obtain the camera 2D coordinate set of the user input object in the physical environment; Obtaining depth information of the physical environment, wherein the depth information includes a depth value of the camera two-dimensional coordinate set that is different from a depth from the user input object; as well as The set of camera two-dimensional coordinates is transformed into a set of display two-dimensional coordinates based on the depth information of the physical environment. 2 . The method of claim 1 , wherein the user input object comprises at least a portion of a user's hand. The method of claim 1 , wherein the user input object comprises a handheld device.
4. The method according to any one of claims 1 to 3, wherein obtaining the camera two-dimensional coordinate set comprises: A set of physical three-dimensional coordinates of the user input object is obtained and the set of physical three-dimensional coordinates is projected onto a camera image plane.
5. The method according to any one of claims 1 to 3, wherein obtaining the camera two-dimensional coordinate set comprises: The user input object is detected in an image of a physical environment.
6. The method according to any one of claims 1 to 5, wherein the depth information of the physical environment is a smoothed depth map.
7. The method according to any one of claims 1 to 6, wherein the depth information of the physical environment is a clamped depth map.
8. The method according to any one of claims 1 to 7, wherein the depth information of the physical environment is a static depth map.
9. The method according to any one of claims 1 to 8, wherein the depth value of the camera two-dimensional coordinate set represents the depth to a static object behind the user input object.
10. The method according to any one of claims 1 to 9, wherein obtaining the depth information of the physical environment comprises: The depth value of the camera two-dimensional set of coordinates is determined via interpolation using depth values of locations surrounding the camera two-dimensional set of coordinates.
11. The method according to any one of claims 1 to 10, wherein obtaining the depth information of the physical environment comprises: The depth value of the camera two-dimensional coordinate set is determined when the user input object is not at the camera two-dimensional coordinate set.
12. The method according to any one of claims 1 to 11, wherein obtaining the depth information of the physical environment comprises: The depth values for the set of camera two-dimensional coordinates are determined based on a three-dimensional model of the physical environment that does not include the user input object.
13. The method of any one of claims 1 to 12, further comprising determining an input three-dimensional coordinate set of the user input object by triangulating the display two-dimensional coordinate set and a second display two-dimensional coordinate set. The method of claim 13 , further comprising determining a user input based on the set of input three-dimensional coordinates.
15. The method of claim 14, further comprising changing a display of virtual content in response to the user input.
16. The method of any one of claims 1 to 15, further comprising displaying virtual content at the set of display two-dimensional coordinates.
17. The method according to any one of claims 1 to 16, further comprising: transforming an image of the environment based on the depth information of the physical environment; as well as The transformed image is displayed.
18. A device, comprising: one or more processors; non-transitory memory; and One or more programs, the one or more programs are stored in the non-transitory memory, and when executed by the one or more processors, the one or more programs cause the device to perform any of the methods according to claims 1 to 17.
19. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device, cause the device to perform any of the methods of claims 1 to 17.
20. A device, comprising: one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods according to claims 1 to 17.
21. A device, comprising: non-transitory memory; and One or more processors configured to: Obtain the camera 2D coordinate set of the user input object in the physical environment; Obtaining depth information of the physical environment; transforming the camera two-dimensional coordinate set into a display two-dimensional coordinate set based on the depth information of the physical environment excluding the user input object; as well as An input three-dimensional coordinate set of the user input object is determined by triangulating the display two-dimensional coordinate set and a second display two-dimensional coordinate set.
22. The device of claim 21, wherein the one or more processors are further configured to determine user input based on the set of input three-dimensional coordinates.
23. An apparatus according to claim 21 or 22, wherein the user input object comprises at least a part of a user's hand.
24. A device as claimed in claim 21 or 22, wherein the user input object comprises a handheld device.
25. The apparatus of any one of claims 21 to 24, wherein the one or more processors are configured to obtain the camera two-dimensional coordinate set by obtaining a physical three-dimensional coordinate set of the user input object and projecting the physical three-dimensional coordinate set to a camera image plane.
26. The apparatus according to any one of claims 21 to 24, wherein obtaining the camera two-dimensional coordinate set comprises: The user input object is detected in an image of a physical environment.
