Dynamically structured light for depth sensing systems

By using dynamic structured light patterns in depth camera components, based on mapping servers and contrast algorithms, the efficiency and accuracy problems of traditional structured light projectors at different distances and contrast ranges are solved, achieving lower power consumption and higher Depth measurement of accuracy.

CN118938252APending Publication Date: 2024-11-12CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410992064.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-05-17
Filing Date
2019-05-20
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional structured light projectors have fixed patterns and are difficult to adapt to different distances and contrast ranges, resulting in high power consumption or low resolution, and the inability to effectively determine depth information.

Method used

By using dynamic structured light patterns in depth camera components, the texture quantity and density of the structured light patterns are adjusted in real time based on the mapping server and contrast algorithm to adapt to the depth information needs of local areas.

Benefits of technology

Reduces power consumption, improves the accuracy and flexibility of depth measurement, and adapts to different environmental conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938252A_ABST
    Figure CN118938252A_ABST
Patent Text Reader

Abstract

The invention relates to dynamically structured light for a depth sensing system. A depth camera assembly (DCA) determines depth information. The DCA projects the dynamically structured light pattern to the local area and captures an image including a portion of the dynamically structured light pattern. The DCA determines a region of interest in which it may be beneficial to increase or decrease the amount of texture added to the region of interest using the dynamically structured light pattern. For example, the DCA may identify the region of interest based on contrast values calculated using a contrast algorithm, or based on parameters received from a mapping server including a virtual model of the local region. The DCA may selectively increase or decrease the amount of texture added by the dynamically structured light pattern in portions of the localized region. By selectively controlling portions of the dynamically structured light pattern, the DCA may reduce power consumption and / or improve accuracy of depth sensing measurements.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an application with a filing date of May 20, 2019, application number 201980034030.5, and invention name “Dynamic structured light for depth sensing systems”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 674,430, filed on May 21, 2018, and U.S. Application Serial No. 16 / 415,202, filed on May 17, 2019, both of which are hereby incorporated by reference in their entireties. background

[0005] The present disclosure relates generally to depth camera assemblies, and more particularly to dynamic structured light for use with depth camera assemblies.

[0006] Very generally, structured light sensing is the process of projecting a known structured light pattern (e.g., a bar) onto a scene using a structured light projector. Depth information for the scene can be calculated using an image of the scene illuminated with the structured light pattern. The effectiveness of the structured light pattern is based in part on the density of the pattern features. Conventional structured light projectors typically have fixed patterns that are not dynamically adjusted. As a result, conventional structured light projectors are typically optimized for a specific range of distances or contrasts, but outside of these parameters, the pattern either becomes too dense (e.g., making power consumption unnecessarily high and / or features indistinguishable from each other) or too sparse (e.g., resulting in very low resolution).

[0007] Overview

[0008] A depth camera assembly (DCA) determines depth information. Embodiments of the present disclosure support methods, computer-readable media, and devices for providing structured light for determining depth information. In some embodiments, a dynamic structured light pattern is determined based on a virtual model of a physical location stored at a mapping server connected to a head-mounted device (headset) via a network. The virtual model describes a plurality of spaces and properties of these spaces, wherein a location in the virtual model corresponds to a physical location of the head-mounted device. The mapping server determines a location of the head-mounted device in the virtual model based on information describing at least a portion of a local area received from the head-mounted device. The mapping server determines a set of parameters associated with the physical location of the head-mounted device based in part on the determined location in the virtual model and any parameters associated with the determined location. The head-mounted device generates a dynamic structured light pattern using the set of parameters received from the mapping server.

[0009] DCA may determine regions of interest where it may be beneficial to increase or decrease the amount of texture added to the region of interest using a dynamic structured light pattern. For example, DCA may identify regions of interest based on contrast values ​​calculated using a contrast algorithm, or based on parameters received from a mapping server. DCA may selectively increase or decrease the amount of texture added by the dynamic structured light pattern in portions of the local region. By selectively controlling portions of the dynamic structured light pattern, DCA may reduce power consumption and / or improve the accuracy of depth sensing measurements.

[0010] In some embodiments, the DCA includes a structured light (SL) projector, a camera assembly, and a controller. The SL projector is configured to project one or more SL patterns into a local area according to an illumination instruction. The camera assembly is configured to capture an image of a portion of the local area including one or more SL patterns. The controller is configured to determine the amount of contrast in the local area based in part on the captured image. The controller also identifies a first region of interest in the local area and a second region of interest in the local area. The first region of interest has an amount of a first contrast, and the second region of interest has an amount of a second contrast greater than the first amount. The controller updates the illumination instruction based in part on the amount of the first contrast in the first region of interest and the amount of the second contrast in the second region of interest. The controller provides an updated illumination instruction to the SL projector. The updated illumination instruction causes the SL projector to project a first SL pattern into a first region of interest and a second SL pattern into a second region of interest. And the second SL pattern is different from the first SL pattern.

[0011] The present disclosure also relates to the following aspects:

[0012] 1) A depth camera assembly, comprising:

[0013] a structured light SL projector configured to project one or more SL patterns into a local area according to the illumination instructions;

[0014] a camera assembly configured to capture an image of a portion of the local area including the one or more SL patterns; and

[0015] A controller configured to:

[0016] determining an amount of contrast in the local area based in part on the captured image;

[0017] identifying a first region of interest in the local region and a second region of interest in the local region, the first region of interest having a first amount of contrast and the second region of interest having a second amount of contrast greater than the first amount of contrast;

[0018] updating the lighting instructions based in part on the amount of the first contrast within the first region of interest and the amount of the second contrast within the second region of interest; and

[0019] Provide updated lighting instructions to the structured light SL projector, wherein the updated lighting instructions cause the structured light SL projector to project a first SL pattern having a first pattern element density into the first region of interest, and to project a second SL pattern having a second pattern element density into the second region of interest.

[0020] 2) The depth camera assembly according to 1), wherein the controller is further configured to:

[0021] Information describing the local area is transmitted to a mapping server, the information including visual information about at least the portion of the local area.

[0022] 3) The depth camera assembly according to 1), wherein the controller is further configured to:

[0023] The contrast in the local area is calculated using a contrast algorithm.

[0024] 4) The depth camera assembly of 3) wherein the contrast algorithm comprises calculating a sliding window maximum of pixel-level contrast.

[0025] 5) The depth camera assembly according to 1), wherein the controller is configured to identify the first region of interest in the local area based on detecting movement of an object within the first region of interest.

[0026] 6) The depth camera assembly according to 1), wherein the controller is configured to:

[0027] identifying the first region of interest in the local area based on detecting the occluded object in the first region of interest; and

[0028] In response to detecting the occluded object, reducing an amount of texture in the first region of interest, wherein the occluded object is a physical object, and wherein the physical object is occluded by the virtual object.

[0029] 7) The depth camera assembly according to 1), wherein the controller is configured to:

[0030] determining an expected position of a virtual object within the local area; and

[0031] A power level of the first SL pattern is selected based on the expected position of the virtual object.

[0032] 8) The depth camera assembly according to 1), wherein the second SL pattern is different from the first SL pattern.

[0033] 9) The depth camera assembly according to 1), wherein the controller is configured to receive parameters describing the local area from a mapping server.

[0034] 10) The depth camera assembly according to 1), wherein the controller is configured to:

[0035] subtracting a first image captured without any SL pattern from a second image containing one of the SL patterns; and

[0036] Based on the subtraction, the background level of IR illumination is determined.

[0037] 11) A method comprising:

[0038] determining an amount of contrast in a local region;

[0039] identifying a first region of interest in the local region and a second region of interest in the local region, the first region of interest having a first amount of contrast and the second region of interest having a second amount of contrast greater than the first amount of contrast;

[0040] updating lighting instructions based in part on the amount of the first contrast within the first region of interest and the amount of the second contrast within the second region of interest; and

[0041] Provide updated lighting instructions to the structured light SL projector, wherein the updated lighting instructions cause the structured light SL projector to project a first SL pattern having a first pattern element density into the first region of interest, and to project a second SL pattern having a second pattern element density into the second region of interest.

[0042] 12) The method according to 11), further comprising:

[0043] Information describing the local area is transmitted to a mapping server, the information including visual information about at least a portion of the local area.

[0044] 13) The method according to 11) further comprises:

[0045] The contrast in the local area is calculated using a contrast algorithm.

[0046] 14) The method according to 13), wherein the contrast algorithm includes a sliding window for calculating pixel-level contrast.

[0047] 15) The method according to 11) further comprises:

[0048] The first region of interest in the local area is identified based on detecting movement of an object within the first region of interest.

[0049] 16) The method according to 11), further comprising:

[0050] The first region of interest in the local area is identified based on detecting the occluded object in the first region of interest.

[0051] 17) The method according to 16) further comprises:

[0052] In response to detecting the occluded object, an amount of texture in the first region of interest is reduced.

[0053] 18) The method according to 17), wherein the occluded object is a physical object, and wherein the physical object is occluded by the virtual object.

[0054] 19) The method according to 11) further comprises:

[0055] Parameters describing the local area are received from a mapping server.

[0056] 20) A non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform operations comprising:

[0057] determining an amount of contrast in a local region;

[0058] identifying a first region of interest in the local region and a second region of interest in the local region, the first region of interest having a first amount of contrast and the second region of interest having a second amount of contrast greater than the first amount of contrast;

[0059] updating lighting instructions based in part on the amount of the first contrast within the first region of interest and the amount of the second contrast within the second region of interest; and

[0060] Provide updated lighting instructions to the structured light SL projector, wherein the updated lighting instructions cause the structured light SL projector to project a first SL pattern having a first pattern element density into the first region of interest, and to project a second SL pattern having a second pattern element density into the second region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1Ais a perspective view of a head mounted device including a depth camera assembly according to one or more embodiments.

[0063] Figure 1B According to one or more embodiments Figure 1A Schematic diagram of the DCA of the headset.

[0064] Figure 2 A schematic diagram illustrates a user wearing a head mounted device in a room in accordance with one or more embodiments.

[0065] Figure 3 is a block diagram of a DCA in accordance with one or more embodiments.

[0066] Figure 4 is a block diagram of a mapping server according to one or more embodiments.

[0067] Figure 5 is an example of a virtual model describing a local area and parameters describing a configuration of the local area according to one or more embodiments.

[0068] Figure 6 is a flow chart illustrating a process for generating a dynamic structured light pattern in accordance with one or more embodiments.

[0069] Figure 7 is a flow chart illustrating a process for determining an amount of contrast in a local area in accordance with one or more embodiments.

[0070] Figure 8 is a flow chart illustrating a process of identifying a region of interest based on object location in accordance with one or more embodiments.

[0071] Fig. 9 is a block diagram of a system environment including a head mounted device and a mapping server according to one or more embodiments.

[0072] The accompanying drawings depict embodiments of the present disclosure for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles or benefits of the present disclosure described herein. Detailed Description

[0074] A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern onto a local area and captures an image including a portion of the dynamic structured light pattern. The distortion of the dynamic structured light pattern caused by a three-dimensional object in the local area allows the DCA to calculate three-dimensional depth information. The DCA can apply a certain amount of texture to an object in the local area using a dynamic structured light pattern. The DCA can determine an area of ​​interest in which it may be beneficial to change the texture added to the area of ​​interest using a dynamic structured light pattern. The amount of texture added using a dynamic structured light pattern can affect the accuracy of the depth measurement (e.g., more texture can improve accuracy). The texture is affected by the properties of the dynamic structured light pattern (e.g., intensity, density, pattern shape (e.g., dots, bars, grids), polarization, flicker rate, etc.). Changing one property of the dynamic structured light pattern can change the texture added to the local area by the dynamic structured light pattern. For example, the DCA can identify an area of ​​interest based on a contrast value calculated using a contrast algorithm, or based on parameters received from a mapping server including a virtual model of the local area. The DCA can selectively increase or decrease the amount of texture added by the dynamic structured light pattern in some parts of the local area.

[0075] The DCA may independently change the dynamic structured light pattern in different portions of the local area. For example, in response to a change in conditions at a given moment, the DCA may increase the intensity of the dynamic structured light pattern in a first region of interest, and the DCA may simultaneously decrease the intensity of the dynamic structured light pattern in a second region of interest. For each portion of the local area, the DCA may adjust any suitable properties of the dynamic structured light pattern, such as intensity, density, pattern shape (e.g., dots, bars, grid), polarization, blinking rate, etc. In some embodiments, the dynamic structured light pattern may be time multiplexed so that different patterns are projected onto different areas at different times.

[0076] Conventional systems may project a static structured light pattern in order to determine depth information. However, the static structured light pattern may provide more texture than necessary in certain areas of the local area, which consumes excessive power. Additionally, the static structured light pattern may not provide enough texture to obtain accurate depth information in certain areas of the local area, such as in low contrast areas. In contrast, by obtaining information about the local area from a mapping server, and by selectively controlling portions of the dynamic structured light pattern, DCA may reduce power consumption and / or improve the accuracy of depth sensing measurements.

[0077] Embodiments of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted devices, head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, near-eye displays (NEDs), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0078] The head mounted device may be, for example, an NED, an HMD, or some other type of head mounted device. The head mounted device may be part of an artificial reality system. The head mounted device also includes a display and an optical assembly. The display of the head mounted device is configured to emit image light. The optical assembly of the head mounted device is configured to direct the image light to an eye box of the head mounted device corresponding to the position of the wearer's eyes. In some embodiments, the image light may include depth information of a local area around the head mounted device.

[0079] Figure 1A 1 is a perspective view of a head mounted device including a DCA 120 according to one or more embodiments. Figure 1A ), the head mounted device 100 is implemented as a near eye display (NED). In an alternative embodiment ( Figure 1A), the head mounted device 100 is implemented as a head mounted display (HMD). Typically, the head mounted device 100 can be worn on the face of a user so that content (e.g., media content) is presented using one or two lenses 110 of the head mounted device 100. However, the head mounted device 100 can also be used so that media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 100 include one or more images, videos, audio, or some combination thereof. The head mounted device 100 may include a frame 105, lenses 110, one or more speakers 115, a depth camera assembly (DCA) 120, and a position sensor 125, among other components.

[0080] The DCA 120 may be part of a simultaneous localization and mapping (SLAM) sensor mounted on the head mounted device 100 to capture visual information of a local area around part or all of the head mounted device 100. Figure 1A The components of the head mount 100 are shown in example locations on the head mount 100, but the components may be located elsewhere on the head mount 100, on a peripheral device that is paired with the head mount 100, or some combination thereof.

[0081] The head mounted device 100 may correct or enhance the user's vision, protect the user's eyes, or provide images to the user. The head mounted device 100 may be glasses that correct the user's visual defects. The head mounted device 100 may be sunglasses that protect the user's eyes from the sun. The head mounted device 100 may be safety glasses that protect the user's eyes from collisions. The head mounted device 100 may be a night vision device or infrared goggles to enhance the user's night vision. The head mounted device 100 may be a near-eye display that generates artificial reality content for the user.

[0082] The frame 105 holds the other components of the head mounted device 100. The frame 105 includes a front portion that holds the lens 110 and an end piece that attaches to the user's head. The front portion of the frame 105 rests on top of the user's nose. The end piece (e.g., temples) is the portion of the frame 105 to which the user's temples are attached. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that curls behind the user's ear (e.g., temple tip, ear piece).

[0083] The lens 110 provides (e.g., generates and / or transmits) light to a user wearing the head mounted device 100. The lens 110 may be a prescription lens (e.g., a single vision lens, a bifocal lens, and a trifocal lens or a progressive lens) to help correct the user's imperfect vision. The prescription lens transmits ambient light to the user wearing the head mounted device 100. The transmitted ambient light may be altered by the prescription lens to correct the user's imperfect vision. The lens 110 may be a polarized lens or a tinted lens to protect the user's eyes from sunlight. The lens 110 may be one or more waveguides that are part of a waveguide display, wherein image light is coupled to the user's eyes through the end or edge of the waveguide. The lens 110 may include an electronic display for providing image light, and may also include an optical block for amplifying the image light from the electronic display.

[0084] Speakers 115 produce sound for the user's ears. Although speakers 115 are shown outside of frame 105, speakers 115 may be enclosed in frame 105. In some embodiments, instead of individual speakers for each ear, headgear 100 includes a speaker array (e.g., an array of speakers integrated into the end pieces of frame 105). Figure 1A ), to enhance the directionality of the presented audio content.

[0085] The DCA 120 determines depth information for a local area (e.g., a room) surrounding the head mounted device 100. The DCA 120 includes a light projector (e.g., a structured light projector and / or flash lighting for time-of-flight), an imaging device, and a controller ( Figure 1A ). The structured light projector can generate a dynamic structured light pattern in a local area, which can be adjusted according to various conditions, as further described herein.

[0086] The DCA 120 may capture data, such as an image of light projected onto a local area by a light projector captured by an imaging device. In one embodiment, the DCA 120 may include a controller and two or more imaging devices, the imaging devices being oriented to capture portions of the local area in a stereoscopic manner. The captured data may be an image of the local area captured in a stereoscopic manner by the two or more imaging devices. The controller of the DCA 120 uses the captured data and depth determination techniques (e.g., structured light, time of flight, stereo imaging, etc.) to calculate depth information of the local area. Based on the depth information, the controller of the DCA 120 determines absolute position information of the head mounted device 100 within the local area. The controller of the DCA 120 may also generate a model of the local area. The DCA 120 may be integrated with the head mounted device 100, or may be located within the local area outside the head mounted device 100.

[0087] In some embodiments, some cameras of the DCA 120 may also be configured to function as a passive camera assembly (PCA) that includes one or more passive imaging devices that generate color (e.g., RGB) image data. The passive imaging devices capture light from the environment of a local area to generate color image data. Rather than the pixel values ​​defining the depth or distance from the imaging device, the pixel values ​​of the color image data may define the visible color of an object captured in the image data. In some embodiments, the passive imaging devices are controlled by a controller that generates the color image data based on the light captured by the passive imaging devices.

[0088] DCA 120 may correct for ambient IR signals in an image frame that includes an IR structured light pattern. Ambient IR signals may be any IR signals in an image frame that are not provided by the IR structured light pattern. DCA 120 may project a dynamic structured light pattern during capture of a first image, and DCA 120 may deactivate the dynamic structured light pattern during capture of a second image. DCA 120 may subtract pixel values ​​of the second image from pixel values ​​of the first image, which may remove ambient IR signals from the image and enhance the ability of DCA 120 to calculate depth information.

[0089] The position sensor 125 generates one or more measurement signals in response to the movement of the head mounted device 100. The position sensor 125 may be located on a portion of the frame 105 of the head mounted device 100. The position sensor 125 may include a position sensor, an inertial measurement unit (IMU), or both. Some embodiments of the head mounted device 100 may include or exclude the position sensor 125, or may include more than one position sensor 125. In an embodiment where the position sensor 125 includes an IMU, the IMU generates IMU data based on the measurement signals from the position sensor 125. Examples of the position sensor 125 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor 125 may be located outside the IMU, inside the IMU, or some combination thereof.

[0090] Based on one or more measurement signals, the position sensor 125 estimates the current position of the head mounted device 100 relative to the initial position of the head mounted device 100. The estimated position may include the position of the head mounted device 100 and / or the orientation of the head mounted device 100 or the head of the user wearing the head mounted device 100, or some combination thereof. The orientation may correspond to the position of each ear relative to a reference point. In some embodiments, the position sensor 125 estimates the current position of the head mounted device 100 using depth information and / or absolute position information from the DCA 120. The position sensor 125 may include multiple accelerometers that measure translational motion (forward / backward, up / down, left / right) and multiple gyroscopes that measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the head mounted device 100 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometer over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated position of a reference point on the head mounted device 100. A reference point is a point that can be used to describe the position of the head mounted device 100. Although a reference point can generally be defined as a point in space; however, in practice, a reference point is defined as a point within the head mounted device 100.

[0091] Figure 1B According to an embodiment Figure 1A 150 of a DCA 120 of a head mounted device 100 projecting light into a localized area 155. Some embodiments of the DCA 120 have components different from those described herein. Similarly, in some cases, functionality may be distributed among components in a manner different from that described herein.

[0092] The DCA 120 includes a structured light projector 160, a camera assembly 165, and a controller 170. The structured light projector 160 illuminates the local area 155 with structured light. In some embodiments, the structured light projector 160 includes a light source, a diffractive optical unit, and a projection assembly. The light source emits light. The diffractive optical unit includes a plurality of diffractive optical elements (DOEs). The DOEs may be, for example, one or more diffraction gratings, diffusers, spatial light modulators, some other elements that form structured light, or some combination thereof. Each DOE of the diffractive optical unit converts light from the light source into a structured light pattern. The structured light pattern may be, for example, a dot pattern, a line pattern, a sinusoidal pattern, a multi-(spatial) tone pattern and a grid pattern, diffuse light (e.g., for time of flight or active stereo depth determination), some other light that can be used to determine depth information, or some combination thereof. Dynamic light pattern projection is achieved by dynamic control of the light source, DOE, or projection assembly, or a combination thereof. In embodiments where the structured light pattern is diffuse light, the DOE can be used to generate different fields of view for the DCA 120. The DCA 120 selects the DOE to adjust its field of view. The projection assembly projects the structured light pattern into the local area 155. For example, the projection assembly includes one or more optical elements (e.g., lenses, polarizers, gratings, etc.) that collect the structured light and project the structured light pattern into part or all of the local area 155.

[0093] The camera assembly 165 captures an image of a portion of the local area 155 and outputs the captured image to the controller 170. The camera assembly 165 includes one or more cameras.

[0094] In some embodiments, the controller 170 is configured to use the captured image from the camera assembly 165 to determine the depth information of the object in the local area 155. The controller 170 also controls how the structured light projector 160 projects the structured light and controls how the camera assembly 165 captures the image light. For example, the controller 170 instructs the structured light projector 160 to select one or more DOEs and / or control the light source and instructs the camera assembly 165 to capture the image. In some embodiments, the controller 170 sends a signal to the console (not in the Figure 1B ) and / or appropriate modules of the head mounted device 100 (e.g., Figure 1B The console and / or the head mounted device 100 may utilize the depth information to, for example, generate content for presentation. Figure 3 Additional details regarding the DCA 120 and its components are discussed in detail.

[0095] Figure 2A schematic diagram of a user 210 wearing a head mounted device 200 in a room 220 is shown in accordance with one or more embodiments. The head mounted device 200 includes a DCA (e.g., DCA 120). The head mounted device 200 may be an embodiment of the head mounted device 100. The head mounted device 200 may determine depth information describing a portion or the entire room 220 and / or objects within the room 220. A structured light projector of the DCA may project a dynamic structured light pattern into the room 220. A camera component of the DCA may obtain an image including the dynamic structured light pattern, and distortion in the dynamic structured light pattern may provide information about objects within the room 220.

[0096] The head mounted device 200 may determine visual information describing at least a portion of the room 220 and provide the visual information to the mapping server 230. For example, the head mounted device 200 may include at least one DCA that generates depth information for at least the portion of the room 220. The DCA on the head mounted device 200 may also include at least one passive camera assembly (PCA) that generates color image data for at least the portion of the room 220. In some embodiments, the DCA and PCA of the head mounted device 200 are part of a simultaneous localization and mapping (SLAM) sensor mounted on the head mounted device 200 for determining visual information of the room 220. Therefore, the depth information captured by the at least one DCA and / or the color image data captured by the at least one PCA may be referred to as visual information determined by the SLAM sensor of the head mounted device 200.

[0097] The head mounted device 200 can transmit the visual information to the mapping server 230 via the network 240 for determining a set of parameters of the room 220. The network 240 can include any combination of local area networks and / or wide area networks using wireless and / or wired communication systems. For example, the network 240 can include the Internet and a mobile phone network. In one embodiment, the network 240 uses standard communication technologies and / or protocols. Therefore, the network 240 can include links using technologies such as Ethernet, 802.11, World Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 240 can include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged through the network 240 can be represented using techniques and / or formats including binary image data (e.g., portable network graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or part of the link can be encrypted using conventional encryption techniques (e.g., secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.). The network 240 can also connect multiple head-mounted devices located in the same or different rooms to the same mapping server 230.

[0098] In some embodiments, in addition to the visual information used to determine the parameter set, the head mounted device 200 also provides its location information (e.g., the global positioning system (GPS) location of the room 220) to the mapping server 230. In some embodiments, the head mounted device 200 provides the location information to the mapping server 230 for determining the parameter set. The parameter set can be used to represent various properties of a specific configuration in the room 220, which together define a depth map in the room 220. The parameters can include the size of the room, the number of objects in the room, the location of the objects, the reflectivity of the surface, the contrast value, the brightness, the lighting conditions, whether and what type of texture (e.g., the density of the SL pattern, the type of the SL pattern, the brightness of the SL pattern, etc.) should be added to the location in the room, etc., for each location in the virtual model.

[0099] The configuration and associated conditions in the room 220 may change based on, for example, at least one of a change in the position of the head mounted device 200 in the room 220, a change in the position of a physical or virtual object in the room 220, an increase or decrease in the number of objects in the room 220, a change in the lighting in the room, etc. The head mounted device 200 may dynamically change all or a portion of the dynamic structured light pattern based on information available to the head mounted device 200. The dynamic structured light pattern may be formed by a plurality of segments or tiles, each of which is projected into a portion of the room 220, together forming the entire dynamic structured light pattern. In some embodiments, each segment of the dynamic structured light pattern may contain a different structured light pattern and may be adjusted individually. For example, in response to a change in the conditions in the room 220 at a given moment, the head mounted device 200 may increase the intensity of the dynamic structured light pattern in a first segment, and the head mounted device 200 may simultaneously reduce the intensity of the dynamic structured light pattern in a second segment. For each portion of the room 220, the head mounted device 200 may adjust any suitable properties of the dynamic structured light pattern, such as intensity, density, pattern shape (e.g., dots, bars, grid), polarization, blinking rate, etc. In some embodiments, the dynamic structured light pattern may be time multiplexed such that different patterns are projected into different portions of the room 220 at different times.

[0100] The head mounted device 200 can adjust the dynamic structured light pattern in the low contrast area 260. For example, the head mounted device can change the intensity, density and / or shape (e.g., from bars to dots) of the dynamic structured light pattern in the direction of the low contrast area 260 to add texture to the low contrast area 260. In some embodiments, the depth camera assembly can detect the low contrast area 260, and the depth camera assembly can adjust the dynamic structured light pattern in the direction of the low contrast area 260 to improve the quality of depth estimation in that area.

[0101] The mapping server 230 includes a database storing virtual models describing a plurality of spaces, wherein a position in the virtual model corresponds to the current configuration of the room 220. The mapping server 230 receives visual information describing at least a portion of the room 220 and / or position information of the room 220 from the head mounted device 200 via the network 240. The mapping server 230 determines, based on the received visual information and / or position information, a position in the virtual model associated with the current configuration of the room 220. The mapping server 230 determines (e.g., retrieves) a set of parameters associated with the current configuration of the room 220 based in part on the determined position in the virtual model and any parameters associated with the determined position.

[0102] Mapping server 230 may provide information about the parameter set to head mounted device 200 (e.g., via network 240) to enable head mounted device 200 to determine an appropriate light pattern for the structured light projector. For example, mapping server 230 may transmit information about the location of low contrast area 260 in room 220 to head mounted device 200. Alternatively, mapping server 230 may generate information describing a structured light pattern using the parameter set and provide the information to head mounted device 200, which may then use the information to generate a structured light pattern. In some embodiments, some components of mapping server 230 may be connected to a network via a wired connection ( Figure 2 The device may be integrated with another device (e.g., a console) connected to the head mounted device 200 (not shown).

[0103] The head mounted device 200 may display a virtual object, such as the virtual object 270 or the virtual object 280, to the user 210. Figure 2 As shown, virtual object 280 can be located in the line of sight between head mounted device 200 and physical object 290. Head mounted device 200 can obtain information describing the location of physical object 290 via mapping server 230 or depth camera assembly.

[0104] The head mounted device 200 may not need to obtain depth information for physical objects that are occluded by the virtual object 280. However, it may be desirable for the head mounted device 200 to obtain depth information for any object that may be located between the head mounted device 200 and the virtual object 280. Therefore, the head mounted device 200 may reduce the amount of texture in the region of interest containing the occluded object by reducing the intensity or density of the structured light projected in the direction of the virtual object 280 so that the texture is sufficient to determine the depth information between the head mounted device 200 and the virtual object 280.

[0105] Similarly, the head mounted device 200 may not need to obtain depth information in the area of ​​the virtual object 270 that is occluded by the physical object 290. Therefore, the head mounted device 200 may reduce the intensity or density of the structured light projected in the direction of the virtual object 270 without providing sufficient texture for the distance of the virtual object 270.

[0106] Figure 3 is a block diagram of a DCA 300 for a head mounted device according to one or more embodiments. In some embodiments, the DCA 300 may be a reference Figure 1A and Figure 1B1. Some embodiments of the DCA 300 have components that differ from those described herein. Similarly, functionality may be distributed among components in a manner different from that described herein. In some embodiments, some functionality of the DCA 300 may be part of different components (e.g., some functionality may be part of the headset and some functionality may be part of the console and / or server).

[0107] The DCA 300 generates depth information for a local area, such as a room. The depth information includes pixel values ​​defining distances from the DCA 300, which provides a map of the locations captured in the depth information, such as a three-dimensional map of the locations captured in the depth information. The DCA 300 includes a structured light projector 310, a camera assembly 320, and a controller 330.

[0108] The structured light projector 310 generates structured light and projects the structured light into a local area. The structured light projector 310 includes one or more illumination sources, one or more templates (pattern plates) and one or more projection components. The illumination source is configured to emit light (e.g., as a beam), and can emit light of multiple wavelengths. The illumination source can emit, for example, light in a visible band (about 380nm to 750nm), an infrared (IR) band (about 750nm to about 1800nm), an ultraviolet band (about 100nm to 380nm), some other part of the electromagnetic spectrum that the camera assembly 320 is configured to detect, or some combination thereof. The illumination source can be, for example, a laser diode (e.g., an edge emitter), an inorganic or organic LED, a vertical cavity surface emitting laser (VCSEL), or some other source. In some embodiments, a single emitter or multiple emitters in a structured light generator can emit light with a structured light pattern. In some embodiments, multiple emitters can be addressed individually or by block / section / group addressing. Light emitted from one or more illumination sources can be, for example, polarized (eg, linearly polarized, circularly polarized, etc.).

[0109] One or more templates generate SL patterns using light from one or more illumination sources. The templates can be, for example, one or more diffractive optical elements, masks, computer-generated holograms, acousto-optic devices (e.g., configured to act as dynamic diffraction gratings), some other element that generates structured light patterns, or some combination thereof.

[0110] The projection assembly projects structured light into the local area. The projection assembly includes one or more optical elements that direct the structured light pattern into the local area. For example, the projection assembly may include multiple lenses. In some embodiments, the projection assembly includes a beam shaping element that changes the profile or general intensity envelope of the structured light diffracted by one or more templates. The SL pattern includes multiple SL elements. Each element is a discrete portion of the SL pattern, such as a line or a dot. For example, each SL element projected by the illuminated template is a dot associated with a specific location on the template. For example, the structured light projector 310 can project multiple SL elements of different types (e.g., lines, grids, or dots) onto a portion of the local area around the head-mounted device.

[0111] The structured light projector 310 can dynamically change the dynamic structured light pattern according to instructions from the controller 330. In some embodiments, the structured light projector 310 can project the same pattern across the entire field of view of the structured light projector 310, or the structured light projector 310 can project different structured light patterns into different parts of the field of view. For example, the dynamic structured light pattern can be formed by a plurality of segments or blocks, each of which is projected into a portion of the field of view of the structured light projector 310, together forming the entire dynamic structured light pattern. In some embodiments, each segment of the dynamic structured light pattern can contain a different structured light pattern and can be adjusted individually. For example, in response to a change in conditions at a given moment, the structured light projector 310 can increase the intensity of the dynamic structured light pattern in the first segment, and the structured light projector 310 can simultaneously reduce the intensity of the dynamic structured light pattern in the second segment. For each portion of the field of view of the structured light projector 310, the structured light projector 310 may adjust any suitable properties of the dynamic structured light pattern, such as intensity, density, pattern shape (e.g., dots, bars, grid), polarization, blinking rate, etc. In some embodiments, the dynamic structured light pattern may be time multiplexed such that different patterns are projected into different portions of the field of view at different times.

[0112] The structured light projector 310 can change the dynamic structured light pattern by making different emitters active, making some or all emitters adjustable, or by using different templates for different structured light patterns. In some embodiments, the dynamic structured light pattern is achieved by controlling each emitter alone or in subsets or sections, where the dynamic properties are manifested in the projected pattern in the form of pattern shape, intensity, polarization, time modulation, field of view, etc. In some embodiments, adjustable optical components such as lenses, gratings, or spatial light modulators are used to achieve the dynamic pattern. In some embodiments, considering the projector size, weight, power, cost, etc., a combination of addressable light sources and adjustable optical devices can be used to achieve the dynamic light pattern.

[0113] The camera assembly 320 is configured to capture an image of a local area. The camera assembly 320 includes one or more imaging devices (e.g., cameras) that can capture images of SL patterns in at least one light band. In some embodiments, one or more imaging devices of the camera assembly 320 and / or other imaging devices can also capture light in the visible light band. In some cases, a captured image of part or all of a local area may include part or all of the SL pattern (e.g., reflected by an object in the local area).

[0114] Controller 330 controls the components of DCA 300. The controller may include a mapping module 340, a contrast module 350, a depth measurement module 360, and an illumination module 370. Some embodiments of controller 330 have components that are different than those described herein. Similarly, functionality may be distributed among components in a manner different than described herein. In some embodiments, some functions of controller 330 may be part of different components (e.g., some functions may be part of the headset and some functions may be part of the console and / or server).

[0115] The mapping module 340 may be configured to communicate with a mapping server (eg, the mapping server 230). The mapping module 340 may transmit information to the mapping server, including the location of the DCA 300 and / or images captured by the DCA 300. The mapping module may also transmit information to the mapping server. Figure 2 The mapping server 230 or some other component provides the depth information.

[0116] The contrast module 350 can calculate the contrast in the local area based on the image captured by the camera assembly 320. The contrast module 350 can use various contrast algorithms, such as Weber contrast, Michelson contrast, root mean square (RMS) contrast, etc. In some embodiments, the contrast module 350 can use an L2 contrast algorithm with a sliding maximum to calculate the contrast, such as reference Figure 7As further described, the contrast module 350 may calculate a contrast for each pixel of the sensor corresponding to a specific region in the local area.

[0117] The lighting module 370 may generate lighting instructions to be provided to the light projector 310 to project a dynamic structured light pattern. The lighting instructions may reduce (and in some cases minimize) the power consumption of the light projector 310 while providing sufficient texture in a local area to calculate depth information. In some embodiments, the lighting module 370 may determine that the local area contains sufficient texture based on the confidence level of the depth measurement (e.g., if the depth measurement is calculated to have a confidence level greater than 95% or greater than 50%, the texture may be sufficient). For example, the contrast module 350 may detect a first region of interest having a relatively high contrast value in the local area, and the lighting module 370 may reduce the amount of texture provided by the light projector 310 to the first region of interest, because the depth measurement module 360 ​​may fully calculate the depth information in the first region of interest without the additional texture provided by the light projector 310. Similarly, the contrast module 350 may detect a second region of interest having a relatively low contrast value in the local area, and the lighting module 370 may increase the amount of texture provided by the light projector 310 to the second region of interest, because the depth measurement module 360 ​​may better calculate the depth information in the second region of interest using the additional texture.

[0118] In some embodiments, as part of frame subtraction, the illumination module 370 can indicate whether the light projector is active (e.g., emitting light) or inactive (e.g., not emitting light) in conjunction with the image captured by the camera assembly 320. For example, the DCA 300 can correct for ambient IR signals in an image frame that includes an IR structured light pattern. The ambient IR signal can be any IR signal in the image frame that is not provided by the IR structured light pattern. The illumination module 370 can instruct the light projector 310 to activate during the capture of the first image by the camera assembly 320 and to deactivate during the capture of the second image by the camera assembly 320.

[0119] The depth measurement module 360 ​​determines the depth information of each pixel of the image based on the image captured by the camera assembly 320, and stores the depth information of each pixel in association with the pixel to generate a depth image. The depth measurement module analyzes the image containing the dynamic structured light pattern. The distortion of the dynamic structured light pattern caused by the three-dimensional object in the local area allows DCA to calculate the three-dimensional depth information.

[0120] In some embodiments, depth measurement module 360 ​​performs frame subtraction using multiple image frames captured by illumination module 370. Depth measurement module 360 ​​may subtract pixel values ​​of a second image captured with the structured light pattern inactive from pixel values ​​of a first image captured with the structured light pattern activated, which may remove ambient IR signals from the image and improve the ability of depth measurement module 360 ​​to calculate depth information. In some embodiments, the intensity of the first image and the second image may be scaled based on the difference in exposure time between the captures.

[0121] In various embodiments, the depth measurement module 360 ​​determines depth information in an image without a dynamic structured light pattern, for example by applying one or more stereo imaging (also referred to as "stereo") processes to a pair of images captured at a common time by the camera assembly 320 to determine the depth information. Example stereo processes include global patch matching and semi-global matching. When the depth measurement module 360 ​​applies a semi-global matching process, the depth measurement module 360 ​​performs dense patch matching on a subset of images captured by a first imaging device of the camera assembly 320 and a subset of additional images captured by a second imaging device of the camera assembly 320 at a common time with the images; for example, the depth measurement module 360 ​​rectifies the image and the additional image and performs dense patch matching along epipolar lines between the rectified image and the additional image. When applying a semi-global matching process, the depth measurement module 360 ​​also propagates the depth information of a pixel to other pixels along a limited number of paths (e.g., 4, 8, 16) on the image. However, in other embodiments, the depth measurement module 360 ​​implements any suitable stereo process, such as via a convolutional neural network. In some embodiments, illumination module 370 may instruct light projector 310 to project a structured light pattern into a scene to add texture to the scene, which may improve the accuracy of depth measurements even without a pre-calibrated structured light pattern.

[0122] Figure 4 is a block diagram of a mapping server 230 according to one or more embodiments. The mapping server 230 determines information such as Figure 1AA set of parameters of the physical space (room) in which the head-mounted device 100 of the head-mounted device is located. The parameters may include the size of the room, the number of objects in the room, the location of the objects, the reflectivity of the surface, the contrast value, the brightness, the lighting conditions, whether and what type of texture should be added to the location in the room, etc. The determined set of parameters can be used to generate a structured light pattern of a local area at the head-mounted device 100. The mapping server 230 includes a virtual model database 410, a communication module 420, a mapping module 430, and an analysis module 440. In other embodiments, the mapping server 230 may have any combination of the listed modules and any additional modules. In some other embodiments, the mapping server 230 includes a combination of Figure 4 One or more modules of the module functions shown. The processor of the mapping server 230 ( Figure 4 ) can run the virtual model database 410, the communication module 420, the mapping module 430, the analysis module 440, one or more other modules or a combination of Figure 4 Part or all of the modules in the module functionality shown in .

[0123] The virtual model database 410 stores virtual models describing a plurality of physical spaces and the properties of these physical spaces. Each position in the virtual model corresponds to a physical position of a head-mounted device in a local area having a specific configuration. The condition represents a condition of a local area having a set of properties represented by a set of parameters. A specific position in the virtual model may correspond to a physical position of a head-mounted device in a local area having a specific configuration. Figure 2 The current physical location within the room 220 of the virtual model. Each location in the virtual model is associated with a parameter set of the corresponding physical space, which parameter set represents a configuration of the local area. The parameter set describes various properties of a specific configuration of the local area. The parameters may include room size, number of objects in the room, object location, reflectivity of the surface, contrast value, brightness, lighting conditions, whether and what type of texture should be added to the location in the room, etc., for each location in the virtual model.

[0124] The communication module 420 is connected via a network (eg, Figure 2The communication module 420 is a module for communicating with the head mounted device (network 240) via the head mounted device. The communication module 420 receives visual information describing at least a portion of the room 220 from the head mounted device. In one or more embodiments, the visual information includes image data of at least a portion of the room 220. For example, the communication module 420 receives depth information captured by the DCA of the head mounted device, the depth information having information about the shape of the room 220 defined by the surfaces of the room 220 (e.g., the surfaces of the walls, floor, and ceiling of the room 220). The communication module 420 may also receive color image data captured by the PCA of the head mounted device. The mapping server 230 may use the color image data to associate different locations and contrast values ​​with the surfaces of the room 220. The communication module 420 may provide the visual information (e.g., depth information and color image data) received from the head mounted device to the mapping module 430.

[0125] The mapping module 430 maps the visual information received from the head mounted device to the position of the virtual model. The mapping module 430 determines the position of the virtual model corresponding to the current physical space where the head mounted device is located (i.e., the current configuration of the local area). The mapping module 430 searches the entire virtual model to find a mapping between (i) visual information including at least, for example, information about the geometry of the surface of the physical space and information about the contrast value of the surface and (ii) the corresponding configuration of the physical space in the virtual model. The mapping is performed by matching the geometry and / or contrast information of the received visual information with the geometry and / or contrast information stored as part of the configuration of the physical space in the virtual model. The corresponding configuration of the physical space in the virtual model corresponds to the model of the physical space where the head mounted device is currently located. If no match is found, this indicates that the current configuration of the physical space has not been modeled in the virtual model. In this case, the mapping module 430 can notify the analysis module 440 that no match is found, and the analysis module 440 determines a parameter set based at least in part on the received visual information.

[0126] The analysis module 440 determines a set of parameters associated with the physical location of the head mounted device based in part on the determined location in the virtual model obtained from the mapping module 430 and any parameters associated with the determined location in the virtual model. The parameters may include the size of the room, the number of objects in the room, the location of the objects, the reflectivity of the surface, the contrast value, the brightness, the lighting conditions, whether and what type of texture should be added to the location within the room, etc. In some embodiments, the analysis module 440 retrieves the set of parameters from the virtual model because the set of parameters is stored at the determined location in the virtual model associated with the particular spatial configuration. In some other embodiments, the analysis module 440 determines the set of parameters by adjusting a set of parameters previously determined for the particular spatial configuration in the virtual model based at least in part on the visual information received from the head mounted device. For example, the analysis module 440 may run an offline simulation using the received visual information to determine the set of parameters.

[0127] In some embodiments, the analysis module 440 determines that the previously generated parameters are inconsistent with the conditions of the current physical location of the head mounted device. The detected mismatch can trigger the regeneration of a new set of parameters at the mapping server 230. Once recalculated, this new set of parameters can be input into the virtual model of the mapping server 230 as a replacement for the previous set of parameters, or as an additional state of the same physical space. The analysis module 440 provides the derived parameter set to the communication module 420, which transmits the parameter set from the mapping server 230 to the head mounted device, which can use the parameter set to generate a structured light pattern.

[0128] In some embodiments, one or more parameters determined by the analysis module 440 represent a new set of parameters that are not part of the virtual model because the current configuration of the room 220 and the corresponding conditions of the room 220 are not modeled by the virtual model. In this case, the virtual model database 410 stores the new set of parameters at a location within the virtual model associated with the current configuration of the room 220, thereby modeling the current conditions of the room 220. Some or all of the one or more parameters may be stored in the virtual model along with a confidence (weight) and an absolute timestamp associated with the parameter, which may be used to recalculate the portion of the parameters.

[0129] Figure 5is an example of a virtual model 500 describing a local area and parameters describing the configuration of the local area. The parameters may include the size of the room, the number of objects in the room, the location of the objects, the reflectivity of the surface, the contrast value, the brightness, the lighting conditions, whether and what type of texture should be added to the location within the room, etc. The virtual model 500 may be stored in the virtual model database 410 of the mapping server 230. The virtual model 500 may represent a geographic information storage area in the virtual model database 410, which stores a geographically bundled information triplet (i.e., a local area identifier (ID) 510, a local area configuration ID 520, and a parameter set 530) of each local area around one or more head-mounted devices.

[0130] The virtual model 500 includes a list of possible local areas S1, S2, ..., Sn, each of which is identified by a local area ID 510. The local area ID 510 identifies a specific type of local area. For example, the local area ID 510 identifies different types of rooms, such as a conference room, a bathroom, a hallway, an office, a bedroom, a dining room, a living room, some other type of physical space, or some combination thereof. Therefore, each local area ID 510 corresponds to a specific type of physical space.

[0131] Each local area ID 510 is associated with one or more local area configuration IDs 520. Each local area configuration ID 520 corresponds to a configuration having specific depth information or contrast information of the local area identified by the local area ID 510. The local area configuration ID 520 may include information identifying the number of objects in the local area, the positioning of objects in the local area, the ambient lighting of the local area, or other conditions in the local area. Different configurations of the local area affect the depth information of different areas of the local area or the contrast information of different areas of the local area. Each local area configuration ID 520 may be represented as a code ID (e.g., a binary code, an alphanumeric code) that identifies the configuration of the local area ID 510. For example, Figure 5 As shown, the local area S1 can be associated with p different spatial configurations S1C1, S1C2, ..., S1Cp, each spatial configuration represents a different configuration of the local area S1; the local area S2 can be associated with q different configurations S2C1, S2C2, ..., S2Cq, each configuration represents a different configuration of the local area S2; the local area Sn can be associated with r different configurations SnC1, SnC2, ..., SnCr, each configuration represents a different configuration of the local area Sn. The mapping module 430 can search the virtual model 500 to find the appropriate local area configuration ID 520 based on the visual information of the local area received from the head-mounted device.

[0132] Each local area configuration ID 520 is associated with a specific parameter set 530 stored in the corresponding location of the virtual model 500. Figure 5 As shown, p different spatial configurations S1C1, S1C2, ..., S1Cp of the same local area S1 are associated with p different parameter sets {P11}, {P12}, ..., {P1p}. Similarly, Figure 5 As further shown in FIG. 5 , q different spatial configurations S2C1, S2C2, ..., S2Cq of the same local area S2 are associated with q different parameter sets {P21}, {P22}, ..., {P2q}; and r different spatial configurations SnC1, SnC2, ..., SnCr of the same local area Sn are associated with r different parameter sets {Pn1}, {Pn2}, ..., {Pnr}. When the mapping module 430 finds the local area configuration ID 520 corresponding to the current configuration of the local area where the head-mounted device is located, the analysis module 440 can retrieve the corresponding parameter set 530 from the virtual model 500. The parameters can be used by a DCA (e.g., DCA 300) to generate a dynamic structured light pattern.

[0133] Figure 6 is a flow chart illustrating a process 600 for generating a dynamic structured light pattern in accordance with one or more embodiments. Figure 6 The process 600 may be performed by a component of a DCA (e.g., DCA 300). In other embodiments, other entities (e.g., Figure 2 The mapping server 230 and / or Fig. 9 The components shown in the figure can perform some or all steps of the process. Likewise, embodiments can include different and / or additional steps, or perform the steps in a different order.

[0134] Based on information describing at least a portion of the local area, the DCA determines 610 a location of a head mounted device (e.g., head mounted device 200) within the local area (e.g., room 220) in a virtual model. The DCA may transmit information to a mapping server. For example, the DCA may transmit a GPS location, depth information that may be used to determine the location of the head mounted device, or the DCA may transmit an image captured by the head mounted device. In some embodiments, the DCA may initially project a structured light pattern and capture an image to determine the depth information, and the DCA may transmit the depth information to a mapping server to generate parameters for a dynamic structured light pattern. The mapping server may identify the location of the head mounted device in a stored virtual model. The stored virtual model describes a plurality of spaces and properties of those spaces, wherein a location in the virtual model corresponds to a physical location of the head mounted device within the local area, as described with reference to FIG. Figure 5Further described. The information describing at least a portion of the local area may include depth information having information about the shape of at least the portion of the local area defined by the surface of the local area (e.g., the surface of the wall, floor, and ceiling) and one or more (real and / or virtual) objects in the local area. The information describing at least a portion of the local area may also include contrast data for associating areas with relatively high or relatively low contrast with the surface of the local area and the surface of one or more objects. In some embodiments, the information describing at least a portion of the local area may include location information of the local area, such as the address of the local area, the GPS location of the local area, information about the latitude and longitude of the local area, etc. In some other embodiments, the information describing at least a portion of the local area includes: depth information, color image data, acoustic data, location information of the local area, some other information, or a combination thereof. In some embodiments, the DCA may generate an initial structured light pattern based in part on the information describing a portion of the local area.

[0135] The DCA determines 620 an amount of contrast in the local area. The DCA may project structured light into the local area and capture an image of a portion of the local area including the structured light pattern. Based in part on the captured image, the head mounted device determines an amount of contrast. The DCA may determine a contrast value corresponding to a pixel value for each portion of the local area. In some embodiments, the DCA may determine an average contrast value for the entire local area in the field of view of the DCA. In some embodiments, as described with reference to Figure 7 As further described, the DCA uses a contrast algorithm to determine the amount of contrast in the local area. In some embodiments, the head mounted device determines the amount of contrast in the local area based on parameters received from a mapping server.

[0136] The DCA determines 630 a first region of interest in the local area and a second region of interest in the local area. The first region of interest has a first amount of contrast and the second region of interest has a second amount of contrast. In some embodiments, the DCA may select the region of interest based on a contrast algorithm that identifies regions with relatively low or relatively high contrast values. In various embodiments, the head mounted device receives information describing the local area from a mapping server and selects one or more regions of interest in the local area based on the information received from the mapping server. For example, the information from the mapping server identifies regions within the local area that previously included an object. The head mounted device selects a region of interest in the image that corresponds to the region identified as previously including the object by the information from the mapping server. As another example, the information received from the mapping server identifies contrast measurements previously determined for different regions of the local area, and the DCA selects the region of interest as a region having at least a threshold contrast measurement value based on the information received from the mapping server. Thus, in various embodiments, the DCA selects the region of interest as a region of the image having one or more specific characteristics corresponding to the information received from the mapping server.

[0137] The DCA updates 640 the lighting instructions based in part on the amount of contrast in the local area. The DCA may also update the lighting instructions based on the determined location in the virtual model and any parameters associated with the determined location. The updated lighting instructions may cause a change in the dynamic structured light pattern. Changing the dynamic structured light pattern may increase or decrease the texture added to the local area by the dynamic structured light pattern. Increasing the texture of the local area helps the DCA determine the distance between the DCA and the object. In some embodiments, the mapping server retrieves a set of parameters from the virtual model based on the determined location in the virtual model associated with the spatial configuration in which the head-mounted device is currently located. In some other embodiments, the mapping server determines the set of parameters by adjusting a previously determined set of parameters in the virtual model based at least in part on information describing at least a portion of the local area received from the DCA. The mapping server may analyze the image received from the DCA to determine whether an existing set of parameters (if available) is consistent with the image analysis or should be recalculated. If the existing parameters are inconsistent with the image analysis, the mapping server may determine a new set of parameters using information describing at least a portion of the local area (e.g., room geometry, lighting).

[0138] The DCA may update the lighting instructions for the identified regions of interest. For example, the lighting instructions may increase the density or intensity of the dynamic structured light pattern in the first region of interest, and the lighting instructions may decrease the density or intensity of the structured light pattern in the second region of interest. In some embodiments, the updated lighting instructions may instruct the light projector to activate or deactivate the structured light pattern in one or more regions of interest.

[0139] The DCA provides 650 updated instructions to the structured light projector, wherein the updated illumination instructions cause the SL projector to project a first SL pattern into the first region of interest, and to project a second SL pattern into the second region of interest, wherein the second SL pattern is different from the first SL pattern. Thus, the DCA dynamically adjusts the structured light pattern used to obtain depth information in the local region. In some embodiments, process 600 may be repeated continuously, or may be repeated in response to an event, such as in response to movement of the head mounted device or movement of an object in the local region. Thus, the DCA may save battery by reducing the amount of structured light projected into areas where additional texture is not needed. Similarly, the DCA may improve depth measurement accuracy by increasing texture in regions of interest where additional texture is beneficial.

[0140] Figure 7 is a flow chart illustrating a process 700 for determining the amount of contrast in a local area according to one or more embodiments. In some embodiments, process 700 may be used as a reference. Figure 6 Process 700 may be performed by a component of a DCA (eg, DCA 300).

[0141] DCA captures 710 one or more images in a local area. DCA calculates the contrast of all or a portion of the image. Contrast is the difference in brightness, color, distance, or other measurable quantities that make an object (or its representation in an image or display) distinguishable. The contrast in the image is related to the local depth performance in the region of interest. For example, if the image contains a high contrast value in the region of interest, DCA can set a high confidence value for the depth measurement in the region of interest. Therefore, DCA can reduce the amount of texture provided by the dynamic structured light pattern in the region of interest, which can save battery power. Conversely, if the image contains a low contrast value in the region of interest, DCA can increase the amount of texture provided by the dynamic structured light pattern to improve the accuracy of the depth measurement. In some embodiments, the contrast can be normalized to the local noise level in the image. Many algorithms for calculating contrast are known in the art, such as Weber contrast, Michelson contrast, RMS contrast, etc. DCA can define the contrast at each pixel (i, j). In some embodiments, DCA can calculate the contrast using an L2 contrast algorithm with a sliding maximum.

[0142] To perform the L2 contrast algorithm with sliding maximum, DCA can first calculate the 720 pixel level contrast. The pixel level contrast can be calculated using the following formula:

[0143]

[0144] Where Ii,j is the number of sensor pixel (i,j), where i and j are integer values ​​representing the i-th column and j-th row of the pixel in the image sensor.

[0145] Second, DCA may calculate 730 the sliding window maximum of the original contrast. The sliding window maximum of the original contrast may be calculated using the following formula:

[0146] Contrast i,j =max[RawContrast k,l ] (2)

[0147] k∈[iw, i+w], l∈[jw, j+w]

[0148] Where w controls the size of the sliding window (the window size is 2w+1 by 2w+1 pixels). k and l are integer values ​​representing the kth column and lth row of pixels within the sliding window. The L2 contrast algorithm with sliding maximum shows good correlation with depth performance. Therefore, as referenced Figure 6 As described in step 640 of FIG. 1 , the DCA may use the results of the contrast algorithm to identify areas that may require higher or lower lighting levels, and the DCA may use the results to update the lighting instructions.

[0149] Figure 8 8 is a flow chart illustrating a process 800 for identifying a region of interest based on an object location according to one or more embodiments. In some embodiments, the process may be a reference to Figure 6 All or part of step 630 described.

[0150] The DCA may identify 810 one or more objects in the local area. In some embodiments, the DCA may receive the location of the object from a mapping server. However, in some embodiments, the DCA may detect the object based on a local depth confidence metric, which may be estimated on the headset, via a connected device, via the cloud, or locally based on stacked sensor logic.

[0151] The DCA may detect 820 movement of an object. In some embodiments, the DCA may detect movement based on changes in depth information in a portion of a local area. In some embodiments, a head mounted device including a DCA may detect motion using other sensors on the head mounted device, such as visible light or infrared image sensors, image sensors used as inputs to SLAM, color RGB sensors for image capture, texture capture, environment reconstruction, audio sensors, telepresence, IMU sensor data, dynamic vision sensors, etc., which detect objects moving in the scene. The movement of the object may indicate to the DCA that the DCA should increase the amount of texture projected in the region of interest containing the object.

[0152] The DCA may detect 830 occlusion of an object. The DCA may determine that two or more objects are in the same line of sight from the DCA. For example, based on information provided by a mapping server, or depth information determined by the DCA, the DCA may determine that the two or more objects are in the same line of sight. Based on the depth information of the two or more objects, the DCA determines that the object closest to the DCA occludes any object in the same line of sight that is farther from the DCA. The occluded object may be behind another object in the local area relative to the line of sight from the head-mounted device. In some embodiments, a virtual object may be located between the head-mounted device and a physical object, occluding the physical object. In some embodiments, a physical object may be located between the head-mounted device and a virtual object, occluding the virtual object. Similarly, in some embodiments, a physical object may occlude another physical object, or a virtual object may occlude another virtual object.

[0153] The DCA may identify 840 a region of interest based on movement of an object or detection of an occluded object. For example, in response to movement of an object, the DCA may identify the object as being within the region of interest. The DCA may determine to increase the amount of texture in the region of interest that includes the moving object. In some embodiments, in response to detection of occlusion of an object, the DCA may identify the occluded object as being within the region of interest. The DCA may determine to reduce the amount of texture in the region of interest that includes the occluded object. For example, rather than projecting structured light with sufficient texture to obtain a depth measurement at the distance of the occluded object, the DCA may reduce the amount of texture to a level sufficient to obtain a measurement at a smaller distance of the object that occludes the occluded object. As described with reference to Figure 6 As described in step 640 , the identified region of interest may be used to update lighting instructions.

[0154] System environment

[0155] Fig. 9 900 is a system environment 900 of a head mounted device 905 according to one or more embodiments. In some embodiments, the head mounted device 905 may be Figure 1A The head mounted device 100. The system 900 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Fig. 9 The system 900 shown includes a head mounted device 905, a network 935, a mapping server 965, and an input / output (I / O) interface 940 coupled to a console 945. Although Fig. 9 The example system 900 is shown to include one head mounted device 905 and one I / O interface 940, but in other embodiments, any number of these components may be included in the system 900. For example, there may be multiple head mounted devices, each with an associated I / O interface 940, where each head mounted device and I / O interface 940 communicate with a console 945. In alternative configurations, different and / or additional components may be included in the system 900. Furthermore, in some embodiments, in combination with Fig. 9 The functionality described by one or more of the components shown may be combined with Fig. 9 The described methods are distributed between the components in different ways. For example, some or all of the functions of the console 945 can be provided by the head mounted device 905.

[0156] The head mounted device 905 includes a display assembly 910, an optical block 915, one or more position sensors 925, a DCA 930, and an inertial measurement unit (IMU) 920. Some embodiments of the head mounted device 905 have a combination of Fig. 9 The components described are different components. In addition, by combining Fig. 9The functionality provided by the various components described may be distributed differently among the components of the head mount 905 in other embodiments, or may be captured in a separate component remote from the head mount 905.

[0157] Display component 910 may include an electronic display that displays 2D or 3D images to a user based on data received from console 945 . Figure 1A Lens 110 of can be an embodiment of display assembly 910. In various embodiments, display assembly 910 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of a user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), some other display, or some combination thereof. Lens 110 is an embodiment of display assembly 910. Note that in some embodiments, lens 110 can also include some or all of the functionality of optical block 915.

[0158] The optical block 915 amplifies the image light received from the electronic display, corrects the optical errors associated with the image light, and presents the corrected image light to the user of the head mounted device 905. In various embodiments, the optical block 915 includes one or more optical elements. Example optical elements included in the optical block 915 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 915 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 915 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0159] The magnification and focusing of the image light by the optical block 915 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content is presented using almost all of the user's field of view (e.g., approximately 110 degrees diagonally), and in some cases all of the field of view. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0160] In some embodiments, the optical block 915 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and when the optical block 915 receives image light generated based on the content from the electronic display, the optical block 915 corrects the distortion.

[0161] The IMU 920 is an electronic device that generates data indicating the position of the head mounted device 905 based on measurement signals received from one or more position sensors 925. The position sensor 925 generates one or more measurement signals in response to the movement of the head mounted device 905. Examples of the position sensor 925 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 920, or some combination thereof. The position sensor 925 can be located outside the IMU 920, inside the IMU 920, or some combination of these locations.

[0162] The DCA 930 generates depth information for a local area such as a room. The DCA 930 may be Figure 3 An example of a DCA 300 of FIG. 300. The depth information includes pixel values ​​that define the distance from the imaging device and thus provides a (e.g., 3D) map of the location captured in the depth information. The DCA 930 includes a light projector, one or more imaging devices, and a controller. The light projector can project dynamic structured light patterns or other light that reflects off objects in a local area and is captured by the imaging device to generate depth information.

[0163] For example, a light projector can project multiple structured light (SL) elements (e.g., lines, grids, or dots) of different types onto a portion of a local area around the head mounted device 905. In various embodiments, the light projector includes an illumination source and a template. The illumination source may include one or more emitters configured to illuminate the template with light (e.g., infrared light). The illuminated template projects an SL pattern including multiple SL elements into the local area. For example, each SL element projected by the illuminated template is a dot associated with a specific location on the template.

[0164] Each SL element projected by the DCA 930 includes light in the infrared portion of the electromagnetic spectrum. In some embodiments, the illumination source is a laser configured to illuminate the template with infrared light so that it is invisible to a person. In some embodiments, the illumination source can be pulsed. In some embodiments, the illumination source can be visible and pulsed so that the light is invisible to the eye.

[0165] The SL pattern projected into the local area by the DCA 930 deforms when encountering various surfaces and objects in the local area. One or more imaging devices are each configured to capture one or more images of the local area. Each of the captured one or more images may include multiple SL elements (e.g., points) projected by the light projector and reflected by objects in the local area. Each of the one or more imaging devices may be a detector array, a camera, or a video camera.

[0166] The DCA 930 generates depth information based on the light captured by the DCA 930. The controller 930 may further provide the depth information to a console 945, an audio controller, or some other component.

[0167] The DCA may include a PCA that includes one or more passive imaging devices that generate color (e.g., RGB) image data. The PCA captures light from the environment of a local area to generate image data. Rather than defining pixel values ​​of depth or distance from the imaging device, the pixel values ​​of the image data may define the visible color of the object captured in the imaging data. In some embodiments, the PCA includes a controller that generates color image data based on the light captured by the passive imaging device. In some embodiments, the DCA 930 and the PCA share a common controller. For example, the common controller may map each of one or more images captured in the visible spectrum (e.g., image data) and the infrared spectrum (e.g., depth information) to each other. In one or more embodiments, the common controller is configured to provide one or more images of the local area to the audio controller or console 945 in addition or alternatively.

[0168] The I / O interface 940 is a device that allows a user to send an action request and receive a response from the console 945. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end capturing image or video data, or an instruction to perform a specific action within an application. The I / O interface 940 may include one or more input devices. Example input devices include a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 945. The action request received by the I / O interface 940 is transmitted to the console 945, and the console 945 performs the action corresponding to the action request. In some embodiments, as further described above, the I / O interface 940 includes an IMU 920, which captures calibration data indicating the estimated position of the I / O interface 940 relative to the initial position of the I / O interface 940. In some embodiments, the I / O interface 940 can provide tactile feedback to the user according to the instruction received from the console 945. For example, tactile feedback is provided when an action request is received, or when console 945 transmits instructions to I / O interface 940 that cause I / O interface 940 to generate tactile feedback when console 945 performs the action.

[0169] The console 945 provides content to the headset 905 for processing based on information received from one or more of the DCA 930, the headset 905, and the I / O interface 940. Fig. 9 In the example shown, the console 945 includes an application store 950, a tracking module 955, and an engine 960. Some embodiments of the console 945 have a Fig. 9 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Fig. 9 The described approach is distributed among the components of the console 945. In some embodiments, the functions discussed herein with reference to the console 945 can be implemented in the headset 905 or in a remote system.

[0170] The application storage 950 stores one or more applications for execution by the console 945. An application is a set of instructions that, when executed by the processor, generates content for presentation to the user. The content generated by the application may be in response to input received from the user via the movement or I / O interface 940 of the head mounted device 905. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0171] The tracking module 955 uses one or more calibration parameters to calibrate a local area of ​​the system 900, and may adjust one or more calibration parameters to reduce errors in the determination of the position of the head mounted device 905 or the I / O interface 940. For example, the tracking module 955 transmits the calibration parameters to the DCA 930 to adjust the focus of the DCA 930 to more accurately determine the position of the SL elements captured by the DCA 930. The calibration performed by the tracking module 955 also takes into account information received from the IMU 920 in the head mounted device 905 and / or the IMU 920 included in the I / O interface 940. In addition, if tracking of the head mounted device 905 is lost (e.g., the DCA 930 loses sight of at least a threshold number of projected SL elements), the tracking module 955 can recalibrate part or all of the system 900.

[0172] The tracking module 955 tracks the movement of the head mounted device 905 or the I / O interface 940 using information from the DCA 930, one or more position sensors 925, the IMU 920, or some combination thereof. For example, the tracking module 955 determines the location of a reference point of the head mounted device 905 in a map of the local area based on the information from the head mounted device 905. The tracking module 955 may also determine the location of an object or a virtual object. Additionally, in some embodiments, the tracking module 955 may use the portion of the data from the IMU 920 indicating the location of the head mounted device 905 and the representation of the local area from the DCA 930 to predict the future location of the head mounted device 905. The tracking module 955 provides the estimated or predicted future location of the head mounted device 905 or the I / O interface 940 to the engine 960.

[0173] The engine 960 executes the application and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the head mounted device 905 from the tracking module 955. Based on the received information, the engine 960 determines the content to be provided to the head mounted device 905 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 960 generates content for the head mounted device 905 that reflects the user's movement in a virtual local area or in a local area that enhances the local area with additional content. In addition, the engine 960 performs an action within an application executed on the console 945 in response to an action request received from the I / O interface 940, and provides feedback to the user that the action is performed. The feedback provided may be visual or auditory feedback via the head mounted device 905, or tactile feedback via the I / O interface 940.

[0174] Additional configuration information

[0175] Embodiments according to the invention are specifically disclosed in the attached claims relating to methods, devices and storage media, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., device, storage medium, system and computer program product). The dependencies or back-references in the attached claims are selected for formal reasons only. However, any subject matter resulting from an intentional back-reference (especially multiple references) to any previous claim may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of other features in the claims. In addition, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature of the attached claims.

[0176] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to (or within) any of the above-mentioned embodiments.

[0177] In an embodiment, a system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, the processor being operable to perform a method according to any of the above-mentioned embodiments (or within these embodiments) when executing the instructions.

[0178] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, is operable to perform a method according to any of the above mentioned embodiments (or within these embodiments) when executed on a data processing system.

[0179] The foregoing description of the embodiments of the present disclosure is presented for illustrative purposes; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art can recognize that many modifications and variations are possible in light of the above disclosure.

[0180] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0181] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0182] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that uses a multi-processor design to increase computing power.

[0183] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information produced by the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0184] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the inventive subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the disclosure, which is set forth in the appended claims.

Claims

1. A depth camera assembly, comprising: a structured light SL projector configured to project one or more SL patterns into the local area according to the illumination instructions, each SL pattern having a corresponding pattern element density; a camera assembly configured to capture an image of a portion of the local area including the one or more SL patterns; and A controller configured to: providing a first lighting instruction to the structured light SL projector, wherein the first lighting instruction causes the structured light SL projector to project a first SL pattern having a first pattern element density into the first area of ​​interest based on an amount of a first contrast of brightness or color in the first area of ​​interest, and Providing a second lighting instruction to the structured light SL projector, wherein the second lighting instruction causes the structured light SL projector to project a second SL pattern having a second pattern element density into the second area of ​​interest based on an amount of second contrast in brightness or color in the second area of ​​interest, wherein the amount of the second contrast is greater than the amount of the first contrast and the second pattern element density is different from the first pattern element density.

2. The depth camera assembly of claim 1, wherein: The controller is also configured to: transmitting a description of the location of the depth camera assembly to a mapping server, wherein the mapping server stores a virtual model describing a plurality of physical spaces; receiving from the mapping server an amount of the first contrast of the first region of interest; and The second amount of contrast for the second region of interest is received from the mapping server.

3. The depth camera assembly of claim 2, wherein: The mapping server is configured to determine an amount of the first contrast based on a virtual model of the local area.

4. The depth camera assembly of claim 2, wherein: The controller is configured to identify the first region of interest and the second region of interest based on parameters received from the mapping server.

5. The depth camera assembly of claim 1, wherein: The controller is configured to identify the first region of interest in the local area based on detecting movement of an object within the first region of interest.

6. The depth camera assembly of claim 1, wherein: The controller is configured to: identifying the first region of interest in the local area based on detecting the occluded object in the first region of interest; and In response to detecting the occluded object, reducing an amount of texture in the first region of interest, wherein the occluded object is a physical object, and wherein the physical object is occluded by the virtual object.

7. The depth camera assembly of claim 1, wherein: The controller is configured to: determining an expected position of a virtual object within the local area; and A power level of the first SL pattern is selected based on the expected position of the virtual object.

8. The depth camera assembly of claim 1, wherein: The intensity of the second SL pattern is different from the intensity of the first SL pattern.

9. The depth camera assembly of claim 1, wherein: The controller is configured to transmit visual information describing the portion of the local area to a mapping server.

10. The depth camera assembly of claim 1, wherein: The controller is configured to: subtracting a first image captured without any SL pattern from a second image containing one of the SL patterns; and Based on the subtraction, the background level of IR illumination is determined.

11. A method comprising: providing a first lighting instruction to a structured light SL projector, wherein the first lighting instruction causes the structured light SL projector to project a first SL pattern having a first pattern element density into the first region of interest based on an amount of a first contrast of brightness or color in the first region of interest, and Providing a second lighting instruction to the structured light SL projector, wherein the second lighting instruction causes the structured light SL projector to project a second SL pattern having a second pattern element density into the second area of ​​interest based on an amount of second contrast in brightness or color in the second area of ​​interest, wherein the amount of the second contrast is greater than the amount of the first contrast and the second pattern element density is different from the first pattern element density.

12. The method according to claim 11, further comprising: transmitting to a mapping server a location describing a local area, the location including the first region of interest and the second region of interest, wherein the mapping server stores a virtual model describing a plurality of physical spaces; receiving from the mapping server an amount of the first contrast of the first region of interest; and The second amount of contrast for the second region of interest is received from the mapping server.

13. The method according to claim 12, wherein: The mapping server is configured to determine an amount of the first contrast based on a virtual model of the local area.

14. The method according to claim 12, further comprising: The first area of ​​interest and the second area of ​​interest are identified based on parameters received from the mapping server.

15. The method according to claim 12, further comprising: The first region of interest in the local area is identified based on detecting movement of an object within the first region of interest.

16. The method according to claim 12, further comprising: The first region of interest in the local area is identified based on detecting the occluded object in the first region of interest.

17. The method according to claim 16, further comprising: In response to detecting the occluded object, an amount of texture in the first region of interest is reduced.

18. The method according to claim 17, wherein: The occluded object is a physical object, and wherein the physical object is occluded by the virtual object.

19. The method according to claim 11, further comprising: Visual information describing portions of the local area is transmitted to a mapping server.

20. A non-transitory computer-readable storage medium having encoded thereon instructions that, when executed by a processor, cause the processor to perform operations comprising: providing a first lighting instruction to a structured light SL projector, wherein the first lighting instruction causes the structured light SL projector to project a first SL pattern having a first pattern element density into the first region of interest based on an amount of a first contrast of brightness or color in the first region of interest, and providing a second illumination instruction to the structured light SL projector, wherein the second illumination instruction causes the structured light SL projector to project a second SL pattern having a second pattern element density into the second region of interest based on an amount of second contrast of brightness or color in the second region of interest, wherein, An amount of the second contrast is greater than an amount of the first contrast and a second pattern element density is different from the first pattern element density.