Decoding images for active depth sensing to combat optical distortion

Abstract

Aspects of the disclosure relate to decoding images for active depth sensing. An example method includes receiving an image. The image includes one or more reflections of a light distribution. The method also includes sampling a first region of the image using a first sampling grid to generate first image samples, sampling a second region of the image using a second sampling grid different from the first sampling grid to generate second image samples, determining a first depth value based on the first image samples, and determining a second depth value based on the second image samples.

Description

Technical Field

[0001] This disclosure generally relates to active depth sensing systems and devices, such as decoding images for active depth sensing to address the effects of optical distortion. Background Technology

[0002] Many devices include active depth sensing systems. For example, a smartphone may include a front-facing active depth sensor transmitter for projecting light (such as for face unlock or other applications that use depth information) and an image sensor for capturing the reflections of the light projected by the transmitter. The transmitter can project a predefined light distribution, and the depth of objects in the scene can be determined based on the reflections of the light distribution captured by the image sensor. This active depth sensing technology can be called structured light depth sensing. Summary of the Invention

[0003] The present invention is provided to present a selection of concepts in a simplified form, which will be further described in the following detailed description. The present invention is not intended to represent key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] An example device for active depth sensing includes a memory and one or more processors. The one or more processors are configured to receive an image. The image includes one or more reflections of a light distribution. The one or more processors are further configured to: sample a first region of the image using a first sampling grid; sample the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and select the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value.

[0005] An example method for active depth sensing is provided. The method includes: receiving an image including one or more reflections of a light distribution; sampling a first region of the image using a first sampling grid; sampling the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; determining a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and selecting the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value.

[0006] An example of non-transitory computer-readable medium storage instructions, when executed by one or more processors of a device, cause the device to: receive an image including one or more reflections of a light distribution; sample a first region of the image using a first sampling grid; sample the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and select the first sampling grid to determine a first depth value of the first region based on the first confidence value being greater than the second confidence value.

[0007] Another example device for active depth sensing includes: components for receiving an image including one or more reflections of a light distribution; components for sampling a first region of the image using a first sampling grid; components for sampling the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; components for determining a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and components for selecting the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value.

[0008] In some respects, this light distribution is a spot distribution.

[0009] In some aspects, the methods, apparatus, and computer-readable media described above further include: determining a first image sample based on sampling a first region of the image using the first sampling grid; and determining a first depth value of the first region based on the first image sample.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further include: identifying a first codeword in an array of light distributions in a first region based on the first image sample; and determining a first parallax based on the position of the first codeword in the array, wherein the determination of the first depth value is based on the first parallax.

[0011] In some aspects, the methods, apparatus, and computer-readable media described above also include: sampling a second region of the image using a third sampling grid to generate a second image sample; and determining a second depth value based on the second image sample.

[0012] In some respects, the arrangement of sampling points in the second sampling grid differs from that in the first sampling grid.

[0013] In some aspects, the arrangement of sampling points of the first sampling grid includes a first spacing between sampling points of the first sampling grid, and the arrangement of sampling points of the second sampling grid includes a second spacing between sampling points of the second sampling grid.

[0014] In some respects, the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis, which is associated with the transmitter that transmits the light distribution and the receiver that captures the image.

[0015] In some respects, the total number of sampling points in the second sampling grid differs from the total number of sampling points in the first sampling grid.

[0016] In some respects, the first sampling grid is an isotropic sampling grid, while the second sampling grid is an anisotropic sampling grid.

[0017] In some aspects, the above-described methods, apparatus, and computer-readable media further include: determining a first image sample based on sampling a first region of the image using a first sampling grid; determining a second image sample based on sampling the first region of the image using a second sampling grid; comparing the first image sample with the second image sample; and selecting a first image sample to be used to determine the first depth value based on comparing the first image sample with the second image sample.

[0018] In some examples, to determine a first confidence value associated with the first sampling grid, the methods, apparatus, and computer-readable medium described above may include determining a first confidence value for the first image sample. In some examples, to determine a second confidence value associated with the second sampling grid, the methods, apparatus, and computer-readable medium described above may include determining a second confidence value for the second image sample. In some examples, the methods, apparatus, and computer-readable medium described above may include selecting the first sampling grid for determining a first depth value for the first region, wherein the one or more processors are configured to select the first image sample based on the first confidence value being greater than the second confidence value.

[0019] In some aspects, the device includes a receiver configured to capture the image.

[0020] In some aspects, the device includes a transmitter configured to transmit the light distribution, wherein the transmitter and the receiver are separated by a baseline distance along a baseline axis.

[0021] In some aspects, the device includes one or more signal processors configured to process the image and then decode the processed image.

[0022] In some aspects, the methods, apparatus, and computer-readable media described above also include generating a depth map based on the image, wherein the depth map includes a plurality of depth values, the plurality of depth values ​​including the first depth value, and wherein the plurality of depth values ​​indicate one or more depths of one or more objects in a scene captured in the image.

[0023] In some aspects, the device is, is part of, and / or includes the following: mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), cameras, personal computers, laptop computers, server computers, vehicles or computing devices or components of vehicles, robotic devices or systems, televisions, or other devices. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0024] The content of this invention is neither intended to identify key or essential features of the claimed subject matter nor to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the complete specification of this patent, any or all of the accompanying drawings, and each claim.

[0025] The foregoing, along with other features and embodiments, will become more apparent from the following description, claims, and drawings. Attached Figure Description

[0026] Aspects of this disclosure are shown by way of example rather than limitation in the accompanying drawings, and the same reference numerals in the drawings refer to similar elements.

[0027] Figure 1 The illustration shows a depiction of an example active depth sensing system using a predetermined light distribution, based on some examples.

[0028] Figure 2 A depiction of an exemplary distribution for active depth sensing is shown, based on some examples.

[0029] Figure 3 The illustration shows a depiction of an exemplary distribution including pincushion distortion, based on some examples.

[0030] Figure 4A block diagram of an example device for active depth sensing is shown, based on some examples.

[0031] Figure 5 A block diagram of an example decoding process for active depth sensing is shown, based on some examples.

[0032] Figure 6 A depiction of an example sampling grid is shown based on some examples.

[0033] Figure 7 The illustration depicts an exemplary distribution of light spots in a corrected image, based on several examples.

[0034] Figure 8 An example depiction of locations identified in the projected distribution of an image during active depth sensing is shown, based on some examples.

[0035] Figure 9 An illustrative flowchart illustrating an exemplary process for decoding an image for active depth sensing, based on some examples, is shown.

[0036] Figure 10 An example depiction of a first sampling grid and a second sampling grid with different spacing between adjacent sampling points is shown, based on some examples.

[0037] Figure 11 An example depiction of a first and second sampling grid with different skews is shown, based on some examples.

[0038] Figure 12 A block diagram of an example decoding process using different sampling grids is shown, based on some examples.

[0039] Figure 13 An exemplary graph showing the relationship between the theoretical spacing between depicted sampling points and the parallax measurement used to accurately sample image regions, based on some examples, is shown.

[0040] Figure 14 Examples of depictions of square and hexagonal light spot arrays are shown, based on some examples.

[0041] Figure 15 An example depiction of the displacement of light spots in the distribution caused by distortion is shown, based on some examples.

[0042] Figure 16 An illustrative flowchart illustrating an exemplary process for decoding an image for active depth sensing, based on some examples, is shown. Detailed Implementation

[0043] Various aspects of this disclosure can be used in active depth sensing systems and devices. For structured light depth sensing, one or more components of the transmitter may cause optical distortion in the distribution of light emitted by the transmitter. This optical distortion may affect the location on the image sensor where the reflection of the light distribution is received. For example, a reflection of a portion of the light distribution might be expected to be received at a first portion of the image sensor, but this reflection might be received at a second portion of the image sensor due to optical distortion (thus shifting the reflection on the image sensor from the first location to the second location). The light distribution may also be distorted due to optical distortion (such as a light distribution including pincushion distortion). Due to optical distortion, one or more depth values ​​may be undetermined or incorrectly determined during active depth sensing. Some aspects of this disclosure include decoding to reduce the impact of optical distortion on the determination of depth values ​​for active depth sensing.

[0044] In the following description, numerous specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupling" means a direct connection to or a connection via one or more intermediate components or circuits. Additionally, specific terms are set forth in the following description and for illustrative purposes to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that practicing the teachings disclosed herein may not require these specific details. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of this disclosure. Some portions of the following detailed description are presented in the form of programs, logic blocks, processes, and other symbolic representations of operations on data bits within computer memory. In this disclosure, programs, logic blocks, processes, etc., are considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not essential, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system.

[0045] However, it should be remembered that all these or similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise explicitly stated in the discussion below, it should be understood that throughout this application, discussions using terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “standardize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” and “respond” refer to the actions and processes of a computer system or similar electronic computing device used to manipulate data representing physical (electronic) quantities in the registers and memories of the computer system and transform them into other data representing physical quantities similarly represented in the computer system’s memory or registers or other such information storage, transmission, or display devices. In some embodiments, as used herein, “determine,” “generate,” or other similar terms may be used interchangeably.

[0046] In the accompanying drawings, a single block may be described as performing one or more functions; however, in practice, one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure. Moreover, the example device may include components other than those shown, including well-known components such as processors and memory.

[0047] The aspects of this disclosure apply to any suitable electronic device for decoding information from an image to perform active depth sensing. The device may include any number of image sensors (including zero image sensors for enabling the device to receive image frames from another device or component) configured to capture images or any number of transmitters (including zero transmitters for devices separate from the transmitting device or component used for active depth sensing) configured for active depth sensing. Example devices include security systems, smartphones, tablet computers, laptop computers, digital cameras, driverless or autonomous vehicles, and the like. While many examples described herein depict devices including transmitters and image sensors, the device may have one, two, or no components, or multiple instances of any of these components. Therefore, this disclosure is not limited to devices having a specific number of image sensors, active depth sensing transmitters, components, component orientations, etc.

[0048] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. Similarly, the term "system" is not limited to one or a specific number of physical objects (such as one or more devices, one or more smartphones, one or more camera controllers, one or more processing systems, etc.). As used herein, a system can be any number of devices or part of a device that can implement at least some parts of this disclosure. Although the following description and examples may use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a particular configuration, type, or number of objects. Thus, "device" and "system" can be used interchangeably to refer to similar aspects of this disclosure.

[0049] One type of active depth sensing system involves emitting a predefined (known) light distribution toward objects in a scene and capturing reflections of that light distribution in an image. The image is analyzed to identify the reflections of the light distribution, and the identified reflections are used to determine the depth of one or more objects in the scene. Depth values ​​can be determined based on the location of a portion of the reflections in the image, and the depth values ​​can represent or indicate depth (such as a number corresponding to a distance in meters, feet, or other suitable units of measurement, a variable used to identify distance, etc.).

[0050] Figure 1 A depiction of an example active depth sensing system 100 using a predetermined (known) light distribution 104 is shown. The active depth sensing system 100 (also referred to herein as a structured light system or structured light depth sensing system) can be used to determine one or more depths of objects in scene 106. The depth of the objects can then be used for any suitable application. For example, scene 106 may include a face, and the active depth sensing system 100 can be used to identify or authenticate the face for screen unlocking or security purposes.

[0051] The active depth sensing system 100 may include a transmitter 102 and a receiver 108. The transmitter 102 may be referred to as a "transmitter," "projector," etc., and is not limited to a specific transmitting component. Throughout the following disclosure, the terms projector and transmitter may be used interchangeably. The receiver 108 may be referred to as a "detector," "sensor," "image sensor," "sensing element," "photodetector," etc., and is not limited to a specific receiving component.

[0052] Although this disclosure refers to the distribution as a light distribution, any suitable wireless signal of other frequencies (such as radio frequency waves, sound waves, etc.) can be used. Furthermore, although this disclosure refers to the distribution as comprising multiple light spots, the light can be focused into any suitable size and dimension. For example, the light can be projected in the form of a line, a square, or any other suitable size.

[0053] Distribution 104 may be a codeword distribution, where the defining portion of the distribution (such as a predefined image patch of light points) is referred to as a codeword. If the light point distribution is known, the codewords of the distribution can be known. In some embodiments, the memory may include a codeword library for the codewords included in distribution 104 emitted by transmitter 102. The codeword library can then be used to identify codewords in the reflection of light emitted by transmitter 102 as received by receiver 108, and the position of the codeword on the receiver's sensor (indicated by the position of the codeword in the image captured by the receiver's sensor) can be used to determine one or more depths in the scene. For example, image sensor 132 may be configured to capture images including reflections of the codeword distribution emitted by the associated transmitter 102. A codeword library corresponding to the codeword distribution of light emitted by transmitter 102 can be used to identify codewords in the reflection of the codeword distribution in the image from image sensor 132, and the position is used to determine the depth of one or more objects in scene 106. The distribution of the transmitted wireless signals can be organized and used in any way, and this disclosure should not be limited to a particular type of distribution or a particular type of wireless signal.

[0054] As shown, emitter 102 can be configured to project a light spot distribution 104 onto scene 106. Black circles in distribution 104 can indicate where light is not projected for possible point locations, while white circles in distribution 104 can indicate where light is projected for possible point locations. In some exemplary embodiments, emitter 102 may include one or more light sources 124 (such as one or more lasers), lens 126, and light modulator 128. Light source 124 may include any suitable light source. In some exemplary embodiments, light source 124 may include one or more distributed feedback (DFB) lasers. In some other exemplary embodiments, light source 124 may include one or more vertical cavity surface-emitting lasers (VCSELs). In some examples, one or more light sources 124 include a VCSEL array, a DFB laser array, or another suitable laser array of multiple lasers. In some other examples, one or more light sources 124 include any suitable array of suitable light sources or wave sources, such as a light-emitting diode (LED) array, an ultrasonic transducer array, or an antenna array (such as for transmitting radio frequency or other suitable wave frequencies). Although this example may describe light source 124 as including an array of lasers in order to clearly explain various aspects of this disclosure, this disclosure is not limited to a particular configuration or type of light source or wave source.

[0055] The laser of light source 124 can be configured to emit infrared (IR) light. As used herein, IR light can include portions of the visible spectrum and / or portions of the spectrum invisible to the naked eye. In one example, IR light can include near-infrared (NIR) light, which may or may not include light within the visible spectrum, and / or IR light outside the visible spectrum (such as far-infrared (FIR) light). The term IR light should not be limited to light having a specific wavelength in or near a wavelength range. Furthermore, infrared light is provided as an exemplary emission for active depth sensing. In the following description, light of other suitable wavelengths can be emitted by light source 124 (or captured by image sensor 132 or otherwise used for active depth sensing). Thus, active depth sensing is not limited to the use of IR light or IR light of a specific frequency.

[0056] Emitter 102 includes an aperture 122 from which emitted light exits emitter 102 and reaches scene 106. In some embodiments, emitter 102 includes one or more diffractive optical elements (DOEs) to diffract the emission from light source 124 into additional emissions. In some aspects, light modulator 128 (which can adjust the emission intensity) may include one or more DOEs. The DOE includes material located in the projection path of light spots from one or more lasers from light source 124, and the material may be configured to split the light spots into additional light spots. For example, the material of the DOE may be a translucent or transparent polymer with a known refractive index. The surface of the DOE may include peaks and valleys (changing the depth of the DOE) such that when light passes through the DOE, one light spot is split into multiple light spots. The DOE can receive one or more light spots from one or more lasers and project a greater number of light spots to cover a larger area of ​​scene 106 than an area of ​​scene covered only by one or more light spots from one or more lasers. When projecting the light spot distribution 104 onto the scene 106, the emitter 102 can project one or more light spots from the light source 124 onto the scene 106 through the lens 126 and via a DOE. In this way, the distribution 104 can include repetitions of the same light spot distribution at different portions of the distribution 104. For example, the distribution 104 can include a pattern of m rows multiplied by n columns of light emitted by the light source 124 (for integers m and n greater than or equal to one).

[0057] As mentioned above, the light projected by transmitter 102 can be IR light. IR light is provided as an example emission from transmitter 102. In the description below, other suitable light wavelengths can be used. For example, transmitter 102 can output light in portions of the visible light spectrum outside the IR light wavelength range or ultraviolet light. Alternatively, other signals with different wavelengths, such as microwaves, radio frequency signals, and other suitable signals, can be used.

[0058] Scene 106 may include objects at different depths from the structured light system (such as from transmitter 102 and receiver 108). For example, objects 106A and 106B in scene 106 are at different depths. Receiver 108 may be configured to receive reflections 110 of the transmitted light spot distribution 104 from scene 106. To receive reflections 110, image sensor 132 of receiver 108 may capture an image. When capturing an image, receiver 108 receives reflections 110, as well as (i) other reflections from other parts of the light spot distribution 104 at different depths, and (ii) ambient light. Active depth sensing system 100 may be configured to filter or reduce ambient light interference to isolate reflections of distribution 104 in the captured image (e.g., by using a bandpass filter or other suitable components to allow reflections to be received at image sensor 132 of receiver 108).

[0059] As shown, transmitter 102 and receiver 108 can be located on the same reference plane, and transmitter 102 and receiver 108 can be separated by a distance referred to as baseline (112). In some other embodiments, transmitter 102 and receiver 108 can be located on different reference planes. For example, transmitter 102 can be located on a first reference plane, while receiver 108 can be located on a second reference plane. The first and second reference planes can be the same reference plane, can be parallel reference planes separated from each other, or can be reference planes intersecting at a non-zero angle. The angle and position of intersection on the reference planes are based on the position and orientation of the reference planes relative to each other. The reference planes can be oriented to be associated with a common side of the device. For example, both reference planes (whether parallel or intersecting) can be oriented to receive light from a common side of the device including the active depth sensing system 100 (such as the front side of a smartphone including a display, the top side of a smartphone, etc.).

[0060] In equipment manufacturing, minute differences or errors in the manufacturing process can cause variations in the orientation or positioning of the first or second reference plane. In one example, mounting transmitter 102 or receiver 108 on a printed circuit board (PCB) may involve an error (within tolerance) where the orientation of transmitter 102 or receiver 108 differs from the orientation of the PCB. In another example, the orientation of different PCBs including transmitter 102 and receiver 108 may differ slightly from the design (such as slight variations in orientation when the PCBs are designed to be along the same reference plane or parallel to each other). The first and second reference planes may be referred to as identical reference planes, parallel reference planes, or intersecting reference planes as intended by the equipment design, regardless of variations in the orientation of the reference planes due to manufacturing, calibration, etc., during equipment production.

[0061] Receiver 108 includes an aperture 120 to receive light (including reflections 110) from scene 106. In some exemplary embodiments, receiver 108 may include a lens 130 to focus or direct the received light (including reflections 110 from objects 106A and 106B) onto image sensor 132 of receiver 108. Assuming an example that only reflections 110 are received, the depths of objects 106A and 106B can be determined based on baseline 112, a shift in the light distribution 104 (such as in codewords) in reflections 110, and the intensity of reflections 110. For example, a difference 134 between position 116 and center 114 of image sensor 132 is used to determine the depth of object 106B in scene 106. Similarly, a difference 136 between position 118 and center 114 of image sensor 132 is used to determine the depth of object 106A in scene 106. Differences 134 or 136 can be measured based on the number of pixels in sensor 132 (such as the number of pixels in a captured image) or based on distance (such as in millimeters).

[0062] In some exemplary embodiments, image sensor 132 may include an array of photodiodes (such as avalanche photodiodes) for capturing images. To capture an image, each photodiode in the array can capture light illuminating a photosensitive surface associated with the photodiode and can provide a value indicating the light intensity (capture value). Therefore, the image can represent the capture value provided by the photodiode array.

[0063] As a complement or alternative to the image sensor 132, which includes an array of photodiodes, the sensor 132 may include a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. To capture an image using a photosensitive sensor such as a CMOS or CCD sensor, each pixel of the sensor can capture the light illuminating the pixel and can provide a value indicating the light intensity. In some exemplary embodiments, the photodiode array may be coupled to the sensor. In this way, electrical pulses generated by the photodiode array can trigger the corresponding pixel of the sensor to provide a captured value (or a value converted into a captured value by an analog front end coupled to the image sensor 132). While this example may describe the sensor as a CMOS sensor for the sake of clarity in explaining various aspects of this disclosure, this disclosure is not limited to a particular sensor type or configuration of components.

[0064] As the object moves closer to receiver 108, the difference on the image sensor 132 associated with the object increases. As shown, difference 134 (corresponding to reflection 110 from object 106B) is smaller than difference 136 (corresponding to reflection 110 from object 106A). Therefore, object 106A is closer to receiver 108 than object 106B. The difference is... Figure 1The image sensor 132 is shown along the line representing it. However, the image sensor 132 receives light along a two-dimensional planar segment (such as a rectangle). Therefore, differences can be visualized in a two-dimensional manner. The component of the difference along the same axis as the baseline 112 can be called parallax. The component of the difference at a 90-degree angle to the axis of the baseline 112 (called orthogonal to the baseline) can be called orthogonal difference. In an ideal sensor that is perfectly aligned with the transmitter and calibrated according to the transmitter so that there is no angular difference between the transmitter and the sensor, the orthogonal difference is zero for objects located at different depths from the sensor (while the parallax varies based on depth). In this way, the parallax component (which is associated with the baseline 112) is used to determine the depth of the object from the receiver 108.

[0065] The disparity component is determined by: identifying codewords in reflections from the image from image sensor 132; determining the position of the identified codewords in the image; determining the position of the identified codewords in the distribution 104 projected by transmitter 102; determining the corresponding position in the diffraction array (e.g., a copy of distribution 104); and determining or measuring the distance (e.g., in pixels or subpixels) between the diffraction array region and the image region along the baseline 112 axis. The disparity component represents the difference between a position in the image and a position in the emitted distribution 104 (or diffraction array). Returning to reference objects 106A and 106B, triangulation based on baseline 112 and the disparity components of differences 134 and 136 can determine the different depths (such as depth values) of objects 106A and 106B in scene 106.

[0066] As mentioned above, one or more DOEs can be used to replicate a distribution (such as a spot distribution from a laser array) to generate a larger distribution (such as a spot distribution projected by emitter 102 that is larger than the spot distribution originally emitted by the laser array). In this way, a smaller light source (such as a smaller VCSEL array) can be used to cover a similarly sized portion of the scene for active depth sensing. However, since the original distribution can be replicated using one or more DOEs, the projected distribution is not unique overall. The unique portion (such as the size of the VCSEL array) can indicate the maximum parallax that can be determined in the image and thus the minimum depth that can be determined using an active depth sensing system. As the unique portion of the distribution (referred to as the original distribution) is repeated, receiver 108 receives reflections of multiple instances of the original distribution (which was replicated by one or more DOEs before being emitted onto scene 106). The following examples use a spot distribution (with a rectangular distribution) emitted by a VCSEL array to describe aspects of this disclosure. However, any suitable type of distribution, emitted light, and light source can be used.

[0067] Figure 2A depiction of an exemplary spot distribution 200 projected onto a scene for active depth sensing is shown. Dashed line 202 indicates the boundary of the projected distribution 200. The projected spot distribution 200 comprises a repetition of an original distribution with M rows and N columns. While the projected distribution 200 is shown as M=5 and N=5 (counting from -2 to 2 for both M and N), the number of repetitions (such as the number of rows and columns) can be any suitable number. Additionally, M and N can be different from each other or the same. In some implementations, the original distribution may be projected at the center of the projected distribution, and copies may be projected at other parts of the distribution. For example, in the projected distribution 200, the original distribution might be located at position 0x0 (the m-th row in M ​​= 0, and the n-th column in N = 0). In this way, the original distribution is located at the center of the projected distribution 200. The location of the repeating distribution or the original distribution in the projected distribution 200 can be referred to as (m,n). In the example above, the original distribution is located at (0,0). A copy of the original distribution may be located at other positions in the projected distribution (such as at (m,n), where at least one of m or n is not equal to 0). For example, a copy of the original distribution at (0,0) may be located at (2,-1), (1,0), and other positions not at (0,0). The original distribution may be referred to as a primitive array (or a 0th-order array or a non-0th-order array), while the copied distribution may be referred to as a diffraction array (or the diffraction order of the primitive array, a non-0th-order array, or a non-0th-order array). In some implementations, the projected distribution is 17 x 7 (M = 17 and N = 7), where the primitive array is located at (0,0), and the diffraction array is located at all other positions (where m of M ranges from -8 to 8 and n of N ranges from -3 to 3).

[0068] Because the projection distribution is not unique as a whole, the reflection of an object in a portion of the distribution captured in an image may be associated with different arrays based on the object's location. For example, the center of the distribution received at the image sensor may be associated with an array of primitives, and different portions of the distribution received at the image sensor may be associated with diffraction arrays. The parallax associated with an image region including identified codewords is based on the codeword's position in the array (such as the difference between the codeword's position in the array and the image center along the baseline). Since the distribution comprises multiple array instances, objects at different locations in the scene may be illuminated by different arrays of light points from the distribution. In this way, parallax can shift from maximum parallax to minimum parallax (such as from 192 image pixels to 0) and vice versa, based on the changing position of objects in the scene.

[0069] Each array in a specific example can be referred to as a "tile". In this way, distribution 200 is 5 tiles x 5 tiles. Each array or tile of distribution 200 can be associated with a portion of an image including the reflection of distribution 200. For example, image sensor 132 ( Figure 1 The image sensor pixel at the top left corner of the distribution 200 can capture reflections from the array (2, -2). The device may include a mapping of image locations from the image sensor 132 to a specific array in the distribution. In this way, the center of the array in the image and the position of the codeword in the array can be determined based on the mapping. In some implementations, the mapping indicates the position in each image corresponding to the center of each array.

[0070] This mapping (or a specific array in the computational distribution) is based on a projected distribution that does not contain any distortion. However, replicating the array of primitives can cause optical distortion in the projected distribution. For example, one or more DOEs in a transmitter can cause the projected distribution to include pincushion distortion. While pincushion distortion is shown in the examples, any other type of distortion (such as distortion caused by objects having different depths along the surface of an object) can be included in the projected distribution. Therefore, while some examples may illustrate the effect of reducing pincushion distortion, the effects of other types of distortion can be reduced based on various aspects of this disclosure.

[0071] Figure 3 A depiction of an exemplary distribution 300 including pincushion distortion is shown. Distribution 300 is 17 tiles x 7 tiles. A primitive array 302 (also referred to as a 0th-order array) is located at the center of distribution 300 (position (0,0)). A diffraction array 304 surrounds the primitive array 302. One or more DOEs used to replicate the primitive array 302 for the diffraction array 304 may cause pincushion distortion in the projected distribution 300. As shown, the diffraction array 304 can become more stretched and skewed as it approaches a corner of distribution 300 from its center.

[0072] Sensor boundary line 306 indicates the boundary of projection distribution 300, against which the image sensor receives reflections from distribution 300. If distribution 300 is not distorted, all diffraction arrays 304 will be located inside sensor boundary line 306. In this way, reflections from each diffraction array 304 can be received by the image sensor. Furthermore, each diffraction array 304 and primitive array 302 can be associated with a position on the image sensor (and therefore with a position in the image captured by the image sensor).

[0073] Because stretching, skewness, or other deformations of the array can cause changes in the position of the light spots, devices performing conventional decoding on images for active depth sensing may fail to recognize codewords in the image. Specifically, the device includes a mapping of codewords to the primitive array, and decoding is based on recognizing the pattern of light spots in the image as codewords in the primitive array. In this way, it is assumed that each diffraction array is sufficiently similar to the primitive array such that minor distortions in the distribution of light spots (captured in the image) do not negatively affect the recognition of light spots in the image. However, as... Figure 3 As shown, the distortion of a diffraction array may be greater than the tolerance allowed by still using a primitive array to identify codewords.

[0074] To address the aforementioned issues, a device might attempt to store a mapping of codewords for all diffraction arrays (and primitive arrays), taking into account the distortion of each array. However, the device needs to identify which array corresponds to the distribution of light spots identified in the image. For example, the device might store a tree structure or a mapping of spatial relationships between arrays (where the root node of the tree structure corresponds to the primitive array, and the diffraction arrays correspond to child nodes and further generation nodes from the root node), and perform a depth-first search through this tree structure to attempt to identify the corresponding diffraction array. As a result, the device recursively attempts to match the identified distribution of light spots with multiple codewords from multiple different arrays until the best match is found. This recursive approach and the use of a mapping of all array codewords in the distribution increase the time and processing resources required to attempt to determine the depth value compared to using primitive arrays for all matches. This increase in time may be unsatisfactory for users (e.g., for latency-constrained applications, including VR or other real-time applications). Additionally, resource-limited devices (such as mobile devices) may not be able to provide the required increase in processing resources. Therefore, the device uses a primitive array mapping of codewords to identify codewords throughout the image, which is more economical in terms of depth sensing time and processing resources compared to mapping using multiple arrays.

[0075] The device can also assume which diffraction array includes the identified light spot matched based on the location being processed in the image. For example, a sampling mask (“mask”) can be applied to a pixel location in the image, and that pixel location is statically associated with a particular array. However, due to distortion in the projection, the device might associate the location of the identified codeword with an incorrect array in the projection (which would result in errors in the parallax). For example, as... Figure 3 As shown, most of the diffraction array 304 with n=-3 or n=3 is located outside the image sensor boundary line 306. As a result, in one example, the device may incorrectly associate a recognized codeword with array (-7,-3) because that location is at the top left corner of the image (based on the pixel position in the image mapping to array (-7,-3)). However, the codeword could actually be part of array (-7,-2) or (-6,-2).

[0076] Furthermore, since distortion of the projection distribution (such as distortion caused by one or more DOEs of the transmitter) may occur during transmission, the amount of distortion in the reflections of the projection received by the image sensor may be based on the depth of the object from the image sensor. For example, as the object moves away from the image sensor, the stretching of the array in the reflections of the projection distribution from the object received by the image sensor may increase. Thus, the distortion caused by the transmitter is different for each image because the distortion of the projection distribution in the captured image may vary based on the depth of objects in the scene. A device that uses a mapping of codewords from the primitive array to decode the entire image may attempt to correct the image distortion before decoding. When correcting distortion before decoding, the device may determine the corrections to be applied to the image (such as a mask to be applied to the image to correct the position of each light spot in the image based on the distortion). However, correcting distortion before decoding (such as determining the mask to be applied) requires the ability to correctly identify each region of the projection distribution in the image. Correctly identifying each region may include identifying multiple codewords in each array of the distribution in the image. However, distortion may cause the device to fail to identify codewords or incorrectly identify the arrays associated with the codewords. As a result, it may be impossible to determine the corrections to apply before decoding. Furthermore, attempting to determine such a mask to reduce distortion may be time- and resource-intensive due to the use of codeword mappings for all arrays and mappings between arrays during decoding, which may be unacceptable for latency- or resource-constrained applications.

[0077] Alternatively, the device can attempt to compensate for distortion after decoding. For example, before determining the depth value, the increment (“delta”) of the determined disparity or depth value (determined for the image region) caused by known pincushion distortion can be subtracted from the disparity. However, pincushion distortion may vary based on depth and may be due to other distortions present in objects within the scene. Without knowing the exact distortion, the device cannot determine the increment to correct the disparity or depth value. Furthermore, distortion may prevent some projected portions of the image from being recognized for disparity or depth value determination. Therefore, the problems with pre-decoding correction and post-decoding correction are that distortion must be corrected first for successful image decoding, but the image must first be successfully decoded before distortion can be corrected.

[0078] In some aspects of this disclosure, the device is capable of decoding images for active depth sensing in the presence of projection distortion (such as pincushion distortion or distortion that may be caused by tilted or curved surfaces of objects in the scene). In some embodiments, the device adjusts sampling of one or more regions of the image to compensate for the distortion. Because the device is capable of decoding images in the presence of distortion, it can determine the correct depth values ​​of objects in the scene without attempting to correct the distortion before or after decoding.

[0079] Figure 4 A block diagram of an example device 400 for active depth sensing is shown. The example device 400 can be configured to perform structured light depth sensing. The example device 400 may include or be coupled to a transmitter 401. The transmitter 401 may be similar to... Figure 1 The transmitter 402 is shown in the image. For example, transmitter 401 is configured to project a light distribution for structured light depth sensing. Example device 400 may also include or be coupled to receiver 402, which is separated from transmitter 401 via baseline 403. Receiver 402 may be similar to... Figure 1 Receiver 108 in the image. For example, receiver 402 includes an image sensor configured to receive IR light (or light of other frequencies) emitted by transmitter 401 and reflected by one or more objects in the scene. Transmitter 401 and receiver 402 may be an active depth sensing system (such as a light controller 410 and / or processor 404) controlled by light controller 410 and / or processor 404. Figure 1 It is part of system 100.

[0080] An image sensor configured to receive IR light may be referred to as an IR image sensor. In some embodiments, an IR image sensor is configured to receive light with a frequency range greater than IR. For example, an image sensor not coupled to a color filter array may be able to measure the light intensity from a wide range of frequencies, such as color frequencies and IR frequencies. In some other embodiments, an IR image sensor is configured to receive light specific to IR light frequencies. For example, an IR image sensor may include or be coupled to a bandpass filter to filter light outside a frequency range unrelated to IR light. As used herein, IR light may include portions of the visible spectrum and / or portions of the spectrum invisible to the naked eye. In one example, IR light may include: near-infrared (NIR) light, which may or may not include light within the visible spectrum; and / or IR light (such as far-infrared (FIR) light), which is outside the visible spectrum. The term IR light should not be limited to light having a specific wavelength within or near the wavelength range of IR light. Furthermore, infrared light is provided as an exemplary emission for active depth sensing. In the following description, other suitable wavelengths of light may be captured by the image sensor or used for active depth sensing, and the IR image sensor or active depth sensing is not limited to IR light or IR light of a specific frequency.

[0081] Example device 400 also includes a processor 404, a memory 406 storing instructions 408 and a codeword library 409, an optical controller 410, and a signal processor 412. Device 400 may optionally include (or be coupled to) a display 414, multiple input / output (I / O) components 416, and a power supply 418. Device 400 may also include additional features or components not shown. For example, a wireless interface that may include multiple transceivers and a baseband processor may be included for enabling wireless communication devices to perform wireless communication. In another example, device 400 may include one or more cameras (such as a contact image sensor (CIS) camera or other suitable cameras for capturing images using visible light).

[0082] Memory 406 may be a non-transitory or non-temporary computer-readable medium storing computer-executable instructions 408 to perform all or part of one or more operations described in this disclosure. If the light distribution projected by transmitter 401 is divided into codewords, memory 406 may store a codeword library 409 for the light distribution. Codeword library 409 may indicate which codewords are present in the distribution and the relative positions between the codewords in the distribution. For example, since the distribution may include repetitions of primitive arrays, codeword library 409 may indicate codewords and their arrangement in the array. Codeword library 409 may also include a mapping of one or more image sensor locations to array locations in the light distribution (such as diffraction arrays and primitive arrays referencing the positions of images captured by image sensors). Codeword library 409 can therefore be used to decode images from receiver 402.

[0083] Processor 404 may include one or more suitable processors to perform aspects of this disclosure for decoding an image to perform active depth sensing in response to optical distortion. In some aspects, processor 404 may include one or more general-purpose processors capable of executing scripts or instructions of one or more software programs (such as instructions 408 stored in memory 406) or otherwise causing device 400 to perform any number of functions or operations. In additional or alternative aspects, processor 404 may include integrated circuits or other hardware to enable device 400 to perform functions or operations without the use of software. In some embodiments, processor 404 is configured to decode one or more regions of an image from receiver 402 to determine one or more depth values. For example, processor 404 may perform aspects of this disclosure to decode an image in response to optical distortion. Processor 404 may also be configured to provide instructions to light controller 410 for controlling transmitter 401.

[0084] The light controller 410 is configured to control the operation of the transmitter 401. The light controller 410 can indicate whether the transmitter is enabled or disabled based on whether the device 400 is in an active depth sensing mode. The light controller 410 can also instruct the transmitter 401 to adjust the intensity of the projection distribution (e.g., by adjusting the current to the VCSEL or another suitable light source of the transmitter). In some embodiments, the light controller 410 includes one or more suitable processors to execute programs or instructions (such as instructions 408 in memory 406). In additional or alternative aspects, the light controller 410 may include an integrated circuit or other hardware to control the transmitter 401. The light controller 410 may be controlled by a processor 404. For example, the processor 404 may provide the light controller 410 with general instructions regarding the operation of the transmitter 401 (e.g., the transmitter 401 will project a distribution). The light controller 410 can translate these general instructions into component-specific instructions recognized by the transmitter 401 to control the operation of the transmitter 401. Although the light controller 410 is depicted as separate from the processor 404, in some embodiments, the light controller 410 may be included within the processor 404. For example, the optical controller 410 may be embodied in the core of the processor 404. In another example, the optical controller 410 may be embodied in software (such as in instruction 408) that, when executed by the processor 404, enables the processor 404 to control the operation of the transmitter 401.

[0085] Signal processor 412 may include one or more processors to process images captured by receiver 402. For example, signal processor 412 may include one or more image signal processors (ISPs) as part of an image processing pipeline to apply one or more filters to the image from receiver 402, which is then decoded by processor 404. Exemplary filters that may be applied by signal processor 412 may include brightness uniformity correction filters, noise reduction filters, or other suitable image processing filters. In some aspects, signal processor 412 may execute instructions from memory (such as instructions 408 from memory 406 or instructions stored in a separate memory coupled to signal processor 412). In other aspects, signal processor 412 may include integrated circuits or other specific hardware for operation. Signal processor 412 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions. While signal processor 412 is depicted as processing the image before processor 404 decodes the image from receiver 402, in some embodiments, processor 404 may receive the image from receiver 402 (the device may not include signal processor 412 for further image processing). In some other embodiments, the signal processor 412 may be configured to perform decoding on the image from the receiver 402. For example, the signal processor 412 may perform aspects of this disclosure to decode the image.

[0086] Display 414 may include any suitable display or screen that allows the user to interact with and / or present items (such as depth maps, preview images of scenes, etc.) for the user to view. In some aspects, display 414 may be a touch-sensitive display. I / O component 416 may include any suitable mechanism, interface, or device for receiving input (such as commands) from the user and providing output to the user. For example, I / O component 416 may include a graphical user interface (GUI), keyboard, mouse, microphone and speaker, squeezable bezels or borders of device 400, physical buttons located on device 400, etc.

[0087] Despite Figure 4The example shown is coupled to each other via processor 404, but in various arrangements, processor 404, memory 406, optical controller 410, signal processor 412, display 414, and I / O components 416 may be coupled to each other. For example, processor 404, memory 406, optical controller 410, signal processor 412, display 414, and / or I / O components 416 may be coupled to each other via one or more fundamental buses (not shown for simplicity). Although some components of device 400 are shown, device 400 may include other components that are not shown for clarity in describing aspects of this disclosure. For example, device 400 may include an analog front end between receiver 402 and signal processor 412. The analog front end may convert analog signals of images captured by receiver 402 into an array of digital values ​​of the image. Conversely, some components of device 400 are shown but are not essential for performing aspects of this disclosure. For example, signal processor 412 may not be required to process images from receiver 402. In another example, processor 404 and memory 406 can receive images from a separate device including transmitter 401 and receiver 402. In this way, device 400 may not include light controller 410, transmitter 401, receiver 402, or signal processor 412. In another example, device 400 may include receiver 402 but not transmitter 401. Furthermore, as mentioned above, device 400 may not include display 414 and / or I / O components 416. While the following example of image decoding for active depth sensing (such as structured light depth sensing) is described with reference to device 400, any suitable device can be used to perform aspects of this disclosure. Therefore, this disclosure is not limited to a specific device configuration or component configuration for performing aspects of this disclosure.

[0088] Device 400 (such as processor 404) can decode an image from receiver 402, including sampling regions of the image, identifying portions of an array (such as identification codewords) within the sampled regions of the image, determining disparity based on the position of the identified portions in the array, and determining depth values ​​based on the determined disparity. Decoding a region of the image can be associated with a metric or function indicating the confidence level of the result determined during decoding (such as the identified spot position, the identified portion of the projection distribution based on the arrangement of the identified spots, the determined disparity, or the determined depth value). The metric or function indicating the confidence level can be referred to as a confidence value or a cost function. For example, during decoding, a confidence value can indicate the likelihood that the identified codeword for the image region is correct. Device 400 can use the confidence value to determine whether a depth value should be determined for the region or whether the determined depth value is assumed to be correct. The confidence value can also be used to determine which sampling grid should be applied to the region to identify one or more spots in the region.

[0089] Figure 5 A block diagram of an example decoding process 500 for active depth sensing is shown. The decoding process 500 can be handled by a processor 404 ( Figure 4 In some other embodiments, the decoding process 500 may be performed by the signal processor 412 or other suitable components of the device 400. As shown, the decoding process 500 does not require recursion or other resource-intensive operation flows. The decoding process 500 may be a linear operation, wherein the operation is performed once (not multiple recursive executions, as required in other possible solutions for dealing with distortion in the projection).

[0090] In the decoding process 500, the sampling grid stage 504 includes sampling the image 502 to generate image samples for analysis. Sampling may include identifying the distribution of light spots in image regions. In some embodiments, device 400 (such as processor 404) receives the image 502 (such as from receiver 402, from memory, or from another device including an active depth sensing system) for decoding active depth sensing. During the sampling grid stage 504, device 400 samples regions of image 502. The sampling grid is used to sample regions of image pixels from image 502 to generate image samples (wherein the image samples are analyzed to attempt to identify positions in an array to determine the parallax of the image regions). The sampling grid may be used to sample different regions of image 502 to attempt to identify different arrangements of light spots in the array (such as to attempt to identify codewords in each image region of image 502). For example, the sampling grid may be used to identify the location of image patches of a projected distribution. As used herein, an image patch may refer to a P×Q portion of the distribution (where P indicates the number of rows of possible light spots and Q indicates the number of columns of possible light spots). For example, the sampling grid can be associated with a 4x4 portion or image patch of the distribution, which may include 16 possible spot locations (4 rows x 4 columns). In some implementations, the size of the sampling grid for image 502 can be associated with the size of the codewords of the array. For example, if the array is associated with 5x5 codewords, the size of the sampling grid can be associated with 5x5 codewords. However, the size of the sampling grid can be any size suitable for sampling.

[0091] The sampling grid can be large enough that the associated image patch is unique within the array compared to all other image patches of similar size. For example, the sampling grid is independent of image patches of size 1x1 or 2x1, since multiple instances of such image patches exist in the array. In the example, the sampling grid is depicted and described as being associated with an image patch of size 4x4 (codeword size 4x4). However, the sampling grid can be associated with any suitable image patch to uniquely identify the image patch in the array for decoding.

[0092] Transmitter 401 can be configured to project a static distribution of light spots. For example, the light source of transmitter 401 and one or more DOEs can be fixed in position within transmitter 401 such that the projection distribution remains unchanged. In this way, the spacing between the light spots in the light source distribution is known (including the spacing between light spots along a baseline, which may be referred to as the interval). For example, the spacing between VCSELs in a VCSEL array (such as the interval) is known. In this example, for the sake of clarity in explaining various aspects of this disclosure, it is assumed that the interval between light spots is constant and without distortion throughout the array. However, in some embodiments, the interval may vary based on the position in the array (such as different portions of a VCSEL array having different spacings between VCSELs).

[0093] The sampling grid can be larger than (in image pixels of image 502) the size of the image patch distributed at the projection point of transmitter 401 (such as the codeword size of the array). For example, a 4x4 codeword can be associated with a sampling grid larger than 4 image pixels x 4 image pixels. Each light spot in the distribution can be associated with a point spread function, and the light spot spreads as it travels to an object in the scene and reflects back to receiver 402. As a result, multiple pixels of the image sensor can receive the light associated with the light spot. In addition, the spacing between light spots, crosstalk between components, thermal noise, distortion of the light spot in the optical path (such as perspective distortion), and scattering of the light spot at objects in the scene can all cause the light spot to be received at multiple pixels of the image sensor of receiver 402. In this way, the sampling grid size (in image pixels) to be applied to image 502 can be based on the spacing between light spots, known distortions (such as perspective distortion), and the baseline of the active depth sensing system.

[0094] Figure 6 A depiction 600 of an example sampling grid overlaid on a portion 604 of an image is shown. As illustrated, the sampling grid comprises a 4x4 arrangement of sampling points 606 used to sample 16 portions of the image to determine the presence of light spots within the image portion 604 at any location of the 16 sampling points 606. While sampling points 606 are described as being used to sample a single image pixel, each sampling point 606 can be used to sample one or more image pixels (such as a 2x1 or 2x2 group of image pixels).

[0095] Image portion 604 is increased in size to represent individual pixels of the image. During image capture, brighter (whiter) image pixels indicate that more light is received at the associated image sensor pixel than at the image sensor pixel associated with a darker (blacker) image pixel. For example, as mentioned above... Figure 1 The transmitter 102 can emit or project a light spot distribution 104 (which includes a codeword distribution) onto a scene. The light spots can be reflected back from one or more objects in the scene. The image sensor 132 can be configured to capture an image including the reflections of the light spots emitted by the transmitter 102. As indicated by the exemplary portion 604 of the image, light from a single light spot is received at multiple image sensor pixels. For example, light from a single light spot can be received at a 3x3 pixel group or a 4x4 pixel group of the image sensor. In the example, the light spot distribution within the image portion 604 can have similar spacing between the light spots in both the vertical and horizontal directions. Figure 6 As shown, this distribution does not include optical distortion. In this way, the spacing between the light spots in section 604 can be the same across the entire image.

[0096] When the sampling grid comprises 16 sampling points 606, generating image samples using the sampling grid can include sampling the luminance values ​​of image pixels located at sampling points 606. In this way, image samples can include vectors or other data structures of luminance values ​​(where positions in the vector correspond to the positions of associated sampling points 606 relative to other sampling points 606). Each vector can be associated with a position in the image (such as the row and column positions of pixels in the image at the center of the sampling grid). This position can be included as an entry in the vector, indicated by a storage location in the vector's memory, or indicated in any other suitable manner. The sampling grid can therefore be used to sample regions of an image, and such vectors can be image samples of image regions.

[0097] The size of the sampling grid can be based on the spacing between sampling points 606 (e.g., in pixels). In the example, each sampling point 606 includes itself and three image pixels between it and its neighboring sampling points 606 (where the sampling points 606 are in an equidistant 4x4 arrangement). In this way, each sampling point 606 can be associated with a set of 4x4 image pixels of a portion 604, and the sampling grid including the 4x4 equidistant arrangement of sampling points 606 can be associated with an image region of size 16 image pixels x 16 image pixels. As depicted, the sampling grid is used to sample a region 602 (which can have a size of 16x16 image pixels of the image), where sampling includes sampling 16 image pixels in region 602 located at the 16 sampling points 606 of the sampling grid. In the example, the sampling grid can be referred to as having a size of 16 image pixels x 16 image pixels. Although a quadrilateral shape is shown and described in the example of the sampling grid, other shapes can be used for the sampling grid (such as hexagons). This shape can be based on the arrangement of light spots in the distribution.

[0098] In the example, region 602 comprises a 4x4 image patch with a projected distribution. For example, region 602 may include 4x4 codewords from light spots in an array. In some implementations, device 400 may move a sampling grid throughout the image to generate image samples. For example, device 400 may move the sampling grid pixel-by-pixel or region-by-region. Figure 6 As shown, a sampling grid can be used to sample region 602, and the image sample can be processed to attempt to identify a location in the projected distribution (such as to identify a 4x4 codeword). The sampling grid can then be used to sample adjacent regions, and the image sample can be processed to attempt to identify a location in the projected distribution associated with the image sample (e.g., in the primitives of a primitive array 302 such as 3). In one example, device 400 can shift the sampling grid in the image by one or more image pixels and generate another image sample for the new region. If one image pixel is shifted, the image region can be sampled for virtually every image pixel in the image.

[0099] In some implementations of the sampling method, the brightness value of the image pixel located at sampling point 606 can be compared with a brightness threshold to determine whether a light spot of that distribution exists at the image pixel. For example, device 400 can determine whether the image pixel at sampling point 606 has a brightness greater than a threshold. In some other implementations, device 400 can determine whether the brightness value of the image pixel at sampling point 606 is greater than the brightness value of adjacent image pixels. In some further implementations, device 400 can combine the brightness values ​​of the image pixels at sampling point 606 and surrounding pixels to determine whether the combined value is greater than a threshold. In this way, sampling point 606 does not need to be precisely located at the center of the image pixel containing the brightness value of the light spot to identify the light spot (e.g., if the image pixel with the light spot off-center has a brightness value greater than a brightness threshold). Figure 6 In the example depicted, sampling point 606 is aligned with the position of the light spot in region 602, such that 8 out of 16 possible locations in region 602 include the light spot. In some embodiments, the vector of brightness values ​​generated from sampling region 602 may alternatively include a binary value indicating the presence of the light spot at the location of sampling point 606 (e.g., 0 indicating no light spot was identified, and 1 indicating a light spot was identified). In some other embodiments, the image samples may indicate the arrangement of light spots identified when sampling the image region in any suitable manner. Although Figure 6Sixteen sampling points are shown, but any suitable number of sampling points can exist. For example, every pixel in region 602 (such as every pixel in a 16×16 pixel area) can be sampled, or any suitable subset of the pixels in the region can be sampled. Thus, while the example provided herein is described with reference to the pixel located at sampling point 606 (or a similar sampling point), any suitable pixel in the image region can be sampled.

[0100] Return to reference Figure 5 During the decoder cost function phase 508, device 400 attempts to identify locations in the array using image samples generated during the sampling grid phase 504. For example, device 400 attempts to identify codewords in the array based on the arrangement of identified light points indicated by the image samples. Device 400 may also determine a confidence value associated with the identified location (which may refer to the identified codeword in the array). In some implementations, device 400 may compare the image samples with a reference mask 506. The reference mask 506 may indicate the arrangement of light points for an image block of the array. For example, if the sampling grid includes a 4x4 arrangement of sampling points (such as...) Figure 6 As shown, reference mask 506 can indicate the arrangement of light spots for each 4x4 image block of the primitive array for the projected distribution. In some embodiments, reference mask 506 indicates codewords in the array. Reference mask 506 can include vectors (such as vectors of binary values) or other suitable indications of light spots at specific locations of codewords. Device 400 can compare generated image samples with one or more reference masks 506 for the primitive array to attempt to find a match. As used herein, reference mask 506 can refer to a portion or region of a whole reference mask for the entire primitive array. For example, reference mask 506 can be a 4x4 region of a whole reference mask for the entire primitive array (associated with codewords of the primitive array). While examples herein may be described as using multiple reference masks for clarity in describing aspects of this disclosure, such examples can refer to using multiple separate reference masks or can refer to using different portions of a whole reference mask for the primitive array.

[0101] If the reference mask 506 indicates the spot position of a codeword, the codeword library 409 can store multiple reference masks 506 for comparison. The device 400 can identify the reference mask 506 that best matches the spot arrangement identified in the image sample. Since each reference mask 506 is associated with a position in the primitive array, the device 400 can identify positions in the array associated with regions of image 502. Based on the position of a region in image 502, the device 400 can also identify which array of the projected distribution is associated with that region (e.g., whether the region is associated with a primitive array or which diffraction array the region is associated with).

[0102] Device 400 can determine a confidence value or cost function at a specific location in the projected distribution. For example, device 400 may not accurately identify all light spots in an image region during sampling. As a result, no reference mask 506 can match an image sample. However, the arrangement of the identified light spots may be sufficient to match multiple reference masks 506, while determining that the remaining reference masks 506 do not match. Some reference masks 506 that may match can also be removed from consideration based on reference masks 506 that match other image samples. For example, a reference mask 506 that matches an adjacent image sample can be used to determine a reference mask 506 associated with an adjacent location in the array (and therefore may be more likely to match the current image sample). In another example, if a possible reference mask 506 matches a different image sample in a portion of an image 502 corresponding to the same distribution array, a reference mask 506 can be removed or reduced from consideration along with a sample that matches the current image. Therefore, the confidence value can be based on the number of possible matching reference masks 506, whether the reference mask 506 has been matched before, or whether the reference mask matches other image samples.

[0103] If device 400 identifies too many reference masks 506 with similar probability of a correct match, device 400 can determine a low confidence value (or the location associated with reference mask 506) associated with the most likely matching reference mask 506. If more points in the image sample are correctly identified, there may be more points matching reference mask 506, fewer possible matching reference masks, and the confidence value may increase. If fewer points in the image sample are correctly identified, there may be fewer points matching reference mask 506, more possible matching reference masks, and the confidence value may decrease.

[0104] In some implementations, the confidence value can be determined by calculating the number of locations identified in the image sample and reference mask 506 that have matching or non-matching light spots. For example, for an image sample comprising an array of 4x4 image blocks, the confidence value could be from 0 to 16 to indicate the number of correctly matched locations (such as whether the locations in the array indicated by reference mask 506 and their corresponding locations in the image sample both contain light spots or neither contains light spots). Such a confidence value could be a Hamming distance, and in such an example, thresholding the brightness value of an image pixel indicates whether a light spot exists at a location in the image sample (whether the brightness value of that pixel is greater than a threshold). In another example, matching only locations that contain light spots is used to determine the confidence value. While the confidence value is described as an integer, the confidence value can be any suitable indication of confidence (such as a percentage, decimal, fraction, or any suitable number used to measure confidence on a recognized scale). Another example of determining a confidence value or determining a match could include determining the cross-correlation between the image sample and reference mask 506. However, any suitable means for determining confidence values ​​or identifying matches can be used.

[0105] In some decoding implementations (such as identifying codewords in an array based on block matching), device 400 identifies locations (such as codewords in the array) in the array by identifying the reference mask 506 associated with the maximum confidence value from a plurality of reference masks 506. In some implementations, if the confidence value is greater than a threshold, the identified location in the array can be determined by device 400 as correct. If all confidence values ​​are less than the threshold, device 400 can determine that the location cannot be determined or that the location in the array is not used to determine the disparity and depth values ​​of a region in the image.

[0106] In some other decoding implementations, the device can determine a signature of the image sample, and determining the signature can also determine a confidence value. For example, if the primitive array includes codewords of size 4x4, the sample region of the image can have a size associated with a 4x4 codeword (e.g., ...). Figure 6 (Showing 16 image pixels × 16 image pixels). The dots in the primitive array can be arranged such that each codeword includes two dots at four positions per column. In this way, the columns of the codeword can be associated with six different combinations of the two dots for the four positions. Each combination of the two dots is associated with a symbol used to generate the signature. In this way, the codeword can be associated with a signature with four symbols (one symbol for each of the four columns), and the signature can have 1,296 (6 4 ( ) possible strings of four symbols.

[0107] Return to reference Figure 6The 16 sampling points 606 of the sampling grid are arranged in four columns (each column having four sampling points 606) corresponding to 4x4 codewords (such as in the example above). As mentioned above, sampling can be used to indicate which points 606 of the sampling grid are associated with projected light spots from image region 602. The device can also generate a signature for the image region based on the sampling. For example, the device determines a symbol for the samples from each column of sampling points. Continuing with the example above where each column of 4x4 codeword includes two light spots, if the device identifies two light spots in a column (such as for each column of sampling points 606 for image region 602), the device can determine that the symbol for that column (from six possible symbols) corresponds to the position of the two light spots in that column. In this way, the device can generate a signature with four symbols for image region 602 (or any suitable region of the image).

[0108] If two light spots are identified for a column, the symbol for that column is determined to have high confidence (such as above a threshold or even 100% confidence, since only one of the six combinations of light spots in that column matches). However, based on the localization of the sampling grid or on distortion in the projection or reflection of the light spots, the device may identify more or fewer than two light spots in an image region. If more or fewer light spots are identified, more than one symbol or no symbol can correspond to that column (because for a codeword, the specific combination of two light spots in that column does not specifically match the combinations of already identified light spots in that column). For example, if three light spots are identified for a column of sampled points in an image region, three different symbols out of six possible symbols may correspond to that column. The device can attempt to determine the best matching symbol by any suitable means (such as determining the two most likely light spots based on the difference between brightness values, based on the cross-correlation between image samples and current sampled values, based on machine learning or neural networks to determine the most suitable symbol, etc.). However, any matching symbol is not determined to have 100% confidence. In a simplified example, if three symbols might correspond to a column based on the identification of three light spots, the determined symbols can be associated with a 50% confidence level based on only three of the six symbols that absolutely do not correspond to that column. However, the confidence level can be based on other information, such as the difference between the brightness values ​​of the identified light spots or other suitable measurements used to determine the probability that a symbol corresponds to that column. Alternatively, if no symbol can be determined (e.g., based on the absence of a light spot identified in the column), a zero-percentage confidence level can be determined. In cases where each determined symbol (or column with no determined symbol) is associated with a confidence level, four confidence levels can be used to determine a confidence value for the determined signature. For example, if the confidence level for each column is a percentage less than or equal to 100% or a decimal or fraction less than or equal to 1, the confidence levels can be multiplied to determine the confidence value of the signature. In this way, determining a signature can also include determining a confidence value for the signature for each image sample. In some implementations, the device determines multiple candidate signatures and confidence values ​​associated with different candidate signatures. The device can then select the candidate with the highest confidence value as the final signature associated with the image region. While some examples are provided for determining a signature and the corresponding confidence value, any suitable means can be used to determine the signature and confidence value.

[0109] To identify codewords associated with an image region in a primitive array based on a signature, the device can match the signature with a signature or token string associated with the codeword. In one example, reference mask 506 can be a symbol string associated with the codeword. The overall reference mask of the primitive array can be a concatenation of the symbol strings of codewords in the primitive array to generate an overall symbol string. In this way, the overall reference mask can include multiple reference masks 506. When attempting to match a codeword, the device can compare a string with four symbols determined for the image region with the overall symbol string of the primitive array, identify the string with four symbols in the overall string, and determine the position of the string with four symbols in the overall symbol string of the primitive array. The position of the string with four symbols in the overall symbol string can indicate the position of the codeword in the primitive array, and the position of the codeword in the primitive array can be used to determine a depth value (based on parallax, as described herein). Although some examples in this disclosure describe block matching methods for identifying codewords from a primitive array for the purpose of clearly describing various aspects of this disclosure, matching codewords can be performed using any suitable means, such as methods based on signature generation for image regions. Therefore, this disclosure is not limited to specific implementations for identifying codewords or positions in a primitive array during processing.

[0110] Although not in Figure 5 As shown, however, stages 504 and 508 can be performed for multiple sampled regions of image 502 to identify associated positions (such as associated codewords) in the primitive array. After determining the position in the primitive array corresponding to the sampled image region, device 400 can determine the corresponding position in a reference diffraction array (based on the replication distribution of the primitive array, such as...). Figure 3 (As shown). After determining the position in the reference diffraction array, device 400 can determine the disparity along a baseline axis between the position in the reference diffraction array and the position of the sampled image region (e.g., the center of the sampled image region). Disparity is the distance in image 502 between the position of the sampled image region and the associated position in the reference diffraction array (along the baseline, such as...). Figure 1 (Baseline 112). In some cases, parallax can be measured in the number of image pixels (or subpixels) along the baseline.

[0111] In addition to pincushion distortion or distortion caused by objects in the scene or by the optical system (including one or more lenses, DOE, etc.), the positioning of transmitter 401 and receiver 402 relative to each other can also introduce perspective distortion into the distribution captured in image 502. For example, transmitter 401 and receiver 402 may be in an in-the-way ("toe-in") configuration relative to each other. Since transmitter 401 projects a light distribution onto the scene from a first viewpoint, and receiver 402 captures an image of the scene from a second viewpoint (and transmitter 401 and receiver 402 are in an in-the-way configuration), a parallel parallax exists between the first and second viewpoints. Parallax causes perspective distortion in the projected distribution captured by receiver 402 in image 502. Perspective distortion can be corrected by adjusting the determined parallax based on the perspective distortion (from the known parallel parallax) at the associated location in image 502.

[0112] During correction phase 510, device 400 adjusts one or more parallaxes to reduce perspective distortion. Image correction is the process of adjusting one or more images so that the viewpoints of multiple images are a common viewpoint. Correction for active depth sensing can be visualized as a process similar to image correction (to adjust the viewpoint of image 502 from the receiver's viewpoint to the transmitter's viewpoint). Since parallel parallax is known, parallax can be perspective distorted in a predefined manner based on its position in image 502. Therefore, the transformation used to adjust parallax can be predefined based on the image position, since parallel parallax is known.

[0113] For perspective distortion known by a predefined parallel parallax, device 400 can use distortion map 512 to reduce the effect of perspective distortion. Distortion map 512 may include multiple values, each associated with a location in image 502. In one example of applying distortion map 512 to adjust parallax, the parallax determined for an image region can be multiplied by the value in the distortion map corresponding to that image region.

[0114] Following the correction phase 510 for reducing perspective or optical distortion, the device 400 may determine one or more depth values ​​516 during the parallax-to-depth value conversion phase 514. In some embodiments, this conversion is a predefined mapping of the number of image pixels to depth values ​​based on baseline 403.

[0115] Returning to the correction stage 510, as mentioned above, distortion of the projected distribution can cause a shift in the light spot located in image 502. For example, diffracting the primitive array into a diffraction array by the DOE can cause pincushion distortion in the distribution. Distortion diagram 512 is based on a projected distribution that does not include distortions other than perspective distortion based on parallel parallax. Even assuming that the projected distribution (which includes pincushion distortion or other types of distortion) has no other distortions, correction stage 510 can cause different distortions in the distribution manifested in parallax. For example, if the projected distribution from transmitter 401 includes pincushion distortion, and image 502 including the projected distribution is corrected to adjust the viewing angle of image 502, the distribution in the corrected image (to remove perspective distortion) may appear to include barrel distortion instead of pincushion distortion.

[0116] Figure 7 A depiction of an exemplary distribution 700 of light spots in a corrected image of an example is shown. In this example, the initially projected distribution includes pincushion distortion (such as...). Figure 3 (Depicted in the middle). After correcting the image capturing the distribution with pincushion distortion, the spot distribution 700 includes barrel distortion. A portion 702 of distribution 700 shows the skew between the spots caused by pincushion distortion. The skew between the spots in the upper left corner of distribution 700 and distribution 300 (…) Figure 3 Compare the skewness between the light spots in the upper left corner. The skewness between the light spots is different between distributions of 700 and 300.

[0117] Because the distortion of the projection distribution from transmitter 401 differs from the distortion of the distribution in the corrected image, it may not be possible to determine the distortion caused by transmitter 401 (such as pincushion distortion) or by tilted objects in the scene based on comparing the corrected image with the projection distribution of the light spots. Therefore, distortion correction transformations (to correct pincushion distortion or other distortions) may not be determined or used for decoding. In some implementations, instead of attempting to remove optical distortion from image 502 for decoding (or post-decoding processing), device 400 may use a decoding process that takes into account the effects of optical distortion.

[0118] Return to reference Figure 6 Image portion 604 does not include distortion from the projected distribution. Therefore, a fixed sampling grid may be sufficient for sampling the image. For example, as... Figure 3 As shown, the distortion of the primitive array 302 is less than that of the diffraction array 304. Therefore, the example sampling grid (with an isotropic 4x4 pattern of sampling points 606) Figure 6The sample grid can be successfully used to sample regions of an image that include reflections from the primitive array 302 (and possibly adjacent diffraction arrays 304). However, due to the stretching and skewing of the diffraction array 304 (and the arrangement of light spots) caused by pincushion distortion or tilted objects in the scene, it may be difficult to sample regions of an image that include reflections from the diffraction array 304 that are far from the primitive array 302 (such as those toward the edges of distribution 300) using the sample grid.

[0119] Regarding tilted objects, the object plane in the scene used for active depth sensing, parallel to the plane defined by the projection plane of the image sensor and / or transmitter, is best suited to use an example sampling grid (such as...). Figure 6 The sampling is performed using a sampling grid with isotropic sampling points 606. Without considering other possible optical distortions, the spacing between the captured light spots in the image may be the same because the depth of the entire object is the same (therefore, the light from the transmitter, reflected by the object, and received at the receiver travels the same distance). For objects in the scene with different depths from the receiver 402 at different points on the object surface (which may be referred to as tilted objects), the spacing between the captured light spots in the image may be different (because the paths of the light from the transmitter, reflected by the object, and received at the receiver are different). Due to the difference in depth between different parts of the tilted object, a fixed sampling grid with isotropic position patterns (such as...) Figure 5 The mask 506 in the image may not be suitable for decoding parts of a scene with tilted objects.

[0120] Due to distortion in the projection distribution, it may be impossible to identify locations in the array (such as codewords in the primitive array) for some image regions during decoding. For example, pincushion distortion in the projection distribution may cause device 400 to fail to identify codewords in the array at the corner of the image, and therefore device 400 may be unable to determine one or more depth values ​​for the region at the corner of the image.

[0121] Figure 8 An example depiction 800 of an image region whose position in the array is not identified is shown. The darker portions of depiction 800 indicate regions where the position is not identified (such as unrecognized codewords in a primitive array), and therefore the parallax (and depth values) are not determined. The brighter portions of depiction 800 indicate regions where the position is identified (such as recognized codewords in a primitive array), and the parallax (and depth values) are determined. In some implementations, the brightness of the region in depiction 800 may be based on a confidence value of the identified position in the projection distribution of the region in the image. Depiction 800 may be based on a projection distribution including pincushion distortion. For example, depiction 800 may be... Figure 3The distribution of light spots 300 in the image is associated with this distribution. The portion 802 depicting 800 can therefore be associated with the upper left corner of the distribution 300 in the sensor boundary line 306. Due to the skewness of the light spots in the upper left portion of the distribution 300, the device 400 may be unable to identify the location or determine the parallax (e.g., in the upper left corner of the image) in a large area. Figure 8 (The black area in part 802 is depicted). As shown, the corners depicted in 800 can indicate large areas of the image whose parallax cannot be determined by device 400. Because less parallax is determined at the corners, the depth value determined for the corners of the image may be less than the depth value determined for other parts of the image. If a depth map is generated, most of the corners will indicate the lack of depth values ​​determined for the corners of the captured image.

[0122] For regular decoding, the sampling grid has a fixed size (such as...). Figure 6 The example sampling grid in the image consists of 16 image pixels x 16 image pixels, and the sampling grid typically includes a fixed number and arrangement (including spacing) of sampling points for decoding (such as...). Figure 6 (The sampling points 606 are arranged at equal intervals, with each sampling point spaced three image pixels apart). As used in the following example, referring to the projected distribution, the sampling grid associated with the PxQ image patch of the distribution (such as having PxQ sampling points) can be referred to as a PxQ sampling grid or a sampling grid of size PxQ.

[0123] As mentioned above, conventional decoding can be adjusted to reduce the impact of optical distortion on the determination of depth values. In some implementations, device 400 can be configured to adjust the sampling grid for sampling the image during decoding. For example, device 400 can adjust the arrangement of sampling points (such as adjusting the spacing between sampling points) for the sampling grid. Alternatively, device 400 can adjust the number of sampling points in the sampling grid. Device 400 can adjust the sampling grid to match the distortion of the distribution of light spots captured in the region of the sampled image. In some cases, a sampling grid can be selected from a plurality of available sampling grids for sampling each region associated with (e.g., pixel-centered) a pixel in the image (e.g., a first sampling grid can be selected for a first region in the image, a second sampling grid can be selected for a second region in the image, a first sampling grid can be selected for a third region in the image, etc.).

[0124] Figure 9An illustrative flowchart depicting an exemplary process 900 for decoding an image using active depth sensing is shown. The decoding process includes sampling different regions of the image using different masks. In some embodiments, different masks may refer to individual masks (such as individual signatures or blocks based on decoding methods for different codewords of a primitive array). In some other embodiments, different masks may refer to different regions or portions of a single mask of an array (e.g., a single overall signature or reference mask of a primitive array). At operation 902, device 400 receives an image. In some embodiments, device 400 uses a receiver (such as receiver 402) of an active depth sensing system to capture the image (corresponding to operation 904). In some other embodiments, device 400 (such as processor 404) receives the image from memory (such as memory 406) or from another device; in such embodiments, device 400 may or may not include receiver 402.

[0125] At operation 906, device 400 samples a first region of the image using a first sampling grid to generate a first image sample. In some embodiments, the process of sampling the first region is as described in reference [reference needed]. Figure 6 As described. At operation 908, device 400 samples a second region of the image using a second sampling grid different from the first sampling grid to generate a second image sample. Similar to operation 906, the process of sampling the second region is as described in reference [reference missing]. Figure 6 As described.

[0126] In some implementations, the fact that the second sampling grid is different from the first sampling grid indicates that the sampling point arrangement of the second sampling grid is different from that of the first sampling grid (corresponding to...). Figure 9 (Operation 910). For example, the spacing between sampling points of the second sampling grid (e.g., the number of pixels in the image sensor array) may differ from the spacing between sampling points of the first sampling grid. In another example, the skewness (e.g., tilt or orientation) of the sampling points of the second sampling grid may differ from the skewness of the sampling points of the first sampling grid. In yet another example, the spacing and skewness of the sampling points of the second sampling grid may differ from the spacing and skewness of the sampling points of the first sampling grid.

[0127] As a supplement or alternative to a different sampling point arrangement, the second sampling grid differing from the first sampling grid can indicate that the total number of sampling points in the second sampling grid is different from the total number of sampling points in the first sampling grid (corresponding to operation 912). In an illustrative example, the first sampling grid may include 16 sampling points (such as a 4x4 arrangement of sampling points), while the second sampling grid may include 25 sampling points (such as a 5x5 arrangement of sampling points).

[0128] At operation 914, device 400 can determine a first depth value based on a first image sample. At operation 916, device 400 can determine a second depth value based on a second image sample. For example, returning to reference... Figure 5 The device 400 can identify one or more light spots distributed in an image region (such as during the sampling grid phase 504), identify the position in the primitive array based on the arrangement of the identified light spots in the region (such as during the decoder cost function phase 508), determine the disparity based on the identified position, adjust the disparity during the correction phase 510, and determine the depth value based on the disparity during the disparity-to-depth conversion phase 514.

[0129] In some implementations, device 400 may attempt to determine a depth value for each region of the image. For example, sampling may occur at the region associated with each image pixel. If a location in the array is accurately identified for that region (e.g., the identified codeword is associated with a confidence value greater than a threshold), disparity and depth values ​​can be determined. If no location is accurately identified (e.g., each codeword in the array is associated with a confidence value less than a threshold), device 400 may shift an image pixel (e.g., shift one pixel up, down, left, or right in the image) to sample the next region without generating a depth value for the previous region. While shifting one pixel is described in some examples, shifting or moving within the image in any suitable manner can sample different regions (e.g., shifting multiple pixels in the image).

[0130] Return to reference Figure 9 In operation 910, the arrangement of sampling points in the second sampling grid can differ from that in the first sampling grid. In some implementations, the spacing between sampling points in the first and second sampling grids can be different. For example, the first sampling grid can be similar to... Figure 6 The example sampling grid in the image (with 3 image pixels between adjacent sampling points 606). A second sampling grid may include sampling points with more than 3 image pixels between adjacent points.

[0131] Figure 10An example depiction 1000 of a first sampling grid 1002 and a second sampling grid 1004 with different spacing between adjacent sampling points is shown. Depicted to clearly illustrate the difference in spacing between sampling points, the first sampling grid 1002 and the second sampling grid 1004 are applied to the same region 1008 of the image portion 1006 to sample region 1008. However, in some embodiments, the first sampling grid 1002 and the second sampling grid 1004 may also be applied to the same region of the image (e.g., to determine whether to use the first sampling grid 1002 or the second sampling grid 1004 for that region, or to determine whether to use image sample results from the first sampling grid 1002 or the second sampling grid 1004 for that region).

[0132] The first sampling grid 1002 includes a first spacing between adjacent sampling points 1010, while the second sampling grid 1004 includes a second spacing between adjacent sampling points 1012. The first spacing is smaller than the second spacing. In other words, the first spacing is associated with fewer image pixels between sampling points 1010 compared to the second spacing between sampling points 1012.

[0133] As shown in the figure, the spacing between the light spots in region 1008 is greater than the spacing between sampling points 1010. However, the spacing between the light spots in region 1008 may be similar to the spacing between sampling points 1012. As a result, sampling using the second sampling grid 1004 can correctly identify more light spots present in region 1008 compared to sampling using the first sampling grid 1002. In this way, the second confidence value associated with the second sampling grid 1004 (e.g., based on applying the second sampling grid 1004 and then determining the second confidence value) can be greater than the first confidence value associated with the first sampling grid 1002 (e.g., based on applying the first sampling grid 1002 and then determining the first confidence value) for sampling region 1008 (such as based on determining the position in an array associated with image samples of region 1008 generated using the first sampling grid 1002 and the second sampling grid 1004).

[0134] The light spots in image portion 1006 are depicted as skewed relative to the horizontal and vertical axes. Consequently, region 1008, comprising a distributed 4x4 image patch, is skewed (e.g., the region is not a square or rectangle in this example). This skewness may be caused by pincushion distortion in the projected distribution. The skewness of the sampling points in sampling grids 1002 and 1004 may not be the same as the skewness of region 1008. However, in some embodiments, adjusting the spacing of the sampling points (ensuring the sampling points are not skewed) may be sufficient to sample an image region (such as region 1008) for decoding.

[0135] As described above, for device 400 to identify a light spot at an image pixel located at a sampling point, the sampling point does not need to be located at the center of the light spot in the image. For example, if the brightness of an image pixel is greater than a threshold, device 400 can identify a light spot at the image pixel located at the sampling point. As used herein, a brightness value can refer to any suitable measurement of the light intensity received at an image sensor pixel. Exemplary brightness values ​​may include values ​​in lumens, luminance, a white value defined for an image, a red-green-blue (RGB) value defined for an image, or other suitable values.

[0136] In this way, even if region 1008 is skewed, the second sampling grid 1004 (with a spacing of sampling points 1012 similar to the spacing of the light spots in the skewed region 1008) can still be used to successfully identify the position in the array based on the identified light spots at one or more sampling points 1012. Therefore, the disparity and depth values ​​of region 1008 can be determined based on sampling using the unskewed sampling grid 1004. However, in some embodiments, the first and second sampling grids may differ relative to the skewness in the arrangement of the sampling points, as a supplement or alternative to the difference in the spacing between the sampling points.

[0137] Figure 11 An example depiction 1100 of a first sampling grid 1102 and a second sampling grid 1104 is shown, which have different skews applied in the image portion 1106. Although sampling points are not shown for sampling grids 1102 and 1104, the outlines of sampling grids 1102 and 1104 are depicted to illustrate the skew in the arrangement of the sampling points. As used herein, skew of a sampling grid can refer to any stretching, twisting, or other adjustment to the position of the sampling points such that the arrangement of sampling points differs between sampling grids (other than variations in the spacing between sampling points). For example, the sampling point arrangement of the first sampling grid 1102 may be rectangular, while the shape of the sampling point arrangement of the second sampling grid 1104 may be parallelogram, trapezoid, or other suitable quadrilateral. In some embodiments, skew may cause changes in the number of sides of the grid shape, the curvature of the sides, or any other suitable variation in the arrangement of the sampling points. A sampling grid with a sampling point skew similar to the skew of light spots in an image region may be more suitable for sampling that region than other sampling grids with different skews. For example, the second sampling grid 1104 may be more suitable for sampling a region in portion 1106 than the first sampling grid 1102. As used herein, "more suitable" can refer to sampling using a more suitable sampling grid, resulting in a higher confidence value being determined compared to using other sampling grids.

[0138] Return to reference Figure 9Operation 912 in the text, as a supplement or replacement for the arrangement of sampling points in the second sampling grid that differs from the arrangement of sampling points in the first sampling grid, allows the total number of sampling points in the second sampling grid to differ from the total number of sampling points in the first sampling grid. For example, the first sampling grid may include 16 sampling points (such as a 4x4 equidistant arrangement of sampling points, such as in...). Figure 6 The sampling grid in Figure 10 (In sampling grid 1002 or 1004). The second sampling grid may include a total number of sampling points greater than or less than 16. For example, the second sampling grid may include 20 sampling points (such as in a 4x5 or 5x4 arrangement), 25 sampling points (such as in a 5x5 arrangement), or any other suitable number of sampling points.

[0139] Device 400 can use different sampling grids to sample the same area multiple times to generate image samples of that area. These image samples can be used to attempt to identify locations within the array (such as those based on reference mask 506). Figure 5 The device 400 can identify codewords by matching them with the image region and can determine a confidence value for each sampling grid used for the image region. For example, the device 400 can apply sampling grids to the image region and then calculate the confidence value of the sampling grids. In some embodiments, the block matching method can be used to determine codewords for the image region and / or confidence values ​​associated with codewords for the image region. In some other embodiments, the signature generation method can be used to determine codewords and / or confidence values ​​associated with codewords for the image region (as referenced above). Figure 5 (Description). For example, each sampling grid can be used to determine a signature, and each signature can be associated with a confidence value. The highest confidence value indicates the sampling grid used to sample a region in an attempt to determine a depth value. The position of a light spot in an image may depend on the light spot's position in the light spot's projection distribution, the distortion of the projection distribution, and the depth of objects in the scene reflecting the light spot from the projection distribution. Therefore, different sampling grids can be associated with different confidence values ​​based on the position of an image region in the image. In this way, different sampling grids can be used to sample different regions of the image during decoding to determine depth values ​​from the image.

[0140] Figure 12 This illustrates the use of an image sensor or receiver (e.g., Figure 4The block diagram illustrates an example decoding process 1200 for an image captured by a receiver 402 using different sampling grids. During the sampling grid phase 1204, the device 400 may sample one or more regions of the received image 1202 using multiple sampling grids 1 to X (where X is an integer greater than 1). In some embodiments, the device 400 may store sampling grids 1 to X for sampling during decoding. In some other embodiments, the device 400 may include one or more template sampling grids (such as a template mapping for the arrangement of sampling points for a portion of the sampled image). The template sampling grids may be adjusted to generate one or more different sampling grids 1 to X for sampling during decoding. Although some examples of persisting or generating sampling grids are described, sampling grids may be generated or persisted in any suitable manner so that they can be used to sample one or more regions of the image 1202 during the decoding process 1200. Although some examples may be described with reference to X = 2, any suitable number of sampling grids may be used during the sampling grid phase 1204. The number of sampling grids can be determined to balance improved sampling accuracy (by having more sampling grids) with reduced processing time and computational resources (by having fewer sampling grids). Each sampling grid is different from the other sampling grids used when sampling an image region. For example, each sampling grid may have a unique combination of sampling point arrangement (e.g., spacing, skew, etc.) and / or the total number of sampling points. For example, in some implementations, sampling grids are different from each other based on a unique spacing between sampling points.

[0141] In one example of a unique sampling grid used for decoding, the sampling grids are distinct from each other entirely based on the spacing between the sampling points, and the spacing ranges from 4 to 6 image pixels. In this particular example, the number of unique sampling grids can be 3 unique sampling grids to be used for sampling.

[0142] In the specific example above with three unique sampling grids, pincushion distortion and the maximum disparity, which can be determined based on the size of the array along the baseline of the distribution, can indicate the minimum number of unique spacings that may be beneficial during sampling. Exemplary implementations of three unique sampling grids can be based on one or more constraints (or similar constraints) of the example active depth sensing system in the specific example. In one illustrative example, the constraint is that there are 48 spot locations along the baseline of the array of the projected distribution. Another exemplary constraint is that the spacing (relative to the image) of the locations along the baseline is 4 image pixels. With 48 locations along the baseline in the array and a spacing of 4 image pixels, the maximum measurable disparity is 192 image pixels (48 x 4). The spacing between locations in the array increases as one moves along the baseline from the primitive array at the center of the distribution to the diffraction array at the edge of the exemplary distribution. As a result, the spacing of the possible locations of the spots increases for the diffraction array toward the edge of the distribution (which could affect disparity if calculated based on an undistorted distribution). However, based on the specific pincushion distortion and the expected depth range of the active depth sensing system, a maximum disparity of less than 600 image pixels may be sufficient.

[0143] Figure 13 An exemplary graph 1300 is shown depicting the relationship between the theoretical spacing between sampling points of a sampling grid and a disparity measurement used for accurate sampling of an image region. Graph 1300 shows an example depiction of the theoretical spacing (the interval of the sampling grid) of sampling points along the baseline of the sampling grid for accurate sampling of an image region, including image patches with a pincushion distortion distribution. "Accurate sampling" of a region can refer to correctly identifying all light points within the image region. The vertex of parabola 1302 indicates that for a distribution interval of 4 and a sampling grid interval of 4, the measured disparity (D) based on the maximum depth (such as a depth close to infinity) is 0. The vertex of parabola 1304 indicates a measured disparity (D) of 96 pixels based on half the depth between the maximum depth and the depth associated with the maximum disparity that can be measured (which may be referred to as the minimum depth). The vertex of parabola 1306 indicates a measured disparity (D) of 192 pixels based on the minimum depth. The minimum depth can be based on the size of the baseline and the size of the array along the baseline.

[0144] In the absence of distortion and with a distribution interval of 4 image pixels, the interval of sampling points in grid sampling (4 image pixels) is similarly preferred. This is illustrated by the vertices of parabolas 1302 to 1306 for different depths. As depth decreases, parallax increases (as illustrated by parabolas 1304 to the left of parabolas 1302 and parabolas 1306 to the left of parabolas 1304). While some image regions may be associated with an optimal grid interval that is not an integer, the grid interval of the sampling grid is an integer. However, the tolerance based on the light spots existing on multiple image pixels allows the integer closest to the grid interval to be used for the sampling grid. In this way, for a depth range, the preferred interval between sampling points (which may also be referred to as the grid interval) can be an integer defined by parabolas 1302 and 1306. As shown, a grid interval with 4 to 6 image pixels may be sufficient for accurate decoding of each region of the image.

[0145] While the examples above illustrate the use of isotropic sampling grids (where the number of columns of sampling points is the same as the number of rows), in some implementations, the decoding process (such as...) is performed... Figure 12 The device 400 in the decoding process 1200 can also use an anisotropic sampling grid. For example, the multiple sampling grids to be used can include an arrangement of 4x4 sampling points, 4x5 sampling points, 5x6 sampling points, etc. In another example, the multiple sampling grids to be used can include an arrangement of 4x4 sampling points with different spacing between sampling points in the horizontal and vertical directions (such as 4 pixels vertically and 5 pixels horizontally between points, 5 pixels vertically and 6 pixels horizontally between points, 5 pixels vertically and 4 pixels horizontally between points, etc.). As a supplement or alternative, although the examples describe the arrangement of sampling points for the sampling grid as rectangular or square, for one or more sampling grids, the arrangement can be skewed (such as the one mentioned above). Figure 11 As described above, the number of sampling grids to be used can be based on a balance between accuracy and performance.

[0146] While some examples describe the uniqueness of a sampling grid entirely based on the spacing between sampling points, in other implementations, sampling grids are distinguished from each other based on a unique combination of the spacing between sampling points and the skewness of the sampling points. However, any suitable property that makes each sampling grid unique can be used.

[0147] Return to reference Figure 12 The sampling of a region of the image using any of the sampling grids 1 to X (during sampling grid phase 1204) can be performed by device 400, similar to the above reference. Figure 5The sampling grid stage 504 is described in the text. Figure 12 The sampling grid stage 1204 in the middle may be different Figure 5 In the sampling grid stage 504, because the region can be sampled X times using different sampling grids (instead of... Figure 5 (Sampling once as in sampling grid stage 504) generates multiple image samples 1 to X for the image region. See reference... Figure 5 As mentioned, generating image samples may include identifying light spots in the image region at the location of sampling points in the sampling grid (or identifying light spots in the image region using any other suitable sampling method).

[0148] In an example where two sampling grids are used for sampling (e.g., X equals or is greater than 2), a first sampling grid can be used to determine the depth value of a first region of image 1202, and a second sampling grid can be used to determine the depth value of a second region of image 1202. As described in more detail herein, the first sampling grid used for the first region can be determined based on a confidence value determined for the first sampling grid being greater than a confidence value determined for the other sampling grids when applied to the first region. Similarly, the second sampling grid used for the second region can be determined based on a confidence value determined for the second sampling grid being greater than a confidence value determined for the other sampling grids when applied to the second region. In this way, device 400 can sample a first region of an image using the first sampling grid (e.g., to generate a first image sample) and sample a second region of an image using a second sampling grid different from the first sampling grid (e.g., to generate a second image sample). Device 400 can then determine a first depth value based on the first image sample and can determine a second depth value based on the second image sample. For example, to determine the first depth value, device 400 can identify an array (e.g., a primitive array or a diffraction array, such as...) based on the first image sample. Figure 3 The first position is determined in the primitive array 302 or the diffraction array 304, and a first disparity is determined based on the first position. The first disparity can then be converted into a first depth value (as described herein).

[0149] Both sampling grids can be used for both the first and second regions of image 1202 during sampling grid phase 1204. In this way, device 400 can also use the second sampling grid to sample the first region of the image (e.g., to generate a third image sample). In some cases, device 400 can compare the first sampling grid with the second sampling grid. Device 400 can select the first sampling grid to be used to determine the first depth value based on this comparison. In some cases, device 400 can compare the first image sample with the third image sample, and can select the first image sample to be used to determine the first depth value based on this comparison. During decoder cost function phase 1208, device 400 can determine a confidence value associated with the sampling grid (e.g., sampling grid 1, sampling grid 2, sampling grid X). The confidence value is... Figure 12 The confidence score is represented as score 1 associated with sampling grid 1, score 2 associated with sampling grid 2, and so on, up to score X associated with sampling grid X used for the image region. In some embodiments, comparing the first sampling grid with the second sampling grid or comparing the first image sample with the third image sample may include comparing confidence scores that indicate the probability of successfully identifying a spot or location in an array of recognition (e.g., a primitive array) within the image region. In some cases, device 400 may determine the confidence score for each sampling grid (e.g., a first confidence score for sampling grid 1, a second confidence score for sampling grid 2, and a first confidence score for sampling grid X). In some cases, device 400 may determine the confidence score for each image sample generated using each sampling grid (e.g., a first confidence score for the first image sample generated using sampling grid 1, a second confidence score for the second image sample generated using sampling grid 2, and a first confidence score for the third image sample generated using sampling grid X).

[0150] In some implementations during the decoder cost function phase 1208 (e.g., if a block matching method is used to determine codewords), device 400 may attempt to identify locations in a distribution array for each image sample based on the arrangement of the identified light spots. In some implementations, locations in the identification array (e.g., a primitive array) include image blocks in the identification array (e.g., codewords in the identification primitive array 302). In some other implementations, device 400 may generate a signature for each image sample (e.g., as described above). In some cases, the signature for each image sample is associated with a confidence value. In such implementations, locations in the identification array may include a symbol string of the identification primitive array that matches a signature generated for one or more image samples (e.g., for an image sample associated with the highest confidence value of an image region). In some implementations, each identification location for each image sample is associated with a confidence value (e.g., the confidence value is associated with a corresponding sampling grid, image sample, and / or signature). In some cases, device 400 may determine a confidence value associated with each identification location for each image sample.

[0151] (During decoder cost function phase 1208) Identifying locations in the array and generating confidence values ​​for image regions (e.g., confidence values ​​for each image sample or sampling grid) can be compared with those for... Figure 5 The process is similar to that described in stage 508 of the decoder cost function. For example, one or more reference masks 1206 used for distribution can be compared with image samples to attempt to match the reference masks 1206 with the image samples. If the reference mask 1206 indicates a codeword of the array, the device 400 determines the position of the codeword in the image (which can be used to determine the parallax based on the position of the center of the associated array in the image). Figure 12 The decoder cost function in stage 1208 may differ from... Figure 5 In the decoder cost function stage 508, multiple confidence values ​​can be determined (such as generating a confidence value for each of the image samples using sampling grids 1 to X, generating a confidence value for each of the sampling grids 1 to X applied to the region, etc.), instead of in... Figure 5 During stage 508 of the decoder cost function, a confidence value is determined for the region.

[0152] In some other implementations, a confidence value is determined when determining each signature for each sampling grid of the image region. The signature with the highest confidence value can be selected, and the selected signature is used to attempt to determine the location in the codeword or array (as referenced above). Figure 5 (As described).

[0153] In the above example of comparing a first sampling grid with a second sampling grid, device 400 can determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid. Based on the fact that the first confidence value is greater than the second confidence value, device 400 can select the first sampling grid to determine the depth value of the first region. For example, device 400 can compare the first sampling grid and the second sampling grid at least in part by comparing the first confidence value with the second confidence value. Based on this comparison, device 400 can determine that the first confidence value is greater than the second confidence value.

[0154] In the above example of comparing a first image sample with a third image sample, device 400 can determine a first confidence value associated with a first sampling grid of a first region based on the first image sample, and can determine a second confidence value associated with a second sampling grid of the first region based on the third image sample. Device 400 can compare the first confidence value with the second confidence value (as described above). Device 400 can select the first image sample based on the first confidence value being greater than the second confidence value.

[0155] Therefore, device 400 can generate multiple image samples and associated confidence values ​​for an image region during decoder cost function phase 1208 or sampling grid phase 1204. During selection phase 1209, device 400 can select sampling grids and / or image samples to determine the disparity of the region (and thus the depth value). For example, device 400 can select multiple recognition locations (such as the location associated with the maximum confidence value) in an array (e.g., a primitive array) based on the confidence value, or it can select a signature with the highest confidence value to determine the location in the array. For example, a sampling mask associated with the maximum confidence value can be used to determine the disparity of the region during decoding. Since confidence values ​​may depend on different factors (such as the depth of the object, tilted objects, or distortion in the distribution), a first sampling grid can be selected for a first region of image 1202, and a second sampling grid can be selected for a second region of image 1202.

[0156] Similar to the reference above Figure 5As described, although not shown, the decoding process 1200 may include performing a sampling grid phase 1204, a decoder cost function phase 1208, and a selection phase 1209 on multiple regions of image 1202. For example, device 400 may sample (using multiple sampling masks) one or more pixels in image 1202 and shift them. In some embodiments, sampling of a unique region of image 1202 may be performed on each image pixel of image 1202 (where the new region is an image pixel shifted from a previous region in a certain direction). Although phases 1204, 1208, and 1209 are described as being performed recursively on multiple regions of image 1202, in some embodiments, phases 1204, 1208, and 1209 may be performed simultaneously on at least two or more regions of image 1202. Therefore, the phase order depicted in the figures or examples may not necessarily be required.

[0157] After selecting a location within the array of projection distributions (during selection phase 1209), device 400 can determine the disparity associated with that area. Device 400 can then determine one of one or more depth values ​​1216 based on the disparity (such as during disparity-to-depth value conversion phase 1214). In some embodiments, portion 1218 of the decoding process 1200 is related to... Figure 5 The decoding process in step 500 is the same. For example, the correction stage 1210 can be the same as... Figure 5 The correction stage 510 is the same as that in the distortion image 1212, which can be compared with the previous stage. Figure 5 The distortion map 512 is the same, and the parallax to depth value conversion stage 1214 can be the same as... Figure 5 The parallax-to-depth value conversion stage 514 is the same. In this way, the decoding process 1200 can differ from the sampling grid stage 1204, the decoder cost function stage 1208, and the new selection stage 1209. Figure 5 The decoding process 500 is described above. In some implementations, one or more depth values ​​1216 may be used to generate a depth map that includes one or more depth values ​​1216. For example, a first depth value of a first region and a second depth value of a second region are included in the depth map, and the depth map indicates one or more depths of one or more objects in the scene. The depth map may be displayed to a user on a display 414 and may be used for one or more depth sensing applications (such as facial recognition for display unlocking or security applications, obstacle detection or avoidance, ranging or distance measurement, or augmented reality applications), or may be used for other suitable applications.

[0158] While the above examples of projection distributions are described with reference to a square lattice of light points, the projection distribution can be any suitable distribution. For example, the shape of the light in this distribution does not have to be a point; it can be an arc, a straight line, a curve, a square, etc. In another example, the lattice of light points does not have to be a square lattice. For example, the distribution of light can include a hexagonal lattice of light points. Figure 14 Example depictions 1400 of square dot array 1402 and hexagonal dot array 1404 are shown for comparison. An active depth sensing system can reduce the size of the primitive array using a projection distribution with a hexagonal dot array while including the same number of dot locations in the array. In this way, smaller components (such as smaller image sensors and smaller DOEs) can be used, which can reduce the size of the active depth sensing system or reduce the cost of manufacturing it. In this way, the sampling grid can be based on a distribution including hexagonal dot arrays instead of square (or rectangular) dot arrays.

[0159] As mentioned above, determining depth values ​​based on the identified location within the array involves determining the disparity and converting the disparity into a depth value. When identifying a location, device 400 can use the location of an image region to identify the position of an image patch (such as a codeword) within the primitive array. The location within the image region can be the coordinates (such as (x, y) coordinates) of the region associated with the identified codeword in the image. If the baseline axis is the x-axis, the disparity can be x... location -x center , where x location It is the x-coordinate of the codeword's position in the image, and x center It is the x-coordinate of the center position of the associated array in the distribution.

[0160] However, the DOE can replicate the primitive array along the baseline, such that the projected distribution includes a series of diffraction arrays centered on the primitive array. For example, there might be three diffraction arrays on either side of the primitive array along the baseline in the projected distribution, and eight diffraction arrays on either side of the primitive array along an axis at a 90-degree angle to the baseline. Therefore, the position of a codeword could correspond to any of these arrays. While the mapping from image position to distribution position (determined for an undistorted distribution) can be used to attempt to identify the associated array for a recognized codeword, distortion (such as pincushion distortion) can cause this mapping to associate one or more recognized codewords with incorrect arrays. For example, the mapping might indicate an array where, due to distortion, the correct array is adjacent to the indicated array.

[0161] As mentioned above, as the position of an object changes within the scene, the associated disparity (as described above) calculated based on the position of the identified codeword in the image may reverse from minimum disparity to maximum disparity, or vice versa (indicating a change in the array of projected distributions corresponding to the position). Reversal can also occur in a direction 90 degrees to the baseline axis (e.g., if measuring the difference in y-coordinate between the position of the codeword in the image and the center of the array in the image).

[0162] To accurately determine parallax, device 400 can determine which array corresponds to the image region containing the identified codewords. For example, device 400 can determine the distribution (such as...) Figure 3 The distribution 300 in the array (including the primitive array 302 and various diffraction arrays 304) is determined by a row of the array and a column of the array.

[0163] When determining a location, device 400 determines the codeword in the array (as described above) and which array of the projected distribution corresponds to a region in the image. As mentioned above, device 400 may include a mapping from different locations in the image from receiver 402 to corresponding arrays in the projected distribution. This mapping may be calculated based on a distribution that does not include pincushion distortion or other optical distortions, or otherwise. In some embodiments, the codeword at 90 degrees to the baseline is large enough that the maximum displacement of the light spot caused by distortion is less than the size of the codeword along this direction.

[0164] Figure 15 An example depiction of spot displacement caused by distortion (such as pincushion distortion) in distribution 1500 (such as the distribution captured in the image) is shown. In this example, the baseline is in the horizontal direction of distribution 1500. The array may include columns (indicated by the columns in box 1502) with seven unique codewords (such as 4x4 codewords) in the vertical direction. The columns in box 1502 indicate the 7 codewords corresponding to the first array (array(m,n)) in distribution 1500, while the columns in box 1506 indicate the same 7 codewords corresponding to the array (array(m,n-1)) adjacent to the first array. The maximum disparity that can be calculated is D. max (as referenced above) Figure 13 (The example uses 192 image pixels). When parallax = D max The position of column 1502 in the image is shifted along the baseline by a factor of D. max The number of image pixels (described by column 1504). Parallax can be between 0 and D. maxThe positions of columns in box 1506 and 1504 in the image should be reversed if there is no distortion. The displacement of the light spot caused by distortion can be visualized by comparing columns 1504 with those in box 1506 (shifted left and up from column 1504 in the image due to distortion).

[0165] When determining the corresponding array for an image region, device 400 can determine the columns of the array and the rows of the array in the projected distribution. In some implementations, the maximum vertical displacement of the light spot is less than the vertical displacement of the codeword in the image (where the baseline is along the horizontal direction in the image). For example, when comparing the column of box 1506 with column 1504, the vertical displacement of the codeword in 1504 is less than the height of the codeword in the image. Since the vertical displacement is less than the height of the codeword, if there is no distortion, device 400 can determine that the row of the array of the distortion distribution corresponding to the image region is the nearest row of the array corresponding to the image region. For example, the rows of the array for the image region (with the identified codeword) can be determined using an array-to-image-part mapping (which can be calculated based on the absence of distortion in the distribution) without any transformation or further calculation.

[0166] Device 400 can also determine the columns of the array corresponding to the image region. Since parallax is calculated along the horizontal axis (baseline), the above mapping may indicate incorrect columns of the array. For example, the mapping may be used to determine array (m, n), but the codeword in the image region may actually correspond to an adjacent array (m, n-1). In some embodiments, device 400 uses the mapping to determine two adjacent columns of the array as possible columns of the image region. If the maximum vertical displacement of the codeword is less than, for example, half the height of the primitive array in the image (e.g., less than the height of the codeword in the image), then the nearest row of the array indicated by the mapping corresponds to the image region. In this way, device 400 can use the mapping to determine two adjacent arrays in the distribution (e.g., a left array and a right array) as possible arrays associated with the image region including the identified codeword.

[0167] In some implementations, to select an associated array from two arrays, device 400 may determine a first disparity based on the first array and a second disparity based on the second array (as described above regarding determining disparity). In some implementations, device 400 selects the array associated with the smaller of the first or second disparity. In some other implementations, device 400 does not select the array associated with a disparity greater than the maximum disparity. In this way, a disparity can be determined for each recognized codeword in the image, and a depth value can be determined for each disparity (taking into account optical distortion of the distribution). As mentioned above, the depth value can be used to generate a depth map. The depth map can be used in applications based on active depth sensing (such as face recognition, object detection, obstacle avoidance, augmented reality, etc.).

[0168] Figure 16 This is a flowchart illustrating an example of a process 1600 for determining the exposure duration of multiple frames using the techniques described herein. At block 1602, process 1600 includes receiving an image that includes one or more reflections of a light distribution. For example, the light distribution may include a distribution of light spots. In an illustrative example, the light distribution may include... Figure 3 The distribution is 300.

[0169] At box 1604, process 1600 includes sampling a first region of the image using a first sampling grid. At box 1606, process 1600 includes sampling the first region of the image using a second sampling grid, which is different from the first sampling grid. In some cases, the arrangement of the sampling points of the second sampling grid differs from the arrangement of the sampling points of the first sampling grid. In some examples, the arrangement of the sampling points of the first sampling grid includes a first spacing between the sampling points of the first sampling grid. In such examples, the arrangement of the sampling points of the second sampling grid may include a second spacing between the sampling points of the second sampling grid. In some aspects, the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis. As described herein, the baseline axis (e.g., Figure 1 The baseline 112 shown is compared with the transmitter that transmits the light distribution (e.g., Figure 1 The transmitter 102) and the receiver that captures the image (e.g., Figure 1 The receiver 108 is associated with this. In some examples, the total number of sampling points in the second sampling grid differs from the total number of sampling points in the first sampling grid. In some cases, the first sampling grid is an isotropic sampling grid, while the second sampling grid is an anisotropic sampling grid.

[0170] At block 1608, process 1600 includes determining a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid. For example, as described herein, process 1600 may include applying the first sampling grid to a first region of an image and then calculating or determining the first confidence value. Process 1600 may also include applying a second sampling grid to a first region of an image and then calculating or determining the second confidence value. In some examples, process 1600 may determine the first confidence value associated with the first sampling grid at least in part by determining the first confidence value of a first image sample. In some examples, process 1600 may determine the second confidence value associated with the second sampling grid at least in part by determining the second confidence value of a second image sample.

[0171] At box 1610, process 1600 includes selecting the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value. In some examples, process 1600 may select the first sampling grid to determine the first depth value for the first region at least in part by selecting a first image sample based on the first confidence value being greater than the second confidence value.

[0172] In some examples, process 1600 may include determining a first image sample based on sampling a first region of the image using the first sampling grid. Process 1600 may also include determining a first depth value for the first region based on the first image sample. In some cases, process 1600 may include identifying an array of light distributions in the first region based on the first image sample (e.g., Figure 3 The first codeword is located in either the primitive array 302 or the diffraction array 304 of the light distribution 300. Process 1600 may include determining a first disparity based on the position of the first codeword in the array, wherein the determination of the first depth value is based on the first disparity. In some aspects, process 1600 may include sampling a second region of the image using a third sampling grid to generate a second image sample. Process 1600 may include determining a second depth value based on the second image sample. Process 1600 may continue in such a process to determine any number of depth values ​​for the received image.

[0173] In some examples, process 1600 may include determining a first image sample based on sampling a first region of the image using a first sampling grid. Process 1600 may include determining a second image sample based on sampling a first region of the image using a second sampling grid. In some cases, process 1600 may include comparing the first image sample with the second image sample. Process 1600 may include selecting a first image sample to be used to determine the first depth value based on comparing the first image sample with the second image sample.

[0174] In some cases, process 1600 may include generating a depth map based on the image. For example, the depth map may include multiple depth values, including a first depth value. The multiple depth values ​​indicate one or more depths of one or more objects in the scene captured in the image.

[0175] In some examples, the processes described herein (e.g., process 900, process 1200, process 1600, and / or other processes described herein) may be executed by a computing device or apparatus. Figure 4 Equipment 400 Figure 1 The active depth sensing system 100 (e.g., in or implemented by device 400) and / or other devices or systems configured to perform the operation of process 900, process 1200, process 1600 and / or other processes described herein shall be executed.

[0176] Computing devices may include any suitable device, such as mobile devices (e.g., mobile phones), desktop computing devices, tablet computing devices, extended reality devices (e.g., virtual reality (VR) headsets such as head-mounted displays (HMDs), augmented reality (AR) headsets such as HMDs, AR glasses, or other wearable devices), wearable devices (e.g., connected watches or smartwatches), server computers, vehicles or vehicle computing devices, robotic devices, televisions, and / or any other computing device having the resource capability to perform the processes described herein (including processes 900, 1200, and 1600). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components that may be configured to perform one or more of the operations of the processes described herein. In some examples, the computing device may include a receiver or sensor configured to capture images. In some cases, the computing device may include a transmitter configured to transmit a light distribution. For example, the transmitter may be along a baseline axis (e.g., Figure 1The baseline 112) is separated from the receiver by a baseline distance. In some examples, the computing device may include one or more signal processors configured to process the image and then decode the processed image. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP)-based data or other types of data. In some aspects, the device or computing device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0177] Components of a computing device may be implemented in a circuit. For example, the component may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry); and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0178] Processes 900, 1200, and 1600 are shown as logic flowcharts, whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, the operation represents a computer-executable instruction stored on one or more computer-readable storage media that performs the described operation when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and the process can be implemented in any order and / or in any combination of the described operations.

[0179] Furthermore, processes 900, 1200, 1600, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is jointly executed on one or more processors, executed by hardware, or a combination thereof. As mentioned above, this code may be stored, for example, in the form of a computer program comprising multiple instructions executable by one or more processors, on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium may be non-transitory.

[0180] Unless specifically described as being implemented in a particular manner, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the technology may be at least partially contained in a non-transitory processor-readable storage medium (such as...) Figure 4 The example device 400 (memory 406) implements instructions that, when executed by a processor (or signal processor or another suitable component), cause the device to perform one or more of the methods described above. A non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0181] Non-volatile processor-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Alternatively or additionally, this technology may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures, and can be accessed, read, and / or executed by a computer or other processor.

[0182] The various illustrative logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be generated by one or more processors (such as...) Figure 4 The processor (or signal processor 412) in example device 400 executes the commands. This type of processor may include, but is not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. As used herein, the term "processor" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured as described herein. Furthermore, the techniques may be fully implemented in one or more circuit or logic elements. A general-purpose processor may be a microprocessor, but optionally, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0183] Although this disclosure illustrates illustrative aspects, it should be noted that various changes and modifications may be made herein without departing from the scope of the appended claims. For example, while two sampling grids may be described in some examples, any suitable number of sampling grids may be used to perform aspects of this disclosure. Furthermore, while two regions of an image may be described for sampling and attempting to determine depth values ​​or other measurements (such as parallax, signature, confidence values, etc.), any number of regions of an image may be sampled. Additionally, unless explicitly stated otherwise, the functions, steps, or actions of the method claims according to the aspects described herein need not be performed in any particular order. For example, one or more steps of the described exemplary operations may be performed in any order and at any suitable frequency. Moreover, although elements or components may be described or claimed in the singular, the plural form is conceivable unless explicitly limited to the singular.

[0184] For clarity, in some cases, this technology may be represented as including individual functional blocks, which include devices, device components, steps, or routines comprising a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0185] The various embodiments described above can be presented as processes or methods, depicted as flowcharts, data flow diagrams, structural diagrams, or block diagrams. While a flowchart can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process can correspond to a method, function, program, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0186] The processes and methods according to the examples above can be implemented using computer-executable instructions stored in or available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. The portion may be accessible via a network of the computer resources used. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include hard disks, optical disks, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.

[0187] Apparatus implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments for performing necessary tasks (e.g., computer program products) may be stored on a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small-form-factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or expansion cards. By further example, this functionality may also be implemented on a circuit board between different chips or different processes executed in a single chip.

[0188] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary components for providing the functionality described in this disclosure.

[0189] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application have been described in detail herein, it should be understood that the inventive concept may be implemented and employed in other ways, and the appended claims are intended to be construed as including such variations other than those limited by the prior art. Various features and aspects of the above-described applications may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0190] Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively.

[0191] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform operations, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform operations, or any combination thereof.

[0192] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0193] The claim language or other language that states "at least one" and / or "one or more" in a set indicates that one or more members of the set (in any combination) satisfy the claim. For example, the claim language stating "at least one of A and B" and / or "at least one of A or B" means A, B, or A and B. In another example, the claim language stating "at least one of A, B, and C" and / or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. "At least one" and / or "one or more" in a set does not limit the set to items listed in that set. For example, the claim language stating "at least one of A and B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0194] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0195] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication handheld terminals, or multi-purpose integrated circuit devices, including applications in wireless communication handheld terminals and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the technology can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. The computer-readable medium can include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Alternatively or concurrently, the technology may be implemented at least in part by a computer-readable communication medium (such as a propagating signal or wave) that carries or transmits code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0196] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding or incorporated into a combined video encoder-decoder (CODEC).

[0197] The illustrative aspects of this disclosure include:

[0198] Aspect 1: A device for active depth sensing, comprising: a memory; and one or more processors configured to: receive an image, the image including one or more reflections of a light distribution; sample a first region of the image using a first sampling grid; sample the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and select the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value.

[0199] Aspect 2: The device according to aspect 1, wherein the light distribution is a light spot distribution.

[0200] Aspect 3: The device according to any one of Aspect 1 or 2, wherein the one or more processors are configured to: determine a first image sample based on sampling a first region of the image using the first sampling grid; and determine a first depth value of the first region based on the first image sample.

[0201] Aspect 4: The device according to aspect 3, wherein the one or more processors are further configured to: identify a first codeword in an array of light distributions in the first region based on the first image sample; and determine a first parallax based on the position of the first codeword in the array, wherein the determination of the first depth value is based on the first parallax.

[0202] Aspect 5: The device according to any one of Aspect 3 or 4, wherein the one or more processors are configured to: sample a second region of the image using a third sampling grid to generate a second image sample; and determine a second depth value based on the second image sample.

[0203] Aspect 6: The device according to any one of Aspects 1 to 5, wherein the arrangement of the sampling points of the second sampling grid is different from the arrangement of the sampling points of the first sampling grid.

[0204] Aspect 7: The device according to aspect 6, wherein the arrangement of the sampling points of the first sampling grid includes a first spacing between the sampling points of the first sampling grid, and wherein the arrangement of the sampling points of the second sampling grid includes a second spacing between the sampling points of the second sampling grid.

[0205] Aspect 8: The device according to aspect 7, wherein the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis, the baseline axis being associated with a transmitter that transmits the light distribution and a receiver that captures the image.

[0206] Aspect 9: The device according to any one of Aspects 1 to 8, wherein the total number of sampling points of the second sampling grid is different from the total number of sampling points of the first sampling grid.

[0207] Aspect 10: The device according to any one of Aspects 1 to 9, wherein the first sampling grid is an isotropic sampling grid and the second sampling grid is an anisotropic sampling grid.

[0208] Aspect 11: The device according to any one of aspects 1 to 10, wherein the one or more processors are further configured to: determine a first image sample based on sampling a first region of the image using the first sampling grid; determine a second image sample based on sampling the first region of the image using the second sampling grid; compare the first image sample with the second image sample; and select the first image sample to be used to determine the first depth value based on comparing the first image sample with the second image sample.

[0209] Aspect 12: The apparatus according to aspect 11, wherein: in order to determine a first confidence value associated with the first sampling grid, the one or more processors are configured to determine a first confidence value of the first image sample; in order to determine a second confidence value associated with the second sampling grid, the one or more processors are configured to determine a second confidence value of the second image sample; and in order to select the first sampling grid for determining a first depth value of the first region, the one or more processors are configured to select the first image sample based on the first confidence value being greater than the second confidence value.

[0210] Aspect 13: The device according to any one of aspects 1 to 12 further includes a receiver configured to capture the image.

[0211] Aspect 14: The device according to aspect 13 further includes a transmitter configured to transmit the light distribution, wherein the transmitter and the receiver are separated by a baseline distance along a baseline axis.

[0212] Aspect 15: The device according to any one of aspects 1 to 14 further includes one or more signal processors configured to process the image and then decode the processed image by the one or more processors.

[0213] Aspect 16: The device according to any one of Aspects 1 to 15, wherein the one or more processors are configured to generate a depth map based on the image, wherein the depth map includes a plurality of depth values, the plurality of depth values ​​including a first depth value, and wherein the plurality of depth values ​​indicate one or more depths of one or more objects in a scene captured in the image.

[0214] Aspect 17: A method for active depth sensing, comprising: receiving an image including one or more reflections of a light distribution; sampling a first region of the image using a first sampling grid; sampling the first region of the image using a second sampling grid, the second sampling grid being different from the first sampling grid; determining a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid; and selecting the first sampling grid to determine a first depth value for the first region based on the first confidence value being greater than the second confidence value.

[0215] Aspect 18: The method according to aspect 17, wherein the light distribution is a light spot distribution.

[0216] Aspect 19: The method according to any one of Aspects 17 or 18 further includes: determining a first image sample based on sampling a first region of the image using the first sampling grid; and determining a first depth value of the first region based on the first image sample.

[0217] Aspect 20: The method according to aspect 19 further includes: identifying a first codeword in an array of light distributions in the first region based on the first image sample; and determining a first parallax based on the position of the first codeword in the array, wherein the determination of the first depth value is based on the first parallax.

[0218] Aspect 21: The method according to any one of Aspects 19 or 20 further includes: sampling a second region of the image using a third sampling grid to generate a second image sample; and determining a second depth value based on the second image sample.

[0219] Aspect 22: The method according to any one of aspects 17 to 21, wherein the arrangement of the sampling points of the second sampling grid is different from the arrangement of the sampling points of the first sampling grid.

[0220] Aspect 23: According to the method of aspect 22, the arrangement of the sampling points of the first sampling grid includes a first spacing between the sampling points of the first sampling grid, and the arrangement of the sampling points of the second sampling grid includes a second spacing between the sampling points of the second sampling grid.

[0221] Aspect 24: According to the method of aspect 23, wherein the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis, the baseline axis being associated with a transmitter that transmits the light distribution and a receiver that captures the image.

[0222] Aspect 25: The method according to any one of aspects 17 to 24, wherein the total number of sampling points of the second sampling grid is different from the total number of sampling points of the first sampling grid.

[0223] Aspect 26: The method according to any one of Aspects 17 to 25, wherein the first sampling grid is an isotropic sampling grid and the second sampling grid is an anisotropic sampling grid.

[0224] Aspect 27: The method according to any one of aspects 17 to 26 further includes: determining a first image sample based on sampling a first region of the image using the first sampling grid; determining a second image sample based on sampling the first region of the image using the second sampling grid; comparing the first image sample with the second image sample; and selecting the first image sample to be used to determine the first depth value based on comparing the first image sample with the second image sample.

[0225] Aspect 28: The method according to aspect 27, wherein: determining a first confidence value associated with the first sampling grid includes determining a first confidence value of the first image sample; determining a second confidence value associated with the second sampling grid includes determining a second confidence value of the second image sample; and selecting the first sampling grid for determining a first depth value of the first region includes selecting the first image sample based on the first confidence value being greater than the second confidence value.

[0226] Aspect 29: The method according to any one of aspects 17 to 28 further includes capturing the image using a receiver.

[0227] Aspect 30: The method according to aspect 29 further includes transmitting the light distribution using a transmitter, wherein the transmitter and the receiver are separated by a baseline distance along a baseline axis.

[0228] Aspect 31: The method according to any one of aspects 17 to 30 further includes using one or more signal processors to process the image and then decoding the processed image.

[0229] Aspect 32: The method according to any one of aspects 17 to 31 further includes generating a depth map based on the image, wherein the depth map includes a plurality of depth values, the plurality of depth values ​​including a first depth value, and wherein the plurality of depth values ​​indicate one or more depths of one or more objects in a scene captured in the image.

[0230] Aspect 33: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the one or more processors, when executed, to perform any one of the operations of aspects 1 to 32.

[0231] Aspect 34: An apparatus comprising components for performing any of the operations described in aspects 1 to 32.

Claims

1. A device for active depth sensing, comprising: Memory; as well as One or more processors, said one or more processors being configured to: Receive an image, the image including one or more reflections of a light distribution; The first region of the image is sampled using the first sampling grid; A first region of the image is sampled using a second sampling grid, which is different from the first sampling grid; Determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid, wherein the first confidence value and the second confidence value are measures or functions indicating the confidence in the first sampling grid and the second sampling grid, respectively; as well as Based on the fact that the first confidence value is greater than the second confidence value, the first sampling grid is selected to determine the first depth value of the first region.

2. The device according to claim 1, wherein the light distribution is a light spot distribution.

3. The device of claim 1, wherein the one or more processors are configured to: A first image sample is determined based on sampling a first region of the image using the first sampling grid; and The first depth value of the first region is determined based on the first image sample.

4. The device of claim 3, wherein the one or more processors are further configured to: Based on the first image sample, identify the first codeword in the array of light distribution in the first region; and A first disparity is determined based on the position of the first codeword in the array, wherein the first depth value is determined based on the first disparity.

5. The device of claim 4, wherein the one or more processors are configured to: A second image sample is generated by sampling a second region of the image using a third sampling grid; and The second depth value is determined based on the second image sample.

6. The device according to claim 1, wherein the arrangement of sampling points in the second sampling grid is different from the arrangement of sampling points in the first sampling grid.

7. The device of claim 6, wherein the arrangement of the sampling points of the first sampling grid includes a first spacing between the sampling points of the first sampling grid, and wherein the arrangement of the sampling points of the second sampling grid includes a second spacing between the sampling points of the second sampling grid.

8. The device of claim 7, wherein the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis, the baseline axis being associated with a transmitter transmitting the light distribution and a receiver capturing the image.

9. The device according to claim 1, wherein the total number of sampling points in the second sampling grid is different from the total number of sampling points in the first sampling grid.

10. The device of claim 1, wherein the first sampling grid is an isotropic sampling grid and the second sampling grid is an anisotropic sampling grid.

11. The device of claim 1, wherein the one or more processors are further configured to: The first image sample is determined by sampling a first region of the image using the first sampling grid. The second image sample is determined based on sampling a first region of the image using the second sampling grid; Compare the first image sample with the second image sample; as well as Based on comparing the first image sample with the second image sample, the first image sample is selected to be used to determine the first depth value.

12. The device according to claim 11, wherein: In order to determine a first confidence value associated with the first sampling grid, the one or more processors are configured to determine a first confidence value for the first image sample; In order to determine a second confidence value associated with the second sampling grid, the one or more processors are configured to determine a second confidence value for the second image sample; as well as In order to select the first sampling grid for determining the first depth value of the first region, the one or more processors are configured to select the first image sample based on the first confidence value being greater than the second confidence value.

13. The device of claim 1, further comprising a receiver configured to capture the image.

14. The device of claim 13, further comprising a transmitter configured to transmit the light distribution, wherein the transmitter and the receiver are separated by a baseline distance along a baseline axis.

15. The device of claim 1, further comprising one or more signal processors configured to process the image and then decode the processed image by the one or more processors.

16. The device of claim 1, wherein the one or more processors are configured to generate a depth map based on the image, wherein the depth map includes a plurality of depth values, the plurality of depth values ​​including the first depth value, and wherein the plurality of depth values ​​indicate one or more depths of one or more objects in a scene captured in the image.

17. A method for active depth sensing, comprising: Receive an image, the image including one or more reflections of a light distribution; The first region of the image is sampled using the first sampling grid; A first region of the image is sampled using a second sampling grid, which is different from the first sampling grid; Determine a first confidence value associated with the first sampling grid and a second confidence value associated with the second sampling grid, wherein the first confidence value and the second confidence value are measures or functions indicating the confidence in the first sampling grid and the second sampling grid, respectively; as well as Based on the fact that the first confidence value is greater than the second confidence value, the first sampling grid is selected to determine the first depth value of the first region.

18. The method of claim 17, wherein the light distribution is a light spot distribution.

19. The method of claim 17, further comprising: The first image sample is determined by sampling a first region of the image using the first sampling grid. as well as The first depth value of the first region is determined based on the first image sample.

20. The method of claim 19, further comprising: Based on the first image sample, identify the first codeword in the array of light distribution in the first region; as well as A first disparity is determined based on the position of the first codeword in the array, wherein the first depth value is determined based on the first disparity.

21. The method of claim 20, further comprising: A second image sample is generated by sampling a second region of the image using a third sampling grid, wherein the third sampling grid is different from at least one of the first sampling grid and the second sampling grid; as well as The second depth value is determined based on the second image sample.

22. The method of claim 17, wherein the arrangement of the sampling points of the second sampling grid is different from the arrangement of the sampling points of the first sampling grid.

23. The method of claim 22, wherein the arrangement of the sampling points of the first sampling grid includes a first spacing between the sampling points of the first sampling grid, and wherein the arrangement of the sampling points of the second sampling grid includes a second spacing between the sampling points of the second sampling grid.

24. The method of claim 23, wherein the first spacing and the second spacing are along a baseline axis and an axis orthogonal to the baseline axis, the baseline axis being associated with a transmitter transmitting the light distribution and a receiver capturing the image.

25. The method of claim 17, wherein the total number of sampling points in the second sampling grid is different from the total number of sampling points in the first sampling grid.

26. The method of claim 17, wherein the first sampling grid is an isotropic sampling grid and the second sampling grid is an anisotropic sampling grid.

27. The method of claim 17, further comprising: The first image sample is determined by sampling a first region of the image using the first sampling grid. The second image sample is determined based on sampling a first region of the image using the second sampling grid; Compare the first image sample with the second image sample; as well as Based on comparing the first image sample with the second image sample, the first image sample is selected to be used to determine the first depth value.

28. The method of claim 27, wherein: Determining a first confidence value associated with the first sampling grid includes determining a first confidence value for the first image sample; Determining the second confidence value associated with the second sampling grid includes determining the second confidence value of the second image sample; and Selecting the first sampling grid to determine the first depth value of the first region includes selecting the first image sample based on the first confidence value being greater than the second confidence value.

29. The method of claim 17, further comprising: The light distribution is transmitted.

30. The method of claim 17, further comprising: A depth map is generated based on the image, wherein the depth map includes a plurality of depth values, the plurality of depth values ​​including the first depth value, and wherein the plurality of depth values ​​indicate one or more depths of one or more objects in the scene captured in the image.

31. A computer-readable medium having program code recorded thereon, wherein, The program code may be executed by one or more processors of an active depth sensing device to cause the processors to perform the method described in any one of claims 17-30.

32. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 17-30.

Images