Polarized multi-channel sensor with sparse polarized pixels

Abstract

A polarized multi-channel sensor with sparse polarized pixels is described. The sensor has a plurality of macro pixels. Each macro pixel includes a corresponding plurality of pixels configured to detect an intensity of light in different color channels, and one or more polarization pixels. Polarizing pixels may be configured to detect the intensity of light linearly polarized to a particular orientation in the same color channel. Where a plurality of polarized pixels are present within the macro pixel, the plurality of polarized pixels may be configured to detect the intensity of light in the same color channel but linearly polarized in different orientations. Further, the number of polarized pixels within the macro pixel is sparse compared to the total number of pixels within the macro pixel. The detected intensity information from the polarized pixels may be used to determine one or more attributes of the object.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority and benefit of U.S. Patent Application No. 18 / 520,947, filed on November 28, 2023, the entire content of which is incorporated herein by reference. Technical field

[0003] The present disclosure generally relates to polarization sensors, and more particularly, to polarization multi - channel sensors having sparse polarization pixels. Background art

[0004] The polarization of light can be used to determine useful information about the shape and material properties of an object. Conventional color polarization image sensors use a wire grid matrix filter that covers the entire color sensor (e.g., a sensor covered with a Bayer filter). This structure is not ideal because it reduces the optical power signal (the wire grid reflects about 50% of the light), for example, and this structure reduces the resolution. Summary of the invention

[0005] Described herein is a polarization multi - channel sensor (sensor) having sparse polarization pixels. The sensor has a plurality of macro - pixels. Light from a local area including an object can be incident on some or all of the macro - pixels. Each macro - pixel includes a corresponding plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, blue), and one or more polarization pixels. The polarization pixels can be configured to detect the intensity of linearly polarized light in a specific orientation in the same color channel (e.g., green). In the case where there are multiple polarization pixels within a macro - pixel, the multiple polarization pixels can be configured to detect the intensity of linearly polarized light in different orientations in the same color channel. Additionally, the number of polarization pixels within a macro - pixel is relatively sparse compared to the total number of pixels within the macro - pixel. A controller can use the detected intensity information from pixels having the same color channel as the polarization pixels and the detected intensity information from the polarization pixels to determine one or more properties of the object (e.g., position, shape, etc.).

[0006] In some embodiments, a sensor is described. The sensor includes a plurality of macro pixels. Each macro pixel includes a plurality of pixels and a first polarization pixel. The plurality of pixels are configured to detect the intensity of light in different color channels. The plurality of pixels includes a first pixel configured to detect the intensity of light in a first color channel among the different color channels. The first polarization pixel is configured to detect the intensity of light linearly polarized in a first direction in the first color channel. The light incident on one of the plurality of macro pixels is from an object, and a controller may be configured to determine an attribute of the object based in part on a first intensity signal of the first pixel in the macro pixel and a first polarization signal of the first polarization pixel in the macro pixel.

[0007] In some embodiments, a system is described. The system may be a polarization camera assembly. The system includes a camera and a controller. The camera includes a plurality of macro pixels that receive light from a local area including an object. One of the plurality of macro pixels includes: a plurality of pixels and a first polarization pixel. The plurality of pixels are configured to detect the intensity of light in different color channels. The plurality of pixels includes a first pixel configured to detect the intensity of light in a first color channel (among the different color channels) as a first intensity signal. The first polarization pixel may be configured to detect the intensity of light linearly polarized in a first direction in the first color channel as a first polarization signal. The controller may be configured to determine an attribute of the object based in part on the first intensity signal and the first polarization signal.

[0008] In some embodiments, a sensor is described. The sensor includes a plurality of macro pixels. Each macro pixel includes a plurality of pixels and a first polarization pixel. The plurality of pixels may be configured to detect the intensity of light in different color channels. The first polarization pixel may be configured to detect the intensity of light linearly polarized in a first direction in a first color channel among the different color channels. The light incident on one of the plurality of macro pixels is from an object, and a controller may be configured to determine an attribute of the object based in part on a first polarization signal of the first polarization pixel in the macro pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a perspective view of a headset implemented as a glasses device according to one or more embodiments.

[0010] Figure 2 is a block diagram of a polarization camera assembly viewing a local area according to one or more embodiments.

[0011] Figure 3AAn example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of four different orientations.

[0012] Figure 3B An example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where according to one or more embodiments, each macro-pixel is configured to detect linearly polarized light of four different orientations.

[0013] Figure 3C An example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where according to one or more embodiments, each macro-pixel is configured to detect linearly polarized light of four different orientations.

[0014] Figure 4A An example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of three different orientations.

[0015] Figure 4B An example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of three different orientations.

[0016] Figure 4C An example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of three different orientations.

[0017] Figure 5A An example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of two different orientations.

[0018] Figure 5B An example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of two different orientations.

[0019] Figure 5C An example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro-pixel is configured to detect linearly polarized light of two different orientations.

[0020] Figure 6An example polarization system according to one or more embodiments, the example polarization system using a single oriented polarized light and a polarization multi-channel sensor having sparse polarization pixels, wherein each macro-pixel is configured to detect a single oriented linearly polarized light.

[0021] Figure 7A An example polarization system according to one or more embodiments, the example polarization system using two oriented polarized lights and a polarization multi-channel sensor having sparse polarization pixels, wherein each macro-pixel is configured to detect two different oriented linearly polarized lights.

[0022] Figure 7B is Figure 7A the second layout of the sensor in

[0023] Figure 7C is Figure 7A the third layout of the sensor in

[0024] Figure 8A Illustrates an example operation of a metasurface that guides light according to polarization according to one or more embodiments.

[0025] Figure 8B An example first layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect four different oriented linearly polarized lights.

[0026] Figure 8C An example second layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect four different oriented linearly polarized lights.

[0027] Figure 9 An example layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect three different oriented linearly polarized lights.

[0028] Figure 10A An example first layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect two different oriented linearly polarized lights.

[0029] Figure 10B An example second layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect two different oriented linearly polarized lights.

[0030] Figure 11 A system including a head-mounted device according to one or more embodiments.

[0031] Each of the figures depicts various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. Detailed Description

[0032] A polarization multi-channel sensor (sensor) with sparse polarization pixels is described herein. The sensor is part of a camera of a polarization camera assembly (PCA). The sensor has a plurality of macro-pixels. Each macro-pixel includes a corresponding plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, blue), and one or more polarization pixels (e.g., wire grid or metasurface). The polarization pixels in each macro-pixel can be configured to detect the intensity of linearly polarized light in the same color channel (e.g., green). And in the case where there are multiple polarization pixels within a macro-pixel, the polarization pixels can be configured to detect the intensity of linearly polarized light in the same color channel but with different orientations. For example, a macro-pixel can include two polarization pixels, one polarization pixel configured to detect linearly polarized light in a first orientation, and the other polarization pixel configured to detect linearly polarized light in a second orientation, where the angle between the first orientation and the second orientation is 45 degrees. A controller of the PCA can use the detected intensity information from pixels having the same color channel as the color channel of the polarization pixels, and the detected intensity information from the polarization pixels, to determine one or more attributes (e.g., position, shape, etc.) of the object.

[0033] Note that the number of polarization pixels within a macro-pixel is relatively sparse compared to the total number of pixels within the macro-pixel. Sparse as used herein means that no more than 50% of the pixels within a macro-pixel are polarization pixels. Additionally, since the human eye is most sensitive to green light, the green channel typically includes the most pixels, and for a given macro-pixel, the polarization pixels can replace some (or in some embodiments, all) of the green pixels among the plurality of green pixels. This sparsity in the number of polarization pixels helps the sensor to have a higher signal-to-noise ratio compared to a conventional polarization sensor where all pixels are polarization pixels (i.e., automatically losing half of the incident unpolarized light).

[0034] Embodiments of the present invention may include an artificial reality system or may be implemented in combination with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of the foregoing may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives a viewer a three-dimensional effect). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof that are used to create content in artificial reality and / or otherwise be used in artificial reality. An artificial reality system that provides artificial reality content may be implemented on various platforms, including wearable devices (e.g., head-mounted devices) connected to a host computer system, standalone wearable devices (e.g., head-mounted devices), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0035] Figure 1 is a perspective view of a head-mounted device 100 implemented as a glasses device according to one or more embodiments. In some embodiments, the glasses device is a near eye display (NED). Generally, the head-mounted device 100 may be worn on a user's face such that content (e.g., media content) is presented using a display component and / or an audio system. However, the head-mounted device 100 may also be used such that media content is presented to the user in a different manner. Examples of media content presented by the head-mounted device 100 include one or more images, videos, audio, or some combination thereof. The head-mounted device 100 includes a frame 110 and a PCA 140 and may include other components such as a display component (which includes one or more display elements 120), an audio system, and a position sensor 190. Although Figure 1 shows an example location of the components of the head-mounted device 100 on the head-mounted device 100, these components may be located at other locations on the head-mounted device 100, on a peripheral device paired with the head-mounted device 100, or some combination thereof. Similarly, there may be more components or fewer components on the head-mounted device 100 than Figure 1 shown.

[0036] The frame 110 holds other components of the head-mounted device 100. The frame 110 includes a front component and end components (e.g., temple arms), the front component holding the one or more display elements 120, and the end components being attached to the user's head. The front component of the frame 110 spans across the top of the user's nose. The length of the end components can be adjustable (e.g., adjustable temple arm length) to fit different users. The end components may also include portions that curve behind the user's ears (e.g., temple tips, earpieces).

[0037] The one or more display elements 120 provide light to a user wearing the head-mounted device 100. As shown, the head-mounted device includes one display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light that is provided to the eyebox of the head-mounted device 100. The eyebox is the location in the space occupied by the user's eyes when wearing the head-mounted device 100. For example, the display element 120 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is internally coupled into the one or more waveguides, and the one or more waveguides output light in a manner such that pupil replication exists in the eyebox of the head-mounted device 100. One or more diffraction gratings can be used to accomplish the internal coupling of light into the one or more waveguides and / or the external coupling of light from the one or more waveguides. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a mirror, etc.) that scans the light when the light from the light source is internally coupled into the one or more waveguides. Note that in some embodiments, one or both of the two display elements 120 are opaque and do not transmit light from a local area around the head-mounted device 100. The local area is the area around the head-mounted device 100. For example, the local area can be the room in which the user wearing the head-mounted device 100 is located, or the user wearing the head-mounted device 100 can be outdoors, and the local area is the outdoor area. In this context, the head-mounted device 100 generates VR content. Alternatively, in some embodiments, one or both of the two display elements 120 are at least partially transparent, such that light from the local area can be combined with light from the one or more display elements to generate AR content and / or MR content.

[0038] In some embodiments, the display element 120 does not generate image light. Instead, the display element 120 is a lens that transmits light from a local area to the eye zone. For example, one or both of the two display elements 120 can be an uncorrected (non-prescription) lens or a prescription lens (e.g., a single vision lens, a bifocal lens, a trifocal lens, or a progressive lens) to help correct a user's vision defect. In some embodiments, the display element 120 can be polarized and / or colored to protect the user's eyes from sun damage.

[0039] The PCA 140 determines one or more attributes (e.g., position, shape, etc.) of an object within the local area of the PCA 140. The following describes the PCA 140 in detail with reference to Figures 2 to 11 .

[0040] The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller. However, in other embodiments, the audio system can include different components and / or additional components. Similarly, in some cases, the functions described with respect to the components in the audio system can be distributed among multiple components in a manner different from that described herein. For example, some or all of the functions of the controller can be performed by a remote server.

[0041] The transducer array presents sound to the user. The transducer array includes a plurality of transducers. The transducers can be speakers 160 or tissue transducers (e.g., bone conduction transducers or cartilage conduction transducers). Although the speaker 160 is shown outside the frame 110, the speaker 160 can be enclosed within the frame 110. In some embodiments, the head-mounted device 100 includes a speaker array instead of a separate speaker for each ear, the speaker array including a plurality of speakers integrated into the frame 110 to improve the directivity of the presented audio content. The tissue transducer is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to generate sound. The number and / or position of the transducers can be different from Figure 1 the number and / or position shown in

[0042] The sensor array detects sound within the local area of the head-mounted device 100. The sensor array includes a plurality of acoustic sensors 180. The acoustic sensors 180 collect sound emitted from one or more sound sources in the local area (e.g., a room). Each acoustic sensor is configured to detect sound and convert the detected sound into an electronic format (analog or digital). The acoustic sensors 180 can be acoustic wave sensors, microphones, sound transducers, or similar sensors suitable for detecting sound.

[0043] In some embodiments, one or more acoustic sensors 180 may be placed in the ear canal of each ear (e.g., as a stereo microphone). In some embodiments, the acoustic sensors 180 may be placed on the outer surface of the head-mounted device 100, on the inner surface of the head-mounted device 100, separate from the head-mounted device 100 (e.g., as part of some other device), or some combination thereof. The number and / or location of the acoustic sensors 180 may be different from Figure 1 the number and / or location shown in. For example, the number of acoustic detection locations may be increased to increase the amount of audio information collected and the sensitivity and / or accuracy of the information. The acoustic detection locations may be oriented such that the microphones can detect sounds in a wide range of directions around the user wearing the head-mounted device 100.

[0044] The audio controller processes information from the sensor array that describes the sounds detected by the sensor array. The audio controller may include a processor and a computer-readable storage medium. The audio controller may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and / or a head-related transfer function), track the location of a sound source, form a beam in the direction of the sound source, classify the sound source, generate a sound filter for the speaker 160, or some combination thereof.

[0045] The position sensor 190 generates one or more measurement signals in response to the movement of the head-mounted device 100. The position sensor 190 may be located on a portion of the frame 110 of the head-mounted device 100. The position sensor 190 may include an inertial measurement unit (IMU). Examples of the position sensor 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting movement, a type of sensor for error correction of the IMU, or some combination thereof. The position sensor 190 may be located outside the IMU, inside the IMU, or some combination thereof.

[0046] In some embodiments, the head-mounted device 100 may provide simultaneous localization and mapping (SLAM) for the localization of the head-mounted device 100 and the update of the model of the local area. For example, the head-mounted device 100 may include a passive camera assembly that generates color image data. The passive camera assembly may include one or more RGB cameras that capture images of part or all of the local area. The images captured by the PCA and the position information determined by the PCA 140 may be used to determine the parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the position sensor 190 tracks the localization (e.g., position and orientation) of the head-mounted device 100 within the room. Additional details regarding the components of the head-mounted device 100 are discussed below in conjunction with Figure 2 to discuss additional details regarding the components of the head-mounted device 100.

[0047] Figure 2 is a block diagram of the PCA200 viewing the local area 205. The PCA 140 is an embodiment of the PCA200. In Figure 2 the embodiment of, the PCA200 includes a camera 220, a controller 230, and in some embodiments, also includes a polarized light projector 240. Some embodiments of the PCA200 have components different from those described herein. Similarly, in some cases, the various functions may be distributed among the components in a manner different from that described herein.

[0048] The camera 220 detects light from the local area 205. The camera 220 includes a polarized multi-channel sensor with sparse polarized pixels (the "sensor") and may include one or more optical elements (e.g., lenses). The one or more optical elements may, for example, focus the light from the local area 205 onto the sensor.

[0049] The sensor detects intensity information of the light from the local area 205. The sensor includes a plurality of macro pixels. Each macro pixel includes a corresponding plurality of pixels and one or more polarized pixels, and the plurality of pixels are configured to detect the intensity of light in different color channels. These color channels may be, for example, red, green, and blue. In other embodiments, these color channels may include some other color and / or correspond to some other color combination.

[0050] One or more polarized pixels in each macro pixel are configured to detect the intensity of linearly polarized light in the same color channel. For example, in some embodiments, one or more polarized pixels in each macro pixel are configured to detect the intensity of linearly polarized green light. In embodiments where there are multiple polarized pixels within a macro pixel, the multiple polarized pixels may be configured to detect the intensity of linearly polarized light in the same color channel but with different orientations. For a given sensor layout, the orientation of the linearly polarized light detected by one or more polarized pixels in each macro pixel is the same. The sensor may have different layouts. For example, the linearly polarized pixels are configured to detect linearly polarized light with 2, 3, or 4 different orientations. The following refers to, for example Figures 3A to 10B Examples describing various layouts are provided. A polarized pixel is a pixel in a specific color channel covered with a linear polarizer device. The linear polarizer can be, for example, a wire grid, a metasurface.

[0051] The polarized light projector 240 outputs linearly polarized light. The polarized light projector 240 includes one or more illuminators that emit linearly polarized light. In some embodiments, the one or more illuminators emit linearly polarized light in a single orientation (e.g., 0 degrees), which is the same for all of the one or more illuminators, and the orientation is aligned 90 degrees with respect to the polarized pixels on the camera 220. In other embodiments, the polarized light projector 240 emits linearly polarized light in a first orientation (0 degrees) (the first orientation is aligned 90 degrees with respect to some of the polarized pixels on the camera 220) and linearly polarized light in a second orientation (e.g., 45 degrees) (the second orientation is aligned 90 degrees with respect to some of the polarized pixels on the camera 220) in a time multiplexed manner and / or in a sequential manner. The polarized light projector 240 may include a first illuminator that emits light in the first orientation and a second illuminator that emits light in the second orientation. In other embodiments, a single illuminator can emit linearly polarized light with different orientations at different times.

[0052] The controller 230 determines one or more attributes of an object (e.g., object 250) within the local region 205 of the PCA 140. The attributes of the object can be, for example, shape, position, one or more material attributes (e.g., roughness, diffuse albedo, etc.) or some combination thereof. How the controller 230 processes information from the sensor and / or controls the polarized light projector 240 to determine one or more attributes of the object is partially based on the specific sensor layout and is discussed below with reference to, for example Figures 3A to 10B is discussed.

[0053] Figure 3AAn example first layout of a portion with sparse polarization pixels of a polarization multi-channel sensor according to one or more embodiments, where each macro pixel is configured to detect linearly polarized light in four different orientations. A portion of the sensor shown includes four macro pixels (e.g., macro pixel 310), and the layout of the pixels and polarization pixels of the four macro pixels is the same. For example, macro pixel 310 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), and polarization pixels 320A, 320B, 320C, and 320D (collectively referred to as polarization pixels 320). Polarization pixel 320A is configured to detect the intensity of light linearly polarized in a first direction (0 degrees) in the green channel. Polarization pixel 320B is configured to detect the intensity of light linearly polarized in a second direction (-π / 4 or -45 degrees) in the green channel. Polarization pixel 320C is configured to detect the intensity of light linearly polarized in a third direction (π / 2 or 90 degrees) in the green channel. Polarization pixel 320D is configured to detect the intensity of light linearly polarized in a fourth direction (π / 4 or 45 degrees) in the green channel. As shown, the respective directions are shown in the subscripts of the polarization pixels in the figure.

[0054] A controller (e.g., controller 230) can use the detected intensities from these polarization pixels 320 to determine the Stokes parameters S 0 , S 1 and S 2 . S 0 is the following Stokes parameter: which refers to the total intensity of light. S 1 is the following Stokes parameter: which refers to the horizontal preference of linear polarization relative to the vertical preference, e.g., the difference between the amount of light polarized along 0 degrees and the amount of light polarized at 90 degrees. S 2 is the following Stokes parameter: which refers to the diagonal preference of linear polarization relative to the anti-diagonal preference, e.g., the difference between the amount of light polarized along 45 degrees and the amount of light polarized along -45 degrees. The controller can determine S 0 , S 1 and S 2 by the following equations:

[0055]

[0056] S 1 = A G0 - A Gπ / 2 (2)

[0057] S 2 = A Gπ / 4 - A G-π / 4 (3)

[0058] where, A G0is the intensity (i.e., absorption) detected at polarized pixel 320A, A G-π / 4 is the intensity detected at polarized pixel 320B, A Gπ / 2 is the intensity detected at polarized pixel 320C, A Gπ / 4 is the intensity detected at polarized pixel 320D, and A G is the intensity (i.e., absorption) detected at an unpolarized green pixel in macro pixel 310 (i.e., not one of the polarized pixels 320). In this example, A can be obtained in four different pixels of each macro pixel G . In some embodiments, A is determined based on the value of the pixel closest to A G (e.g., by interpolation). The controller can use methods found in the art and solve for the material properties using Stokes parameters of other factors (e.g., shape). G

[0059] Note that with the layout of Figure 3A , the intensity of the detected green pixels (i.e., unpolarized green pixels) is not required to determine S 0 . However, the intensity of the detected unpolarized green pixels can still be used to maintain a green quantum efficiency (QE) higher than that of the other color channels (red and blue). Thus, the sensor can provide intensity information of the polarized pixels as if the intensity of these unpolarized green pixels were not provided. Additionally, an alternative method of obtaining A G is useful for interpolation purposes on the matrix. This is a significant improvement over traditional polarization sensors that cover all pixels of the sensor (which may lose approximately half of the incident signal on all color channels).

[0060] Figure 3B is an example second layout of a portion of a polarization multi-channel sensor with sparse polarized pixels according to one or more embodiments, where according to one or more embodiments, each macro pixel is configured to detect linearly polarized light in four different orientations. Except that each macro pixel (e.g., macro pixel 330) includes a smaller number of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), the portion of the sensor shown is substantially the same as the portion of the sensor in Figure 3A . The controller (e.g., controller 230) can use equations (1) through (3) as described above to determine the Stokes parameters S 0 , S 1 and S 2 .

[0061] Figure 3C ​is an example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels, where according to one or more embodiments, each macro pixel is configured to detect linearly polarized light in four different orientations. Except that each macro pixel (e.g., macro pixel 340) includes polarization pixels 320A, 320B, 320C, 320D separated from each other by at least one unpolarized pixel, a portion of the sensor shown is substantially the same as the portion of the sensor in Figure 3A The controller (e.g., controller 230) can use equations (1) to (3) as described above to determine the Stokes parameters S 0 , S 1 and S 2 .

[0062] Figure 4A is an example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels, where each macro pixel is configured to detect linearly polarized light in three different orientations. A portion of the sensor shown includes four macro pixels (e.g., macro pixel 410), and the layout of the pixels and polarization pixels of these four macro pixels is the same. For example, macro pixel 410 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), and polarization pixels 420A, 420B, and 420C (collectively referred to as polarization pixels 420). Polarization pixel 420A is configured to detect the intensity of light linearly polarized in a first direction (π / 8 or 22.5 degrees) in the green channel. Polarization pixel 420B is configured to detect the intensity of light linearly polarized in a second direction (-π / 8 or -22.5 degrees) in the green channel. Polarization pixel 420C is configured to detect the intensity of light linearly polarized in a third direction (3π / 8 or 67.5 degrees) in the green channel. As shown, the respective directions are shown in the subscripts of the polarization pixels in the figure.

[0063] The controller (e.g., controller 230) can use the intensities detected from these polarization pixels 420 to determine the Stokes parameters S 0 , S 1 and S 2 . The controller can determine S 0 , S 1 and S 2 through the following equations:

[0064] S 0 = A G = A G-π / 8 + A G3π / 8 (4)

[0065]

[0066] Wherein, A Gπ / 8 is the intensity detected at polarization pixel 420A, A G-π / 8 is the intensity detected at polarization pixel 420B, and A G3π / 8 is the intensity detected at polarization pixel 420C. Then, the controller can use methods found in the art to solve for the material properties using Stokes parameters.

[0067] Note that with the Figure 4A layout, the intensity of the detected green pixels (i.e., non-polarized green pixels) is not required to determine S 0 , as the detected intensity can be determined by Equation 4. However, this detected intensity can still be used to maintain a green QE that is higher than that of the other color channels (red and blue). Thus, the sensor can provide intensity information of the polarization pixels as if the intensity of these non-polarized green pixels were not provided. Additionally, an alternative method of obtaining A G is useful for interpolation purposes on the matrix. This represents a significant improvement over traditional polarization sensors that cover all pixels of the sensor (which may lose approximately half of the incident signal on all color channels).

[0068] Figure 4B is an example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro pixel is configured to detect linearly polarized light of three different orientations. Except that each macro pixel (e.g., macro pixel 430) includes a smaller number of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), a portion of the sensor shown is substantially the same as the Figure 4A portion of the sensor in (e.g., the portion containing polarization pixel 420). The controller (e.g., controller 230) can use Equations (4) to (6) as described above to determine the Stokes parameters S 0 , S 1 and S 2 .

[0069] Figure 4C is an example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where each macro pixel is configured to detect linearly polarized light of three different orientations. In this layout, a portion of the sensor shown is a single macro pixel 440. Except that each of the polarization pixels 420A, 420B, 420C, which are separated from each other by at least one non-polarized pixel, and only a single macro pixel (i.e., macro pixel 440) is shown, a portion of the sensor shown is the same as the Figure 4AThe controller (eg, controller 230) may determine the Stokes parameter S using equations (4) to (6) as described above. 0 , S 1 and S 2 .

[0070] Figure 5A It is an example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, wherein each macropixel is configured to detect linear polarized light of two different orientations. A portion of the sensor shown includes four macropixels (e.g., macropixel 510), and the layout of the pixels and polarization pixels of the four macropixels is the same. For example, macropixel 510 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), and polarization pixels 520A and 520B (collectively referred to as polarization pixels 520). Polarization pixel 520A is configured to detect the intensity of linearly polarized light in a first direction (π / 8 or 22.5 degrees) in a green channel. Polarization pixel 520B is configured to detect the intensity of linearly polarized light in a second direction (-π / 8 or -22.5 degrees) in a green channel. As shown in the figure, the various directions are shown in the subscripts of the polarization pixels in the figure.

[0071] A controller (eg, controller 230) may use the detected intensity from polarization pixel 520 to determine Stokes parameter S 0 , S 1 and S 2 The controller can determine S by the following equation 0 , S 1 and S 2 :

[0072] S 0 =A G (7)

[0073]

[0074] Among them, A Gπ / 8 is the intensity detected at polarization pixel 520A, and A G-π / 8 is the intensity detected at polarization pixel 520B. The controller can then use methods found in the art to solve for the material properties using the determined Stokes parameters.

[0075] Figure 5Bis an example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels, where each macro pixel is configured to detect linearly polarized light of two different orientations. Except that each macro pixel (e.g., macro pixel 530) includes a smaller number of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), a portion of the sensor shown is substantially the same as the portion of the sensor in Figure 5A (e.g., the portion including polarization pixel 520). A controller (e.g., controller 230) can use equations (7) to (9) as described above to determine the Stokes parameters S 0 、S 1 and S 2 .

[0076] Figure 5C is an example third layout of a portion of a polarization multi-channel sensor with sparse polarization pixels, where each macro pixel is configured to detect linearly polarized light of two different orientations. Except that each macro pixel (e.g., macro pixel 540) includes two polarization pixels 520A and two polarization pixels 520B separated from each other by at least one non-polarization pixel, a portion of the sensor shown is substantially the same as the portion of the sensor in Figure 5A . A controller (e.g., controller 230) can use equations (7) to (9) as described above to determine the Stokes parameters S 0 、S 1 and S 2 .

[0077] Note that in each of Figures 3A to 5C , a specific orientation of the polarization pixels is used. These orientations can be rotated by a common offset such that for a given layout, the relative differences in the orientations of the polarization pixels are maintained. Additionally, the above layouts can be used to determine S 0 、S 1 and S 2 such that they can be used with other factors (e.g., shape) to determine all material properties. Note that the layouts described above with reference to Figures 3A to 5C do not use a polarization light projector (e.g., polarization light projector 240) and instead only use the intensity of light (in a single color channel) detected at the sensor of the camera to determine the Stokes parameters, and then these Stokes parameters can be used to determine the material properties of an object in a local area.

[0078] In other embodiments, the PCA uses both a camera and a polarized light projector to determine the material properties of an object. These embodiments are based in part on properties associated with specular and diffuse reflection. Specular reflection occurs at a surface, e.g., at microfacets along the surface of the object. While the total amount of specular reflection is not intended to vary, its angular distribution is a function of roughness. Through a distribution function, roughness is the main material property affecting specular reflection. The distribution function describes the density of facets as a function of the zenith angle of the facet relative to the surface normal. The particularity of specular reflection is that it completely preserves the polarization of the light source after reflection. Thus, if the light source is polarized at the same orientation angle for any position on the object, the specular reflection is completely polarized. In contrast, diffuse reflection is the result of a certain amount of light propagating into the object and then being backscattered, shown as reflected light. The amount of this backscattering is called the diffuse albedo (between 0 and 1). This diffusely reflected light is depolarized inside the object, so this diffusely reflected light can only be polarized by transmitting it back through the air-object interface. For the direction perpendicular to the surface normal, this transmission is exactly unpolarized. Figures 6 to 7C Embodiments of the PCA are described that use a polarized light projector to determine different material properties of an object.

[0079] Figure 6 is an example polarization system according to one or more embodiments, the example polarization system using a single oriented polarized light and a polarization multi-channel sensor having sparse polarization pixels, where each macro-pixel is configured to detect a single oriented linearly polarized light. A portion of the shown sensor includes four macro-pixels (e.g., macro-pixel 610), and the layout of the pixels and polarization pixels of the four macro-pixels is the same. For example, macro-pixel 610 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), and polarization pixel 620. Polarization pixel 620 is configured to detect the intensity of light linearly polarized in a first direction (0 degrees) in the green channel. The sensor is integrated into a camera (e.g., camera 220) of the PCA (e.g., PCA200), and the PCA further includes a polarized light projector (e.g., polarized light projector 240). The polarized light projector is configured to emit light linearly polarized in a second direction (e.g., horizontal direction) orthogonal to the first direction.

[0080] The controller (e.g., controller 230) instructs the polarized light projector to illuminate a local area with polarized light. Multiple portions of the polarized light are reflected and / or scattered by an object within the local area and incident on the sensor.

[0081] The controller can be configured to determine the degree of linear polarization in the second direction of a given macro-pixel (e.g., 610) through the following equation:

[0082]

[0083] where A G = S 0 = S 0-sp + S 0-diff , S 0-sp is the specular reflection component of S 0 , and S 0-diff is the diffuse reflection component of S 0 , and A G0 is the intensity detected by the polarization pixel (e.g., 620) of the macro pixel.

[0084]

[0085] The controller can be configured to determine the degree of linear polarization (“DOLP highlight ”) at highlight for a given macro pixel (e.g., 610) through the following equation:

[0086]

[0087] DOLP highlight refers to the degree of linear polarization at maximum irradiance (characterized by the viewing direction being exactly orthogonal to the surface normal of the object). The controller uses the DOLP Horizontal value to determine material properties such as roughness and / or diffuse albedo, and S 0 can be used to determine the object position.

[0088] DOLP highlight (equal to the horizontal DOLP) is the ratio of specular irradiance to total irradiance (partial specular irradiance) and provides information about the ratio of diffuse albedo (“w diff ”) to the peak of the distribution function (peak of the distribution function at the starting angle) D 0 . The ratio of diffuse albedo to the peak of the distribution function can be obtained through the following equation:

[0089]

[0090] where n is the refractive index (e.g., 1.5). The distribution function can be obtained through one of the following: highlight sharpness, or the minimum value near the highlight of DOLP Horizontal . For example, the minimum value of DOLP Horizontal can be obtained through the following equation:

[0091]

[0092] where α is the angle, α minis the angle at which the corresponding function is at a minimum. The controller can use the above equations 13 and 14 to determine the distribution function (e.g., describing the object roughness) and the diffuse albedo of the color channel (e.g., green) in which the polarization pixel is located. Then, the controller can determine the diffuse albedo of these other color channels (e.g., blue and / or red) based on the relative intensities of these other color channels with respect to the color channel in which the polarization pixel is located.

[0093] The controller can determine the object position based in part on the determined material properties and the detected intensities. For example, the distance to the object ("d high ") can be calculated by the following equation:

[0094]

[0095] where S 0-highlight is the Stokes parameter S 0 at the highlight, and I light is the source power (e.g., the power of the light emitted by the polarization light projector 240). Thus, the controller is able to use the equations shown above to determine roughness, diffuse albedo, and position.

[0096] Figure 7A is an example polarization system according to one or more embodiments, the example polarization system using two oriented polarization lights and a polarization multi-channel sensor with sparse polarization pixels, where each macro-pixel is configured to detect two different oriented linearly polarized lights. A portion of the shown sensor includes four macro-pixels (e.g., macro-pixel 710), and the layout of the pixels and the polarization pixels of the four macro-pixels is the same. For example, macro-pixel 710 includes a plurality of pixels configured to detect the intensities of light in different color channels (e.g., red, green, and blue), and polarization pixels 720A and 720B (collectively referred to as polarization pixels 720). Polarization pixel 720A is configured to detect the intensity of the light linearly polarized in the first direction (0 degrees) in the green channel. Polarization pixel 720B is configured to detect the intensity of the light linearly polarized in the second direction (π / 4 or 45 degrees) in the green channel. The sensor is integrated into the camera (e.g., camera 220) of a PCA (e.g., PCA 200), and the PCA also includes a polarization light projector (e.g., polarization light projector 240). The polarization light projector is configured to (e.g., in a time-division multiplexing manner and / or a sequential manner) emit the light 730A linearly polarized in a direction orthogonal to the first direction (e.g., the horizontal direction) and the light 730B linearly polarized in a direction orthogonal to the second direction (e.g., -45 degrees) during different corresponding time periods.

[0097] The controller of the PCA can, for example, use the above equations to determine the object position and material properties. Figures 7A to 7CThe embodiments described herein can reconstruct an object shape by extracting the diffuse reflection part of light. The diffuse reflection part of light is described by Stokes parameters and is referred to as S 0-diff 、S 1-diff and S 2-diff . The diffuse reflection component includes the shape information of the object. For a given macro-pixel (e.g., macro-pixel 710), the controller can completely extract the diffuse reflection component containing the shape information through the following equation:

[0098]

[0099]

[0100] where the superscript (1) refers to the time period during which the sensor collects the light 730A reflected from the object in the local area, and the superscript (2) refers to the time period during which the sensor collects the light 730B reflected from the object in the local area. Thus, is the intensity of the light 730A detected at the green pixel (non-polarized green pixel), is the intensity of the light 730B detected at the green pixel (non-polarized green pixel), is the intensity of the light 730B detected at the polarized pixel (e.g., 720A), is the intensity of the light 730A detected at the polarized pixel (e.g., 720B), and is the intensity of the light 730B detected at the polarized pixel (e.g., 720B).

[0101] Once these diffuse reflection components are extracted, the controller can determine one or more surface normal orientations (i.e., shape) of the object. The controller can use one or more methods known in the art to make the determination.

[0102] Figure 7B is Figure 7A the second layout of the sensor in. Except that each macro-pixel includes a greater number of pixels configured to detect the intensity of light in different color channels (e.g., red, green, and blue), a part of the shown sensor is substantially the same as Figure 7A a part of the sensor in (e.g., the part including the polarized pixel 720). The controller (e.g., controller 230) can use the equations described with reference to, for example Figure 6 and Figure 7A to determine the properties, position, and shape of the material.

[0103] Figure 7C is Figure 7AThe third layout of the sensors in. Except that each macro pixel (e.g., macro pixel 740) includes two polarization pixels 720A and two polarization pixels 720B separated from each other by at least one non-polarization pixel, a part of the shown sensor is the same as that in Figure 7A A part of the sensors in. The controller (e.g., controller 230) can use the equations described above with reference to, for example Figure 6 and Figure 7A to determine the properties, positions, and shapes of the materials.

[0104] Note that in each of Figures 6 to 7C a specific orientation of the polarization pixels is used. These orientations can be rotated by a common offset so that for a given layout, the relative differences in the orientations of the polarization pixels are maintained. In addition, various sensor layouts provide different trade-offs. For example, relative to a part of the sensors shown in Figure 7B and Figure 7C a part of the sensors shown in Figure 7A has fewer pixels in each macro pixel. Therefore, a part of the sensors shown in Figure 7A can be implemented more simply, have a lower power budget, etc., at the cost of a lower resolution compared to the embodiments described in Figure 7B and Figure 7C .

[0105] Although all the inventions are described using wire-grid polarization filters, there is another component that can be used, a metasurface polarization router. These metasurfaces are typically dielectric (while wire-grids are metallic) and are capable of routing the light to the correct pixels according to the polarization state of the light. This further reduces photon loss compared to wire-grids.

[0106] Figure 8A An example operation of a metasurface 800 that guides light according to polarization according to one or more embodiments is shown. The metasurface includes a plurality of surface features (e.g., nanostructured patterns with subwavelength thicknesses) that are customized to guide light in different directions according to the polarization of the light. In this way, the metasurface can guide first linearly polarized light to a first pixel, second linearly polarized light to another pixel, and so on. If the angles are orthogonal, the light will be completely separated, otherwise the light will not be completely separated (for wire-grids, complete extinction can only be obtained at orthogonal polarization angles).

[0107] For example, as shown in Figure 8AAs shown, light 805 is incident on the metasurface 800. The light 805 is unpolarized light. As shown in the figure, the metasurface 800 is configured to direct the portion of the light 805 having a first linear polarization to the pixel 810, and direct the portion of the light 805 having a second linear polarization orthogonal to the first linear polarization to the pixel 820. Note that the absorption and / or reflection of the metasurface to the light 805 is negligible. In contrast, a wire-grid polarizer (e.g., depending on the type of wire-grid polarizer, by reflection or absorption) will lose about 50% of the incident light. Therefore, a multi-channel color sensor can also use a metasurface to obtain polarization information of a color channel without non-polarized pixels of a specific color channel.

[0108] Figure 8B FIG. 4 is an example first layout of a portion of a polarization multi-channel sensor having sparse polarization pixels according to one or more embodiments. These sparse polarization pixels use a metasurface, and each macro-pixel is configured to detect linearly polarized light in four different orientations. A portion of the sensor shown includes four macro-pixels (e.g., macro-pixel 830), and the layout of the pixels and polarization pixels of the four macro-pixels is the same. For example, the macro-pixel 830 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red and blue), and polarization pixels 835A, 835B, 835C, and 835D (collectively referred to as polarization pixels 835) including a metasurface. The metasurface of the polarization pixels 835 is used to direct the incident light to different polarization pixels according to the polarization of the incident light. The polarization pixel 835A is configured to detect the intensity of the light linearly polarized in a first direction (0 degrees) in the green channel. The polarization pixel 835B is configured to detect the intensity of the light linearly polarized in a second direction (-π / 4 or -45 degrees) in the green channel. The polarization pixel 835C is configured to detect the intensity of the light linearly polarized in a third direction (π / 2 or 90 degrees) in the green channel. The polarization pixel 835D is configured to detect the intensity of the light linearly polarized in a fourth direction (π / 4 or 45 degrees) in the green channel. Note that the macro-pixel 830 does not include any non-polarized green pixels. This is because, as referred to above Figure 8A as described, the metasurface (e.g., relative to a wire-grid polarizer) does not absorb / reflection a large amount of incident light. Therefore, the intensity value equivalent to that of a non-polarized green pixel can be easily obtained according to the intensity values of the polarization pixels 835. A controller (e.g., controller 230) can use the detected intensity to determine the attributes of an object in the same manner as described above with reference to, for example Figure 3A FIG. 2.

[0109] Figure 8Cis an example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect linearly polarized light in four different orientations. Except that each macro-pixel (e.g., macro-pixel 840) includes a fewer number of pixels configured to detect the intensity of light in different color channels (e.g., red and blue), a portion of the sensor shown is substantially the same as the portion of the sensor in Figure 8B (e.g., the portion including polarization pixel 835). A controller (e.g., controller 230) can use the detected intensities to determine properties of an object in the same manner as described above with reference to, for example, Figure 3A .

[0110] Figure 9 is an example layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, the sparse polarization pixels using a metasurface, and each macro-pixel being configured to detect linearly polarized light in three different orientations. A portion of the sensor shown includes four macro-pixels (e.g., macro-pixel 910), and the layout of the pixels and polarization pixels of the four macro-pixels is the same. For example, macro-pixel 910 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red and blue), and two sets of polarization pixels 920A, 920B, and 920C (collectively referred to as polarization pixels 920), and each polarization pixel in the two sets of polarization pixels 920A, 920B, and 920C includes a metasurface. Polarization pixel 920A is configured to detect the intensity of light linearly polarized in a first direction (-π / 8 or -22.5 degrees) in the green channel. Polarization pixel 920B is configured to detect the intensity of light linearly polarized in a second direction (π / 8 or 22.5 degrees) in the green channel. Polarization pixel 920C is configured to detect the intensity of light linearly polarized in a third direction (3π / 8 or 67.5 degrees) in the green channel. Note that macro-pixel 910 does not include any non-polarized green pixels. As described above, the light loss of the metasurface is negligible, and S can be obtained by adding the intensities detected from polarization pixels 920A and 920C (orthogonal polarizations) using the detected intensities. 0 A controller (e.g., controller 230) can use the detected intensities to determine properties of an object in the same manner as described above with reference to, for example, Figure 4A .

[0111] Figure 10Ais an example first layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where the sparse polarization pixels use a metasurface and each macro-pixel is configured to detect linearly polarized light of two different orientations. The portion of the sensor shown includes four macro-pixels (e.g., macro-pixel 1010), and the layout of the pixels and polarization pixels is the same for these four macro-pixels. For example, macro-pixel 1010 includes a plurality of pixels configured to detect the intensity of light in different color channels (e.g., red and blue), and two sets of polarization pixels 1020A and 1020B (collectively referred to as polarization pixels 1020). Polarization pixel 1020A is configured to detect the intensity of light linearly polarized in a first direction (0 degrees) in the green channel. Polarization pixel 1020B is configured to detect the intensity of light linearly polarized in a second direction (π / 2 or 90 degrees) in the green channel. As described above, the light loss of the metasurface is negligible, and using the detected intensities, S can be obtained by adding the detected intensities from polarization pixels 1020A and 1020B (orthogonal polarizations). 0 。

[0112] A controller (e.g., controller 230) can use the detected intensities to determine properties of an object, at least in part based on the manner described above for Figure 6 Note that the sum of polarization pixel 1020A and polarization pixel 1020B gives the same result as that of a non-polarized green pixel (e.g., A G ).

[0113] Figure 10B is an example second layout of a portion of a polarization multi-channel sensor with sparse polarization pixels according to one or more embodiments, where the sparse polarization pixels use a metasurface and each macro-pixel is configured to detect linearly polarized light of two different orientations. Except that each macro-pixel (e.g., macro-pixel 1030) includes polarization pixels 1040A and 1040B (collectively referred to as polarization pixels 1040), and each macro-pixel also includes a non-polarized green pixel (to obtain S 0 information), the portion of the sensor shown is similar to the portion of the sensor in Figure 10A in that the portion of the sensor shown includes a plurality of polarization pixels configured to detect linearly polarized light of two different orientations. Polarization pixel 1040A is configured to detect the intensity of light linearly polarized in a first direction (π / 8 or 22.5 degrees) in the green channel. Polarization pixel 1040B is configured to detect the intensity of light linearly polarized in a second direction (-π / 8 or -22.5 degrees) in the green channel. A controller (e.g., controller 230) can use equations (7) to (9) described above with reference to Figure 5A to determine the Stokes parameter S 0 、S 1and S 2 。

[0114] Figure 11 System 1100 including a head-mounted device 1105 according to one or more embodiments. In some embodiments, the head-mounted device 1105 can be the Figure 1 head-mounted device 100 in. System 1100 can operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 11 The system 1100 shown in includes a head-mounted device 1105, an input / output (I / O) interface 1110 coupled to a console 1115, a network 1120, and a map building server 1125. Although Figure 11 the example system 1100 is shown including one head-mounted device 1105 and one I / O interface 1110, in other embodiments, the system 1200 can include any number of these components. For example, there can be multiple head-mounted devices, each of the multiple head-mounted devices having an associated I / O interface 1110, where each head-mounted device and I / O interface 1110 communicate with the console 1115. In an alternative configuration, the system 1100 can include different components and / or additional components. Additionally, in some embodiments, the functions described in connection with Figure 11 one or more of the components shown in can be distributed among the components in a different manner than described in connection with Figure 11 . For example, some or all of the functions of the console 1115 can be provided by the head-mounted device 1105.

[0115] The head-mounted device 1105 includes a display component 1130, an optical block 1135, one or more position sensors 1140, and a PCA 200. Some embodiments of the head-mounted device 1105 have components different from those described in connection with Figure 11 . Additionally, in other embodiments, the functions provided by the various components described in connection with Figure 11 can be distributed differently among the components of the head-mounted device 1105, or embodied in separate components remote from the head-mounted device 1105.

[0116] The display component 1130 displays content to a user based on data received from the console 1115. The display component 1130 uses one or more display elements (e.g., display element 120) to display the content. The display element can be, for example, an electronic display. In various embodiments, the display component 1130 includes a single display element or multiple display elements (e.g., one display for each eye of the user). Examples of electronic displays include: liquid crystal display (LCD), organic light emitting diode (OLED) display, active-matrix organic light-emitting diode display (AMOLED), waveguide display, some other display, or some combination thereof. Note that in some embodiments, the display element 120 may also include some or all of the functions of the optical block 1135.

[0117] The optical block 1135 can magnify the image light received from the electronic display, correct the optical errors associated with the image light, and present the corrected image light to one or both eyesight areas of the head-mounted device 1105. In various embodiments, the optical block 1135 includes one or more optical elements. Example optical elements included in the optical block 1135 include: aperture, Fresnel lens, convex lens, concave lens, filter, reflective surface, or any other suitable optical element that affects the image light. Additionally, the optical block 1135 can include a combination of different optical elements. In some embodiments, one or more of the optical elements in the optical block 1135 can have one or more coatings, such as a partial reflection coating or an anti-reflection coating.

[0118] The magnification and focusing of the image light by the optical block 1135 allows the electronic display to be physically smaller, lighter in weight, and lower in power consumption than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that almost all (e.g., approximately 110 degrees diagonal), and in some cases all of the user's field of view is used to present the displayed content. Additionally, in some embodiments, the magnification amount can be adjusted by adding or removing optical elements.

[0119] In some embodiments, the optical block 1135 may be designed to correct one or more types of optical errors. Examples of optical errors include barrel distortion or pincushion distortion, longitudinal chromatic aberration or lateral chromatic aberration. Other types of optical errors may also include: spherical aberration, chromatic aberration, or errors caused by lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided for display to the electronic display is pre-distorted, and the optical block 1135 corrects the distortion when it receives the image light generated based on the content from the electronic display.

[0120] The position sensor 1140 is an electronic device that generates data indicating the position of the head-mounted device 1105. The position sensor 1140 generates one or more measurement signals in response to the movement of the head-mounted device 1105. The position sensor 190 is an embodiment of the position sensor 1140. Examples of the position sensor 1140 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor for detecting movement, or some combination thereof. The position sensor 1140 may include: multiple accelerometers for measuring translational movement (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational movement (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the head-mounted device 1105 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate the velocity vector, and integrates the velocity vector over time to determine the estimated position of a reference point on the head-mounted device 1105. The reference point is a point that can be used to describe the position of the head-mounted device 1105. Although the reference point can generally be defined as a point in space, in fact, the reference point is defined as a point within the head-mounted device 1105.

[0121] The PCA 200 is configured to determine the attributes of an object in a local area of the head-mounted device 1105. As described above with reference to Figures 1 to 10B the operation and structure of the PCA 200 are described.

[0122] The audio system 1150 provides audio content to the user of the head-mounted device 1105. The audio system 1150 is substantially the same as that described above with reference to Figure 1is the same as the described audio system. The audio system 1150 may include one or more acoustic sensors, one or more transducers, and an audio controller. The audio system 1150 may provide spatialized audio content to a user. In some embodiments, the audio system 1150 may request acoustic parameters from the map building server 1125 via the network 1120. The acoustic parameters describe one or more acoustic properties of a local area (e.g., room impulse response, reverberation time, reverberation level, etc.). The audio system 1150 may provide information, such as from the PCA 200, that describes at least a portion of the local area, and / or location information of the head-mounted device 1105 from the position sensor 1140. The audio system 1150 may use one or more of the received acoustic parameters from the map building server 1125 to generate one or more sound filters and use these sound filters to provide audio content to the user.

[0123] The I / O interface 1110 is a device that allows a user to send action requests to the console 1115 and receive responses from the console 1115. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or stop acquiring image data or video data, or an instruction to perform a specific action within an application. The I / O interface 1110 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and transmitting these action requests to the console 1115. The action requests received by the I / O interface 1110 are transmitted to the console 1115, which performs the action corresponding to the action request. In some embodiments, the I / O interface 1110 includes an IMU that acquires calibration data indicating an estimated position of the I / O interface 1110 relative to an initial position of the I / O interface 1110. In some embodiments, the I / O interface 1110 may provide haptic feedback to the user according to instructions received from the console 1115. For example, haptic feedback is provided upon receiving an action request, or the console 1115 transmits an instruction to the I / O interface 1110 such that the I / O interface 1110 generates haptic feedback when the console 1115 performs an action.

[0124] The console 1115 provides content for processing to the head-mounted device 1105 based on information received from one or more of the following: the PCA 200, the head-mounted device 1105, and the I / O interface 1110. In Figure 11 the example shown, the console 1115 includes an application repository 1155, a tracking module 1160, and an engine 1165. Some embodiments of the console 1115 have modules or components different from the modules or components described in connection with Figure 11 those described. Similarly, the various functions further described below may be implemented in a manner different from that described in connection withFigure 11 distributed among the components of the console 1115 in ways different from those described. In some embodiments, the various functions described herein with respect to the console 1115 may be implemented in the head-mounted device 1105 or a remote system.

[0125] The application repository 1155 stores one or more applications for execution by the console 1115. An application is a set of instructions that, when executed by a processor, generate content for presentation to a user. The content generated by an application may be in response to input received from a user in response to movement via the head-mounted device 1105 or the I / O interface 1110. Examples of applications include: game applications, conferencing applications, video playback applications, or other suitable applications.

[0126] The tracking module 1160 uses information from the PCA 200, one or more position sensors 1140, or some combination thereof to track the movement of the head-mounted device 1105 or the I / O interface 1110. For example, the tracking module 1160 determines the position of a reference point of the head-mounted device 1105 in the construction of a map of a local area based on information from the head-mounted device 1105. The tracking module 1160 may also determine the position of an object or a virtual object. Additionally, in some embodiments, the tracking module 1160 may use a portion of the data indicating the position of the head-mounted device 1105 from the position sensor 1140 and a representation of the local area from the PCA 200 to predict the future position of the head-mounted device 1105. The tracking module 1160 provides an estimated future position of the head-mounted device 1105 or the I / O interface 1110, or a predicted future position of the head-mounted device 1105 or the I / O interface 1110, to the engine 1165.

[0127] The engine 1165 executes applications and receives the following information about the head-mounted device 1105 from the tracking module 1160: position information, acceleration information, velocity information, predicted future position, or some combination thereof. The engine 1165 determines the content to be provided to the head-mounted device 1105 for presentation to the user based on the received information. For example, if the received information indicates that the user has looked to the left, the engine 1165 generates the following content for the head-mounted device 1105: the content reacts to the movement of the user in the virtual local area, or reacts to the movement of the user in the local area to enhance the local area with additional content. Additionally, the engine 1165 executes an action within an application executed on the console 1115 in response to an action request received from the I / O interface 1110 and provides feedback to the user that the action has been executed. The feedback provided may be visual feedback or auditory feedback via the head-mounted device 1105, or tactile feedback via the I / O interface 1110.

[0128] Network 1120 couples the head-mounted device 1105 and / or the console 1115 to the map building server 1125. Network 1120 can include any combination of local area networks and / or wide area networks using wireless communication systems and / or wired communication systems. For example, network 1120 can include the Internet and mobile phone networks. In one embodiment, network 1120 uses standard communication technologies and / or protocols. Thus, network 1120 can include links using technologies such as, for example: Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), Infinite Bandwidth, PCI Express Advanced Switching, etc. Similarly, the networking protocols used on network 1120 can include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Technologies and / or formats can be used to represent data exchanged over network 1120: these technologies and / or formats include image data in binary form (e.g., Portable Network Graphic (PNG)), Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc.Alternatively, traditional encryption technologies (e.g., secure sockets layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc.) can be used to encrypt all or some of the links.

[0129] The map building server 1125 can include a database that stores virtual models describing multiple spaces, where a location in the virtual model corresponds to the current configuration of a local area of the head-mounted device 1105. The map building server 1125 receives, via the network 1120, information describing at least a portion of the local area and / or location information of the local area from the head-mounted device 1105. The user can adjust privacy settings to allow or block the head-mounted device 1105 from transmitting information to the map building server 1125. The map building server 1125 determines the location in the virtual model associated with the local area of the head-mounted device 1105 based on the received information and / or location information. The map building server 1125 determines (e.g., retrieves) one or more acoustic parameters associated with the local area, at least in part based on the determined location in the virtual model and any acoustic parameters associated with the determined location. The map building server 1125 can send the location of the local area and any values of acoustic parameters associated with the local area to the head-mounted device 1105.

[0130] One or more components of the system 1100 can include a privacy module that stores one or more privacy settings for user data elements. The user data elements describe the user or the head-mounted device 1105. For example, the user data elements can describe the physical characteristics of the user, actions performed by the user, the location of the user of the head-mounted device 1105, the location of the head-mounted device 1105, the head-related transfer function (HRTF) of the user, etc. The privacy settings (or "access settings") for the user data elements can be stored in any suitable manner, e.g., stored in association with the user data elements, stored in an index on an authorization server, stored in another suitable manner, or any suitable combination thereof.

[0131] Privacy settings for user data elements specify how the user data elements (or specific information associated with the user data elements) can be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, displayed, or identified). In some embodiments, privacy settings for user data elements can specify a "blacklist" of entities that may not have access to certain information associated with the user data elements. Privacy settings associated with user data elements can specify any suitable granularity of access permitted or denied. For example, some entities may have the right to view the existence of a particular user data element, some entities may have the right to view the content of a particular user data element, and some entities may have the right to modify a particular user data element. Privacy settings can allow a user to permit other entities to access or store user data elements for a limited period of time.

[0132] Privacy settings can allow a user to specify one or more geographical locations from which the user data elements can be accessed. Access or denial of access to user data elements can depend on the geographical location of the entity attempting to access the user data elements. For example, a user can permit access to user data elements and specify that an entity can only access the user data elements when the user is in a particular location. If the user leaves that particular location, the entity may no longer be able to access the user data elements. As another example, a user can specify that only entities within a threshold distance of the user (e.g., another user of a head-mounted device within the same local area as the user) can access the user data elements. If the user then changes location, entities that had access to the user data elements may lose access, while a new set of entities can gain access when they come within the user's threshold distance.

[0133] System 1100 can include one or more authorization servers / privacy servers for implementing privacy settings. A request from an entity for a particular user data element can identify the entity associated with the request, and if the authorization server determines, based on the privacy settings associated with the user data element, that the entity is authorized to access the user data element, the user data element can be sent only to that entity. If the requesting entity is not authorized to access the user data element, the authorization server can prevent the requested user data element from being retrieved or can prevent the requested user data element from being sent to the entity. Although this disclosure describes implementing privacy settings in a particular manner, this disclosure contemplates implementing privacy settings in any suitable manner.

[0134] Additional configuration information

[0135] The foregoing description of the embodiments has been presented for purposes of illustration; the foregoing description is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Those skilled in the relevant art will appreciate that, given the above disclosure, many modifications and variations are possible.

[0136] Some portions of this description describe various embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. Although these operations are described functionally, computationally, or logically, these operations should be understood to be implemented by a computer program or equivalent circuitry, microcode, etc. Additionally, it has proven convenient at times, without loss of generality, to refer to arrangements of these operations as modules. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combination thereof.

[0137] Any steps, operations, or processes described herein may be performed or implemented singly or in combination with other devices by one or more hardware or software modules. In one embodiment, a software module is implemented with a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0138] The embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes, and / or the apparatus may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Additionally, any computing system mentioned in this specification may include a single processor, or may be an architecture employing a multi-processor design for increased computing power.

[0139] The embodiments may also relate to a product produced by the computing processes described herein. Such a product may include information obtained from the computing process, where the information is stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of the computer program product or other data combinations described herein.

[0140] Finally, the language used in this specification has been principally selected for readability and instructional purposes, and the language may not have been selected to circumscribe or limit the patent rights. Accordingly, the scope of the patent rights is not intended to be limited by the specific embodiments described herein, but rather is to be limited by any claims published based on the application as presented herein. Accordingly, the disclosure of each embodiment is intended to illustrate rather than limit the scope of the patent rights, which scope is set forth in the appended claims.

Claims

1. A sensor, comprising: A plurality of macro pixels, wherein each macro pixel comprises: a plurality of pixels configured to detect intensities of light in different color channels, the plurality of pixels including a first pixel configured to detect intensities of light in a first color channel of the different color channels, and a first polarization pixel configured to detect an intensity of light linearly polarized in a first direction in the first color channel, The light incident on a macropixel among the plurality of macropixels is from an object, and the controller is configured to determine a property of the object based in part on a first intensity signal of a first pixel of the macropixel and a first polarization signal of a first polarization pixel of the macropixel.

2. The sensor according to claim 1, wherein: Each macropixel also includes: a second polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a second direction different from the first direction, Wherein, the determination of the property of the object is also partially based on the second polarization signal of the second polarization pixel of the macro pixel.

3. The sensor according to claim 2, wherein: The angle separating the first direction and the second direction is forty-five degrees.

4. The sensor according to claim 3, wherein: Each macropixel also includes: a third polarization pixel configured to detect an intensity of light linearly polarized in a third direction in the first color channel, wherein an angle separating the first direction from the third direction is 67.5 degrees, Wherein, the determination of the attribute of the object is also partially based on the third polarization signal of the third polarization pixel of the macro pixel.

5. The sensor according to claim 2, wherein: The macropixel is sequentially illuminated by first polarized light and second polarized light both in the first color channel, and an angle between the first polarized light and the second polarized light is 45 degrees.

6. The sensor according to claim 2, wherein: The controller is configured to: For the macro pixel, using the first intensity signal, the first polarization signal and the second polarization signal to determine values ​​of Stokes parameters S0, S1 and S2, Wherein the property of the object is based in part on the determined value.

7. The sensor according to claim 2, wherein: The angle separating the first direction and the second direction is 90 degrees, wherein each macro pixel further includes: a third polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a third direction, wherein the third direction is -45 degrees relative to the first direction; and a fourth polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a fourth direction, wherein the fourth direction is +45 degrees relative to the first direction; The attribute of the object is partially based on the third polarization signal of the third polarization pixel of the macropixel and the fourth polarization signal of the fourth polarization pixel of the macropixel.

8. The sensor according to claim 2, wherein: For the macropixel, the first polarization pixel and the second polarization pixel include a metasurface that guides light linearly polarized in the first direction to the first polarization pixel and guides light linearly polarized in the second direction to the second polarization pixel.

9. The sensor according to claim 1, wherein: The properties of the object are selected from the group consisting of: shape, position, and one or more material properties, wherein the material properties are selected from the group consisting of roughness and diffuse albedo.

10. The sensor according to claim 9, wherein: The macropixel is illuminated by a first polarized light, and the controller is configured to: determining the degree of linear polarization DOLP using the first intensity signal, Therein, the properties of the object are also based in part on the determined DOLP.

11. A system comprising: A camera, the camera comprising a plurality of macro pixels, the plurality of macro pixels receiving light from a local area including an object, wherein a macro pixel of the plurality of macro pixels comprises: a plurality of pixels configured to detect intensities of light in different color channels, the plurality of pixels including a first pixel configured to detect the intensity of light in a first color channel of the different color channels as a first intensity signal, and a first polarization pixel configured to detect an intensity of light linearly polarized in a first direction in the first color channel as a first polarization signal, and A controller is configured to determine a property of the object based in part on the first intensity signal and the first polarization signal.

12. The system according to claim 11, wherein: The macro pixel also includes: a second polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a second direction different from the first direction, Wherein, the determination of the property of the object is also partially based on the second polarization signal of the second polarization pixel of the macro pixel.

13. The system according to claim 12, wherein: The angle separating the first direction and the second direction is 45 degrees.

14. The system according to claim 13, wherein: The macro pixel also includes: a third polarization pixel configured to detect an intensity of light linearly polarized in a third direction in the first color channel, wherein an angle separating the first direction from the third direction is 67.5 degrees, Wherein, the determination of the attribute of the object is also partially based on the third polarization signal of the third polarization pixel of the macro pixel.

15. The system according to claim 12, further comprising: a projector configured to sequentially illuminate the object with a first polarized light and a second polarized light both in the first color channel, and an angle between the first polarized light and the second polarized light is 45 degrees, A portion of the first polarized light reflected by the object and a portion of the second polarized light reflected by the object are incident on the macro pixel.

16. The system of claim 15, wherein: The controller is configured to: For the macro pixel, using the first intensity signal, the first polarization signal and the second polarization signal to determine values ​​of Stokes parameters S0, S1 and S2, Wherein, the property of the object is also based in part on the determined value.

17. The system of claim 12, wherein: The angle separating the first direction and the second direction is 90 degrees, wherein the macro pixel further includes: a third polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a third direction, wherein the third direction is -45 degrees relative to the first direction; and a fourth polarization pixel configured to detect an intensity of light in the first color channel that is linearly polarized in a fourth direction, wherein the fourth direction is +45 degrees relative to the first direction; The attribute of the object is also partially based on the third polarization signal of the third polarization pixel of the macropixel and the fourth polarization signal of the fourth polarization pixel of the macropixel.

18. The system of claim 12, wherein: For the macropixel, the first polarization pixel and the second polarization pixel include a metasurface that guides light linearly polarized in the first direction to the first polarization pixel and guides light linearly polarized in the second direction to the second polarization pixel.

19. A sensor comprising: A plurality of macro pixels, wherein each macro pixel comprises: a plurality of pixels configured to detect intensities of light in different color channels, and a first polarization pixel configured to detect an intensity of light linearly polarized in a first direction in a first color channel of the different color channels, The light incident on a macropixel among the plurality of macropixels is from an object, and the controller is configured to determine a property of the object based in part on a first polarization signal of a first polarization pixel of the macropixel.

20. The sensor according to claim 19, wherein For the macropixel, the first polarization pixel includes a metasurface that guides light linearly polarized in the first direction to the first polarization pixel and guides light linearly polarized in the second direction to the second polarization pixel.

Images