Distributed sensor systems

Abstract

In one example, a mobile device includes: a physical link; a plurality of image sensors, each image sensor configured to send image data via the physical link; and a controller coupled to the physical link, whereby the physical link, the plurality of image sensors, and the controller form a multi-branch network. The controller is configured to: send control signals to configure image sensing operations at the plurality of image sensors; receive image data from at least a subset of the plurality of image sensors via the physical link; combine image data from at least a subset of the plurality of image sensors to obtain an extended field of view (FOV); determine information of the surrounding environment of the mobile device captured within the extended FOV; and provide the information to an application to generate content based on the information.

Description

[0001] Related Applications

[0002] This patent application claims priority to U.S. Provisional Patent Application Serial No. 62,928,233, filed on October 30, 2019, entitled “DISTRIBUTED SENSOR SYSTEM,” which has been assigned to the present assignee and is incorporated herein by reference in its entirety for all purposes.

[0003] field

[0004] The present disclosure relates generally to sensor networks and, more particularly, to sensor networks in wearable electronic devices such as head mounted displays (HMDs).

[0005] background

[0006] Wearable electronic devices may include many sensors to support different applications of the device. For example, wearable virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems may include many image sensors, audio sensors, motion sensors, and the like. Sensors may be used to collect sensor data of the user's physical environment to support various applications, such as simultaneous localization and mapping (SLAM) algorithms that track the location of users of wearable electronic devices, object detection / measurement applications, and the like. Based on the sensor data, the VR / AR / MR system may generate and update, for example, virtual image data for display to the user via a near-eye display, audio data for output to the user via a speaker, and the like, to provide an interactive experience for the user.

[0007] In order to improve the sensing of the surrounding environment, a wearable electronic device may include one or more high-resolution image sensor modules. Each image sensor module may include a lens stack and a high-resolution image sensor to capture a high-resolution image. However, it may be challenging to integrate such a network of high-resolution sensor modules into a wearable electronic device. Specifically, a high-resolution sensor module typically requires a large silicon area with a relatively large form factor, and the field of view (FOV) lens stack may have a considerable vertical height, all of which makes it difficult to integrate the module into a wearable electronic device with very limited space. In addition, the generation of high-resolution image data and the transmission and processing of high-resolution image data typically consume a lot of power, and mobile devices typically operate with a very limited power budget. All of this makes it challenging to implement a wearable device that provides high-quality sensing of the surrounding environment.

[0008] Overview

[0009] The present disclosure relates to sensor networks. More particularly, but not limited to, the present disclosure relates to sensor networks that can be used in wearable electronic devices such as HMDs.

[0010] In one example, a mobile device includes: a physical link; a plurality of image sensors, each of the plurality of image sensors being configured to send image data via the physical link; and a controller coupled to the physical link, whereby the physical link, the plurality of image sensors, and the controller form a multi-drop network. The controller is configured to: send control signals to configure image sensing operations at the plurality of image sensors; receive image data from at least a subset of the plurality of image sensors via the physical link; combine image data from at least a subset of the plurality of image sensors to obtain an extended field of view (FOV) larger than that provided by each image sensor in the subset of image sensors; determine information of an environment surrounding the mobile device captured within the extended FOV; and provide the information to an application to generate content based on the information.

[0011] In some aspects, the controller is configured to select a subset of the plurality of image sensors to transmit image data and disable the remainder of the plurality of image sensors based on the transmit control signal.

[0012] In some aspects, the image data is first image data. The controller is configured to: control a subset of the plurality of image sensors to transmit the first image data at a first resolution; and control the remainder of the plurality of image sensors to transmit the second image data at a second resolution based on sending the control signal. The first resolution is higher than the second resolution.

[0013] In some aspects, the image data is first image data. The controller is configured to: control a subset of the plurality of image sensors to send the first image data at a first frame rate; and control the remainder of the plurality of image sensors to send the second image data at a second frame rate based on sending the control signal, wherein the first frame rate is higher than the second frame rate.

[0014] In some aspects, the information of the surrounding environment includes tracking results of the object of interest. The controller is configured to select the subset of the plurality of image sensors based on determining that image data generated by the subset of the plurality of image sensors may contain one or more images of the object.

[0015] In some aspects, the controller is configured to determine that image data generated by a subset of the plurality of image sensors may contain one or more images of the object based on detecting features of the object in previous image data from the subset of the plurality of image sensors.

[0016] In some aspects, the controller is configured to: determine a previous trajectory of relative movement between the object and the mobile device based on previous image data from the plurality of image sensors. The controller is configured to: predict a trajectory of the object based on the previous trajectory; and determine that the image data generated by a subset of the plurality of image sensors may contain one or more images of the object based on the predicted trajectory.

[0017] In some aspects, each image sensor of the plurality of image sensors includes a pixel cell array. The controller is configured to selectively configure image capture operations of a particular subset of pixel cells in the pixel cell array in one or more image sensors of the plurality of image sensors based on sending a control signal.

[0018] In some aspects, the controller is configured to: enable a first subset of pixel cells in a pixel cell array of a first image sensor among multiple image sensors to send first image data to the controller via a physical link; and enable a second subset of pixel cells in a pixel cell array of a second image sensor among multiple image sensors to send second image data to the controller via a physical link based on sending a control signal. The first subset and the second subset are different. The controller is configured to: enable a first subset of pixel cells in a pixel cell array of a first image sensor among multiple image sensors to generate first image data at a first resolution; enable a second subset of pixel cells in a pixel cell array of the first image sensor to generate second image data at a second resolution based on sending a control signal. The first resolution is higher than the second resolution.

[0019] In some aspects, the controller is configured to: set a first dynamic range of a first subset of pixel cells in a pixel cell array of a first image sensor among the plurality of image sensors based on sending a control signal; and set a second dynamic range of a second subset of pixel cells in the pixel cell array of the first image sensor. The first dynamic range is higher than the second dynamic range.

[0020] In some aspects, for each image sensor in the subset of the plurality of image sensors, the control signal identifies a pixel cell in a particular subset of pixel cells in the array of pixel cells.

[0021] In some aspects, the control signal includes a guidance signal. A first image sensor of the plurality of image sensors is configured to locally determine a particular subset of pixel cells in the array of pixel cells based on the guidance signal.

[0022] In some aspects, the guidance signal specifies a feature of the object of interest.The first image sensor is configured to: determine a region of interest including the object of interest based on the guidance signal; and determine a particular subset of pixel cells in the array of pixel cells based on the region of interest.

[0023] In some aspects, the particular subset of pixel cells in the array of pixel cells is determined based on at least one of: tracking results of the object of interest or movement of the mobile device.

[0024] In some aspects, image data from at least a first image sensor and a second image sensor of the plurality of image sensors are combined. The first image sensor and the second image sensor face different directions.

[0025] In some aspects, image data from at least a first image sensor and a second image sensor of the plurality of image sensors are combined. The first image sensor is configured to capture light of a first frequency range. The second image sensor is configured to capture light of a second frequency range different from the first frequency range.

[0026] In some aspects, the physical link comprises at least one of: a bus based on the I3C specification or an optical link.

[0027] In one example, a method includes: sending control signals to configure image sensing operations at multiple image sensors of a mobile device; receiving image data from each image sensor in a subset of the multiple image sensors via a physical link, wherein the multiple image sensors and the physical link form a multi-branch network; combining image data from at least a subset of the multiple image sensors to obtain an extended field of view (FOV) that is larger than a field of view (FOV) provided by each image sensor in the subset of the image sensors; determining information of the surrounding environment of the mobile device captured within the extended FOV; and providing the information to an application to generate content to be output by the mobile device based on the information.

[0028] In some aspects, the method further includes selecting a subset of the plurality of image sensors to transmit the image data based on the transmit control signal and disabling the remainder of the plurality of image sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Illustrative embodiments are described with reference to the following figures.

[0031] Figure 1A and Figure 1B is a schematic diagram of an embodiment of a near-eye display.

[0032] Figure 2 is an embodiment of a cross-section of a near-eye display.

[0033] Figure 3 An isometric view of an embodiment of a waveguide display having a single source assembly is shown.

[0034] Figure 4 A cross section of an embodiment of a waveguide display is shown.

[0035] Figure 5is a block diagram of an embodiment of a system including a near-eye display.

[0036] Fig. 6A , Figure 6B , Figure 6C and Fig.6D Examples of image sensors and their operation are shown.

[0037] Fig. 7A , Figure 7B , Figure 7C and Fig.7D An example of a mobile device having a sensor network and its operation is shown.

[0038] Fig. 8A , Figure 8B , Figure 8C , Fig.8D and Fig. 8E shows the sensor network and Figure 7A-7D Examples of sensor networks and their operation.

[0039] Fig. 9A , Fig. 9B , Fig. 9C and Fig.9D Shows Figure 8A-8E Example internal components of a sensor network.

[0040] Fig.10 A flow chart illustrating an example process for generating content to be output by a mobile device is shown.

[0041] The accompanying drawings depict embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown may be employed without departing from the principles of the present disclosure or the benefits advocated in the present disclosure.

[0042] In the accompanying drawings, similar components and / or features may have the same reference number. In addition, components of the same type may be distinguished by following the reference number with a dash and a second reference that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any one of the similar components having the same first reference number, regardless of the second reference number.

[0043] Detailed Description

[0044] In the following description, specific details are set forth for the purpose of explanation in order to provide a thorough understanding of certain inventive embodiments. However, it will be apparent that various embodiments can be implemented without these specific details. The drawings and descriptions are not intended to be limiting.

[0045] An example according to the present disclosure relates to a mobile device, such as an HMD. The mobile device includes a physical link, a plurality of image sensors, and a controller, each image sensor being oriented in a particular direction and coupled to the physical link, and the controller being coupled to the physical link. The image sensor, the controller, and the physical link may form a multi-branch network, wherein each of the plurality of image sensors is configured to capture image data of the surrounding environment of the mobile device, and the image data is sent to the controller via the physical link. The controller may determine an operating condition (e.g., positioning, orientation, etc.) of the mobile device. Furthermore, based on the operating condition, the controller may select a subset of the plurality of image sensors, and generate control data based on the selection. The controller may send control data to a subset of the image sensors via the physical link to configure the image capture operation of the subset of the image sensors. The controller may also receive first image data captured by a subset of the image sensors based on the control data via the physical link, and generate an output based on the first image data.

[0046] Specifically, the controller can execute an application that determines information about the surrounding environment based on the first image data and generates content (display content, audio signals, etc.) based on the information. In one example, the application may include a simultaneous localization and mapping (SLAM) algorithm to track, for example, the location of a user of a mobile device, the user's orientation, and / or the user's movement path in the environment. As another example, the application may include a mobile object tracking algorithm to track the location of a moving object (e.g., a moving hand). In yet another example, the application may include a scene context understanding algorithm to detect / track objects (e.g., other people who are not in the user's line of sight). In all of these examples, the application can detect certain image features of objects of interest (e.g., objects in a scene, hands, people's faces, etc.) in the image data, determine their image positions (if detected), and generate information about the surrounding environment based on the detected image features.

[0047] The controller can configure the image capture operations of the image sensors to reduce the amount of image data sent over the multi-branch network while providing the image data required by the application. For example, the controller can determine that the image sensors may capture images of one or more objects of interest at a given time based on, for example, detecting features of the object from previous images captured by a subset of the image sensors. The controller can then enable a subset of the image sensors to generate and send image data while disabling the remaining image sensors. In the event that the object of interest moves relative to the mobile device, the controller can determine a predicted movement trajectory of the object of interest and, based on the predicted trajectory, enable different subsets of the image sensors at different times.

[0048] In addition to enabling a subset of image sensors, the controller may also control a subset of image sensors to perform sparse sensing operations to further reduce the amount of image data sent over the network. Specifically, each image sensor includes an array of pixel cells. For each image sensor in the subset of enabled image sensors, the controller may determine a region of interest (ROI) of pixels that may include an object of interest. In some examples, the controller may control each image sensor in the subset of image sensors to enable only a subset of pixel cells corresponding to the ROI, or to send image data from a subset of pixel cells without sending image data from the remaining pixel cells. In addition, the controller may also configure other aspects of the image capture operation of the subset of pixel cells (such as increasing quantization resolution, increasing exposure cycle, and increasing dynamic range, etc.) to improve the quality of the image data. In some examples, the controller may send a guidance signal to guide the sensor to determine the ROI instead of sending control data specifying a subset of pixel cells to be enabled at each image sensor. The guidance signal includes information identifying features of the object of interest, a rough estimate of the location and size of the ROI, etc. Each image sensor may then determine the ROI locally based on the guidance signal. Such an arrangement can reduce the amount of control data sent by the controller to the image sensor because the controller does not need to send updated ROI information to the image sensor between image frames to account for movement of the mobile device and / or object of interest, which can further reduce power and bandwidth requirements of the network.

[0049] A multi-branch network can be implemented using various technologies. In some examples, a multi-branch network can be implemented using a shared bus, such as a bus implemented based on the I3C specification. Specifically, each component coupled to the bus (including an image sensor and a controller) can drive the bus in turn to send data via the bus. Each component can also monitor the bus to receive data. Due to the reduction in the amount of image data and control data, a relatively low-bandwidth and low-power shared bus can be used to send data. In some examples, a multi-branch network can also be implemented using an optical link, in which each image sensor and controller are coupled to the optical link via an optical modulator to send and receive data. The image sensor can modulate the light in the optical link in turn, or modulate different components of the light simultaneously to send image data.

[0050] Using the disclosed examples, a mobile device may include multiple image sensors, each oriented in a particular direction to capture image data of the surrounding environment. Since each image sensor has a FOV aligned in a particular direction, the effective FOV of the mobile device can be extended by combining the image data captured by the image sensors. Such an arrangement can reduce the need to increase the resolution and FOV of each image sensor, which may take up a lot of space and consume a lot of power, both of which are very limited in mobile devices. In addition, by using a multi-branch network instead of multiple point-to-point interconnects to transmit image data and control data between the controller and the image sensor, the space required to implement the network and its power consumption can be further reduced. All of this allows the wearable device to provide high-quality sensing of the surrounding environment, which in turn can improve the performance of applications that rely on the output of the sensing operation.

[0051] The disclosed technology may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof for, for example, creating content in artificial reality and / or being used in other ways in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including HMDs connected to a host computer system, standalone HMDs, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0052] Figure 1Ais a schematic diagram of an embodiment of a near-eye display 100. The near-eye display 100 presents media to a user. Examples of media presented by the near-eye display 100 include one or more images, videos, and / or audio. In some embodiments, the audio is presented via an external device (e.g., a speaker and / or a headphone), which receives audio information from the near-eye display 100, a console, or both, and presents audio data based on the audio information. The near-eye display 100 is typically configured to operate as a virtual reality (VR) display. In some embodiments, the near-eye display 100 is modified to operate as an augmented reality (AR) display and / or a mixed reality (MR) display.

[0053] The near-eye display 100 includes a frame 105 and a display 110. The frame 105 is coupled to one or more optical elements. The display 110 is configured to allow a user to see content presented by the near-eye display 100. In some embodiments, the display 110 includes a waveguide display component for directing light from one or more images to the user's eyes.

[0054] The near-eye display 100 also includes image sensors 120a, 120b, 120c, and 120d. Each of the image sensors 120a, 120b, 120c, and 120d may include a pixel array configured to generate image data representing different fields of view along different directions. For example, sensors 120a and 120b may be configured to provide image data representing two fields of view along the Z axis toward direction A, while sensor 120c may be configured to provide image data representing a field of view along the X axis toward direction B, and sensor 120d may be configured to provide image data representing a field of view along the X axis toward direction C.

[0055] In some embodiments, the sensors 120a-120d may be configured as input devices to control or affect the display content of the near-eye display 100, thereby providing an interactive VR / AR / MR experience to a user wearing the near-eye display 100. For example, the sensors 120a-120d may generate physical image data of the physical environment in which the user is located. The physical image data may be provided to a positioning tracking system to track the user's position and / or movement path in the physical environment. The system may then update the image data provided to the display 110 based on, for example, the user's position and orientation to provide an interactive experience. In some embodiments, as the user moves within the physical environment, the positioning tracking system may run a SLAM algorithm to track a set of objects in the physical environment and within the user's field of view. The positioning tracking system may build and update a map of the physical environment based on the set of objects, and track the user's position within the map. By providing image data corresponding to multiple fields of view, the sensors 120a-120d may provide a more comprehensive view of the physical environment to the positioning tracking system, which may result in more objects being included in the construction and updating of the map. With this arrangement, the accuracy and robustness of tracking the user's position within the physical environment can be improved.

[0056] In some embodiments, the near-eye display 100 may also include one or more active illuminators 130 to project light into the physical environment. The projected light may be associated with different spectrums (e.g., visible light, infrared light, ultraviolet light, etc.) and may be used for various purposes. For example, the illuminator 130 may project light in a dark environment (or in an environment with low intensity infrared light, ultraviolet light, etc.) to help the sensors 120a-120d capture images of different objects in the dark environment, thereby, for example, enabling location tracking of the user. The illuminator 130 may project certain markers onto objects in the environment to help the location tracking system identify objects for map construction / updates.

[0057] In some embodiments, the illuminator 130 can also implement stereoscopic imaging. For example, one or more of the sensors 120a or 120b may include both a first pixel array for visible light sensing and a second pixel array for infrared (IR) light sensing. The first pixel array may be covered with a color filter (e.g., a Bayer filter), and each pixel of the first pixel array is configured to measure the intensity of light associated with a specific color (e.g., one of red, green, or blue). The second pixel array (for IR light sensing) may also be covered with a filter that only allows IR light to pass, and each pixel of the second pixel array is configured to measure the intensity of IR light. The pixel array can generate a red-green-blue (RGB) image and an IR image of the object, wherein each pixel of the IR image is mapped to each pixel of the RGB image. The illuminator 130 can project a set of IR markers onto the object, and the image of the object can be captured by the IR pixel array. Based on the distribution of the IR markers of the object shown in the image, the system can estimate the distance of different parts of the object from the IR pixel array and generate a stereoscopic image of the object based on the distance. Based on the stereoscopic image of the object, the system can determine, for example, the relative position of the object with respect to the user, and can update the image data provided to the display 100 based on the relative position information to provide an interactive experience.

[0058] As discussed above, the near-eye display 100 may operate in environments associated with a very wide range of light intensities. For example, the near-eye display 100 may operate in an indoor environment or an outdoor environment and / or at different times of the day. The near-eye display 100 may also operate with or without the active illuminator 130 turned on. As a result, the image sensors 120a-120d may need to have a wide dynamic range in order to be able to operate correctly (e.g., generate output related to the intensity of the incident light) over a very wide range of light intensities associated with the different operating environments of the near-eye display 100.

[0059] Figure 1B is a schematic diagram of another embodiment of a near-eye display 100 . Figure 1B 1 shows a side of the near-eye display 100 that faces the eyeball 135 of the user wearing the near-eye display 100. Figure 1B As shown, the near-eye display 100 may further include a plurality of illuminators 140a, 140b, 140c, 140d, 140e, and 140f. The near-eye display 100 further includes a plurality of image sensors 150a and 150b. The illuminators 140a, 140b, and 140c may face a direction D (with respect to the direction D). Figure 1AThe system may emit light of a certain frequency range (e.g., near infrared ("NIR")) in the opposite direction A of the user's left eye. The emitted light may be associated with a certain pattern and may be reflected by the user's left eyeball. Sensor 150a may include a pixel array to receive the reflected light and generate an image of the reflected pattern. Similarly, illuminators 140d, 140e, and 140f may emit NIR light carrying a pattern. The NIR light may be reflected by the user's right eyeball and may be received by sensor 150b. Sensor 150b may also include a pixel array to generate an image of the reflected pattern. Based on the images of the reflected patterns from sensors 150a and 150b, the system may determine the user's gaze point and update the image data provided to display 100 based on the determined gaze point to provide an interactive experience to the user.

[0060] As discussed above, in order to avoid damaging the user's eyes, the illuminators 140a, 140b, 140c, 140d, 140e, and 140f are generally configured to output very low intensity light. Figure 1A In the case of the same sensor device as image sensors 120a-120d, when the intensity of the incident light is very low, image sensors 120a-120d may need to be able to generate outputs related to the intensity of the incident light, which may further increase the dynamic range requirements of the image sensors.

[0061] In addition, the image sensors 120a-120d may need to be able to generate outputs at high speeds to track eye movement. For example, a user's eyes may make very rapid movements (e.g., saccade movements), in which they may quickly jump from one eye position to another. In order to track the rapid movement of the user's eyes, the image sensors 120a-120d may need to generate images of the eyes at high speeds. For example, the rate at which the image sensor generates image frames (frame rate) may at least need to match the speed at which the eye moves. A high frame rate requires that the total exposure time of all pixel units involved in generating the image frame be short, and that the speed at which the sensor output is converted into digital values ​​for image generation be fast. In addition, as discussed above, the image sensor may also need to be able to operate in an environment with low light intensity.

[0062] Figure 2 1 is an embodiment of a cross-section 200 of the near-eye display 100 shown in FIG. The display 110 includes at least one waveguide display assembly 210. The exit pupil 230 is the location of a single eyeball 220 of a user in the eyebox region when the user wears the near-eye display 100. For illustration purposes, Figure 2A cross section 200 is shown associated with an eyeball 220 and a single waveguide display assembly 210, but with a second waveguide display for the user's second eye.

[0063] The waveguide display assembly 210 is configured to direct image light to a window located at an exit pupil 230 and to an eyeball 220. The waveguide display assembly 210 may be composed of one or more materials (e.g., plastic, glass) having one or more refractive indices. In some embodiments, the near-eye display 100 includes one or more optical elements between the waveguide display assembly 210 and the eyeball 220.

[0064] In some embodiments, the waveguide display assembly 210 includes a stack of one or more waveguide displays, including but not limited to a stacked waveguide display, a zoom waveguide display, and the like. A stacked waveguide display is a multicolor display (e.g., an RGB display) created by stacking waveguide displays whose corresponding monochromatic sources have different colors. A stacked waveguide display is also a multicolor display (e.g., a multiplane color display) that can be projected on multiple planes. In some configurations, a stacked waveguide display is a monochromatic display (e.g., a multiplane monochromatic display) that can be projected on multiple planes. A zoom waveguide display is a display that can adjust the focus position of image light emitted from a waveguide display. In alternative embodiments, the waveguide display assembly 210 may include a stacked waveguide display and a zoom waveguide display.

[0065] Figure 3 An isometric view of an embodiment of a waveguide display 300 is shown. In some embodiments, the waveguide display 300 is a component of the near-eye display 100 (e.g., the waveguide display assembly 210). In some embodiments, the waveguide display 300 is part of some other near-eye display or other system that directs image light to a specific location.

[0066] The waveguide display 300 includes a source component 310, an output waveguide 320, and a controller 330. For purposes of illustration, Figure 3 The waveguide display 300 is shown associated with a single eyeball 220, but in some embodiments, another waveguide display that is separate or partially separate from the waveguide display 300 provides image light to the user's other eye.

[0067] The source component 310 generates image light 355. The source component 310 generates image light 355 and outputs it to the coupling element 350 located on the first side 370-1 of the output waveguide 320. The output waveguide 320 is an optical waveguide that outputs the expanded image light 340 to the eyeball 220 of the user. The output waveguide 320 receives the image light 355 at one or more coupling elements 350 located on the first side 370-1 and guides the received input image light 355 to the guide element 360. In some embodiments, the coupling element 350 couples the image light 355 from the source component 310 into the output waveguide 320. The coupling element 350 can be, for example, a diffraction grating, a holographic grating, one or more cascade reflectors, one or more prismatic surface elements, and / or a holographic reflector array.

[0068] The guiding element 360 redirects the received input image light 355 to a decoupling element 365, such that the received input image light 355 is decoupled from the output waveguide 320 via the decoupling element 365. The guiding element 360 is part of a first side 370-1 of the output waveguide 320 or is fixed to the first side 370-1 of the output waveguide 320. The decoupling element 365 is part of a second side 370-2 of the output waveguide 320 or is fixed to the second side 370-2 of the output waveguide 320 such that the guiding element 360 is opposite to the decoupling element 365. The guiding element 360 and / or the decoupling element 365 may be, for example, a diffraction grating, a holographic grating, one or more cascade reflectors, one or more prismatic face elements, and / or a holographic reflector array.

[0069] The second side 370-2 represents a plane along the x-dimension and the y-dimension. The output waveguide 320 can be composed of one or more materials that facilitate total internal reflection of the image light 355. The output waveguide 320 can be composed of, for example, silicon, plastic, glass, and / or a polymer. The output waveguide 320 has a relatively small form factor. For example, the output waveguide 320 can be about 50 mm wide along the x-dimension, about 30 mm long along the y-dimension, and about 0.5 mm-1 mm thick along the z-dimension.

[0070] The controller 330 controls the scanning operation of the source component 310. The controller 330 determines the scanning instructions for the source component 310. In some embodiments, the output waveguide 320 outputs the expanded image light 340 to the user's eyeball 220 with a large FOV. For example, the expanded image light 340 is provided to the user's eyeball 220 with a diagonal FOV (in x and y) of 60 degrees and / or greater and / or 150 degrees and / or less. The output waveguide 320 is configured to provide a viewing window having a length of 20 mm or greater and / or equal to or less than 50 mm; and / or a width of 10 mm or greater and / or equal to or less than 50 mm.

[0071] In addition, the controller 330 also controls the image light 355 generated by the source component 310 based on the image data provided by the image sensor 370. The image sensor 370 can be located on the first side 370-1 and can include, for example Figure 1A 370-1. The image sensors 120a-120d can be operated to perform 2D sensing and 3D sensing of, for example, an object 372 in front of a user (e.g., facing the first side 370-1). For 2D sensing, each pixel unit of the image sensors 120a-120d can be operated to generate pixel data representing the intensity of light 374 generated by the light source 376 and reflected by the object 372. For 3D sensing, each pixel unit of the image sensors 120a-120d can be operated to generate pixel data representing the time-of-flight measurement results of light 378 generated by the illuminator 325. For example, each pixel unit of the image sensors 120a-120d can determine a first time when the illuminator 325 is enabled to project light 378 and a second time when the pixel unit detects light 378 reflected from the object 372. The difference between the first time and the second time can indicate the flight time of the light 378 between the image sensors 120a-120d and the object 372, and the flight time information can be used to determine the distance between the image sensors 120a-120d and the object 372. The image sensors 120a-120d can be operated to perform 2D and 3D sensing at different times and provide 2D and 3D image data to a remote control console 390, which may (or may not) be located within the waveguide display 300. The remote control console can combine the 2D and 3D images to, for example, generate a 3D model of the user's environment, track the user's location and / or orientation, etc. The remote control console can determine the content of the image to be displayed to the user based on the information derived from the 2D and 3D images. The remote control console can send instructions related to the determined content to the controller 330. Based on these instructions, the controller 330 can control the source component 310 to generate and output image light 355 to provide an interactive experience to the user.

[0072] Figure 4An embodiment of a cross-section 400 of a waveguide display 300 is shown. The cross-section 400 includes a source component 310, an output waveguide 320, and an image sensor 370. Figure 4 In the example of , the image sensor 370 may include a group of pixel units 402 located on the first side 370-1 to generate an image of the physical environment in front of the user. In some embodiments, a mechanical shutter 404 and an optical filter array 406 may be inserted between the group of pixel units 402 and the physical environment. The mechanical shutter 404 can control the exposure of the group of pixel units 402. In some embodiments, as will be discussed below, the mechanical shutter 404 can be replaced by an electronic shutter gate. As described below, the optical filter array 406 can control the optical wavelength range of light to which the group of pixel units 402 are exposed. Each pixel unit 402 can correspond to a pixel of the image. Although Figure 4 Not shown, but understood, each pixel unit 402 may also be covered with a filter to control the optical wavelength range of light to be sensed by the pixel unit.

[0073] After receiving the instruction from the remote console, the mechanical shutter 404 may open and expose the group of pixel units 402 in an exposure period. During the exposure period, the image sensor 370 may obtain light samples incident on the group of pixel units 402, and generate image data based on the intensity distribution of the incident light samples detected by the group of pixel units 402. The image sensor 370 may then provide the image data to the remote console, which determines the display content, and provides the display content information to the controller 330. The controller 330 may then determine the image light 355 based on the display content information.

[0074] Source assembly 310 generates image light 355 according to instructions from controller 330. Source assembly 310 includes source 410 and optical system 415. Source 410 is a light source that generates coherent light or partially coherent light. Source 410 can be, for example, a laser diode, a vertical cavity surface emitting laser, and / or a light emitting diode.

[0075] The optical system 415 includes one or more optical components that condition the light from the source 410. Conditioning the light from the source 410 may include, for example, expanding, collimating, and / or adjusting the orientation according to instructions from the controller 330. The one or more optical components may include one or more lenses, liquid lenses, mirrors, apertures, and / or gratings. In some embodiments, the optical system 415 includes a liquid lens with multiple electrodes that allows the light beam to be scanned with a threshold scanning angle to move the light beam to an area outside the liquid lens. The light emitted from the optical system 415 (and the source assembly 310) is referred to as image light 355.

[0076] The output waveguide 320 receives the image light 355. The coupling element 350 couples the image light 355 from the source component 310 into the output waveguide 320. In embodiments where the coupling element 350 is a diffraction grating, the pitch of the diffraction grating is selected such that total internal reflection occurs in the output waveguide 320, and the image light 355 propagates internally in the output waveguide 320 (e.g., by total internal reflection) toward the decoupling element 365.

[0077] The guiding element 360 redirects the image light 355 to the decoupling element 365 for decoupling from the output waveguide 320. In embodiments where the guiding element 360 is a diffraction grating, the pitch of the diffraction grating is selected so that the incident image light 355 exits the output waveguide 320 at an angle oblique to the surface of the decoupling element 365.

[0078] In some embodiments, the guide element 360 and / or the decoupling element 365 are similar in structure. The expanded image light 340 exiting the output waveguide 320 is expanded along one or more dimensions (e.g., can be elongated along the x dimension). In some embodiments, the waveguide display 300 includes a plurality of source components 310 and a plurality of output waveguides 320. Each source component 310 emits a monochromatic image light of a specific wavelength band corresponding to a primary color (e.g., red, green, blue). Each output waveguide 320 can be stacked together at a certain spacing distance to output a multi-color expanded image light 340.

[0079] Figure 5 is a block diagram of an embodiment of a system 500 including a near-eye display 100. The system 500 includes the near-eye display 100, an imaging device 535, an input / output interface 540, and image sensors 120a-120d and 150a-150b, each of which is coupled to a control circuit 510. The system 500 can be configured as a head-mounted device, a mobile device, a wearable device, etc.

[0080] The near-eye display 100 is a display that presents media to the user. Examples of media presented by the near-eye display 100 include one or more images, videos, and / or audio. In some embodiments, the audio is presented via an external device (e.g., a speaker, a headset), which receives audio information from the near-eye display 100 and / or the control circuit 510 and presents audio data to the user based on the audio information. In some embodiments, the near-eye display 100 can also act as AR eyewear glasses. In some embodiments, the near-eye display 100 uses computer-generated elements (e.g., images, videos, sounds, etc.) to enhance the view of the physical, real-world environment.

[0081] The near-eye display 100 includes a waveguide display assembly 210, one or more position sensors 525, and / or an inertial measurement unit (IMU) 530. The waveguide display assembly 210 includes a source assembly 310, an output waveguide 320, and a controller 330.

[0082] The IMU 530 is an electronic device that generates rapid calibration data based on measurement signals received from one or more position sensors 525 , the rapid calibration data indicating an estimated position of the near-eye display 100 relative to an initial position of the near-eye display 100 .

[0083] The imaging device 535 can generate image data for various applications. For example, the imaging device 535 can generate image data based on calibration parameters received from the control circuit 510 to provide slow calibration data. The imaging device 535 can include, for example Figure 1A The image sensors 120a-120d are used to generate image data of the user's physical environment for performing location tracking of the user. The imaging device 535 may further include, for example Figure 1B The image sensors 150a-150b are used to generate image data for determining the user's gaze point to identify objects of interest to the user.

[0084] The input / output interface 540 is a device that allows a user to send an action request to the control circuit 510. An action request is a request to perform a specific action. For example, the action request may be to start or end an application, or to perform a specific action within an application.

[0085] The control circuit 510 provides media to the near-eye display 100 for presentation to the user based on information received from one or more of the imaging device 535, the near-eye display 100, and the input / output interface 540. In some examples, the control circuit 510 can be housed within the system 500 configured as a head mounted device. In some examples, the control circuit 510 can be a stand-alone console device that is communicatively coupled to other components of the system 500. Figure 5 In the example shown, the control circuit 510 includes an application storage 545 , a tracking module 550 , and an engine 555 .

[0086] Application storage 545 stores one or more applications for execution by control circuit 510. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0087] Tracking module 550 uses one or more calibration parameters to calibrate system 500 , and may adjust the one or more calibration parameters to reduce errors in the determination of the position of near-eye display 100 .

[0088] The tracking module 550 uses the slow calibration information from the imaging device 535 to track the movement of the near-eye display 100. The tracking module 550 also uses the position information from the fast calibration information to determine the position of a reference point of the near-eye display 100.

[0089] The engine 555 executes applications within the system 500 and receives position information, acceleration information, velocity information, and / or predicted future position of the near-eye display 100 from the tracking module 550. In some embodiments, the information received by the engine 555 can be used to generate a signal (e.g., a display instruction) to the waveguide display assembly 210, which determines the type of content presented to the user. For example, in order to provide an interactive experience, the engine 555 can determine the content to be presented to the user based on the user's position (e.g., provided by the tracking module 550), the user's gaze point (e.g., based on image data provided by the imaging device 535), and the distance between the object and the user (e.g., based on image data provided by the imaging device 535).

[0090] Fig. 6A , Figure 6B , Figure 6C and Fig.6D An example of an image sensor 600 and its operation is shown. Fig. 6A As shown, the image sensor 600 may include a pixel unit array, including a pixel unit 601, and may generate digital intensity data corresponding to a pixel of an image. The pixel unit 601 may be Figure 4 A portion of the pixel unit 402. Fig. 6A As shown, the pixel unit 601 may include a photodiode 602, an electronic shutter switch 603, a transfer switch 604, a charge storage device 605, a buffer 606, and a quantizer 607. The photodiode 602 may include, for example, a PN junction diode, a PIN junction diode, a pinned diode, etc., and the charge storage device 605 may be a floating drain node of the transfer switch 604. When receiving light during an exposure period, the photodiode 602 may generate and accumulate residual charge. When the residual charge is saturated during the exposure period, the photodiode 602 may output overflow charge to the charge storage device 605 through the transfer switch 604. The charge storage device 605 may convert the overflow charge into a voltage, which may be buffered by the buffer 606. The quantizer 607 may quantize the buffered voltage to generate measurement data 608, for example, representing the intensity of light received by the photodiode 602 during the exposure period.

[0091] The quantizer 607 may include a comparator for comparing the buffered voltage to different thresholds for different quantization operations associated with different intensity ranges. For example, for a high intensity range where the amount of overflow charge generated by the photodiode 602 exceeds the saturation limit of the charge storage device 605, the quantizer 607 may perform a saturation time (TTS) measurement operation by detecting whether the buffered voltage exceeds a static threshold representing the saturation limit, and if so, measuring the time it takes for the buffered voltage to exceed the static threshold. The measured time may be inversely proportional to the light intensity. In addition, for a medium intensity range where the photodiode is saturated with residual charge but the overflow charge remains below the saturation limit of the charge storage device 605, the quantizer 607 may perform an FD ADC operation to measure the amount of overflow charge stored in the charge storage device 605. In addition, for a low intensity range where the photodiode is not saturated with residual charge and no overflow charge is accumulated in the charge storage device 605, the quantizer 607 may perform a PD ADC operation to measure the amount of residual charge accumulated in the photodiode 602. The output of one of the TTS, FD ADC, or PD ADC operations may be output as measurement data 608 to represent the intensity of the light.

[0092] Figure 6B An example operation sequence of pixel unit 601 is shown. Figure 6B As shown, the exposure period can be defined based on the timing of the AB signal that controls the electronic shutter switch 603, which can direct away the charge generated by the photodiode 602 when enabled, and based on the timing of the TG signal that controls the transfer switch 604, which is controlled to transfer the overflow charge to the charge storage device 605 and then transfer the remaining charge to the charge storage device 605 for readout. For example, referring to Figure 6B, the AB signal may be asserted inactive at time T0 to allow the photodiode 602 to generate charge. T0 may mark the beginning of an exposure period. During the exposure period, the TG signal may set the transfer switch 604 to a partially on state to allow the photodiode 602 to accumulate at least some of the charge as residual charge until the photodiode 602 is saturated, after which the overflowing charge may be transferred to the charge storage device 605. Between times T0 and T1, the quantizer 607 may perform a TTS operation to determine whether the overflowing charge at the charge storage device 605 exceeds a saturation limit, and then between times T1 and T2, the quantizer 607 may perform an FD ADC operation to measure the amount of overflowing charge at the charge storage device 605. Between times T2 and T3, the TG signal may be asserted active to bias the transfer switch 604 in a fully on state, thereby transferring the residual charge to the charge storage device 605. At time T3, the TG signal may be asserted inactive to isolate the charge storage device 605 from the photodiode 602, and the AB signal may be asserted active to direct away the charge generated by the photodiode 602. Time T3 may mark the end of the exposure period. Between time T3 and T4, the quantizer 607 may perform a PD operation to measure the amount of remaining charge.

[0093] AB and TG signals can be controlled by the controller ( Fig. 6A The controller 601 may be a part of the pixel unit 601 to control the duration of the exposure period and the sequence of quantization operations. The controller may also detect whether the charge storage device 605 is saturated and whether the photodiode 602 is saturated to select an output from one of the TTS, FD ADC, or PD ADC operations as the measurement data 608. For example, if the charge storage device 605 is saturated, the controller may provide a TTS output as the measurement data 608. If the charge storage device 605 is not saturated, but the photodiode 602 is saturated, the controller may provide a FD ADC output as the measurement data 608. If the photodiode 602 is not saturated, the controller may provide a PD ADC output as the measurement data 608. The measurement data 608 from each pixel unit of the image sensor 600 generated during the exposure period may form an image frame. The controller may repeat the above steps in subsequent exposure periods. Figure 6B to generate subsequent image frames.

[0094] Image frame data from image sensor 600 may be sent to a host processor (at Figure 6A-6D612 , performing depth sensing of the object 612 relative to the image sensor 600, etc. For all these applications, only a subset of the pixel cells provide relevant information (e.g., pixel data of the object 612), while the remaining pixel cells do not provide relevant information. Figure 6C , at time T0, a group of pixel cells 620 of image sensor 600 receive light reflected by object 612, and at time T6, object 612 may have shifted (e.g., due to movement of object 612, movement of image sensor 600, or both), and a group of pixel cells 630 of image sensor 600 receive light reflected by object 612.

[0095] In some examples, the image sensor 600 may determine a region of interest (ROI) including pixel data from pixel unit group 620 (at time T0) and pixel data from pixel unit group 630 (at time T6), and only send pixel data from the ROI to the host processor to reduce the amount of pixel data transmitted. In some examples, the image sensor 600 may also cause all pixels to send pixel data, but the pixel units corresponding to the ROI may have different configurations than the pixel units outside the ROI. For example, pixel unit groups 620 and 630 may generate and output pixel data at a higher quantization resolution to represent an image of object 612, while the remaining pixel units may generate and output pixel data at a lower resolution. As another example, pixel unit groups 620 and 630 may have a longer exposure period than other pixel units. As another example, pixel cell groups 620 and 630 may have a wider dynamic range (e.g., based on performing TTS, FD ADC, and PD ADC operations), while the remaining pixel cells may have a narrower dynamic range (e.g., based on disabling one or more of the TTS, FD ADC, and PD ADC operations). All of these arrangements may allow for the generation and transmission of higher resolution images without a corresponding increase in power and bandwidth. For example, a larger pixel cell array including more pixel cells may be used to image object 612 to improve image resolution, and when only a subset of pixel cells including pixel cells providing pixel data for object 612 generates high-quality pixel data and sends the high-resolution pixel data to a host processor, while the remaining pixel cells either do not generate / send pixel data, or generate / send pixel data at relatively low quality, the bandwidth and power required to provide the improved image resolution may be reduced. In addition, although image sensor 600 may be operated to generate images at a higher frame rate, the increase in bandwidth and power may be reduced when each image includes only a small set of pixel values ​​that are high resolution and represented by a large number of bits, while the remaining pixel values ​​have very low resolution and are represented by fewer bits.

[0096] In the case of 3D sensing, the amount of pixel data sent can also be reduced. Fig.6D , the illuminator 640 can project a pattern 642 of structured light onto the object 650. The structured light can be reflected on the surface of the object 650, and the pattern 652 of the reflected light can be captured by the image sensor 600 to generate an image. The host processor can match the pattern 652 with the pattern 642 and determine the depth of the object 650 relative to the image sensor 600 based on the image position of the pattern 652 in the image. For 3D sensing, only the pixel unit groups 660, 662, 664 and 666 contain relevant information (e.g., pixel data of the pattern 652). In order to reduce the amount of pixel data transmitted, the image sensor 600 can be configured to send only pixel data from the ROI including the pixel unit groups 660, 662, 664 and 666 to the host processor, or to send pixel data from the pixel unit groups 660, 662, 664 and 666 to the host processor at a high resolution, and send the remaining pixel data to the host processor at a low resolution.

[0097] Fig. 7A An example of a mobile device 700 including multiple sensors is shown. The mobile device 700 may include Figure 1A A near-eye display 100 is provided. Fig. 7A The left figure of FIG. 7 shows a three-dimensional view of an example of a mobile device 700. Fig. 7A As shown on the left side of , the mobile device 700 may be in the form of a head mounted display (HMD). A plurality of sensors may be distributed on the mobile device 700, wherein each sensor (or sensor group) is oriented in a specific direction. For example, sensors 702a and 702b may be located on the front surface of the mobile device 700 facing direction A, sensor 704 may be located on the side surface of the mobile device 700 facing direction B, sensor 706 may be located on the other side surface of the mobile device 700 facing direction C, and sensor 708 may be located on the rear side of the mobile device 700 facing direction D (e.g., located on the headband 710).

[0098] Each set of sensors 702a, 702b, 704, 706, and 708 may include Fig. 6AThe image sensor 600 (including an array of pixel cells) and other types of sensors (e.g., audio sensors and motion sensors) and illuminators (e.g., infrared illuminators) of the mobile device 700 may be provided. Each group of sensors may have a field of view (FOV) facing the direction that the group of sensors respectively faces, and the image captured by each group of sensors may represent a portion of the surrounding environment within the corresponding FOV. For example, sensor 702a and sensor 702b may have FOV 712a and FOV 712b, respectively, both of which face direction A. In addition, sensor 704 may have FOV 714 facing direction B, sensor 706 may have FOV 716 facing direction C, and sensor 708 may have FOV 718 facing direction D. The controller of the mobile device 700 may process the images captured by the sensors to obtain information about the surrounding environment, which allows the mobile device to have a combined FOV of the surrounding environment, which combines FOVs 712a, 712b, 714, 716, and 718. With the controller processing images from sensors 702a, 702b, 704, 706, and 708, the controller can obtain images captured from the front, back, and both sides of the mobile device 700, which can result in a combined FOV of nearly 360 degrees.

[0099] In some examples, each set of sensors 702a, 702b, 704, 706, and 708 can have a relatively small pixel cell array with a relatively low (e.g., less than 1 megapixel) resolution. Due to the reduced size of the pixel cell array, each set of sensors 702a, 702b, 704, 706, and 708 can have a reduced silicon area and a reduced form factor, which allows the sensors to be distributed at different locations of the mobile device 700 where available space is very limited. On the other hand, when the images from the sensors are combined to combine the FOV provided by each set of sensors, the mobile device can still provide a wide FOV.

[0100] The wide FOV provided by the mobile device 700 can enhance the performance of various applications that rely on image data provided by the image sensor, such as SLAM operations, context determination applications, hand tracking applications, etc., all of which can determine content to be output to the user to provide an interactive AR / VR / MR experience.

[0101] Specifically, in SLAM operation, when the user moves in the environment, certain significant features of physical objects in the user's surrounding environment can be tracked relative to time. Tracking can be based on identifying a set of features across multiple image frames captured by the image sensor at different times, and determining the pixel position of the features in those image frames. Based on the results of the tracking, the position of the physical object with the image features relative to the user can be determined. A map of the environment and the position of the user in the environment at those times can also be determined. In order to make the SLAM operation more robust, more significant features can be tracked to reduce the impact of tracking errors of specific features on the overall accuracy of the SLAM operation. The extended FOV provided by image sensors 702a, 702b, 704, 706 and 708 allows more significant features to be tracked to improve the robustness of the SLAM operation.

[0102] Figure 7B An exemplary SLAM operation 720 is shown. Figure 7B As shown, to support SLAM operation 720, image sensor 702a can capture images including physical object 724 and its features 725 within FOV 712a, image sensor 704 can capture images including physical object 726 and its features 727 within FOV 714, and image sensor 706 can capture images including physical object 728 and its features 729 within FOV 716. Compared to the case where mobile device 700 only captures images of physical objects in front of it, the extended FOV provided by multiple image sensors allows mobile device 700 to capture images of physical objects around the mobile device, which in turn allows tracking of more significant features to make SLAM operation 720 more robust. In some examples, the FOVs of image sensors facing the same direction (e.g., image sensors 702a and 702b) can be combined so that the image sensors form a stereo pair. The image sensors of the stereo pair can detect light in the same frequency range (e.g., visible light) or light in different frequency ranges (e.g., visible light and infrared) to support depth sensing.

[0103] In addition, the expanded FOV also allows the mobile device 700 to capture features that would otherwise be obscured by another physical object. Figure 7B In the example, object 724 is located between image sensor 702a and object 730, and object 724 may block feature 731 from being captured by image sensor 702a. However, feature 731 is also within the FOV of image sensor 702b, which allows feature 731 to be imaged and tracked to improve the robustness of SLAM operation 720.

[0104] Figure 7C740 that may be supported by the mobile device 700. The scene context determination allows the mobile device 700 to obtain information about a scene that is not within the line of sight of the user of the mobile device and determine the context of the scene based on the information. Figure 7C , a user of mobile device 700 interacts with person 742 standing in front of the user (e.g., in direction A), where person 742 is within the user's line of sight. However, there are other people in the scene that are not within the user's line of sight, including people 744, 746, and 748. Images of people 744, 746, and 748 can be captured by image sensors 704, 706, and 708, respectively. In some examples, mobile device 700 can display images of people 744, 746, and 748 to the user to enable the user to determine the context of the scene. For example, as part of a contextual artificial intelligence (AI) operation, the postures, facial expressions, or other activities of these people can be determined. In addition, if a person is speaking, the position of the speaker can be determined for directional audio processing. As another example, mobile device 700 can execute a scene context determination application that determines the context based on images of people 744, 746, and 748. Based on the context, output content can be generated.

[0105] Fig.7D An example of object tracking operations 760 that may be supported by mobile device 700 is shown. Fig.7D In the example of FIG. 7 , an object tracking operation 760 may be performed to track the hand of a user 762 ( Fig.7D Due to the different orientations of image sensors 704, 706, and 708, mobile device 700 can capture images of the user's hand in a variety of locations, not just in front of user 762 and mobile device 700. For example, Fig.7D As shown, when the user 762 moves the hand behind his / her head or body, or when the user 762 moves the hand to the side of the mobile device 700, the mobile device 700 can capture an image of the hand.

[0106] In some examples, images of hands captured by image sensors 704, 706, and 708 can be combined to improve the robustness of object tracking operation 760. For example, the controller can control image sensors 704, 706, and 708 to capture images of hands in a particular position. The images can then be combined in the controller to form a stereoscopic image of the user's hand, thereby improving the depth estimation of the hand relative to the user 762. As another example, the controller can also select an image captured from one of image sensors 704, 706, or 708 to perform object tracking. The selection can be based on various criteria. For example, the controller can detect that the image captured by image sensor 704 includes certain target features (e.g., target gestures) that are unlikely to be detected in images from other image sensors, and the controller can determine that the image from image sensor 704 will be used to track the object.

[0107] The sensors (including image sensors 702a, 702b, 704, 706, and 708) and the controller of the mobile device 700 may be connected together by a multi-branch network through which the sensors and the controller communicate. Fig. 8A An example of a sensor network 800 that may be part of a mobile device 700 is shown. Fig. 8A As shown, sensor network 800 may include physical link 802. Sensor network 800 also includes multiple sensors (including 804, 806, 808, and 810, etc.) and a host controller 812, each of which is coupled to physical link 802. Sensors 804, 806, 808, and 810 may include, for example Fig. 7A The host controller 812 may control the sensing operations at the sensors 804, 806, 808, and 810 and process the sensor data obtained by the sensors to support applications such as SLAM operations 720, scene context determination operations 740, and object tracking operations 760. For example, the host controller 812 may combine the image data generated by the sensors 804, 806, 808, and 810 to provide an expanded FOV of the surrounding environment of the mobile device 700.

[0108] The sensor network 800 can be in the form of a multi-branch network in which sensors 804-810 and a host controller 812 communicate with each other using the same physical link 802. For example, each of the sensors 804, 806, 808, and 810 can generate sensor data 814, 816, 818, and 820, respectively, and send the sensor data to the host controller 812 via the physical link 802. The sensor data 814, 816, 818, and 820 can include, for example, image data, audio data, and motion data. In addition, the host controller 812 can generate control data 822 to control the sensing operation at the sensors 804-810, and send the control data 822 to the sensors via the physical link 802.

[0109] Physical link 802 may be implemented using various techniques. Figure 8B An example of a physical link 802 is shown. Figure 8B In the embodiment of the present invention, the physical link 802 may include a bus that can be configured based on the I2C or I3C specification and includes a serial data line (SDL) 824 and a serial clock line (SCL) 826. Each of the SDL 824 and the SCL 826 is configured as an open-drain line coupled to pull-up resistors 828 and 830, respectively. When the SDL line and the SCL line are not driven, they can be pulled up to the power supply voltage by the pull-up resistors. In the I2C / I3C specification, the master node generates a clock and initiates communication with the slave node, and the slave node receives the clock and responds when addressed by the master node. In the sensor network 800, the host controller 812 is configured as a master node, and the sensors 804-810 are configured as slave nodes. To send the control data 822, the host controller 812 may pull down the SCL 826 based on the clock signal pattern, and pull down the SDL 824 based on a serial data pattern including the control data 822 and the address of the target sensor(s). The target sensor(s) may then drive another serial data pattern including the sensor data on the SDL 824. Since the sensors share the same SDL 824, only one of the sensors may send sensor data to the host controller 812 via the SDL 824 at a time. When multiple sensors attempt to drive the SDL 824 simultaneously, the timing at which each sensor sends the sensor data may be controlled by the host controller 812 via the control data 822 defined based on a predetermined transmission schedule, and / or based on back-off delays.

[0110] Figure 8C Another example of a physical link 802 is shown. Figure 8CAs shown, the physical link 802 may include an optical link 840 that can transmit light 841 from a light source 842 to a receiver system 843 coupled to a host controller 812. The optical link 840 may include an optical fiber, a waveguide, or any medium that propagates light. Each of the sensors 804, 806, 808, and 810 may be coupled to the optical link 840 via optical modulators 844, 846, 848, and 850, respectively. In order to send sensor data to the host controller 812, each sensor may control its respective optical modulator to modulate the light 841. The receiver system 843 may include a photodetector (e.g., a photodiode) to convert the modulated light 841 into an electrical signal that can be received and processed by the host controller 812 to obtain sensor data. In some examples, the sensor may modulate the light 841 based on a time division multiplexing scheme, where each sensor modulates the light 841 in turn to send the sensor data to the host controller 812. In some examples, the sensor may also simultaneously modulate different frequency components of the light 841 in a frequency division multiplexing scheme. In addition, the host controller 812 and each sensor can be coupled to a bus 852, and the host controller 812 can send control data 822 through the bus 852. The bus 852 can include an I2C bus, an I3C bus, or other bus technologies. In this way, the optical link 840 and the bus 852 can be combined to provide a physical link 802.

[0111] In some examples, to reduce the amount of image data transmitted over physical link 802 (which can reduce the power and bandwidth of physical link 802), host controller 812 can configure the image capture operations of sensors 804-810 to reduce the amount of image data transmitted over physical link 802 while providing the image data required by the application. Fig.8D An example configuration of sensors 804-810 is shown. Fig.8D As shown, host controller 812 may select a subset of sensors (including sensors 804 and 806) to send sensor data 814 and 816 via physical link 802, while unselected sensors (e.g., sensors 808 and 810) are disabled or otherwise not permitted to send sensor data via physical link 802. In some examples, host controller 812 may select a subset of sensors to send image data at a higher resolution and / or a higher frame rate, while unselected sensors may send image data at a lower resolution and / or a lower frame rate.

[0112] The host controller 812 selects the subset of sensors in various ways. For example, the host controller 812 may determine that the subset of sensors may capture images of the object of interest (e.g., the user's hand, other physical objects for location tracking) at a given time based on, for example, detecting features of the object of interest from previous images captured by the subset of sensors. In the case where the object of interest moves relative to the mobile device 700, the host controller 812 may determine the predicted movement trajectory of the object relative to the mobile device 700. Based on the predicted movement trajectory, the host controller 812 may determine which sensors may capture images of the object at a given time and enable different subsets of image sensors at different times. The predicted trajectory may be based on, for example, the pixel position of the object in previous and recent images captured by the sensor relative to time, and the previous and recent positioning and / or orientation of the mobile device 700. For example, based on detecting that the user's head (and the mobile device 700) is rotating to view a flying object, the host controller 812 may predict the trajectory of the flying object relative to the sensors of the mobile device 700, and determine the subset of sensors that are most likely to capture images of the flying object at different points in time as the user continues to rotate his / her head. The host controller 812 can then enable a subset of the sensors at those points in time while disabling the remaining sensors to reduce the amount of image data transmitted over the physical link 802. The host controller 812 can also apply similar techniques to selectively enable / disable the transmission of other types of sensor data, such as audio data.

[0113] Fig. 8E Another example configuration of sensors 804-810 is shown. Fig. 8E As shown, the host controller 812 can control the sensors 804-810 or a subset of the sensors 804-810 selected as described above (e.g., sensor 804) to perform sparse sensing operations to further reduce the amount of image data transmitted over the network. Specifically, the host controller 812 can determine, for each sensor enabled to send image data, an ROI (e.g., Figure 6C and Fig.6D The determination of the ROI may be based on, for example, detecting features of the object of interest from previous images captured by those sensors, predicting the movement trajectory of the object relative to the mobile device 700, as described above in Fig.8D In Fig. 8E, the host controller 812 can determine ROIs 860, 862, and 864 that include pixels of a person's head and hands. The host controller 812 can then control the sensor 804 to enable only a subset of pixel cells corresponding to the ROIs 860, 862, and 864 to detect light and / or send pixel data, while the remaining pixels of the sensor 804 do not send pixel data. As a result, the sensor data 814 includes only pixels within the ROIs 860, 862, and 864.

[0114] In addition to sparse sensing, the host controller 812 can also change other aspects of the image capture operation between pixel cells belonging to the ROI and pixel cells that do not belong to the ROI. For example, the host controller 812 can increase the quantization resolution, exposure period, dynamic range, etc. of the pixel cells belonging to the ROI relative to other pixel cells (as described above in Figure 6C and Fig.6D to improve the quality of image data.

[0115] The host controller 812 can provide ROI information to the sensor in various ways. For example, the control data 822 may include a programming map that specifies the pixel cells (or blocks of pixel cells) that are part of the ROI for each image frame. In the case where the object of interest moves relative to the mobile device 700, the host controller 812 may send updated control data 822, which includes updated ROIs for different image frames. In another example, the sensor may include certain computing capabilities to determine the ROI locally, and the host controller 812 may send a guidance signal to the sensor as part of the control data 822 to guide the determination of the ROI at the sensor. The guidance signal may include information identifying the following: features of the object of interest, a rough estimate of the location and size of the ROI, etc. Based on the guidance data, the sensor may determine the ROI based on detecting specified features, refining the estimate of the ROI, etc. With such an arrangement, the amount of control data sent to the sensor by the host controller 812 can be reduced because the controller does not need to send updated ROI information to the sensor between image frames to account for the movement of the mobile device and / or the object of interest, which can further reduce the power and bandwidth requirements of the network.

[0116] Fig. 9A , Fig. 9B , Fig. 9C and Fig.9D An example of the internal components of the host controller 812 and sensors 804-810 is shown. Fig. 9AAs shown, the sensor 804 includes a sensor computing circuit 906 and a pixel unit array 908. The sensor computing circuit 906 includes an image processor 910 and a programming map generator 912. In some examples, the sensor computing circuit 906 can be implemented as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a hardware processor that executes instructions to implement the functions of the image processor 910 and the programming map generator 912. In addition, the host controller 812 can include a general-purpose central processing unit (CPU) that can execute an application 914.

[0117] Each pixel cell or pixel cell block of the pixel cell array 908 may be individually programmable, for example, to enable / disable the output of a pixel value, set the resolution of the pixel value output by the pixel cell, etc. The pixel cell array 908 may receive a first programming signal 920 from a programming map generator 912 of the sensor computation circuit 906, which may be in the form of a programming map containing programming data for each pixel cell. The pixel cell array 908 may sense light from a scene and generate a first image frame 922 of the scene based on the first programming signal 920. Specifically, the pixel cell array 908 may be controlled by the first programming signal 920 to operate in different sparsity modes, such as in a full frame mode where the first image frame 922 includes pixels of a complete image frame, and / or in a sparse mode where the first image frame 922 includes only a subset of pixels specified by the programming map. In some examples, the programming map generator 912 may be part of the host controller 812, where the pixel cell array 908 receives the first programming signal 920 from the host controller 812.

[0118] In addition to generating the first programming signal 920, the sensor computing circuit 906 may also generate a global signal that is sent to each pixel cell of the pixel cell array 908. The global signal may include, for example, threshold voltages for quantization operations in TTS, FDADC, and PD ADC operations (e.g., global voltage ramps for FD ADC and PD ADC operations, flat voltages for TTS operations, etc.), and global control signals (such as Figure 6B AB and TG signals).

[0119] The pixel cell array 908 can output the first image frame 922 to both the host controller 812 and the sensor computation circuit 906. In some examples, the pixel cell array 908 can also output the first image frame 922 with different pixel sparsities to the host controller 812 and the sensor computation circuit 906. For example, the pixel cell array 908 can output the first image frame 922 with pixels of a complete image frame back to the sensor computation circuit 906, and output the first image frame 922 with sparse pixels defined by the first programming signal 920 to the host controller 812.

[0120] The sensor computing circuit 906 and the host controller 812 and the image sensor 804 may form a two-layer feedback system based on the first image frame 922 to control the image sensor to generate a subsequent image frame 924. In the two-layer feedback operation, the image processor 910 of the sensor computing circuit 906 may perform an image processing operation on the first image frame 922 to obtain a processing result, and then the programming map generator 912 may update the first programming signal 920 based on the processing result. The image processing operation at the image processor 910 may be directed / configured based on the second programming signal 932 included in the control data 822 from the host controller 812, and the host controller 812 may generate the second programming signal 920 based on the first image frame 922. The pixel unit array 908 may then generate a subsequent image frame 924 based on the updated first programming signal 920. The host controller 812 and the sensor computing circuit 906 may then update the first programming signal 920 and the second programming signal 932 based on the subsequent image frame 924, respectively.

[0121] In the aforementioned two-layer feedback system, the second programming signal 932 of the control data 822 from the host controller 812 can be in the form of a teaching / guidance signal, a result of a neural network training operation (e.g., a back-propagation result), etc., to affect the image processing operation and / or programming map generation at the sensor computing circuit 906. The host controller 812 can determine the context of the light sensing operation performed by the image sensor 804 based not only on the first image frame, but also on other sensor data (e.g., other image frames captured by other image sensors, audio information, motion sensor output, input from a user, etc.), and then determine the teaching / guidance signal to generate the teaching / guidance signal. The context may include, for example, the environmental conditions in which the image sensor 804 operates, the positioning of the image sensor 804, the characteristics of the object of interest, or any other requirements of the application 914. Considering that the context usually changes at a much lower rate than the frame rate, the teaching / guidance signal can be updated at a relatively low rate (e.g., lower than the frame rate) based on the context, while the image processing operation and programming map update at the sensor computing circuit 906 can occur at a relatively high rate (e.g., at the frame rate) to adapt to the image captured by the pixel unit array 908.

[0122] although Fig. 9A The pixel cell array 908 is shown sending the first image frame 922 and the second image frame 924 to both the host controller 812 and the sensor computing circuit 906, but in some cases, the pixel cell array 908 may send image frames of different sparsity to the host controller 812 and the sensor computing circuit 906. For example, the pixel cell array 908 may send the first image frame 922 and the second image frame 924 with full pixels to the image processor 910, and send sparse versions of the two image frames to the host controller 812, the sparse version of each image frame including a subset of pixels selected based on the first programming signal 920.

[0123] Fig. 9B An example of the internal components of the pixel cell array 908 is shown. Fig. 9B As shown, the pixel cell array 908 may include a column controller 934, a row controller 936, and a programming signal parser 940. The column controller 934 is connected to a column bus 935 (e.g., 935a, 935b, ..., 935n), and the row controller 936 is connected to a row bus 937 (e.g., 937a, 937b, ..., 937n). One of the column controller 934 or the row controller 936 is also connected to a programming bus 942 to send a pixel level programming signal 943 targeting a specific pixel cell or a group of pixel cells. The pixel cells marked with P 00 , P 01 , P 0jEach block of the above blocks may represent a pixel cell or a group of pixel cells (e.g., a group of 2×2 pixel cells). Each pixel cell or group of pixel cells may be connected to one of the column buses 937, one of the row buses 935, a programming bus 942, and an output data bus ( Fig. 9B Each pixel cell (or each group of pixel cells) may be individually addressed by a column address signal 944 on a column bus 937 provided by a column controller 934 and a row address signal 946 on a row bus 935 provided by a row controller 936 to receive a pixel level programming signal 943 at a time via a pixel level programming bus 942. The column address signal 944, the row address signal 946, and the pixel level programming signal 943 may be generated based on the first programming signal 920 from the programming map generator 912.

[0124] In addition, the pixel cell array 908 also includes a programming signal parser 940, which can extract a pixel-level programming signal 943 from the first programming signal 920. In some examples, the first programming signal 920 can include a programming map, which can include programming data for each pixel cell or each pixel cell group of the pixel cell array 908. Fig. 9C An example of a pixel array programming diagram 948 is shown. Fig. 9C As shown, the pixel array programming map 948 may include a two-dimensional array of pixel-level programming data, wherein each pixel-level programming data of the two-dimensional array targets a pixel cell or a pixel cell group of the pixel cell array 908. For example, in the case where each pixel-level programming data targets a pixel cell, and assuming that the pixel cell array 908 has a width of M pixels (e.g., M columns of pixels) and a height of N pixels (e.g., N rows of pixels), the pixel array programming map 948 may also have a width of M entries (e.g., M column entries) and a height of N entries (e.g., N row entries), wherein each entry stores the pixel-level programming data for the corresponding pixel cell. For example, the pixel-level programming data A at entry (0, 0) of the pixel array programming map 948 00 Take the pixel cell P at the pixel position (0, 0) of the pixel cell array 908 as an example. 00 The pixel level programming data A at entry (0, 1) of the pixel array programming diagram 948 is the target. 01 The pixel cell P at the pixel position (0, 1) of the pixel cell array 908 is 01 Where the pixel level programming data targets a group of pixel cells, the number of entries along the height and width of the pixel array programming map 948 may be scaled based on the number of pixel cells in each group.

[0125] The pixel array programming diagram 948 can be configured to support Fig. 9B. For example, the pixel-level programming data stored at each entry can program each pixel cell (or each pixel cell group) individually to, for example, power on or off, enable or disable the output of pixel data, set the quantization resolution, set the precision of the output pixel data, select a quantization operation (e.g., one of TTS, FD ADC, PD ADC), set the frame rate, etc. As described above, the programming map generator 912 can generate a pixel array programming map 948 based on, for example, a prediction of one or more ROIs, wherein the pixel-level programming data for the pixel cells within the ROI is different from the pixel-level programming data for the pixel cells outside the ROI. For example, the pixel array programming map 948 can enable a subset of pixel cells (or pixel cell groups) to output pixel data, while the remaining pixel cells do not output pixel data. As another example, the pixel array programming map 948 can control a subset of pixel cells to output pixel data at a higher resolution (e.g., using a larger number of bits to represent each pixel), while the remaining pixel cells output pixel data at a lower resolution.

[0126] Return to reference Fig. 9B , the programming map parser 940 can parse the pixel array programming map 948 that can be in the serial data stream to identify the pixel-level programming data for each pixel cell (or each pixel cell group). The identification of the pixel-level programming data can be based on, for example, a predetermined scanning mode (by which the two-dimensional pixel array programming map is converted into a serial format), and the order in which the programming signal parser 940 receives the pixel-level programming data from the serial data stream. For each entry of the programming data, the programming signal parser 940 can generate a row address signal 944 and a column address signal 946, and send the row address signal 944 and the column address signal 946 to the row controller 936 and the column controller 934, respectively, to select the pixel cell, and send the pixel-level programming signal 943 to the selected pixel cell (or pixel cell group).

[0127] Fig.9D Example internal components of a pixel cell 950 of the pixel cell array 908 are shown. The pixel cell 950 may include Fig. 6A At least some components of the pixel unit 601. The pixel unit 950 may include one or more photodiodes (including photodiodes 952a, 952b, etc.), each of which may be configured to detect light of different frequency ranges. For example, photodiode 952a may detect visible light (e.g., monochrome, or one of red, green, or blue), while photodiode 952b may detect infrared light. The pixel unit 950 also includes a switch 954 (e.g., a transistor, a controller barrier) to control which photodiode outputs the charge for pixel data generation.

[0128] In addition, the pixel unit 950 also includes Fig. 6A The electronic shutter switch 603, transfer switch 604, charge storage device 605, buffer 606, quantizer 607, and reset switch 951 and memory 955 are shown. The charge storage device 605 can have a configurable capacitance to set the charge-to-voltage conversion gain. In some examples, for medium light intensity, the capacitance of the charge storage device 605 can be increased to store overflow charge for FD ADC operation, thereby reducing the possibility of the charge storage device 605 being saturated with overflow charge. For low light intensity, the capacitance of the charge storage device 605 can also be reduced to increase the charge-to-voltage conversion gain for PD ADC operation. The increase in charge-to-voltage conversion gain can reduce quantization error and increase quantization resolution. In some examples, the capacitance of the charge storage device 605 can also be reduced during FD ADC operation to increase quantization resolution. The reset switch 951 can reset the charge storage device 605 before capturing an image frame and / or between FD ADC operation and PD ADC operation. Buffer 606 includes a current source 956 whose current can be set by bias signal BIAS1 and a power gate 958 which can be controlled by PWR_GATE signal to turn on / off buffer 606. As part of disabling pixel unit 950, buffer 606 can be turned off.

[0129] In addition, the quantizer 607 includes a comparator 960 and an output logic 962. The comparator 960 can compare the output of the buffer with a reference voltage (VREF) to generate an output. According to the quantization operation (e.g., TTS, FD ADC, PD ADC operation), the comparator 960 can compare the buffered voltage with different VREF voltages to generate an output, and the output is further processed by the output logic 962 to enable the memory 955 to store the value from the free-running counter as a pixel output. The bias current of the comparator 960 can be controlled by the bias signal BIAS2, which can set the bandwidth of the comparator 960, and the bandwidth of the comparator 960 can be set based on the frame rate to be supported by the pixel unit 950. In addition, the gain of the comparator 960 can be controlled by the gain control signal GAIN. The gain of the comparator 960 can be set based on the quantization resolution to be supported by the pixel unit 950. The comparator 960 also includes a power switch 961, which can also be controlled by the PWR_GATE signal to turn on / off the comparator 960. As part of disabling pixel cell 950 , comparator 960 may be turned off.

[0130] In addition, the output logic 962 can select the output of one of the TTS, FD ADC, or PD ADC operations, and based on the selection, determine whether to forward the output of the comparator 960 to the memory 955 to store the value from the counter. The output logic 962 may include an internal memory to store an indication of whether the photodiode 952 (e.g., the photodiode 952a) is saturated with residual charge and whether the charge storage device 605 is saturated with overflowing charge based on the output of the comparator 960. If the charge storage device 605 is saturated with overflowing charge, the output logic 962 may select the TTS output to be stored in the memory 955 and prevent the memory 955 from overwriting the TTS output with the FD ADC / PD ADC output. If the charge storage device 605 is not saturated, but the photodiode 952 is saturated, the output logic 962 may select the FD ADC output to be stored in the memory 955; otherwise, the output logic 962 may select the PD ADC output to be stored in the memory 955. In some examples, instead of a counter value, an indication of whether photodiode 952 is saturated with residual charge and charge storage device 605 is saturated with overflowing charge may be stored in memory 955 to provide minimum precision pixel data.

[0131] In addition, the pixel cell 950 may include a pixel cell controller 970, which may include logic circuits to generate control signals, such as AB, TG, BIAS1, BIAS2, GAIN, VREF, PWR_GATE, etc. The pixel cell controller 970 may also be programmed by the pixel level programming signal 926. For example, to disable the pixel cell 950, the pixel cell controller 970 may be programmed by the pixel level programming signal 926 to declare PWR_GATE invalid to turn off the buffer 606 and the comparator 960. In addition, to increase the quantization resolution, the pixel cell controller 970 may be programmed by the pixel level programming signal 926 to reduce the capacitance of the charge storage device 605, increase the gain of the comparator 960 via the GAIN signal, etc. To increase the frame rate, the pixel cell controller 970 may be programmed by the pixel level programming signal 926 to increase the BIAS1 signal and the BIAS2 signal to increase the bandwidth of the buffer 606 and the comparator 960, respectively. Furthermore, in order to control the accuracy of the pixel data output by the pixel unit 950, the pixel unit controller 970 can be programmed by the pixel level programming signal 926 to, for example, connect only a subset of the bits of the counter (e.g., the most significant bits) to the memory 955 so that the memory 955 stores only a subset of the bits, or store the indication stored in the output logic 962 as pixel data to the memory 955. In addition, the pixel unit controller 970 can be programmed by the pixel level programming signal 926 to control the sequence and timing of the AB and TG signals, for example, to adjust the exposure period and / or select a specific quantization operation (e.g., one of TTS, FD ADC, or PD ADC), while skipping other quantization operations based on operating conditions, as described above.

[0132] Fig.10 A method 1000 of generating content to be output by a mobile device, such as mobile device 700, is shown. The method 1000 may be performed by a controller of the mobile device, such as Figure 8A-8E The mobile device may include multiple image sensors facing different directions (such as Fig. 7A The controller and the sensor are coupled to the physical link and form a multi-branch network. An example of a multi-branch network is shown in FIG. Figure 8B and Figure 8CEach image sensor or group of image sensors may be individually programmable. For example, each image sensor / group of image sensors may be individually enabled or disabled. The resolution and frame rate at which each image sensor generates image data may also be different. In addition, each image sensor includes a plurality of pixel cells, and the image sensing operation of each pixel cell or block of pixel cells may be individually programmable. For example, each pixel (or block of pixel cells) may be individually enabled or disabled. An example of an image sensor is shown in FIG. Figure 9A-9D Shown in.

[0133] In step 1002, the controller may send a control signal (e.g., control data 822) to select a subset of multiple image sensors of the mobile device. In some examples, the control signal may be sent via a physical link, such as an I3C bus including a serial data line (SDL) and a serial clock line (SCL). In order to send the control signal, the controller may pull down the SCL based on a clock signal pattern, and pull down the SDL based on a serial data pattern including the control data 822 and the address of the target sensor (one or more target sensors). In some examples, the mobile device may also include a first physical link (e.g., a bus) for transmitting control data and a second physical link (e.g., another bus, an optical link, etc.) for transmitting image data. In this case, the controller may send the control signal via the first physical link.

[0134] In some examples, the control signal may also include a pixel array programming map (e.g., pixel array programming map 948) that defines programming information for each pixel (or pixel block) in the target sensor (one or more target sensors). The programming information may define, for example, a subset of pixels to be enabled to perform image sensing operations and / or send pixel data, a frame rate at which pixel data is generated and sent, and a resolution for the image sensing operation.

[0135] In step 1004, the controller receives image data from each image sensor in the subset of the plurality of image sensors via a physical link (which may be the same or different physical link as the physical link that transmits the control signal). In some examples, the control signal may enable each image sensor in the plurality of image sensors. In some examples, the control signal may enable only a subset of the image sensors. Each image sensor may also be controlled by the control signal to select only a subset of pixels to generate pixel data, such that the image data includes a sparse image in which only a subset of pixels contain image data and the remaining pixels do not contain image data, such as Fig. 8E A subset of pixels may correspond to, for example, an object of interest. Different sensors may also generate / send image data at different resolutions and / or different frame rates.

[0136] In the case where the physical link is an I3C bus, a subset of the image sensors can drive another serial data pattern including sensor data on the SDL, and only one of the sensors can send sensor data to the controller via the SDL at a time. When multiple sensors attempt to drive the SDL simultaneously, the timing of each sensor sending sensor data can be controlled based on a control signal defined according to a predetermined transmission schedule and / or based on a backoff delay. In the case where the physical link is an optical link, a subset of the image sensors can modulate light in the optical link simultaneously or sequentially to send sensor data.

[0137] In step 1008, the controller may combine image data from at least a subset of the plurality of image sensors to obtain an extended field of view (FOV) that is larger than the field of view provided by each image sensor in the subset of image sensors. Figure 7A-7C The multiple image sensors may include image sensors facing the same direction and image sensors facing different directions. By combining image data from the image sensors, an extended FOV that is larger than the FOV of each image sensor may be achieved. In some examples, the image sensors of the stereo pair may be configured to capture light in different frequency ranges to support depth sensing.

[0138] In step 1008, the controller may determine information about the surrounding environment of the mobile device captured within the expanded FOV. Figure 7B-7C The information may include, for example, the positioning of one or more physical objects in the scene, detection results of the presence of people (and their postures) around the user, tracking results of the user's body parts (such as hands), etc. The information is provided based on the FOV of the extended image sensor.

[0139] In step 1010, the controller may provide information to the application to generate content based on the information. For example, to provide a VR / AR / MR experience, the application may replace detected objects with virtual objects, generate audio / display signals indicating that someone is standing behind the user, etc.

[0140] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, and / or hardware.

[0141] The steps, operations or processes described may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In some embodiments, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0142] Embodiments of the present disclosure may also relate to an apparatus for performing the described operations. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-temporary, tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that uses a multi-processor design to increase computing power.

[0143] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0144] The language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the subject matter of the invention. Therefore, it is intended that the scope of the present disclosure is not limited by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the present disclosure, which is set forth in the appended claims.

Claims

1. A mobile device, comprising: Physical link; a plurality of image sensors, each of the plurality of image sensors being configured to transmit image data via the physical link, at least two of the plurality of image sensors being oriented in particular different directions; and a controller coupled to the physical link, whereby the physical link, the plurality of image sensors, and the controller form a multi-branch network; Wherein, the controller is configured as: determining image sensing operations of the plurality of image sensors; wherein the image sensing operations reduce an amount of image data sent through the multi-branch network by determining that a subset of the plurality of image sensors is likely to capture images of one or more objects of interest based on detecting features of the objects of interest from previous images captured by the sensors; sending control signals to configure image sensing operations at the plurality of image sensors according to the determined image sensing operations; receiving image data from at least a subset of the plurality of image sensors via the physical link based on the control signal; combining image data from said at least a subset of said plurality of image sensors, to obtain an extended field of view (FOV) larger than that provided by each image sensor in the subset of image sensors; determining information about the surrounding environment of the mobile device captured within the expanded FOV; and The information is provided to an application to generate content to be output by the mobile device based on the information.

2. The mobile device according to claim 1, wherein: The controller is configured to select the subset of the plurality of image sensors to transmit the image data and disable the remainder of the plurality of image sensors based on transmitting the control signal.

3. The mobile device according to claim 2, wherein: The image data is first image data; Wherein, the controller is configured to: controlling the subset of the plurality of image sensors to transmit the first image data at a first resolution; and controlling the rest of the plurality of image sensors to transmit second image data at a second resolution; and The first resolution is higher than the second resolution.

4. The mobile device according to claim 2, wherein: The image data is first image data; Wherein, the controller is configured to: controlling the subset of the plurality of image sensors to transmit the first image data at a first frame rate; and controlling the rest of the plurality of image sensors to transmit second image data at a second frame rate; and The first frame rate is higher than the second frame rate.

5. The mobile device according to claim 2, wherein: The information of the surrounding environment includes the tracking result of the object of interest; and Wherein, the controller is configured to select the subset of the plurality of image sensors based on determining that image data generated by the subset of the plurality of image sensors may contain one or more images of the object.

6. The mobile device according to claim 5, wherein: The controller is configured to determine that image data generated by the subset of the plurality of image sensors likely contains one or more images of the object based on detecting features of the object in previous image data from the subset of the plurality of image sensors.

7. The mobile device according to claim 5, wherein: The controller is configured to: determine a previous trajectory of relative movement between the object and the mobile device based on previous image data from the plurality of image sensors; and Wherein, the controller is configured as: predicting a trajectory of the object based on the previous trajectory; and It is determined based on the predicted trajectory that image data generated by the subset of the plurality of image sensors likely includes one or more images of the object.

8. The mobile device according to claim 1, wherein: Each image sensor of the plurality of image sensors comprises an array of pixel cells; and The controller is configured to selectively configure an image capture operation of a specific subset of pixel cells in a pixel cell array in one or more image sensors among the plurality of image sensors based on sending the control signal.

9. The mobile device according to claim 8, wherein: The controller is configured to: enabling a first subset of pixel cells in a pixel cell array of a first image sensor of the plurality of image sensors to send first image data to the controller via the physical link; and enabling a second subset of pixel cells in the pixel cell array of a second image sensor of the plurality of image sensors to send second image data to the controller via the physical link; and The first subset is different from the second subset.

10. The mobile device according to claim 8, wherein: The controller is configured to: enabling a first subset of pixel cells in a pixel cell array of a first image sensor of the plurality of image sensors to generate first image data at a first resolution; and enabling a second subset of pixel cells in the array of pixel cells of the first image sensor to generate second image data at a second resolution; and The first resolution is higher than the second resolution.

11. The mobile device according to claim 8, wherein: The controller is configured to: setting a first dynamic range of a first subset of pixel cells in a pixel cell array of a first image sensor of the plurality of image sensors; setting a second dynamic range for a second subset of pixel cells in the pixel cell array of the first image sensor; and The first dynamic range is higher than the second dynamic range.

12. The mobile device according to claim 8, wherein: For each image sensor in the subset of the plurality of image sensors, the control signal identifies a pixel cell in the particular subset of pixel cells in the array of pixel cells.

13. The mobile device according to claim 8, wherein: The control signal comprises a guidance signal; and Wherein, a first image sensor among the plurality of image sensors is configured to locally determine the specific subset of pixel cells in the pixel cell array based on the guidance signal.

14. The mobile device according to claim 13, wherein: The guidance signal specifies a characteristic of the object of interest; Wherein, the first image sensor is configured as: Based on the guidance signal, determining a region of interest including the object of interest; and Based on the region of interest, the specific subset of pixel cells in the array of pixel cells is determined.

15. The mobile device according to claim 8, wherein: The specific subset of pixel units in the pixel unit array is determined based on at least one of the following: a tracking result of the object of interest, or a movement of the mobile device.

16. The mobile device according to claim 1, wherein: Image data from at least a first image sensor and a second image sensor of the plurality of image sensors are combined; and The first image sensor and the second image sensor face different directions.

17. The mobile device according to claim 1, wherein: Image data from at least a first image sensor and a second image sensor of the plurality of image sensors are combined; wherein the first image sensor is configured to capture light of a first frequency range; and Wherein, the second image sensor is configured to capture light in a second frequency range different from the first frequency range.

18. The mobile device according to claim 1, wherein: The physical link includes at least one of the following: a bus based on the I3C specification, or an optical link.

19. A method comprising: determining an image sensing operation of a plurality of image sensors; wherein: at least two of the plurality of image sensors are oriented in particular different directions, the plurality of image sensors and the physical link form a multi-branch network, the image sensing operation reducing an amount of image data sent through the multi-branch network by determining that a subset of the plurality of image sensors is likely to capture images of one or more objects of interest based on detecting features of the objects of interest from previous images captured by the sensors; sending control signals to configure image sensing operations at a plurality of image sensors of the mobile device according to the determined image sensing operations; receiving image data from each image sensor of a subset of the plurality of image sensors via a physical link based on the control signal; combining image data from at least a subset of the plurality of image sensors to obtain an extended field of view (FOV) that is larger than that provided by each image sensor in the subset of image sensors; determining information about the surrounding environment of the mobile device captured within the expanded FOV; and The information is provided to an application to generate content to be output by the mobile device based on the information.

20. The method according to claim 19, further comprising: Based on transmitting the control signal, a subset of the plurality of image sensors is selected to transmit the image data, and the remainder of the plurality of image sensors is disabled.

Images