Mode sequencer circuit and mode sequencing method

Abstract

A mode sequencer circuit for a time-of-flight system, the time-of-flight system comprising at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transferring data to an application processor over a data bus, the mode sequencer circuit being configured to: set a first output mode and a second output mode of the imaging unit for outputting first image data and second image data over the data bus interface, wherein the first output mode relates to a first region of interest imaged by the imaging unit and the second output mode relates to a second region of interest imaged by the imaging unit, wherein the first output mode is adapted according to at least one requirement of a first subprogram running on the application processor for analyzing the first region of interest and a first limitation of the data bus interface, and the second output mode is adapted according to at least one requirement of a second subprogram running on the application processor for analyzing the second region of interest and a second limitation of the data bus interface.

Description

Technical Field

[0001] This disclosure generally relates to a pattern sequencer circuit for a time-of-flight system and a corresponding pattern sequencing method. Background Technology

[0002] In general, Time-of-Flight (ToF) devices or systems are known for determining the distance to a target in an illuminated scene, or for creating a depth map of an illuminated scene (of a target). Typically, a ToF system includes an illumination unit (such as an array of light-emitting diodes), optical components (such as lenses), and an imaging unit (such as an image sensor, or a current-assisted photon demodulator (CAPD) pixel array) with readout circuitry. It may also include a processing unit (such as a processor), for example, when depth information is generated on the ToF device.

[0003] Time-of-flight (ToF) encompasses various methods, such as measuring the time required for a light wave to travel a distance in a medium. Known ToF systems can obtain depth measurements (and depth information) of targets in a scene for each pixel of a depth image captured using an imaging unit. To capture this depth image, a ToF system typically illuminates the scene with, for example, modulated light waves and images the backscattered / reflected light waves using an optical lens portion within the imaging unit. The imaging unit may have, for example, a pixel array, where the gain of each pixel is modulated according to a demodulated signal, which can be phase-shifted relative to the modulation of the light waves, thereby generating image data indicating the distance to targets in the scene. Typically, the imaging unit outputs the generated image data to a processing unit for image processing and depth information generation.

[0004] Although there are technologies that output image data to processing units in time-of-flight systems, improving existing technologies is generally desirable. Summary of the Invention

[0005] According to a first aspect, the present invention provides a pattern sequencer circuit for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via the data bus, and the pattern sequencer circuit being configured as follows:

[0006] A first output mode and a second output mode of the imaging unit are configured for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first limitation of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second limitation of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0007] According to a second aspect, this disclosure provides a mode sequencing method for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via the data bus, the mode sequencing method including:

[0008] The imaging unit is configured with a first output mode and a second output mode for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first limitation of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second limitation of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0009] Further aspects are set forth in the dependent claims, the following description, and the accompanying drawings. Attached Figure Description

[0010] Embodiments of the present invention are explained by way of example in conjunction with the accompanying drawings, wherein:

[0011] Figure 1 A block diagram illustrating an implementation of the time-of-flight system is shown schematically.

[0012] Figure 2 The implementation of the light modulation signal, reflected light signal, and four demodulation signals of the illumination unit is schematically shown;

[0013] Figure 3 A block diagram of a first embodiment of the pattern sequencer circuit of a time-of-flight system is schematically shown;

[0014] Figure 4A block diagram of a second embodiment of the mode sequencer circuit for a time-of-flight system is schematically shown;

[0015] Figure 5 This schematically illustrates an implementation method for the vehicle's state;

[0016] Figure 6 A first implementation of the output mode and lighting mode sequence is schematically shown;

[0017] Figure 7 A second implementation of the output mode and illumination mode sequence is schematically shown;

[0018] Figure 8 An implementation of the predetermined output mode is illustrated schematically;

[0019] Figure 9 A flowchart illustrating a first embodiment of the pattern sequencing method is shown schematically.

[0020] Figure 10 A flowchart illustrating a second embodiment of the pattern sequencing method is shown schematically.

[0021] Figure 11 A flowchart illustrating a third embodiment of the pattern sequencing method is shown schematically;

[0022] Figure 12 A flowchart illustrating the fourth embodiment of the pattern sequencing method is shown schematically;

[0023] Figure 13 A block diagram schematically illustrates an implementation of a schematic configuration of a vehicle control system; and

[0024] Figure 14 A schematic diagram illustrating an embodiment that helps explain the mounting positions of the vehicle exterior information detection unit and the imaging unit is shown. Detailed Implementation

[0025] In reference Figure 3 Before providing a detailed description of the implementation methods, a general explanation will be given first.

[0026] As mentioned at the beginning, Time-of-Flight (ToF) systems are known to obtain depth measurements (and depth information) of targets in a scene. Such ToF systems can be used for applications such as gesture recognition, facial recognition, and autofocus in smartphones, or in vehicle applications such as occupant cabin monitoring in driver assistance systems and gesture-based operation in infotainment systems.

[0027] In some implementations, to capture depth images, the ToF system illuminates the scene with modulated light waves and images the reflected light waves using an optical lens portion on the imaging unit. In some implementations, the imaging unit includes a pixel array, wherein the gain of the pixels in the pixel array is modulated according to a demodulated signal to generate image data indicating the distance to a target in the scene. This type of ToF system is called an indirect ToF (iToF) system.

[0028] To enhance the overall understanding of the contents of this disclosure, refer to Figure 1 An implementation of the iToF system 1 is discussed, wherein the implementation of the iToF system 1 is schematically shown in the form of a block diagram.

[0029] iToF system 1 is a global iToF system for providing distance measurement values. iToF system 1 includes an illumination unit 2, an imaging unit 3, and an optical lens section 4. In this embodiment, iToF system 1 is integrated into a mobile device such as a smartphone.

[0030] The imaging unit 3 includes a control unit 5 and an image sensor 6.

[0031] The illumination unit 2 includes a light source such as an LED array that emits intensity-modulated light (illuminating the scene 7) towards the scene 7, including the target 8, and the target 8 reflects at least a portion of the (illumination) light. The reflected light from the target 8 is imaged onto the image sensor 6 (or imaging unit 3) by the optical lens portion 4.

[0032] Control unit 5 controls the overall operation of ToF system 1. Control unit 5 controls the light emission time of illumination unit 2 based on the modulation signal provided to illumination unit 2 (e.g., a rectangular modulation signal with modulation period T, i.e., modulation frequency fmod = 1 / T). Control unit 5 applies a demodulated signal to image sensor 6, which corresponds to the modulation signal provided to illumination unit 2.

[0033] Image sensor 6 generates image data, including pixel values ​​of multiple pixels, based on the amount of reflected light imaged on each pixel by optical lens portion 4 and based on the demodulated signal. Control unit 5 reads out the image data generated by image sensor 6.

[0034] The imaging unit 3 has a data bus interface for transmitting the generated image data to the application processor 10 of the mobile device via the data bus 9 (generally, in other embodiments, the application processor 10 may be replaced by an FPGA, ISP, etc.).

[0035] The application processor 10 includes an image processing unit 11, a 3D image reconstruction unit 12, and an application unit 13.

[0036] Image processing unit 11 acquires image data transmitted via data bus 9 from imaging unit 3. Based on the acquired image data, image processing unit 11 determines the phase shift of the detected reflected light relative to the emitted light. Then, image processing unit 11 calculates the distance d or general depth information for scene 7 (e.g., to target 8) based on the determined phase shift.

[0037] As is generally known, the (defined) range of distance measurements for an iToF system is given by the following formula:

[0038]

[0039] Here, c is the speed of light, T is the modulation period of the modulated signal, and fmod is the modulation frequency.

[0040] To further enhance the overall understanding of the contents of this disclosure, refer to Figure 1 and Figure 2 The fundamental principles for determining phase shift and calculating distance or depth information are discussed, and these principles also apply to other implementations described herein. Figure 2 The implementation of the light modulation signal LMS, the reflected light signal RL, and the four demodulation signals DM1-DM4 of the illumination unit 2 is illustrated schematically.

[0041] The light modulation signal LMS of illumination unit 2 is a rectangular modulation signal with a modulation period T (or the corresponding modulation frequency fmod). The intensity of the emitted light from the light source is modulated in time according to the light modulation signal LMS. The emitted light is reflected onto target 8 in scene 7.

[0042] The reflected light signal RL is the intensity of the reflected light at image sensor 6. It is phase-shifted relative to the light modulation signal LMS and varies according to the intensity of the emitted light. This phase is proportional to the distance to target 8 in scene 7.

[0043] Image sensor 6 captures four frames of image data corresponding to demodulated signals DM1, DM2, DM3 and DM4. These four frames can be transmitted individually to the application processor via data bus 9 or in the form of data packets (the imaging unit 3 stores these four frames before transmission).

[0044] The demodulated signal DM1 has a 0° phase shift relative to the optical modulation signal LMS. When the demodulated signal DM1 is high, the image sensor 6 (each of the multiple pixels) accumulates charge Q1 based on the amount of light incident on each pixel and the overlap between the reflected light signal RL and the demodulated signal DM1.

[0045] The demodulated signal DM2 has a 90° phase shift relative to the optical modulation signal LMS. When the demodulated signal DM2 is high, the image sensor 6 (each of the multiple pixels) accumulates charge Q2 based on the amount of light incident on each pixel and the overlap between the reflected light signal RL and the demodulated signal DM2.

[0046] The demodulated signal DM3 has a 180° phase shift relative to the optical modulation signal LMS. When the demodulated signal DM3 is high, the image sensor 6 (each of the multiple pixels) accumulates charge Q3 based on the amount of light incident on each pixel and the overlap between the reflected light signal RL and the demodulated signal DM3.

[0047] The demodulated signal DM4 has a 270° phase shift relative to the optical modulation signal LMS. When the demodulated signal DM4 is high, the image sensor 6 (each of the multiple pixels) accumulates charge Q4 based on the amount of light incident on each pixel and the overlap between the reflected light signal RL and the demodulated signal DM4.

[0048] As is generally known, charges Q1, Q2, Q3 and Q4 are proportional to, for example, voltage signals (electrical signals) of their respective pixels. The image sensor obtains pixel values ​​(digital values) from these signals and outputs them as image data. Therefore, charges Q1, Q2, Q3 and Q4 represent pixel values.

[0049] Then, the phase is given by the following formula:

[0050]

[0051] Q = Q3 - Q4,

[0052] I = Q1 - Q2.

[0053] Here, Q is the quadrature component and I is the in-phase component; both are pixel component values ​​(IQ values), which together constitute the component data.

[0054] Then, the distance d to the target is given by the following formula:

[0055]

[0056] The amplitude of the reflected light signal RL is proportional to the phase amplitude, which is given by the following formula:

[0057]

[0058] Typically, measurement accuracy improves with higher amplitude and higher modulation frequency.

[0059] Back Figure 1The depth information (distance) of scene 7 is fed from the image processing unit 11 to the 3D image reconstruction unit 12, which constructs (generates) a 3D image, 3D depth map or 3D point cloud of scene 7 based on the depth information from the image processing unit 11.

[0060] Application unit 13 implements various advanced functions, such as gesture recognition, face recognition, or autofocus based on generated depth information or depth maps. Other functions of application unit 13 may include memory management, graphics processing, multimedia decoding and encoding, etc.

[0061] However, in such an implementation, it has been recognized that the amount of image data that must be transmitted over the data bus when the data flows to the application processor at full resolution is large.

[0062] Therefore, it has been recognized that it may require high data bandwidth (e.g., greater than 5Gbps) for the application to achieve a certain expected output frame rate (e.g., 60fps), robustness against motion artifacts, and low latency (e.g., less than 10ms).

[0063] It was further recognized that, in order to compute component data for, for example, four or more full-image data frames, these data frames must be temporarily stored in memory, which places a high load on the application processor's memory bus. Performing functions such as gesture recognition and managing various other functions requires additional computation of depth maps, which further increases the computational load on the application processor.

[0064] Furthermore, it has been recognized that different applications or subroutines that analyze image data and depth information (e.g., gesture and facial recognition for unlocking or operating mobile devices) have different requirements for output resolution, field of view or region of interest, output frame rate, etc., in order to ensure reliable feature detection.

[0065] Furthermore, it has been recognized that, for example in vehicle applications such as cabin occupant monitoring, the sequence of subroutines of the application that analyzes image data is repeatedly executed, with each subroutine having different minimum requirements for image data resolution, frame rate, and data type (such as depth or infrared intensity). Moreover, the order of the subroutines may depend on the state of the vehicle performing the in-vehicle monitoring (e.g., parked or driven).

[0066] Therefore, it has been recognized that optimizing along the processing chain by adjusting the image data output according to the minimum requirements of the application or application subroutines that analyze the image data and according to the limitations of the data bus for transmitting image data to the application processor can reduce the bandwidth requirements of the data bus, memory bus load, and computational load.

[0067] Therefore, some implementations relate to a pattern sequencer circuit for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the pattern sequencer circuit being configured as follows:

[0068] The imaging unit is configured with a first output mode and a second output mode for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first limitation of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second limitation of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0069] The pattern sequencer circuitry may be included in or be part of the time-of-flight system. The pattern sequencer circuitry may be included in or be part of a separate electronic device connected to the application processor or the time-of-flight system. The pattern sequencer circuitry may be embedded in the control unit of the time-of-flight system. The pattern sequencer circuitry may be implemented in or part of the imaging unit, which allows the pattern sequencer circuitry to operate without software intervention, thereby offloading the application processor and the data bus interface between the application processor and the imaging unit. In some embodiments, such as when the pattern sequencer circuitry further controls the illumination unit to a considerable extent, this can further reduce the required components.

[0070] The pattern sequencer circuit can be based on, include, or be implemented as integrated circuit logic, or can be implemented by a central processing unit (CPU), application processor, graphics processing unit (GPU), microcontroller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc. The functionality can be implemented by software executed by a processor (such as an application processor).

[0071] Overall, hardware-based control can give video applications a high degree of determinism because it offers high timing accuracy (e.g., within 1 microsecond or less).

[0072] The pattern sequencer circuit may be based on, may include, or may be implemented by typical electronic components configured to perform the functions described herein.

[0073] The pattern sequencer circuit may be based on, include, or be partially implemented by typical electronic components and integrated circuit logic, and partially implemented by software.

[0074] The pattern sequencer circuit may include a data bus interface for transmitting (and receiving) data via a data bus.

[0075] The data bus interface can be a camera serial interface (CSI) conforming to the MIPI (Mobile Industrial Processor Interface) specification (e.g., MIPII CSI-2 interface, etc.), I 2 Interfaces include C (Integrated Circuit Bus) interface, Controller Area Network (CAN) bus interface, FDP-link (Flat Panel Display Link), and GSML (Gigabit Multimedia Serial Link). The data bus conforms to the corresponding interface specifications.

[0076] In some implementations, the data bus (interface) includes a bus between the imaging unit and the application processor for transmitting data via the data bus.

[0077] The data bus interface may also include the memory bus within the application processor. Therefore, in some implementations, the data bus (interface) includes the bus between the imaging unit and the application processor, as well as the memory bus within the application processor.

[0078] The pattern sequencer circuit may include a communication interface configured to communicate and exchange data with a computer or processor (such as an application processor) via a network (such as the Internet) via a wired connection or a wireless connection such as a mobile telecommunications system, which may be based on, for example, UMTS, LTE, etc. (and implement the corresponding communication protocols).

[0079] The pattern sequencer circuit may include data storage capabilities such as a memory to store data, which may be based on semiconductor storage technology (such as RAM, EPROM, etc.) or magnetic storage technology (such as hard disk drive).

[0080] The Time-of-Flight (ToF) system can be an indirect ToF system, including a global iToF system or a point-based iToF system. This ToF system can be embedded in mobile devices (such as smartphones) for purposes such as gesture recognition, or embedded in vehicles (such as in the cabin of a car) for occupant monitoring, etc.

[0081] The ToF system includes at least an illumination unit and an imaging unit.

[0082] The lighting unit includes a light source.

[0083] The light source can be a laser (such as a laser diode) or multiple lasers (such as multiple laser diodes arranged in rows and columns in an array), a light-emitting diode (LED) or multiple LEDs (such as multiple LEDs arranged in rows and columns in an array), etc. The lighting unit can emit visible light or infrared red light, etc.

[0084] The lighting unit (or the light source in the lighting unit) is configured to emit light into the scene or illuminate the scene.

[0085] The illumination unit can be configured to individually control each laser diode or LED in the array to control the (spatial) area of ​​the scene being illuminated. The illumination unit may include optical elements such as optical lens portions, diffractive optical elements, etc., for imaging the emitted light onto the scene or illuminating the scene. The illumination unit may be controlled by an imaging unit.

[0086] The illumination unit is configured to modulate the intensity of the emitted light (in time) according to a modulation signal (e.g., by modulation current or voltage).

[0087] The illumination unit can receive a modulated signal (or modulation signal information indicating the modulated signal) from the imaging unit. The illumination unit can be configured to generate a modulated signal (e.g., pre-configured or based on modulation information). The illumination unit can be configured to output a modulated signal.

[0088] The modulation signal can be a sinusoidal modulation signal with a predetermined frequency (modulation frequency), or a rectangular modulation signal with a predetermined frequency (modulation frequency). For example, the emitted light is turned on during a first predetermined time period and then turned off during a second predetermined time period, and so on, as is commonly known, or similar.

[0089] The imaging unit includes an image sensor.

[0090] The imaging unit includes a control unit (which may be based on or implemented as, or may include, integrated circuit logic, FPGA, ASIC, typical electronic components, etc. to realize the functions described herein, or may be implemented partly by software and partly by electronic components), such as the readout circuit of the image sensor, the circuit for generating modulation and demodulation signals, the circuit for setting the output mode, etc.

[0091] The control unit of the imaging unit can control the light emission of the illumination unit (both spatially and temporally).

[0092] Overall, the image sensor included in the imaging unit generates image data of multiple pixels, which represents the ToF measurement of light reflected from the scene illuminated by the emitted light of the illumination unit.

[0093] The imaging unit is configured to output the generated image data.

[0094] Image sensors may include pixel circuitry having multiple pixels (arranged in a predetermined pattern, e.g., an array of rows and columns in an image sensor) that generate electrical signals based on the amount of light incident on each of the multiple pixels and based on a demodulated signal, such as modulating the gain of the multiple pixels, where the demodulated signal corresponds to the modulation signal of an illumination unit or light source. The demodulated signal of the image sensor can be phase-shifted relative to the modulation signal of the light source, which is commonly known in Time-of-Flight (ToF) systems.

[0095] Multiple pixels can be current-assisted photonic demodulator (CAPD) pixels, photodiode pixels, or active pixels based on technologies such as CMOS (complementary metal-oxide-semiconductor), where, for example, the gain of multiple pixels is based on demodulation signal modulation.

[0096] Multiple pixels can be multiphase pixels, where each multiphase pixel can include, for example, four sub-pixel regions (e.g., CAPD sub-pixel regions), which are demodulated with four different demodulation signals (e.g., modulation signals relative to the light source, phase shifts of 0°, 90°, 180°, and 270°).

[0097] In general, image data may be based on or may include digital values ​​(pixel values) obtained from analog signals (electrical signals, such as voltage or current signals), which are generated by multiple pixels of an image sensor in an imaging unit based on the amount of light incident on each of the multiple pixels and based on a demodulated signal.

[0098] The image data includes pixel values ​​of multiple pixels from multiple demodulated signals generated for different demodulated signals, with a phase shift relative to the demodulated signal. For example, the first demodulated signal may have a 0° phase shift relative to the modulation signal of the illumination unit or light source, and the second demodulated signal may have a 90° phase shift relative to the modulation signal of the illumination unit or light source.

[0099] In some embodiments, the image data includes multiple pixel values ​​corresponding to four different demodulated signals, which have phase shifts of 0°, 90°, 180°, and 270° relative to the modulation signal of the light source, respectively. Of course, this disclosure is not limited to the number of phase shifts (i.e., demodulated signals) or the specific amount of phase shift; in principle, any number and any amount of phase shift can be implemented.

[0100] Therefore, image data can include image data from multiple frames. For example, image data can include pixel values ​​of multiple pixels from four frames, where the demodulated signal corresponding to the first frame has a 0° phase shift relative to the modulation signal of the light source, the demodulated signal corresponding to the second frame has a 90° phase shift relative to the modulation signal of the light source, the demodulated signal corresponding to the third frame has a 180° phase shift relative to the modulation signal of the light source, and the demodulated signal corresponding to the fourth frame has a 270° phase shift relative to the modulation signal of the light source.

[0101] Alternatively, the image data may include a frame of image data. For example, a multiphase sensor may output image data for each subpixel region, each subpixel region including pixel values ​​corresponding to different demodulated signals (e.g., phase shifts of 0°, 90°, 180°, and 270° relative to the modulation signal of the light source).

[0102] The imaging unit has a data bus interface for transmitting data to the application processor via the data bus.

[0103] The data bus interface can be a camera serial interface (CSI) conforming to the MIPI (Mobile Industrial Processor Interface) specification (e.g., MIPII CSI-2 interface, etc.), I 2 Interfaces include C (Integrated Circuit Bus) and Controller Area Network (CAN) bus. The data bus conforms to the corresponding interface specifications.

[0104] The mode sequencer circuit is configured to set a first output mode and a second output mode of the imaging unit for outputting first image data and second image data through a data bus interface, wherein the first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit.

[0105] Basically, the output mode of an imaging unit is characterized by output mode parameters, which include the region of interest associated with the output mode and may include output frame rate, merging (such as pixel value merging of adjacent pixels) and image data output format (pixel values ​​(included in image data) or component values), illumination control (such as optical power, emission waveform, emission wavelength).

[0106] In general, a region of interest (GI) is a region in a scene that is imaged by an imaging unit and analyzed, for example, by a specific computer program or application (the scene illuminated by an illumination unit), and this analysis is used to obtain information about a target (including a person) within the GI (e.g., a facial region used to obtain facial information for face recognition). A GI imaged by an imaging unit is associated with a pixel region of an image sensor in the imaging unit; therefore, when a GI is analyzed, image data corresponding to that pixel region is analyzed. Different GIs may overlap at least partially, and therefore, the corresponding pixel regions may overlap at least partially but not completely (different GIs are not exactly the same). A GI may include more than one region (and associated pixel regions); for example, a GI may include a region in the left part of the scene and a region in the right part of the scene.

[0107] For example, the region of interest can be the entire scene (the entire field of view imaged by the imaging unit), thus the entire pixel region of the image sensor corresponds to the region of interest. For example, the region of interest can be the central region of the scene, and therefore, the central region of the entire pixel region only includes a subset of pixels from the central region of the image sensor that corresponds to the region of interest.

[0108] Setting a first output mode may include transmitting at least one bit or command indicating the first output mode to the imaging unit, such that the imaging unit is set accordingly. Setting a second output mode may include transmitting at least one bit or command indicating the second output mode to the imaging unit, such that the imaging unit is set accordingly.

[0109] Setting the first output mode may include transmitting the output mode parameters (region of interest (relevant pixel region), output frame rate, merging, image data output format) of the first output mode to the imaging unit, so that the imaging unit is configured accordingly. Setting the second output mode may include transmitting the output mode parameters (region of interest (relevant pixel region), output frame rate, merging, image data output format) of the second output mode to the imaging unit, so that the imaging unit is configured accordingly.

[0110] Before setting their respective output modes, at least one bit, command, or output mode parameter for the first and second output modes can be transmitted one by one. At least one bit, command, or output mode parameter for the first and second output modes can be transmitted together in sequence, which indicates the sequence of output modes, and the output modes are set according to the sequence over time.

[0111] Setting a first output mode may include setting at least one bit that indicates a first output mode stored in the imaging unit. Setting a second output mode may include setting at least one bit that indicates a second output mode stored in the imaging unit.

[0112] Setting the first output mode may include setting output mode parameters (region of interest (relevant pixel area), output frame rate, merging, and image data output format) stored in the imaging unit. Setting the second output mode may include setting output mode parameters (region of interest (relevant pixel area), output frame rate, grading, and image data output format) stored in the imaging unit.

[0113] At least one bit of the OR output mode parameter for the first and second output modes can be set sequentially, one after the other, prior to the corresponding output modes. At least one bit of the OR output mode parameter for the first and second output modes can be set together in a stored sequence indicating the output modes, which are set chronologically according to this stored sequence.

[0114] Setting the first and second output modes can involve configuring two register tables and a trigger: configuring a first register table indicating each output mode (e.g., modes A, B, C and their respective output mode parameters), configuring a second register table indicating a sequence of output modes to be repeated N times (positive integers) or continuously, and triggering a pin or register to initiate the sequence. This reduces interaction with the application processor.

[0115] In the first output mode, first image data related to the first region of interest imaged by the imaging unit is output via the data bus interface. In the second output mode, second image data related to the second region of interest imaged by the imaging unit is output via the data bus interface.

[0116] Therefore, the first image data is output according to the first output mode, and the second image data is output according to the second output mode.

[0117] The first output mode is adjusted based on a first constraint of the data bus interface and at least one requirement of a first subroutine running on the application processor, and is used to analyze a first region of interest. The second output mode is adjusted based on a second constraint of the data bus interface and at least one requirement of a second subroutine running on the application processor, and is used to analyze a second region of interest.

[0118] The first and second limitations may be or include at least one of the following: the bandwidth of the data bus (interface), the clock rate of the data bus (interface), and the width of the data bus (e.g., 16-bit, 32-bit).

[0119] Generally speaking, there are limitations on the maximum amount of data (e.g., image data) that a data bus (interface) can transmit within a given time frame. This limitation may be based on bandwidth, clock rate, or data bus width, or a combination thereof.

[0120] Therefore, in some implementations, the first and second restrictions are the same restriction, namely, the restriction on the maximum amount of data transmitted through the data bus within a certain period of time.

[0121] The first and second subroutines are subroutines or functions of an application running on the application processor. The application on the application processor analyzes image data indicating depth information obtained from the imaging unit via the data bus to obtain information about targets in the scene illuminated by the illumination unit, and performs higher-level functions based on the information about the targets in the scene.

[0122] The first subroutine analyzes the first region of interest to obtain information about targets within that region of interest in the scene. This analysis may include, for example, depth map calculation, object detection, object tracking, human gesture recognition, and face recognition. The first subroutine then returns the information to the application.

[0123] The second subroutine analyzes the second region of interest to obtain information about targets within that region of interest in the scene. This analysis may include, for example, depth map calculation, object detection, object tracking, human gesture recognition, and face recognition. The second subroutine then returns the information to the application.

[0124] The application can perform higher-level functions based on the analysis results (returned information) of the first and second subroutines. For example, when the application is for unlocking a mobile device based on facial recognition and a specific gesture, the first subroutine for facial recognition can return a result indicating whether the facial recognition is confirmed, the second subroutine can return a result indicating whether the specific gesture is confirmed, and the application can unlock the mobile device and its functions based on the results of the first and second subroutines.

[0125] The application, the first subroutine, and the second subroutine (and the associated areas of interest) can depend on the state of the device or vehicle in which the application processor, the ToF system, and the pattern sequencer circuitry are embedded. For example, the vehicle may be parked, and therefore, different applications may be executed depending on whether the vehicle is in motion.

[0126] At least one requirement for the first and second subroutines can be the frame rate (temporal resolution) of the image data, the spatial resolution of the image data, and the depth resolution of the image data. For example, face recognition may require a slower image data frame rate than gesture recognition, but may require a higher spatial resolution.

[0127] In general, as mentioned above, the data bus has limitations; that is, the amount of data that can be transmitted through the data bus within a certain time is finite. Therefore, considering at least one requirement of the subroutine, the amount of data transmitted to the subroutine through the data bus can be optimized and timed (within the limit of the maximum possible data volume) by weighing the temporal resolution, spatial resolution, and depth resolution of the image data. In other words, the output mode of the imaging unit is adjusted according to the limitations of the data bus and the requirements of the subroutine.

[0128] In general, these limitations may stem from several factors: such as the physical bandwidth of the iToF sensor-application processor interface, allocating specific time slots to the iToF sensor (imaging unit) in the case of sensor fusion (e.g., iToF+RGB), and the low-latency memory access required by other (iToF-independent) applications running on the application processor.

[0129] Therefore, in some implementations, adjusting the first output mode and the second output mode may include adjusting the output frame rate (temporal resolution) and the merging of image data (spatial resolution).

[0130] Therefore, in the first output mode, only image data corresponding to the first region of interest imaged by the imaging unit is output to the application processor via the data bus. Furthermore, in the first output mode, image data can be output according to the first output frame rate and the first merging.

[0131] Therefore, in the second output mode, only image data corresponding to the second region of interest imaged by the imaging unit is output to the application processor via the data bus. Furthermore, in the second output mode, image data can be output according to the second output frame rate and the second merging.

[0132] Therefore, in some implementations, a first output frame rate of the first image data is set in a first output mode according to at least one requirement of the first subroutine and a first limitation of the data bus interface, and a second output frame rate of the second image data is set in a second output mode according to at least one requirement of the second subroutine and a second limitation of the data bus interface.

[0133] Therefore, in some implementations, a first merging of first image data is set in a first output mode according to at least one requirement of a first subroutine and a first limitation of the data bus interface, and a second merging of second image data is set in a second output mode according to at least one requirement of a second subroutine and a second limitation of the data bus interface.

[0134] Binning is well known to those skilled in the art and involves combining the electrical signals of a cluster of pixels into a single electrical signal or combining the pixel values ​​of a cluster of pixels into a single pixel value, thereby forming, for example, a pixel block. Therefore, first and second binning can be implemented using analog or digital binning. Analog binning can be implemented by parallel readout of, for example, multiple rows (e.g., two rows) and multiple columns (e.g., two columns, resulting in 2×2 binning), wherein, for example, the electrical signals of two rows of pixels in each column are simultaneously placed on the column line, and the combined electrical signals on the column line are also combined via, for example, connected capacitors. Digital binning can be implemented by digitally averaging the pixel values ​​of the binned pixels.

[0135] Therefore, adjusting the output mode of the imaging unit accordingly can reduce the amount of image data transmitted to the application processor via the data bus (interface), thereby reducing the bandwidth required for the data bus interface, the load on the application processor's memory bus, the computational load on the application processor, and the overall power consumption.

[0136] It is further recognized that during the first output mode, only the first region of interest may need to be illuminated by the illumination unit, and during the second output mode, only the second region of interest may need to be illuminated by the illumination unit, since only these regions in the scene are used for analysis.

[0137] Therefore, in some embodiments, the mode sequencer circuit is further configured to set a first illumination mode for the illumination unit during a first output mode of the imaging unit, and to set a second illumination mode for the illumination unit during a second output mode of the imaging unit, wherein a first illumination area of ​​the illumination unit corresponding to a first region of interest is set in the first illumination mode, and a second illumination area of ​​the illumination unit corresponding to a second region of interest is set in the second illumination mode.

[0138] Basically, the first illumination mode of the illumination unit is characterized by the first illumination area, and the second illumination mode of the illumination unit is characterized by the second illumination area, and can be further characterized by the modulation frequency.

[0139] The first and second illumination areas are the illuminated areas in the scene, corresponding to the first and second areas of interest, respectively. The first and second illumination areas can be adjusted, for example, by moving optical elements (e.g., optical lens portions) or by switching individual LEDs in the LED array on and off, so that the first and second areas of interest are illuminated respectively.

[0140] Setting the first and second illumination modes may include transmitting illumination mode parameters (area of ​​interest information, modulation frequency (modulation signal or modulation signal information)) from the imaging unit to the illumination unit.

[0141] Since only the area corresponding to the region of interest is illuminated, this may reduce overall power consumption.

[0142] In some implementations, a first modulation frequency of the lighting unit is set in a first lighting mode according to at least one requirement of a first subroutine, and a second modulation frequency of the lighting unit is set in a second lighting mode according to at least one requirement of a second subroutine.

[0143] As mentioned above, different subroutines may require different depth resolutions, and therefore, different modulation frequencies for the illumination units may be required.

[0144] This can reduce overall power consumption because the modulation frequency can be set lower for subroutines that require lower depth resolution, and thus, losses in the illumination unit (in the light source of the illumination unit) and the imaging unit (in the pixels of the image sensor) due to the rapid switching frequency of the current or voltage of the light modulation signal and demodulation signal can be reduced respectively.

[0145] In some implementations, the pattern sequencer circuit is also configured as follows:

[0146] First image data is obtained according to the first output mode of the imaging unit, and second image data is obtained according to the second output mode of the imaging unit;

[0147] The first component data is calculated based on the obtained first image data, and the second component data is calculated based on the obtained second image data; and

[0148] According to at least one requirement of the first subroutine and a first limitation of the data bus interface, the acquired first image data or the calculated first component data is transmitted, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the acquired second image data or the calculated second component data is transmitted.

[0149] In such an implementation, the pattern sequencer circuit is part of the ToF system or the imaging unit of the ToF system.

[0150] The calculation of the first and second component data is generally known and can be referred to above. Figure 2 The discussion focuses on calculating component data.

[0151] Transmitting image data or component data in both the first and second output modes can reduce the amount of data transmitted on the data bus, the load on the application processor's memory bus, the application processor's computational load, and overall power consumption.

[0152] In some implementations, the mode sequencer circuit is also configured to set a predetermined output mode based on commands obtained from the application processor until a wake-up interrupt occurs.

[0153] The predetermined output mode is characterized by output mode parameters, which include the region of interest associated with the output mode and may include the output frame rate, merging (such as pixel value merging of adjacent pixels), and image data output format (pixel values ​​(included in the image data) or component values). In some embodiments, the mode sequencer circuit is also configured to set a predetermined illumination mode during the predetermined output mode.

[0154] The command may include at least one bit indicating a predetermined output mode or output mode parameter.

[0155] The application processor can transmit commands in response to state changes in the device or vehicle embedded in the ToF system and mode sequencer circuitry. For example, when a mobile device is not used for a predetermined period of time, it can switch to sleep mode, and a command can be issued and transmitted in response to this switch. In this example, a wake-up interrupt can occur, for instance, in response to another state change, when the mobile device is used again.

[0156] In some implementations, a first region of interest and a second region of interest are set based on preset information. In such implementations, the first region of interest and the second region of interest are static and predetermined.

[0157] In some implementations, a first region of interest and a second region of interest are set based on tracking information obtained from an application processor. In such implementations, a first subroutine and a second subroutine, or an application program including them, analyze the size and movement of targets and people within the respective regions of interest (determine tracking information), and determine the region of interest based on the size and movement of the targets and people. Corresponding pixel regions are then set accordingly.

[0158] This allows the first and second regions of interest to be adjusted based on at least one requirement of each subroutine, and thus reduces the amount of data transmitted on the data bus, etc.

[0159] In some implementations, the pattern sequencer circuit is also configured as follows:

[0160] A first region of interest is determined based on the obtained first image data or the calculated first component data, and a second region of interest is determined based on the obtained second image data or the calculated second component data; and

[0161] A first area of ​​concern is set up based on the determined first area of ​​concern, and a second area of ​​concern is set up based on the determined second area of ​​concern.

[0162] In such an implementation, the pattern sequencer circuit is part of the ToF system or the imaging unit of the ToF system.

[0163] Determining the first and second regions of interest can include object detection, tracking information, etc. Corresponding pixel areas are then set accordingly.

[0164] This can reduce the computational load on the application processor.

[0165] Some implementations relate to a mode sequencing method for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the mode sequencing method including:

[0166] The imaging unit is configured with a first output mode and a second output mode for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first limitation of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second limitation of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0167] Pattern sequencing methods can be based on or implemented by electronic components, integrated circuit logic, CPUs, FPGAs, software, or software executed in part by electronic components and part by processors. The method can also be executed by pattern sequencer circuitry, as described herein.

[0168] The methods described herein are, in some embodiments, also implemented as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium storing a computer program product is also provided, which, when executed by a processor (such as the processor described above), causes the methods described herein to be performed.

[0169] Some implementations relate to a time-of-flight system, which includes:

[0170] The lighting unit includes a light source (as described herein) configured to illuminate the scene;

[0171] The optical lens portion is configured to image the light reflected from the scene onto the imaging unit (as described herein);

[0172] The imaging unit includes: an image sensor configured to generate image data representing time-of-flight measurements based on light imaged onto the image sensor; and a data bus interface for transmitting data to an application processor via a data bus.

[0173] A pattern sequencer circuit (as described herein) is configured to set a first output mode and a second output mode of an imaging unit for outputting first image data and second image data via a data bus interface, wherein the first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit, wherein the first output mode is adjusted according to at least one requirement of a first subroutine running on an application processor and a first constraint of the data bus interface, the first subroutine being used to analyze the first region of interest, and the second output mode is adjusted according to at least one requirement of a second subroutine running on an application processor and a second constraint of the data bus interface, the second subroutine being used to analyze the second region of interest; and

[0174] Application processor.

[0175] Back Figure 3 A block diagram schematically illustrates a first embodiment of the pattern sequencer circuit 33-1 for the time-of-flight system 20-1, which will be referred to below. Figure 3 Let's have a discussion.

[0176] The iToF system 20-1 is a global iToF system used to provide distance measurements. The iToF system 20-1 is embedded in vehicle 21 for occupant monitoring. The iToF system 20-1 and vehicle 21 are shown separately for illustrative purposes only. The iToF system 20-1 can be mounted, for example, on a center console or on the ceiling near the front window of vehicle 21.

[0177] The iToF system 20-1 includes an illumination unit 22, an imaging unit 23-1, and an optical lens section 24.

[0178] The imaging unit 23-1 includes a control unit 25-1 and an image sensor 26.

[0179] The illumination unit 22, including an LED array as a light source, emits intensity-modulated light (temporally) onto a scene (illumination scene) inside the vehicle cabin of the vehicle 21, which includes a first region of interest 27, second regions of interest 28a and 28b, and a third region of interest 29a and 29b. Targets and people (not shown) in regions of interest 27, 28a and 28b, and 29a and 29b reflect at least a portion of the (illumination) light. The reflected light from the targets and people is imaged onto the image sensor 26 by the optical lens portion 24.

[0180] Control unit 25-1 controls the overall operation of iToF system 20-1. Control unit 25-1 controls the light emission time of illumination unit 22 based on the modulation signal provided to illumination unit 22 (which can be a rectangular modulation signal with a modulation period T, i.e., modulation frequency fmod = 1 / T). Control unit 25-1 applies a demodulated signal to image sensor 26, which corresponds to the modulation signal provided to illumination unit 22.

[0181] Image sensor 26 generates image data, including pixel values ​​of multiple pixels, based on the amount of reflected light imaged on the image sensor (or imaging unit 23-1) through optical lens portion 24 and according to the demodulated signal. Control unit 25-1 reads out the image data generated by image sensor 6.

[0182] Imaging unit 23-1 has a data bus interface for transmitting generated image data to application processor 31-1 of vehicle 21 via data bus (interface) 30. Here, data bus 30 is a data bus conforming to the MIPI CSI-2 specification.

[0183] The application processor 31-1 includes an application unit 32, a pattern sequencer circuit 33-1, an image processing unit 34-1, and a 3D image reconstruction unit 35. The application unit 32 can perform further advanced functions, such as memory management, graphics processing, multimedia decoding and encoding, etc.

[0184] Application unit 32 executes the first subroutine 32a, the second subroutine 32b, and the third subroutine 32c (in other words, subroutines 32a, 32b, and 32c run on application processor 31-1).

[0185] The first subroutine 32a analyzes the first region of interest 27, the second subroutine 32b analyzes the second regions of interest 28a and 28b, and the third subroutine 32c analyzes the third regions of interest 29a and 29b.

[0186] The first subroutine 32a is a program for analyzing the body posture of the occupants in vehicle 21 (body posture recognition), the second subroutine 32b is a program for analyzing the face of the occupants in the front seat of vehicle 21 (face recognition), and the third subroutine 32c is a program for analyzing the gestures of the occupants in the front seat of vehicle 21 (gesture recognition).

[0187] Here, the pattern sequencer circuit is implemented based on software executed or run by the application processor 31-1.

[0188] The mode sequencer circuit 33-1 sets the first output mode, the second output mode, and the third output mode of the imaging unit 23-1 for outputting the first image data, the second image data, and the third image data to the application processor 31-1 via the data bus 30.

[0189] The first output mode is associated with the first region of interest 27 imaged by the imaging unit 23-1, the second output mode is associated with the second regions of interest 28a and 28b imaged by the imaging unit 23-1, and the third output mode is associated with the third regions of interest 29a and 29b imaged by the imaging unit 23-1.

[0190] The first output mode is adjusted according to at least one requirement of a first subroutine 32a running on the application processor 31-1 and a first constraint of the data bus interface 30, the first subroutine 32a being used to analyze a first region of interest 27; the second output mode is adjusted according to at least one requirement of a second subroutine 32b running on the application processor 31-1 and a second constraint of the data bus interface 30, the second subroutine 32b being used to analyze second regions of interest 28a and 28b; and the third output mode is adjusted according to at least one requirement of a third subroutine 32c running on the application processor 31-1 and a third constraint of the data bus interface 30, the third subroutine 32c being used to analyze third regions of interest 29a and 29b.

[0191] Here, the first, second, and third limitations are the same: the maximum possible amount of image data that can be transmitted via the data bus 30. (Refer to the following...) Figure 6 Let's discuss specific examples further.

[0192] At least one requirement of the first subroutine 32a, the second subroutine 32b, and the third subroutine 32c may be the frame rate of the image data and the spatial (and depth, or depth) resolution of the image data.

[0193] Image processing unit 34-1 is used for image data processing by a first subroutine (32a), a second subroutine (32b), and a third subroutine (32c). Image processing unit 34-1 obtains image data output by imaging unit 23-1 according to a first output mode, a second output mode, and a third output mode. This data is transmitted from imaging unit 23-1 to application processor 31-1 via data bus 30. Based on the obtained image data, image processing unit 34-1 determines the phase shift of the detected reflected light relative to the emitted light. Then, based on the determined phase shift, image processing unit 34-1 calculates the distance d or general depth information for each region of interest (27, 28a and 28b, 29a and 29b).

[0194] The 3D image reconstruction unit 35 is used by the first subroutine (32a), the second subroutine (32b), and the third subroutine (32c) to construct 3D images, 3D depth maps, or 3D point clouds of their respective regions of interest (27, 28a and 28b, 29a and 29b) based on the depth information from the image processing unit 34-1.

[0195] Figure 4 A block diagram of a second embodiment of the mode sequencer circuit of the time-of-flight system 20-2 is schematically shown below, with reference to... Figure 4 Let's have a discussion.

[0196] basically, Figure 4 The second embodiment of this invention and the reference Figure 3 The first embodiment will be discussed in relation to the second embodiment, and therefore the differences between the first and second embodiments will be discussed.

[0197] The iToF system 20-2 includes an illumination unit 22, an imaging unit 23-2, and an optical lens section 24. The imaging unit 23-2 includes a control unit 25-2 and an image sensor 26.

[0198] Although Figure 3 In the first embodiment, the pattern sequencer circuit 33-1 is implemented based on software executed or run by the application processor 31-1, but the pattern sequencer circuit 33-2 in this second embodiment is implemented as hardware, for example, as an integrated circuit or FPGA in the control unit 25-2.

[0199] In addition, the mode sequencer circuit 33-2 obtains first image data, second image data and third image data according to the first output mode, second output mode and third output mode of the imaging unit 23-2.

[0200] Then, the pattern sequencer circuit 33-2 calculates the first component data, the second component data, and the third component data based on the obtained first image data, second image data, and third image data.

[0201] Then, the pattern sequencer circuit 33-2 transmits the obtained first image data or the calculated first component data to the application processor 31-2 according to at least one requirement of the first subroutine 32a and the first limitation of the data bus interface 30; transmits the obtained second image data or the calculated second component data to the application processor 31-2 according to at least one requirement of the second subroutine 32b and the second limitation of the data bus interface 30; and transmits the obtained third image data or the calculated third component data to the application processor 31-2 according to at least one requirement of the third subroutine 32c and the third limitation of the data bus interface 30.

[0202] In addition, the image processing unit 34-2 in the application processor 31-2 is used for image data processing by the first subroutine (32a), the second subroutine (32b), and the third subroutine (32c), and obtains image data or component data according to the data output format (image data or component data) for image data processing.

[0203] Figure 5 schematically shown Figure 3 and Figure 4 The implementation method of the vehicle state of vehicle 21 in the implementation method will be referred to below. Figure 5 discuss.

[0204] Figure 3 and Figure 4 In the implementation, vehicle 21 is in one of vehicle states S1, S2, S3, S4 and S5.

[0205] State S1 corresponds to the parked state (e.g., vehicle 21 is parked and there are no occupants inside), state S2 corresponds to the entry / exit state, state S3 corresponds to the preparation / end of driving state, state S4 corresponds to the parked state (e.g., the driver parks vehicle 21), and state S5 corresponds to the driving state (e.g., the driver is driving vehicle 21 on the road, the driver is accelerating vehicle 21, etc.).

[0206] At point 40a, for example, when an occupant enters vehicle 21, state S1 transitions to state S2. At point 40b, for example, when vehicle 21 is parked and an occupant leaves vehicle 21, state S2 transitions to state S1.

[0207] At point 41a, for example, when the driver fastens their seatbelt or starts the engine, state S2 transitions to state S3. At point 41b, for example, when the engine is turned off, state S3 transitions to state S2.

[0208] At 42a, for example, when the driver begins to drive vehicle 21 out of the parking lot, state S3 transitions to state S4. At 42b, for example, when the driver parks vehicle 21 in the parking lot, state S4 transitions to state S3.

[0209] At point 43a, for example, when vehicle 21 leaves the parking lot and drives on the road, state S4 transitions to state S5. At point 43b, for example, when the driver begins to park vehicle 21 in the parking lot, state S5 transitions to state S4.

[0210] In each of vehicle states S1, S2, S3, S4, and S5, the application processor (e.g., application processors 31-1 and 31-2) can execute different subroutines to analyze image data or depth information of relevant regions of interest, and the subroutines can have different requirements for the image data, as described herein. Furthermore, the region of interest analyzed by the subroutines may depend on vehicle state S1, S2, S3, S4, or S5.

[0211] For example, in state S1, the subroutine might only need to analyze whether someone has entered vehicle 21, but power consumption should be minimal. For example, in state S2, the subroutine might request facial recognition of the occupant entering vehicle 21. For example, in state S3, the subroutine might need to recognize body posture (e.g., to identify whether the driver is wearing a seatbelt) or gestures (e.g., if gesture-based operation of vehicle 21 is enabled). For example, in state S4, the subroutine might request recognition of whether the driver's hands are on the steering wheel. For example, in state S5, the subroutine might request real-time recognition of the driver's body posture and facial expressions (e.g., to identify driver fatigue) to enable advanced driver assistance functions of vehicle 21, such as gesture recognition for infotainment control.

[0212] Figure 6 The first embodiment of the sequence 50-1 of output modes (OM-A, OM-B, OM-C) and illumination modes (IM-A, IM-B, IM-C) is schematically shown below, with reference to... Figure 6 discuss.

[0213] This implementation method is based on reference. Figure 3 The first embodiment of the pattern sequencer circuit 33-1 discussed, wherein vehicle 21 is in reference Figure 5 The driving state S5 is under discussion.

[0214] In this embodiment, the mode sequencer circuit 33-1 is set to a sequence 50-1 of the output modes (OM-A, OM-B, OM-C) of the imaging unit 23-1 and the illumination modes (IM-A, IM-B, IM-C) of the illumination unit 22.

[0215] even though Figure 3 and Figure 6 The implementation mentions three output modes, but this disclosure is not limited to three output modes (and lighting modes) or any specific number of output modes if at least two output modes (and lighting mode) are set. Furthermore, this implementation is not limited to the specific timeline of sequence 50-1 or any other specific output mode sequence; for example, it also includes sequences such as ACBACB... or BCACA... or ABCB... or ABCBDBCB... are all possible.

[0216] Here, Figure 6 The lower part shows the timeline of subroutines (32a, 32b, 32c) running on the analysis interest area (27, 28a and 28b, 29a and 29b) of the application processor 31-1.

[0217] The first subroutine 32a is a procedure for analyzing the body posture (body posture recognition) of the occupants in the vehicle 21 in the first region of interest 27; the second subroutine 32b is a procedure for analyzing the face or facial expression of the occupants in the front seat of the vehicle 21 in the second regions of interest 28a and 28b (face recognition); and the third subroutine 32c is a procedure for analyzing the gestures (gesture recognition) of the occupants in the front seat of the vehicle 21 in the third regions of interest 29a and 29b.

[0218] Here, Figure 6 The upper part shows the timeline of the set output modes (OM-A, OM-B, OM-C) of the imaging unit 23-1 and the set illumination modes (IM-A, IM-B, IM-C) of the illumination unit 22 during the set output modes, where the dashed lines represent the changes in subroutines and corresponding output modes.

[0219] First, a first subroutine 32a for body pose recognition is executed. The pattern sequencer circuit 33-1 sets a first output pattern OM-A for the imaging unit 23-1 to output first image data via the data bus 30, wherein the first output pattern OM-A is associated with the first region of interest 27 analyzed by the first subroutine 32a.

[0220] The mode sequencer circuit 33-1 sets the output mode parameters of the first output mode OM-A: ROI1 (first region of interest 27), first output frame rate (OFR1), and first combining (BIN1). The mode sequencer circuit 33-1 also sets the illumination mode parameters of the first illumination mode IM-A during the first output mode OM-A: the first illumination region corresponding to ROI1 and the first modulation frequency fmod1. The first output frame rate OFR1 is schematically illustrated by the black line in the time axis.

[0221] Imaging unit 23-1 outputs first image data according to the first imaging mode OM-A.

[0222] Then, the second subroutine 32b for face recognition is executed. The second output mode OM-B for the imaging unit 23-1 to output second image data via the data bus 30 is set by the mode sequencer circuit 33-1, wherein the second output mode OM-B is related to the second regions of interest 28a and 28b analyzed by the second subroutine 32b.

[0223] The mode sequencer circuit 33-1 sets the output mode parameters of the second output mode OM-B: ROI2 (the second regions of interest 28a and 28b), the second output frame rate (OFR2) and the second binning (BIN2). The mode sequencer circuit 33-1 sets the illumination mode parameters of the second illumination mode IM-B during the second output mode OM-B: a second illumination region corresponding to ROI2 and a second modulation frequency fmod2. The second output frame rate OFR2 is schematically illustrated by a black line in the timeline.

[0224] The imaging unit 23-1 outputs second image data according to the second imaging mode OM-B.

[0225] Then, the third subroutine 32c for gesture recognition is executed. The third output mode OM-C of the imaging unit 23-1 for outputting third image data via the data bus 30 is set by the mode sequencer circuit 33-1, wherein the third output mode OM-C is associated with the third regions of interest 29a and 29b analyzed by the third subroutine 32c.

[0226] The mode sequencer circuit 33-1 sets the output mode parameters of the third output mode OM-C: ROI3 (the third regions of interest 29a and 29b), the third output frame rate (OFR3) and the third binning (BIN3). The mode sequencer circuit 33-1 sets the illumination mode parameters of the third illumination mode IM-C during the third output mode OM-C: a third illumination region corresponding to ROI3 and a third modulation frequency fmod3. The third output frame rate OFR3 is schematically illustrated by a black line in the timeline.

[0227] The imaging unit 23-1 outputs third image data according to the third imaging mode OM-C.

[0228] The output modes (OM-A, OM-B, OM-C) are set based on at least one limitation of the data bus 30. Herein, the at least one limitation is the maximum amount of image data that can be transmitted through the data bus 30.

[0229] According to at least one requirement of each corresponding subroutine, the first output frame rate OFR1 is set to be lower than the second output frame rate OFR2, and the second output frame rate OFR2 is set to be lower than the third output frame rate OFR3 (OFR1<OFR2<OFR3), because body posture recognition may require a lower output frame rate OFR1 than face recognition (OFR2), while gesture recognition requires the highest output frame rate OFR3.

[0230] The first merged BIN1 and the third merged BIN3 are set to be equal (BIN1 = BIN3 = 2 × 2 pixels), while the second merged BIN2 is set to one (BIN2 = 1 × 1 pixel) because body pose recognition and gesture recognition may require lower spatial resolution, while face recognition requires the highest spatial resolution.

[0231] The first modulation frequency fmod1 is set to be greater than the third modulation frequency fmod3, and the third modulation frequency fmod3 is set to be greater than the second modulation frequency fmod2 (fmod1>fmod3>fmod2) because body pose recognition may require the highest depth resolution, while face recognition may require the lowest depth resolution.

[0232] Therefore, based on the requirements of each subroutine, the output modes (OM-A, OM-B, OM-C) are optimized under the constraint of the maximum possible amount of image data that can be transmitted through the data bus 30 by making trade-offs between the temporal resolution (OFR1-3), spatial resolution (BIN1-3), and depth resolution (fmod1-3) of the image data in each output mode (OM-A, OM-B, OM-C).

[0233] Therefore, a first output mode OM-A is adjusted according to at least one requirement of a first subroutine 32a running on application processor 31-1 and a limitation of data bus interface 30, the first subroutine 32a being used to analyze a first region of interest (27, ROI1); a second output mode OM-B is adjusted according to at least one requirement of a second subroutine 32b running on application processor 31-1 and a limitation of data bus interface 30, the second subroutine 32b being used to analyze a second region of interest (28a and 28b, ROI2); and a third output mode OM-C is adjusted according to at least one requirement of data bus interface 30 and a limitation of data bus interface 30, the third subroutine 32c running on application processor 31-1, the third subroutine 32c being used to analyze a third region of interest (29a and 29b, ROI3).

[0234] Furthermore, in the first lighting mode IM-A, a first lighting area of ​​the lighting unit 22 corresponding to the first area of ​​interest (27, ROI1) is set; in the second lighting mode IM-B, a second lighting area of ​​the lighting unit 22 corresponding to the second areas of interest (28a and 28b, ROI2) is set; and in the third lighting mode IM-C, a third lighting area of ​​the lighting unit 22 corresponding to the third areas of interest (29a and 29b, ROB) is set.

[0235] Furthermore, according to at least one requirement of the first subroutine 32a, the first modulation frequency fmod1 of the lighting unit 22 is set in the first lighting mode IM-A; according to at least one requirement of the second subroutine 32b, the second modulation frequency fmod2 of the lighting unit 22 is set in the second lighting mode IM-B; and according to at least one requirement of the third subroutine 32c, the third modulation frequency fmod3 of the lighting unit 22 is set in the third lighting mode IM-C.

[0236] Figure 7 The second embodiment of the sequence 50-2 of output modes (OM-A', OM-B', OM-C') and illumination modes (IM-A, IM-B, IM-C) is schematically shown below, with reference to... Figure 7 discuss.

[0237] This implementation method is based on reference. Figure 4 The second embodiment of the pattern sequencer circuit 33-2 discussed, wherein vehicle 21 is in the reference position Figure 5 The driving state S5 is under discussion.

[0238] In this embodiment, the mode sequencer circuit 33-2 sets a sequence 50-2 of the output modes (OM-A', OM-B', OM-C') of the imaging unit 23-2 and the illumination modes (IM-A, IM-B, IM-C) of the illumination unit 22, which essentially corresponds to the reference sequence. Figure 6 The implementation methods discussed are sequence 50-1, but the differences between them will be discussed below.

[0239] In this second embodiment, the mode sequencer circuit 33-2 further obtains the first image data, the second image data, and the third image data according to the first output mode (OM-A'), the second output mode (OM-B'), and the third output mode (OM-C') of the imaging unit 23-2.

[0240] In the first output mode (OM-A') and the third output mode (OM-C'), the mode sequencer circuit calculates the first component data (IQ1) and the third component data (IQ3) based on the obtained first image data and third image data.

[0241] In the first output mode OM-A', the mode sequencer circuit transmits the calculated first component data IQ1 according to at least one requirement of the first subroutine 32a. Since the first region of interest (27, ROI1) covers the entire field of view analyzed by the first subroutine 32a, preprocessing before transmission can reduce the computational load on the application processor 31-2.

[0242] In the third output mode OM-C', the mode sequencer circuit transmits the calculated third component data IQ3 according to at least one requirement of the third subroutine 32c. Since the gesture recognition performed by the third subroutine 32c requires the highest output frame rate OFR3, preprocessing before transmission can reduce the computational load on the application processor 31-2.

[0243] When in the second output mode OM-B', the mode sequencer circuit transmits the second image data according to at least one requirement of the second subroutine 32b.

[0244] Figure 8 The implementation of the predetermined output mode (OM-Pred) and predetermined lighting mode (IM-Pred) is illustrated schematically. The following will refer to... Figure 8 discuss.

[0245] This implementation method is based on referring to the respective... Figure 3 and Figure 4 The first and second embodiments of the pattern sequencer circuits 33-1 and 33-2 are discussed, wherein the vehicle 21 is initially in the entry / exit state S2, as shown in the reference. Figure 5 discuss.

[0246] Initially, vehicle 21 is in the entry / exit state S2, meaning the driver has parked vehicle 21 but is still inside vehicle 21.

[0247] Then, the driver leaves vehicle 21, and vehicle 21 is switched to parked state S1.

[0248] In response to the state transition, the application processor issues a command, which is received by the mode sequencer circuit 33-1 / 2. Based on the command received from the application processor 31-1 / 2, the imaging unit 23-1 / 2 is set to the predetermined output mode OM-Pred. Furthermore, the mode sequencer circuit 33-1 / 2 sets the illumination unit 22 to the predetermined illumination mode IM-Pred.

[0249] In response to the state transition, application processor 31-1 / 2 executes the fourth subroutine 32d, which analyzes the fourth region of interest (ROI) 4. The fourth ROI includes the doors and windows of vehicle 21. Subroutine 32d analyzes whether anyone approaches or enters vehicle 21. The predetermined output mode OM-Pred is associated with the fourth ROI (ROI 4).

[0250] The mode sequencer circuit 33-1 / 2 sets the output mode parameters of the predetermined output mode OM-Pred: ROI4, fourth output frame rate (OFR4), and fourth combining (BIN4). During the predetermined output mode OM-Pred, the mode sequencer circuit 33-1 / 2 sets the illumination mode parameters of the predetermined illumination mode IM-Pred: the predetermined illumination area corresponding to ROI4 and the fourth modulation frequency fmod4. The fourth output frame rate OFR4 is schematically illustrated by the black line in the time axis.

[0251] The fourth output frame rate OFR4 is set to be lower than OFR1-3, the fourth merge BIN4 is set to be greater than BIN1-3, and the fourth modulation frequency fmod4 is set to be less than fmod1-3, because the fourth subroutine 32d may only require a low output frame rate OFR4 (low temporal resolution), low spatial resolution (large BIN4), and low depth resolution (low fmod4).

[0252] When the fourth subroutine 32d detects that someone is approaching and entering vehicle 21, the application processor 31-1 / 2 issues a wake-up interrupt 60.

[0253] Then, vehicle 21 transitions to the entry / departure state S2.

[0254] In response to the state transition, application processors 31-1 / 2 execute, for example, the first subroutine 32a (however, it can execute any other subroutine or sequence of subroutines).

[0255] In response to wake-up interrupt 60, the mode sequencer circuit 33-1 / 2 sets the first output mode OM-A / A' of imaging unit 23-1 / 2 and the first illumination mode IM-A / A' of illumination unit 22 during the first output mode OM-A / A'.

[0256] Figure 9 A flowchart of a first embodiment of the pattern sequencing method 200 is schematically shown (e.g., it can be implemented in the application processor or time-of-flight system of this disclosure).

[0257] At point 201, a first output mode and a second output mode of the imaging unit are configured for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on an application processor and a first constraint of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on an application processor and a second constraint of the data bus interface, the second subroutine being used to analyze the second region of interest, as described herein.

[0258] At point 202, in a first output mode, a first output frame rate of the first image data is set according to at least one requirement of the first subroutine and a first limitation of the data bus interface, and in a second output mode, a second output frame rate of the second image data is set according to at least one requirement of the second subroutine and a second limitation of the data bus interface, as described herein.

[0259] At point 203, in a first output mode, a first merging of first image data is set according to at least one requirement of the first subroutine and a first limitation of the data bus interface, and in a second output mode, a second merging of second image data is set according to at least one requirement of the second subroutine and a second limitation of the data bus interface, as described herein.

[0260] At 204, a first illumination mode is set for the illumination unit during a first output mode of the imaging unit, and a second illumination mode is set for the illumination unit during a second output mode of the imaging unit. In the first illumination mode, a first illumination area of ​​the illumination unit corresponding to a first region of interest is set, and in the second illumination mode, a second illumination area of ​​the illumination unit corresponding to a second region of interest is set, as described herein.

[0261] At 205, a first modulation frequency of the lighting unit is set in a first lighting mode according to at least one requirement of the first subroutine, and a second modulation frequency of the lighting unit is set in a second lighting mode according to at least one requirement of the second subroutine, as described herein.

[0262] Figure 10 A second implementation of the pattern sequencing method 300 is illustrated in the form of a flowchart (e.g., it can be implemented in the time-of-flight system of this disclosure).

[0263] At point 301, execute the reference. Figure 9 Steps 201 to 205 of the implementation method discussed are incorporated herein by reference to avoid unnecessary repetition.

[0264] At 302, first image data is obtained according to the first output mode of the imaging unit, and second image data is obtained according to the second output mode of the imaging unit, as described herein.

[0265] At point 303, first component data is calculated based on the obtained first image data, and second component data is calculated based on the obtained second image data, as described herein.

[0266] At 304, the acquired first image data or calculated first component data is transmitted according to at least one requirement of the first subroutine and a first limitation of the data bus interface, and the acquired second image data or calculated second component data is transmitted according to at least one requirement of the second subroutine and a second limitation of the data bus interface, as described herein.

[0267] At 305, a first region of interest is determined based on the obtained first image data or the first calculated component data, and a second region of interest is determined based on the obtained second image data or the calculated second component data, as described herein.

[0268] At point 306, a first region of interest is set according to the determined first region of interest, and a second region of interest is set according to the determined second region of interest, as described in this article.

[0269] Figure 11 A flowchart of a third embodiment of the pattern sequencing method 400 is schematically shown (e.g., it can be implemented in the application processor or time-of-flight system of this disclosure).

[0270] At point 401, execute the reference. Figure 9 Steps 201 to 205 of the implementation method discussed are included here for reference in order to avoid unnecessary repetition.

[0271] At position 402, a first and a second region of interest are set based on preset information, as described in this article.

[0272] At 403, a predetermined output mode is set based on the command obtained from the application processor until a wake-up interrupt occurs, as described herein.

[0273] Figure 12 A flowchart of a fourth embodiment of the pattern sequencing method 500 is schematically shown (e.g., it can be implemented in the application processor or time-of-flight system of this disclosure).

[0274] At point 501, execute the reference. Figure 9 Steps 201 to 205 of the implementation method discussed are included here for reference in order to avoid unnecessary repetition.

[0275] At point 502, a first region of interest and a second region of interest are set based on the tracking information obtained from the application processor, as described herein.

[0276] Figure 13 This schematically illustrates the use of, for example, according to Figure 3 and Figure 4 A block diagram illustrating an embodiment of the vehicle control system 7000 of vehicle 21.

[0277] Figure 13 This is a block diagram illustrating a schematic configuration of a vehicle control system 7000 as an example of a mobile vehicle body control system, to which the technology according to embodiments of this disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units interconnected via a communication network 7010. Figure 13 In the described example, the vehicle control system 7000 includes a driving system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting the multiple control units can be an in-vehicle communication network conforming to any standard, such as Controller Area Network (CAN), Local Interconnect Network (LIN), Local Area Network (LAN), FlexRay (registered trademark), etc.

[0278] The in-vehicle information detection unit 7500 may include a time-of-flight system, a pattern sequencer circuit, and an application processor conforming to this disclosure.

[0279] Each control unit includes: a microcomputer (including, for example, an application processor) that performs arithmetic processing according to various programs; a storage unit for storing programs executed by the microcomputer, parameters for various operations, etc.; and drive circuitry for driving various control target devices. Each control unit also includes: a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F (including, for example, a MIPI CSI-2 interface) for communicating with devices, sensors, etc., inside and outside the vehicle via wired or radio communication. Figure 13 The integrated control unit 7600 shown includes a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiver 7650, an in-vehicle equipment I / F 7660, a voice / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units also include microcomputers, communication I / Fs, and storage units.

[0280] The driving system control unit 7100 controls the operation of devices related to the vehicle's driving system according to various programs. For example, the driving system control unit 7100 functions as a control device for drive force generation devices such as internal combustion engines and drive motors, for drive force transmission mechanisms that transmit drive force to the wheels, for steering mechanisms that adjust the vehicle's steering angle, and for braking devices that generate braking force. The driving system control unit 7100 may also function as a control device for systems such as anti-lock braking systems (ABS) and electronic stability control (ESC).

[0281] The driving system control unit 7100 is connected to the vehicle condition detection unit 7110. For example, the vehicle condition detection unit 7110 includes at least one of the following: a gyroscope sensor for detecting the angular velocity of the vehicle's axial rotational movement; an acceleration sensor for detecting the vehicle's acceleration; and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, engine speed, or wheel speed. The driving system control unit 7100 performs arithmetic processing using signals input from the vehicle condition detection unit 7110 and controls the internal combustion engine, drive motor, electric power steering system, braking system, etc.

[0282] The body system control unit 7200 controls the operation of various devices supplied to the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for keyless entry systems, smart key systems, power window devices, or various lights (such as headlights, reversing lights, brake lights, turn signals, fog lights, etc.). In this case, radio waves emitted from mobile devices, as a substitute for keys or signals for various switches, can be input to the body system control unit 7200. The body system control unit 7200 receives these input radio waves or signals and controls the vehicle's door locking devices, power window devices, lights, etc.

[0283] The battery control unit 7300 controls the secondary battery 7310, which is the power source for the drive motor, according to various programs. For example, the battery control unit 7300 obtains information such as battery temperature, battery output voltage, and remaining battery charge from battery devices, including the secondary battery 7310. The battery control unit 7300 uses these signals to perform arithmetic processing and execute controls to regulate the temperature of the secondary battery 7310 or control the cooling equipment supplied to the battery devices.

[0284] The exterior information detection unit 7400 detects information about the exterior of the vehicle, including the vehicle control system 7000. For example, the exterior information detection unit 7400 is connected to at least one of the imaging unit 7410 and the exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. For example, the exterior information detection unit 7420 includes at least one of an environmental sensor for detecting current atmospheric or weather conditions and a peripheral information detection sensor for detecting other vehicles, obstacles, pedestrians, etc., surrounding the vehicle including the vehicle control system 7000.

[0285] For example, the environmental sensor can be at least one of a raindrop sensor for detecting raindrops, a fog sensor for detecting fog, a sunlight sensor for detecting illuminance, and a snowfall sensor for detecting snowfall. The peripheral information detection sensor can be at least one of an ultrasonic sensor, a radar device, and a lidar (LiDAR) device (light detection and ranging device, or laser imaging detection and ranging device). Each of the imaging unit 7410 and the external information detection unit 7420 can be provided as an independent sensor or device, or as a device integrating multiple sensors or devices.

[0286] Figure 14 A schematic diagram illustrating an embodiment of the mounting positions of the vehicle exterior information detection unit 7420 and the imaging unit 7410 is shown.

[0287] Imaging units 7910, 7912, 7914, 7916, and 7918 are provided at least one of the following locations: the front nose of the vehicle 7900, the side mirrors, the rear bumper, and the rear door, as well as the upper part of the windshield inside the vehicle. Imaging unit 7910, provided to the front nose, and imaging unit 7918, provided to the upper part of the windshield inside the vehicle, primarily obtain a frontal image of the vehicle 7900. Imaging units 7912 and 7914, provided to the side mirrors, primarily obtain side images of the vehicle 7900. Imaging unit 7916, provided to the rear bumper or rear door, primarily obtains a rear image of the vehicle 7900. Imaging unit 7918, provided to the upper part of the windshield inside the vehicle, is primarily used to detect vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0288] Incidentally, Figure 14Examples of the imaging ranges of each imaging unit 7910, 7912, 7914, and 7916 are described. Imaging range a represents the imaging range provided to the imaging unit 7910 for the front nose. Imaging ranges b and c represent the imaging ranges provided to the imaging units 7912 and 7914 for the side mirrors, respectively. Imaging range d represents the imaging range provided to the imaging unit 7916 for the rear bumper or rear door. For example, by superimposing the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view of the vehicle 7900 from above can be obtained.

[0289] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided to the front, rear, sides, corners, and upper part of the windshield inside the vehicle 7900 can be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930 provided to the front nose, rear bumper, rear door, and upper part of the windshield inside the vehicle 7900 can be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect vehicles ahead, pedestrians, obstacles, etc.

[0290] Return to Figure 13 The description will continue. The exterior information detection unit 7400 causes the imaging unit 7410 to image an image of the exterior of the vehicle and receives the image data. Furthermore, the exterior information detection unit 7400 receives detection information from the exterior information detection section 7420 connected to it. If the exterior information detection section 7420 is an ultrasonic sensor, radar device, or LIDAR device, the exterior information detection unit 7400 emits ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. Based on the received information, the exterior information detection unit 7400 can perform target detection processing (such as people, vehicles, obstacles, signs, characters on the road surface, etc.) or distance detection processing. The exterior information detection unit 7400 can perform environmental recognition processing based on the received information, identifying conditions such as rain, fog, and road conditions. The exterior information detection unit 7400 can calculate the distance to the external target based on the received information.

[0291] Furthermore, based on the received image data, the vehicle exterior information detection unit 7400 can perform image recognition processing to identify people, vehicles, obstacles, signs, characters on the road surface, etc., or to detect their distance. The vehicle exterior information detection unit 7400 can process the received image data, such as distortion correction and alignment, and combine image data captured by multiple different imaging units 7410 to generate a bird's-eye view or panoramic image. The vehicle exterior information detection unit 7400 can use image data captured by imaging units 7410 including different imaging components to perform viewpoint transformation processing.

[0292] The in-vehicle information detection unit 7500 detects information about the interior of the vehicle. The in-vehicle information detection unit 7500 may include a time-of-flight system, a pattern sequencer circuit, and an application processor conforming to this disclosure. The in-vehicle information detection unit 7500 may be connected, for example, to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biological information, and a microphone that collects sound inside the vehicle. For example, the biosensor is placed on the seat surface, steering wheel, etc., and detects the biological information of the occupant sitting in the seat or the driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the in-vehicle information detection unit 7500 can calculate the driver's fatigue level or concentration level, or determine whether the driver is dozing off. The in-vehicle information detection unit 7500 can perform processing such as noise cancellation on the audio signals obtained by collecting sound.

[0293] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is implemented by a device capable of input operation by the occupant, such as a touchpad, button, microphone, switch, lever, etc. The integrated control unit 7600 can provide data through voice recognition of voice input via the microphone. The input unit 7800 can be, for example, a remote control device using infrared or other radio waves, or an external connection device supporting operation of the vehicle control system 7000, such as a mobile phone, personal digital assistant (PDA), etc. The input unit 7800 can also be, for example, a camera. In this case, the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on the information input by the occupant or others using the aforementioned input unit 7800, and outputs the generated input signal to the integrated control unit 7600. Passengers and other personnel can input various data or issue processing instructions to the vehicle control system 7000 through the operation input unit 7800.

[0294] The storage unit 7690 may include a read-only memory (ROM) for storing various programs executed by a microcomputer and a random access memory (RAM) for storing various parameters, operation results, sensor values, etc. Furthermore, the storage unit 7690 may be implemented using magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc.

[0295] The Universal Communication I / F 7620 is a widely used communication interface for communicating with various devices located in the external environment 7750. The Universal Communication I / F 7620 can implement cellular communication protocols such as GSM, WiMAX, LTE, and LTE-Advanced (LTE-A), or other wireless communication protocols such as Wi-Fi (also known as Wi-Fi) and Bluetooth. The Universal Communication I / F 7620 can connect, for example, to devices (e.g., application servers or control servers) located on external networks (e.g., the Internet, cloud networks, or corporate private networks) via base stations or access points. Furthermore, the Universal Communication I / F 7620 can connect, for example, to terminals located near the vehicle (e.g., terminals belonging to drivers, pedestrians, or shopkeepers, or machine-type communication (MTC) terminals) using peer-to-peer (P2P) technology.

[0296] The Dedicated Communications I / F 7630 is a communication interface that supports communication protocols developed for use in vehicles. The Dedicated Communications I / F 7630 can implement standard protocols such as Wireless Access in a Vehicle Environment (WAVE) (a combination of IEEE 802.11p as the lower layer and IEEE 1609 as the higher layer), Dedicated Short Range Communications (DSRC), or cellular communication protocols. The Dedicated Communications I / F 7630 typically implements V2X communication concepts, including one or more of the following: vehicle-to-vehicle communication, road-to-vehicle communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0297] For example, the positioning unit 7640 performs positioning by receiving Global Navigation Satellite System (GNSS) signals from Global Navigation Satellite System (GNSS) satellites (e.g., GPS signals from Global Positioning System (GPS) satellites) and generates location information including the vehicle's latitude, longitude, and altitude. Incidentally, the positioning unit 7640 can determine its current location by exchanging signals with a wireless access point, or it can obtain location information from a terminal with positioning capabilities, such as a mobile phone, Personal Handheld Phone System (PHS), or smartphone.

[0298] For example, the beacon receiver 7650 receives radio waves or electromagnetic waves transmitted from radio stations installed on roads, etc., and thereby obtains information about current location, congestion, road closures, and necessary time. Incidentally, the functions of the beacon receiver 7650 can be included in the dedicated communication I / F 7630 described above.

[0299] The in-vehicle device I / F 7660 is a communication interface used to connect the microcomputer 7610 and various in-vehicle devices 7760 present within the vehicle. The in-vehicle device interface 7660 can establish a wireless connection using wireless communication protocols, such as Wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless Universal Serial Bus (WUSB). Alternatively, the in-vehicle device I / F 7660 can establish a wired connection via Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI (registered trademark)), Mobile High Definition Link (MHL), etc., or via a connection terminal not shown in the figure (and cables if necessary). The in-vehicle devices 7760 may, for example, include at least one of mobile devices and wearable devices owned by the occupants, as well as information devices carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device capable of searching for a path to any destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0300] The vehicle network I / F 7680 is an interface used for communication between the microcomputer 7610 and the communication network 7010. The vehicle network I / F 7680 sends and receives signals according to predetermined protocols supported by the communication network 7010.

[0301] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained through at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on the obtained information about the vehicle's interior and exterior, and output control commands to the driving system control unit 7100. For example, the microcomputer 7610 can perform cooperative control aimed at realizing the functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation, following distance-based driving, speed-maintaining driving, collision warning, lane departure warning, etc. Furthermore, the microcomputer 7610 can perform cooperative control for autonomous driving, which enables the vehicle to drive autonomously without relying on driver operation, and controls the drive force generation device, steering mechanism, braking device, etc. based on information obtained about the vehicle's surrounding environment.

[0302] The microcomputer 7610 can generate 3D distance information between the vehicle and surrounding structures, people, etc., based on information obtained through at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. It also generates local map information including information about the vehicle's current location and its surroundings. Furthermore, the microcomputer 7610 can predict hazards, such as vehicle collisions, pedestrian approach, or entry into closed roads, based on the obtained information, and generate warning signals. For example, the warning signal may be a signal to generate a warning sound or to illuminate a warning light.

[0303] The sound / image output unit 7670 transmits at least one of sound and image output signals to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle. Figure 13In the example, audio speaker 7710, display unit 7720, and dashboard 7730 are shown as output devices. For example, display unit 7720 may include at least one of an in-vehicle display and a head-up display. Display unit 7720 may have augmented reality (AR) display functionality. The output device may be a device other than these, such as another device (e.g., headphones), wearable devices (e.g., eyeglasses displays worn by occupants), projectors, lights, etc. When the output device is a display device, the display device visually displays the results obtained from various processes performed by microcomputer 7610 or information received from another control unit in various forms, such as text, images, tables, charts, etc. Furthermore, when the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or sound data into an analog signal and audibly outputs the analog signal.

[0304] Incidentally, in Figure 13 In the described example, at least two control units interconnected via communication network 7010 can be integrated into one control unit. Alternatively, each individual control unit may include multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown in the figures. Additionally, some or all of the functions performed by one control unit described above can be assigned to another control unit. That is, predetermined arithmetic processing can be performed by any one control unit, provided that information is transmitted and received via communication network 7010. Similarly, a sensor or device connected to one control unit can be connected to another control unit, and multiple control units can transmit and receive detection information from each other via communication network 7010.

[0305] It should be understood that this embodiment describes a method with an exemplary sequence of method steps. However, the specific sequence of method steps is for illustrative purposes only and should not be construed as limiting. For example, Figure 9 In the implementation method, the order of 200 and 203 can be interchanged. Additionally, Figure 9 The order of steps 203, 204, and 205 in the implementation can also be interchanged. Other variations in the order of method steps may be readily apparent to a person skilled in the art.

[0306] Please note that the division of application processor 31-1 into units 32 to 35 is for illustrative purposes only, and this disclosure is not limited to any specific functional division within any particular unit.

[0307] All units and entities described in this specification and claimed in the appended claims, unless otherwise stated, may be implemented as integrated circuit logic, for example, on a chip, and the functions provided by these units and entities, unless otherwise stated, may be implemented by software.

[0308] As the above-disclosed embodiments are implemented at least in part using software-controlled data processing equipment, it is understood that the computer program providing such software control and the transmission, storage or other media providing such computer program are contemplated as aspects of this disclosure.

[0309] Please note that this technology can also be configured as described below.

[0310] (1) A pattern sequencer circuit for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the pattern sequencer circuit being configured to:

[0311] The imaging unit is configured with a first output mode and a second output mode for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first limitation of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second limitation of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0312] (2) According to the mode sequencer circuit in (1), wherein, according to at least one requirement of the first subroutine and the first limitation of the data bus interface, the first output frame rate of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and the second limitation of the data bus interface, the second output frame rate of the second image data is set in the second output mode.

[0313] (3) According to the mode sequencer circuit in (1) or (2), wherein, according to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first merging of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second merging of the second image data is set in the second output mode.

[0314] (4) The pattern sequencer circuit according to any one of (1) to (3), wherein the pattern sequencer circuit is further configured to: set a first illumination mode of the illumination unit during a first output mode of the imaging unit, and set a second illumination mode of the illumination unit during a second output mode of the imaging unit, wherein in the first illumination mode a first illumination area of ​​the illumination unit corresponding to the first region of interest is set, and in the second illumination mode a second illumination area of ​​the illumination unit corresponding to the second region of interest is set.

[0315] (5) According to the mode sequencer circuit in (4), wherein, according to at least one requirement of the first subroutine, the first modulation frequency of the lighting unit is set in the first lighting mode, and according to at least one requirement of the second subroutine, the second modulation frequency of the lighting unit is set in the second lighting mode.

[0316] (6) According to any one of (1) to (5) the pattern sequencer circuit, wherein the pattern sequencer circuit is further configured as follows:

[0317] First image data is obtained according to the first output mode of the imaging unit, and second image data is obtained according to the second output mode of the imaging unit;

[0318] The first component data is calculated based on the obtained first image data, and the second component data is calculated based on the obtained second image data; and

[0319] According to at least one requirement of the first subroutine and a first limitation of the data bus interface, the acquired first image data or the calculated first component data is transmitted, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the acquired second image data or the calculated second component data is transmitted.

[0320] (7) According to any one of (1) to (6) the pattern sequencer circuit, wherein the first interest region and the second interest region are set based on preset information.

[0321] (8) The pattern sequencer circuit according to any one of (1) to (6), wherein the first region of interest and the second region of interest are set based on the tracking information obtained from the application processor.

[0322] (9) According to the pattern sequencer circuit in (6), the pattern sequencer circuit is further configured as follows:

[0323] A first region of interest is determined based on the obtained first image data or the calculated first component data, and a second region of interest is determined based on the obtained second image data or the calculated second component data; and

[0324] A first area of ​​concern is set up based on the determined first area of ​​concern, and a second area of ​​concern is set up based on the determined second area of ​​concern.

[0325] (10) The mode sequencer circuit according to any one of (1) to (9), wherein the mode sequencer circuit is further configured to: set a predetermined output mode based on a command obtained from the application processor until a wake-up interrupt occurs.

[0326] (11) A mode sequencing method for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the mode sequencing method comprising:

[0327] The imaging unit is configured with a first output mode and a second output mode for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on the application processor and a first constraint of the data bus interface. The first subroutine is used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second constraint of the data bus interface. The second subroutine is used to analyze the second region of interest.

[0328] (12) According to the mode sequencing method in (11), wherein, according to at least one requirement of the first subroutine and a first limitation of the data bus interface, the first output frame rate of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the second output frame rate of the second image data is set in the second output mode.

[0329] (13) According to any one of (11) to (12) the mode sequencing method, wherein, according to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first merging of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second merging of the second image data is set in the second output mode.

[0330] (14) According to any one of the pattern ordering methods in (11) to (13), it also includes:

[0331] The illumination unit is configured with a first illumination mode during a first output mode of the imaging unit and a second illumination mode during a second output mode of the imaging unit, wherein in the first illumination mode a first illumination area of ​​the illumination unit corresponding to a first region of interest is configured, and in the second illumination mode a second illumination area of ​​the illumination unit corresponding to a second region of interest is configured.

[0332] (15) According to the mode sequencing method in (14), wherein, according to at least one requirement of the first subroutine, the first modulation frequency of the lighting unit is set in the first lighting mode, and according to at least one requirement of the second subroutine, the second modulation frequency of the lighting unit is set in the second lighting mode.

[0333] (16) According to any one of the pattern ordering methods in (11) to (15), it also includes:

[0334] First image data is obtained according to the first output mode of the imaging unit, and second image data is obtained according to the second output mode of the imaging unit;

[0335] The first component data is calculated based on the obtained first image data, and the second component data is calculated based on the obtained second image data; and

[0336] According to at least one requirement of the first subroutine and a first limitation of the data bus interface, the acquired first image data or the calculated first component data is transmitted, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the acquired second image data or the calculated second component data is transmitted.

[0337] (17) According to any one of (11) to (16) the pattern ordering method, wherein the first region of interest and the second region of interest are set based on preset information.

[0338] (18) According to any one of (11) to (16) the pattern ordering method, wherein the first region of interest and the second region of interest are set based on the tracking information obtained from the application processor.

[0339] (19) According to the pattern ordering method in (16), it also includes:

[0340] A first region of interest is determined based on the obtained first image data or the calculated first component data, and a second region of interest is determined based on the obtained second image data or the calculated second component data; and

[0341] A first area of ​​concern is set based on the determined first area of ​​concern, and a second area of ​​concern is set based on the determined second area of ​​concern.

[0342] (20) According to any one of the pattern ordering methods in (11) to (19), it also includes:

[0343] Based on commands obtained from the application processor, a predetermined output mode is set until a wake-up interrupt occurs.

[0344] (21) A time-of-flight system, the time-of-flight system comprising:

[0345] The lighting unit includes a light source configured to illuminate the scene;

[0346] The optical lens section is configured to image the light reflected from the scene onto the imaging unit;

[0347] The imaging unit includes an image sensor configured to generate image data representing time-of-flight measurements based on light imaged onto the image sensor, and a data bus interface for transmitting data to an application processor via a data bus.

[0348] A pattern sequencer circuit is configured to set a first output mode and a second output mode of an imaging unit for outputting first image data and second image data via a data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted according to at least one requirement of a first subroutine running on an application processor and a first constraint of the data bus interface, the first subroutine being used to analyze the first region of interest. The second output mode is adjusted according to at least one requirement of a second subroutine running on the application processor and a second constraint of the data bus interface, the second subroutine being used to analyze the second region of interest.

[0349] Application processor.

[0350] (22) A computer program including program code that, when executed by a computer, causes the computer to perform the method according to any one of (11) to (20).

[0351] (23) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the method according to any one of (11) to (20) to be performed.

Claims

1. A pattern sequencer circuit for a time-of-flight system, the time-of-flight system including at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the pattern sequencer circuit being configured to: The imaging unit is configured with a first output mode for outputting first image data via the data bus interface, and a second output mode for outputting second image data via the data bus interface, wherein... The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted based on at least one requirement of a first subroutine running on the application processor and a first constraint of the data bus interface. The first subroutine performs facial recognition requiring a first temporal resolution and a first spatial resolution, and analyzes the first region of interest. The second output mode is adjusted based on at least one requirement of a second subroutine running on the application processor and a second constraint of the data bus interface, and analyzes the second region of interest. The second subroutine performs gesture recognition requiring a second temporal resolution and a second spatial resolution, where the first temporal resolution is lower than the second temporal resolution, and the first spatial resolution is higher than the second spatial resolution. The first and second subroutines are subroutines of an application running on the application processor. A sequence including the first output mode and the second output mode is set according to the sequence including the first subroutine and the second subroutine, wherein the first output mode and the second output mode are arranged chronologically according to the sequence including the first output mode and the second output mode. The first image data is obtained according to the first output mode of the imaging unit, and The second image data is obtained according to the second output mode of the imaging unit.

2. The pattern sequencer circuit according to claim 1, wherein, According to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first output frame rate of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second output frame rate of the second image data is set in the second output mode.

3. The pattern sequencer circuit according to claim 1, wherein, According to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first merging of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second merging of the second image data is set in the second output mode.

4. The pattern sequencer circuit according to claim 1, wherein, The mode sequencer circuit is further configured to: set a first illumination mode for the illumination unit during the first output mode of the imaging unit, and set a second illumination mode for the illumination unit during the second output mode of the imaging unit, wherein in the first illumination mode a first illumination area of ​​the illumination unit corresponding to the first region of interest is set, and in the second illumination mode a second illumination area of ​​the illumination unit corresponding to the second region of interest is set.

5. The pattern sequencer circuit according to claim 4, wherein, According to at least one requirement of the first subroutine, a first modulation frequency of the lighting unit is set in the first lighting mode, and according to at least one requirement of the second subroutine, a second modulation frequency of the lighting unit is set in the second lighting mode.

6. The pattern sequencer circuit according to claim 1, wherein, The mode sequencer circuit is further configured to: The first component data is calculated based on the obtained first image data, and the second component data is calculated based on the obtained second image data; and According to at least one requirement of the first subroutine and a first limitation of the data bus interface, the obtained first image data or the calculated first component data is transmitted, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the obtained second image data or the calculated second component data is transmitted.

7. The pattern sequencer circuit according to claim 1, wherein, The first region of interest and the second region of interest are set based on preset information.

8. The pattern sequencer circuit according to claim 1, wherein, The first region of interest and the second region of interest are set based on the tracking information obtained from the application processor.

9. The pattern sequencer circuit according to claim 6, wherein, The mode sequencer circuit is further configured to: The first region of interest is determined based on the first image data obtained or the first component data calculated, and the second region of interest is determined based on the second image data obtained or the second component data calculated. as well as The first area of ​​interest is set according to the determined first area of ​​interest, and the second area of ​​interest is set according to the determined second area of ​​interest.

10. The pattern sequencer circuit according to claim 1, wherein, The mode sequencer circuit is also configured to set a predetermined output mode based on commands obtained from the application processor until a wake-up interrupt occurs.

11. A mode sequencing method for a time-of-flight system, the time-of-flight system comprising at least an imaging unit and an illumination unit, the imaging unit having a data bus interface for transmitting data to an application processor via a data bus, the mode sequencing method comprising: The imaging unit is configured with a first output mode for outputting first image data via the data bus interface, and a second output mode for outputting second image data via the data bus interface. The first output mode is associated with a first region of interest imaged by the imaging unit, and the second output mode is associated with a second region of interest imaged by the imaging unit. The first output mode is adjusted based on at least one requirement of a first subroutine running on the application processor and a first constraint of the data bus interface. The first subroutine analyzes the first region of interest and performs face recognition requiring a first temporal resolution and a first spatial resolution. The second output mode is adjusted based on at least one requirement of a second subroutine running on the application processor and a second constraint of the data bus interface. The second subroutine analyzes the second region of interest and performs gesture recognition requiring a second temporal resolution and a second spatial resolution, where the first temporal resolution is lower than the second temporal resolution and the first spatial resolution is higher than the second spatial resolution. The first and second subroutines are subroutines of an application running on the application processor. A sequence including the first output mode and the second output mode is set according to the sequence including the first subroutine and the second subroutine, wherein the first output mode and the second output mode are arranged chronologically according to the sequence including the first output mode and the second output mode. The first image data is obtained according to the first output mode of the imaging unit, and The second image data is obtained according to the second output mode of the imaging unit.

12. The pattern sequencing method according to claim 11, wherein, According to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first output frame rate of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second output frame rate of the second image data is set in the second output mode.

13. The pattern sequencing method according to claim 11, wherein, According to at least one requirement of the first subroutine and a first limitation of the data bus interface, a first merging of the first image data is set in the first output mode, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, a second merging of the second image data is set in the second output mode.

14. The pattern sequencing method according to claim 11, further comprising: The illumination unit is configured with a first illumination mode during the first output mode of the imaging unit and a second illumination mode during the second output mode of the imaging unit, wherein a first illumination area of ​​the illumination unit corresponding to the first region of interest is configured in the first illumination mode, and a second illumination area of ​​the illumination unit corresponding to the second region of interest is configured in the second illumination mode.

15. The pattern sequencing method according to claim 14, wherein, According to at least one requirement of the first subroutine, a first modulation frequency of the lighting unit is set in the first lighting mode, and according to at least one requirement of the second subroutine, a second modulation frequency of the lighting unit is set in the second lighting mode.

16. The pattern sequencing method according to claim 11, further comprising: Calculate first component data based on the obtained first image data, and calculate second component data based on the obtained second image data; and According to at least one requirement of the first subroutine and a first limitation of the data bus interface, the obtained first image data or the calculated first component data is transmitted, and according to at least one requirement of the second subroutine and a second limitation of the data bus interface, the obtained second image data or the calculated second component data is transmitted.

17. The pattern sequencing method according to claim 11, wherein, The first region of interest and the second region of interest are set based on preset information.

18. The pattern sequencing method according to claim 11, wherein, The first region of interest and the second region of interest are set based on the tracking information obtained from the application processor.

19. The pattern sequencing method according to claim 16, further comprising: The first region of interest is determined based on the first image data obtained or the first component data calculated, and the second region of interest is determined based on the second image data obtained or the second component data calculated. and The first area of ​​interest is set according to the determined first area of ​​interest, and the second area of ​​interest is set according to the determined second area of ​​interest.

20. The pattern sequencing method according to claim 11, further comprising: Based on commands obtained from the application processor, a predetermined output mode is set until a wake-up interrupt occurs.

Images