Image and distance sensing system, image and distance sensing control device, and image and distance sensing method

Abstract

Sensing systems, methods, and devices are disclosed. In one example, a sensing system includes a first imaging sensor including an array of light receiving elements and a first light emitter including an array of light emitting elements. A control circuit is configured to control the first imaging sensor and the first light emitter such that an imaging range of a subset of the array of light receiving elements overlaps an illumination range of a subset of the array of light emitting elements.

Description

Technical Field

[0001] The present technology relates to a sensing system, a sensing control device, and a sensing method, and more particularly, to a sensing system, a sensing control device, and a sensing method applicable to a case of using an image sensor in a rolling shutter method.

[0002] Cross - reference to related applications

[0003] This application claims the benefit of Japanese Priority Patent Application JP 2022 - 178860, filed on November 8, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0004] In recent years, various sensors such as cameras and light detection and ranging or laser imaging detection and ranging (LiDAR) have been provided in vehicles to monitor the surrounding environment of the vehicle. As one of such sensors, for example, a device in which a rotary LiDAR and a camera in a rolling shutter method are combined has been proposed (for example, see Patent Document 1).

[0005] In addition, in order to improve the sensing ability in an environment where it is dark around, such as at night, an infrared camera including an image sensor having an imaging element capable of detecting infrared rays and a light - emitting unit capable of emitting illumination light including infrared (IR) light can be used.

[0006] [Citation List]

[0007] [Patent Document]

[0008] [Patent Document 1] JP 2021 - 105611A Summary of the Invention

[0009] [Technical Problem]

[0010] For example, in the case where the imaging element of the infrared camera is in a rolling shutter method, if the light - emitting unit emits light constantly, illumination light is irradiated even during a period other than the exposure period of each row of the imaging element (hereinafter, referred to as a non - exposure period). Therefore, the utilization efficiency of the light - emitting unit (illumination light) is reduced, and the power consumption increases.

[0011] On the other hand, when simply shortening the exposure period of the light - emitting unit in order to improve the utilization efficiency of the light - emitting unit, since the exposure periods of each row of the imaging element are shifted, the time during which the illumination light is irradiated within the exposure period of each row of the imaging element changes. Therefore, for example, noise such as horizontal stripes may appear in the captured image.

[0012] The present technology is proposed in view of this situation, and its purpose is to improve the utilization efficiency of the light-emitting unit in the case of using an image sensor with a rolling shutter method.

[0013] [Solutions to the problems]

[0014] The present invention discloses a sensing system, method, and device. In one example, the sensing system includes a first imaging sensor and a first light emitter. The first imaging sensor includes an array of light-receiving elements, and the first light emitter includes an array of light-emitting elements. A control circuit is configured to control the first imaging sensor and the first light emitter such that the imaging range of a subset of the array of light-receiving elements overlaps with the irradiation range of a subset of the array of light-emitting elements.

[0015] The sensing system according to the first aspect of the present technology includes: a first image sensor that controls exposure for each row; a first light-emitting unit whose irradiation range overlaps at least a part of the imaging range of the first image sensor and that can control its light-emitting timing for each row; and a control unit that comprehensively controls the first image sensor and the first light-emitting unit.

[0016] The sensing control device according to the second aspect of the present technology includes a control unit that comprehensively controls a first image sensor and a light-emitting unit. The first image sensor controls exposure for each row, and the irradiation range of the light-emitting unit overlaps at least a part of the imaging range of the image sensor and can control its light-emitting timing for each row.

[0017] The sensing method according to the second aspect of the present technology includes comprehensively controlling a first image sensor and a light-emitting unit. The first image sensor controls exposure for each row, and the irradiation range of the light-emitting unit overlaps at least a part of the imaging range of the image sensor and can control its light-emitting timing for each row.

[0018] In the first aspect of the present technology, the first image sensor and the first light-emitting unit are comprehensively controlled. The first image sensor controls exposure for each row, the irradiation range of the first light-emitting unit overlaps at least a part of the imaging range of the first image sensor, and it can control its light-emitting timing for each row.

[0019] In the second aspect of the present technology, the first image sensor and the light-emitting unit are comprehensively controlled. The first image sensor controls exposure for each row, the irradiation range of the light-emitting unit overlaps at least a part of the imaging range of the image sensor, and it can control its light-emitting timing for each row.

[0020] According to another aspect of the present technology, a sensing system includes: a first imaging sensor including an array of light receiving elements; a first light emitter including an array of light emitting elements; and a control circuit configured to control the first imaging sensor and the first light emitter such that an imaging range of a subset of the array of light receiving elements overlaps an illumination range of a subset of the array of light emitting elements.

[0021] The subset of the array of light receiving elements may be multiple rows in the array of light receiving elements, and the subset of the array of light emitting elements may also be the same number of rows in the array of light emitting elements. In one example, the number of rows is 1.

[0022] In addition, the control circuit may control an exposure timing of the first imaging sensor based on a light emission timing of the first light emitter, or may control the light emission timing of the first light emitter based on the exposure timing of the first imaging sensor, or may control a combination of both.

[0023] In one example, the control circuit controls the light emission timing of each row of the array of light emitting elements to be related to the exposure timing of each row of the array of light receiving elements. The control circuit may cause each row of the array of light emitting elements to emit light for a predetermined period during the exposure of each row of the array of light receiving elements.

[0024] In another example of this aspect, the sensing system further includes a light detection sensor including an array of light receiving elements. The control circuit is configured to control the first light emitter to emit light in a first mode for the light detection sensor during a non-exposure period, which is a period other than the exposure period of the first imaging sensor.

[0025] Here, the control circuit may control the light emission timing of each row of the array of light emitting elements according to the non-exposure period of each row of the array of light receiving elements in the first imaging sensor.

[0026] These aspects may also be embodied in a sensing control device, a sensing method, and a computer-readable medium storing program code for performing corresponding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram showing a configuration example of a vehicle control system.

[0028] Figure 2 is a diagram showing an example of a sensing area.

[0029] Figure 3 shows an example of the relationship between the exposure timing and the light emission timing during one frame period of an imaging element in a global shutter mode.

[0030] Figure 4 A and Figure 4 B of the rolling shutter type imaging element shows an example of the relationship between the exposure timing and the light emission timing during one frame.

[0031] Figure 5 Another example of the relationship between the exposure timing and the light emission timing during one frame of the rolling shutter type imaging element is shown.

[0032] Figure 6 is a block diagram of a first embodiment of a sensing system to which the present technology is applicable.

[0033] Figure 7 An example of the configuration of a VCSEL is shown.

[0034] Figure 8 is a diagram showing a scanning method of a light emitting unit.

[0035] Figure 9 shows Figure 6 an example of the operation of the sensing system in

[0036] Figure 10 is a diagram showing an example of the installation positions of a camera and a LiDAR.

[0037] Figure 11 is a block diagram of a second embodiment of a sensing system to which the present technology is applicable.

[0038] Figure 12 is a diagram showing an example of the hardware configuration of a sensing unit.

[0039] Figure 13 shows Figure 11 an example of the installation position of the sensing system in

[0040] Figure 14 shows Figure 11 an example of the operation of the sensing system in

[0041] Figure 15 is a block diagram of a third embodiment of a sensing system to which the present technology is applicable.

[0042] Figure 16 shows Figure 15 an example of the imaging range of the image sensor of the sensing system in

[0043] Figure 17 shows Figure 15 an example of the operation of the sensing system in Detailed Description

[0044] In the following, the modes for implementing the present technology will be described. Note that the description will be given in the following order.

[0045] 1. Configuration example of vehicle control system

[0046] 2. Background of the present technology

[0047] 3. First embodiment

[0048] 4. Second embodiment

[0049] 5. Third embodiment

[0050] 6. Modification example

[0051] 7. Others

[0052] <<1. Configuration example of vehicle control system>>

[0053] Figure 1 FIG. is a block diagram showing a configuration example of a vehicle control system 11 which is an example of a mobile device control system to which the present technology is applied.

[0054] The vehicle control system 11 is provided in the vehicle 1 and performs processes related to driving assistance and autonomous driving of the vehicle 1.

[0055] The vehicle control system 11 includes a vehicle control electronic control unit (ECU) 21, a communication unit 22, a map information accumulation unit 23, a position information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving assistance / autonomous driving control unit 29, a DMS (driver monitoring system) 30, an HMI (human machine interface) 31, and a vehicle control unit 32.

[0056] The vehicle control ECU 21, communication unit 22, map information accumulation unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, storage unit 28, driving assistance / autopilot control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are communicably connected to each other via a communication network 41. The communication network 41 includes an in-vehicle communication network or bus that conforms to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 41 can be selectively used according to the type of data to be transmitted. For example, CAN can be applied to data related to vehicle control, while Ethernet can be applied to large-capacity data. Note that the respective units of the vehicle control system 11 can be directly connected without going through the communication network 41 but by wireless communication (such as near-field communication (NFC) or Bluetooth (registered trademark)) assuming relatively short-distance communication, for example.

[0057] Note that hereinafter, when the respective units of the vehicle control system 11 perform communication via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 perform communication via the communication network 41, it is simply described as the vehicle control ECU 21 and the communication unit 22 performing communication.

[0058] The vehicle control ECU 21 includes various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls all or part of the functions of the vehicle control system 11.

[0059] The communication unit 22 communicates with various settings inside and outside the vehicle, other vehicles, servers, base stations, etc., and sends / receives various data. At this time, the communication unit 22 can perform communication using a variety of communication schemes.

[0060] The communication with the outside that can be performed by the communication unit 22 will be schematically described. The communication unit 22 communicates with a server (hereinafter referred to as an external server) existing on an external network via a base station or access point by wireless communication methods such as 5G (Fifth Generation Mobile Communication System), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communication). The external network that communicates with the communication unit 22 is, for example, the Internet, a cloud network, or a network exclusive to an operator. The communication scheme for the communication unit 22 to communicate with the external network is not particularly limited as long as it is a wireless communication scheme capable of performing digital two-way communication at a communication speed equal to or higher than a predetermined speed and a distance equal to or longer than a predetermined distance.

[0061] In addition, for example, the communication unit 22 can communicate with terminals existing near the host vehicle using P2P (peer-to-peer) technology. For example, the terminals existing near the host vehicle are terminals worn by moving bodies moving at a relatively low speed (such as pedestrians and bicycles), terminals installed at fixed positions such as in stores, or MTC (machine type communication) terminals. In addition, the communication unit 22 can perform V2X communication. V2X communication refers to communication between the host vehicle and other objects, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside equipment, etc., vehicle-to-home communication with a home, and vehicle-to-pedestrian communication with terminals owned by pedestrians.

[0062] For example, the communication unit 22 can receive from the outside a program (Over The Air) for updating software that controls the operation of the vehicle control system 11. The communication unit 22 can also receive map information, traffic information, information around the vehicle 1, etc. from the outside. In addition, for example, the communication unit 22 can send information about the vehicle 1 or information around the vehicle 1 to the outside. Examples of the information about the vehicle 1 sent by the communication unit 22 to the outside include data indicating the state of the vehicle 1, the identification result of the identification unit 73, etc. In addition, for example, the communication unit 22 performs communication corresponding to a vehicle emergency call system such as ecall.

[0063] For example, the communication unit 22 receives electromagnetic waves sent by a road traffic information communication system (Vehicle Information and Communication System (VICS) (registered trademark)), such as a radio wave beacon, an optical beacon, or an FM multiplex broadcast.

[0064] Communication with devices inside the vehicle that can be performed by the communication unit 22 will be schematically described. The communication unit 22 can communicate with various devices inside the vehicle using, for example, wireless communication. Through wireless communication such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB), through a communication scheme capable of performing digital two-way communication at a predetermined communication speed or higher, the communication unit 22 can perform wireless communication with devices inside the vehicle. This technology is not limited to this, and the communication unit 22 can also communicate with various devices inside the vehicle using wired communication. For example, the communication unit 22 can communicate with various devices inside the vehicle through a cable connected to a connection terminal (not shown) using wired communication. Through wired communication such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), and MHL (Mobile High-Definition Link), through a communication scheme capable of performing digital two-way communication at a predetermined communication speed or higher, the communication unit 22 can communicate with various devices inside the vehicle.

[0065] Here, the in-vehicle devices refer to, for example, devices not connected to the communication network 41 in the vehicle. As in-vehicle devices, for example, mobile devices or wearable devices carried by passengers such as the driver, or information devices brought into the vehicle and temporarily installed are assumed.

[0066] The map information accumulation unit 23 accumulates one or both of the map acquired from the outside and the map created by the vehicle 1. For example, the map information accumulation unit 23 accumulates a three-dimensional high-precision map, a global map with a lower precision than the high-precision map and covering a wide area, and the like.

[0067] The high-precision map is, for example, a dynamic map, a point cloud map, or a vector map. The dynamic map is, for example, a map including four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is supplied to the vehicle 1 from an external server or the like. The point cloud map is a map including point clouds (point cloud data). The vector map is, for example, a map in which traffic information such as lane and traffic signal positions is associated with the point cloud map and is suitable for ADAS (Advanced Driver Assistance System) or AD (Autonomous Driving).

[0068] The point cloud map and the vector map can be provided from, for example, an external server or the like, or can be created by the vehicle 1 based on the sensing results of the camera 51, the radar 52, the LiDAR 53, etc. as maps for matching with the local map described later, and accumulated in the map information accumulation unit 23. Further, in the case where the high-precision map is provided from an external server or the like, for example, in order to reduce the communication capacity, map data of several hundred square meters related to the planned route on which the vehicle 1 will travel from now on is acquired from the external server or the like.

[0069] The position information acquisition unit 24 receives GNSS signals from GNSS (Global Navigation Satellite System) satellites and acquires the position information of the vehicle 1. The acquired position information is supplied to the driving assistance / autonomous driving control unit 29. Note that the position information acquisition unit 24 is not limited to the method using GNSS signals, and can use, for example, beacons to acquire position information.

[0070] The external recognition sensor 25 includes various sensors for recognizing the external conditions of the vehicle 1, and supplies the sensor data from each sensor to each part of the vehicle control system 11. Any type and number of sensors included in the external recognition sensor 25 can be adopted.

[0071] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The present technology is not limited thereto, and the external recognition sensor 25 may include one or more types of sensors among the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 is not particularly limited as long as they can be actually installed in the vehicle 1. In addition, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of the respective sensors included in the external recognition sensor 25 will be described later.

[0072] Note that the imaging method of the camera 51 is not particularly limited. For example, cameras of various imaging methods such as ToF (Time of Flight) cameras, stereo cameras, monocular cameras, and infrared cameras that can perform distance measurement can be applied to the camera 51 as needed. The present technology is not limited thereto, and the camera 51 may simply acquire the captured image regardless of distance measurement.

[0073] In addition, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment of the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a rain sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0074] In addition, for example, the external recognition sensor 25 includes a microphone for detecting, for example, the sound around the vehicle 1, the position of the sound source, and the like.

[0075] The in-vehicle sensor 26 includes various sensors for detecting in-vehicle information, and supplies sensor data from each sensor to the respective units of the vehicle control system 11. The types and numbers of the various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they are types and numbers that can be actually installed in the vehicle 1.

[0076] For example, the in-vehicle sensor 26 may include one or more sensors such as a camera, radar, seat sensor, steering wheel sensor, microphone, and biometric sensor. As the camera included in the in-vehicle sensor 26, for example, cameras of various imaging methods capable of measuring distance can be used, such as ToF cameras, stereo cameras, monocular cameras, and infrared cameras. This technology is not limited thereto, and the camera included in the in-vehicle sensor 26 can simply acquire the captured image regardless of distance measurement. The biometric sensor included in the in-vehicle sensor 26 is provided on, for example, a seat or a steering wheel, and detects various types of biometric information of passengers such as the driver.

[0077] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to respective parts of the vehicle control system 11. There are no particular limitations on the types and quantities of the various sensors included in the vehicle sensor 27, as long as they are types and quantities that can be actually installed in the vehicle 1.

[0078] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the operation amount of the accelerator pedal, and a brake sensor that detects the operation amount of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the rotation speed of the engine or motor, a tire pressure sensor that detects the tire pressure, a slip rate sensor that detects the slip rate of the tire, and a wheel speed sensor that detects the rotational speed of the wheel. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining amount and temperature of the battery, and a shock sensor that detects an impact from the outside.

[0079] The storage unit 28 includes at least one of a non-volatile storage medium or a volatile storage medium, and stores data and programs. The storage unit 28 is used as, for example, an EEPROM (electrically erasable programmable read-only memory) and a RAM (random access memory), and magnetic storage devices such as a hard disk drive (HDD), semiconductor storage devices, optical storage devices, and magneto-optical storage devices can be used as the storage medium. The storage unit 28 stores various programs and data used by respective units of the vehicle control system 11. For example, the storage unit 28 includes an EDR (event data recorder) and a DSSAD (data storage system for autonomous driving), and stores information about the vehicle 1 before and after an event such as an accident and information acquired by the in-vehicle sensor 26.

[0080] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, a motion planning unit 62, and an operation control unit 63.

[0081] The analysis unit 61 performs a process of analyzing the vehicle 1 and the surrounding conditions. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and an identification unit 73.

[0082] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map accumulated in the map information accumulation unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and matches the local map with the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 is based on, for example, the center of the rear axle.

[0083] The local map is, for example, a three-dimensional high-precision map created using techniques such as SLAM (Simultaneous Localization and Mapping), an occupancy grid map, etc. The three-dimensional high-precision map is, for example, the above-mentioned point cloud map. The occupancy grid map is a map obtained by dividing the three-dimensional or two-dimensional space around the vehicle 1 into grids (cells) of a predetermined size, and shows the occupancy state of objects in units of grids. The occupancy state of an object is indicated, for example, by the presence or absence of the object or the probability of existence. The local map is also used for the detection process and identification process of the external conditions of the vehicle 1 by the identification unit 73.

[0084] Note that the self-position estimation unit 71 can estimate the self-position of the vehicle 1 based on the position information obtained by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0085] The sensor fusion unit 72 performs sensor fusion processing for obtaining new information by combining multiple different types of sensor data (for example, the image data supplied from the camera 51 and the sensor data supplied from the radar 52). The methods for combining different types of sensor data include integration, fusion, and association.

[0086] The identification unit 73 performs a detection process for detecting the external conditions of the vehicle 1 and an identification process for identifying the external conditions of the vehicle 1.

[0087] For example, the identification unit 73 performs a detection process and an identification process of the external conditions of the vehicle 1 based on the information from the external recognition sensor 25, the information from the self-position estimation unit 71, the information from the sensor fusion unit 72, etc.

[0088] Specifically, for example, the recognition unit 73 performs detection processing, recognition processing, etc. on objects around the vehicle 1. The detection processing of an object is, for example, processing for detecting the presence or absence, size, shape, position, and movement of the object. The recognition processing of an object is, for example, processing for recognizing the attributes (e.g., type) of the object or recognizing a specific object. However, the detection processing and the recognition processing are not always clearly separated and may overlap.

[0089] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering, which is used to classify point clouds based on sensor data acquired by the radar 52, LiDAR 53, etc. into point cloud clusters. Thus, it is detected whether there are objects around the vehicle 1, as well as their size, shape, and position.

[0090] For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking to follow the movement of the point cloud clusters classified by clustering. Thus, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0091] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, buildings, roads, traffic lights, traffic signs, road signs, etc. based on the image data supplied from the camera 51. In addition, the recognition unit 73 can recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0092] For example, the recognition unit 73 can perform processing for recognizing traffic rules around the vehicle 1 based on the map accumulated in the map information accumulation unit 23, the estimation result of its own position by the own position estimation unit 71, and the recognition result of the objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the position and state of traffic lights, the content of traffic signs and road signs, the content of traffic rules, the lanes that can be traveled, etc.

[0093] For example, the recognition unit 73 can perform recognition processing on the surrounding environment of the vehicle 1. As the surrounding environment to be recognized by the recognition unit 73, there are assumed to be weather, temperature, humidity, brightness, road surface conditions, etc.

[0094] The action planning unit 62 creates an action plan for the vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route tracking processing.

[0095] Note that route planning (global route planning) is processing for planning a rough route from the starting point to the destination. This route planning is called trajectory planning and includes performing processing for local route planning that takes into account the motion characteristics of the vehicle 1 in the planned route to be able to drive safely and smoothly near the vehicle 1.

[0096] Route tracking is a process of planning an operation for driving safely and accurately on a route planned by route planning within a planned time. The motion planning unit 62 can calculate, for example, the target speed and target angular velocity of the vehicle 1 based on the result of the route tracking process.

[0097] The operation control unit 63 controls the operation of the vehicle 1 to achieve the motion plan created by the motion planning unit 62.

[0098] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 included in the vehicle control unit 32 described later, and performs acceleration and deceleration control and direction control so that the vehicle 1 travels through the trajectory calculated by the trajectory planning. For example, the operation control unit 63 performs coordinated control aimed at realizing the functions of ADAS, such as collision avoidance, impact mitigation, following driving, vehicle speed maintenance driving, collision warning of the host vehicle, and lane departure warning of the host vehicle. For example, the operation control unit 63 performs coordinated control aimed at autonomous driving or the like, in which the vehicle travels autonomously without depending on the driver's operation.

[0099] The DMS 30 performs driver authentication processing, driver state identification processing, etc. based on sensor data from the in-vehicle sensor 26, input data input to the HMI 31 described later, etc. As the state of the driver to be identified, for example, there are physical condition, wakefulness, concentration, fatigue level, line of sight direction, intoxication level, driving operation, posture, etc.

[0100] Note that the DMS 30 can perform authentication processing for passengers other than the driver and perform identification processing on the state of the passengers. In addition, for example, the DMS 30 can perform identification processing on the condition inside the vehicle based on sensor data from the in-vehicle sensor 26. As the condition inside the vehicle to be identified, for example, there are temperature, humidity, brightness, smell, etc.

[0101] The HMI 31 inputs various data, instructions, etc. and presents various data to the driver, etc.

[0102] The data input of the HMI 31 will be schematically described. The HMI 31 includes an input device for a person to input data. The HMI 31 generates an input signal based on the data, instructions, etc. input by the input device, and supplies the input signal to each unit of the vehicle control system 11. The HMI 31 includes operators such as a touch panel, buttons, switches, and a control lever as the input device. This technology is not limited to this, and the HMI 31 may also include an input device capable of inputting information by methods other than manual operation (for example, voice, gesture, etc.). In addition, as the input device, the HMI 31 may use, for example, a remote control device using infrared rays or radio waves or an external connection device corresponding to the operation of the vehicle control system 11 (for example, a movable device or a wearable device).

[0103] The data presentation of the HMI 31 will be schematically described. The HMI 31 generates visual information, auditory information, and tactile information for passengers or outside the vehicle. In addition, the HMI 31 performs output control of the output, output content, output timing, output method, etc. of these generated information. As the visual information, the HMI 31 generates and outputs information indicated by images or light such as an operation screen, a state display of the vehicle 1, a warning display, and a monitoring image indicating the situation around the vehicle 1. In addition, as the auditory information, the HMI 31 generates and outputs information indicated by sounds such as voice navigation, warning sounds, and warning messages. In addition, as the tactile information, the HMI 31 generates and outputs tactile information that gives a tactile sensation to the passenger through, for example, force, vibration, or movement.

[0104] As an output device for the HMI 31 to output visual information, for example, a display device that presents visual information by displaying an image by itself or a projection device that presents visual information by projecting an image can be applied. Note that in addition to a display device with a normal display, the display device may also be a device that displays visual information within the field of view of the passenger, such as a head-up display, a transmissive display, or a wearable device with an AR (augmented reality) function. In addition, in the HMI 31, the display devices included in the navigation device, instrument panel, camera monitoring system (CMS), electronic mirror, lamp, etc. provided in the vehicle 1 can also be used as output devices for outputting visual information.

[0105] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, a headset, or earphones can be applied.

[0106] As an output device for the HMI 31 to output tactile information, for example, a tactile element using tactile technology can be applied. The tactile element is provided at a portion where a passenger of the vehicle 1 touches, for example, provided at the steering wheel or the seat.

[0107] The vehicle control unit 32 controls each unit of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a braking control unit 82, a driving control unit 83, a body system control unit 84, a lighting control unit 85, and a horn control unit 86.

[0108] The steering control unit 81 performs detection, control, etc. of the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism having a steering wheel, etc., an electric power steering device, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0109] The braking control unit 82 performs detection, control, etc. of the state of the braking system of the vehicle 1. The braking system includes, for example, a braking mechanism having a brake pedal, etc., an ABS (antilock braking system), a regenerative braking mechanism, etc. The braking control unit 82 includes, for example, a braking ECU that controls the braking system, an actuator that drives the braking system, etc.

[0110] The driving control unit 83 performs detection, control, etc. of the state of the driving system of the vehicle 1. The driving system includes, for example, a driving force generation device for generating a driving force (e.g., an accelerator pedal, an internal combustion engine, a driving motor, etc.), a driving force transmission mechanism for transmitting the driving force to the wheels, etc. The driving control unit 83 includes, for example, a driving ECU that controls the driving system, an actuator that drives the driving system, etc.

[0111] The body system control unit 84 performs detection, control, etc. of the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, an electric window device, an electric seat, an air conditioner, a safety airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system, an actuator that drives the body system, etc.

[0112] The lighting control unit 85 performs detection, control, etc. of the state of various lights of the vehicle 1. As the lights to be controlled, for example, there are headlights, backlights, fog lights, turn signal lights, brake lights, projections, bumper displays, etc. The lighting control unit 85 includes a lighting ECU that controls the lights, an actuator that drives the lights, etc.

[0113] The horn control unit 86 performs detection, control, etc. of the state of the vehicle horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the vehicle horn, an actuator that drives the vehicle horn, etc.

[0114] Figure 2 shows Figure 1 an example of the sensing areas of the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25 in Figure 2Vehicle 1 is schematically shown as viewed from above, where the left end side is the front (front) side of vehicle 1, and the right end side is the rear (rear) side of vehicle 1.

[0115] Sensing areas 101F and 101B show examples of the sensing areas of ultrasonic sensors 54. The sensing area 101F covers the periphery of the front end of vehicle 1 through a plurality of ultrasonic sensors 54. The sensing area 101B covers the periphery of the rear end of vehicle 1 through a plurality of ultrasonic sensors 54.

[0116] For example, the sensing results in the sensing areas 101F and 101B are used for parking assistance of vehicle 1 and the like.

[0117] Sensing areas 102F to 102B show examples of the short-range or medium-range sensing areas of radar 52. The sensing area 102F covers a position farther than the sensing area 101F in front of vehicle 1. The sensing area 102B covers a position farther than the sensing area 101B behind vehicle 1. The sensing area 102L covers the rear periphery of the left side of vehicle 1. The sensing area 102R covers the rear periphery of the right side of vehicle 1.

[0118] For example, the sensing result in the sensing area 102F is used to detect vehicles, pedestrians, etc. present in front of vehicle 1. For example, the sensing result in the sensing area 102B is used for a collision prevention function behind vehicle 1 and the like. For example, the sensing results in the sensing areas 102L and 102R are used to detect objects in the blind spots on the sides of vehicle 1.

[0119] Sensing areas 103F to 103B show examples of the sensing areas of camera 51. The sensing area 103F covers a position farther than the sensing area 102F in front of vehicle 1. The sensing area 103B covers a position farther than the sensing area 102B behind vehicle 1. The sensing area 103L covers the periphery of the left side of vehicle 1. The sensing area 103R covers the periphery of the right side of vehicle 1.

[0120] The sensing result in the sensing area 103F can be used, for example, for the recognition of traffic lights or traffic signs, a lane departure prevention assistance system, and an automatic headlight control system. For example, the sensing result in the sensing area 103B can be used for parking assistance and a surround view system. For example, the sensing results in the sensing areas 103L and 103R can be used for a surround view system.

[0121] Sensing area 104 shows an example of the sensing area of LiDAR 53. The sensing area 104 covers a position farther than the sensing area 103F in front of vehicle 1. At the same time, the sensing area 104 has a narrower range in the left-right direction than the sensing area 103F.

[0122] For example, the sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles.

[0123] The sensing area 105 shows an example of the sensing area of the long-range radar 52. The sensing area 105 covers a position farther than the sensing area 104 in front of the vehicle 1. At the same time, the sensing area 105 has a narrower range in the left-right direction than the sensing area 104.

[0124] The sensing results in the sensing area 105 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0125] Note that the sensing areas of the respective sensors including the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 included in the external recognition sensor 25 can have Figure 2 various configurations other than the configurations in. Specifically, the ultrasonic sensor 54 can also sense the side of the vehicle 1, or the LiDAR 53 can sense the rear of the vehicle 1. In addition, the mounting position of each sensor is not limited to each of the above examples. In addition, the number of sensors can be one or more.

[0126] <<2. Background of the Present Technology>>

[0127] Next, the background of the present technology will be described with reference to Figures 3 to 5 explanation.

[0128] In the vehicle 1, for example, according to the imaging of the camera 51 or the camera of the in-vehicle sensor 26, the imaging range is irradiated with illumination light including infrared (IR) light, etc.

[0129] Figure 3 Shows an example of the relationship between the exposure timing of the imaging element and the light emission timing of the illumination light during one frame in the case where the camera includes an imaging element of a global shutter method that simultaneously performs exposure (scanning) of all pixels. In the figure, the horizontal axis represents time and the vertical axis represents the rows of the imaging element.

[0130] For example, during the period from time t1a to time t4a, the exposure of all pixels of the imaging element is performed simultaneously.

[0131] On the other hand, for example, during the period from time t2a to time t3a during the exposure period, the imaging range of the camera is irradiated with illumination light. As a result, the imaging range of all pixels of the imaging element is irradiated substantially uniformly with the illumination light.

[0132] In this example, since the irradiation time of the illumination light is shortened, the power consumption of the illumination light is reduced. In addition, since the ratio of the irradiation period of the illumination light during the exposure period of each pixel increases, the effect of the illumination light is fully exerted even in a scene irradiated with sunlight. In addition, rolling shutter distortion does not occur.

[0133] On the other hand, since the imaging element of the global shutter is expensive, the cost of the camera increases.

[0134] Therefore, an imaging element that controls the rolling shutter method for exposure of each row is usually used, although rolling shutter distortion occurs.

[0135] Figure 4 of A and Figure 4 B of shows an example of the relationship between the exposure timing of the imaging element and the light emission timing of the illumination light during one frame period in the case where the camera includes an imaging element of the rolling shutter method. In the figure, the horizontal axis represents time, and the vertical axis represents the rows of the imaging element.

[0136] In Figure 4 the example of A, for example, the exposure of the head row of the imaging element is performed during the period from time t1b to time t2b. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element. Then, the exposure of the final row of the imaging element is performed during the period from time t5b to time t6b.

[0137] On the other hand, for example, during the period from time t3b to time t4b, the imaging range of the camera is irradiated with the illumination light.

[0138] In this case, the illumination light is irradiated only during the exposure period of some pixels. Therefore, horizontal stripes due to the illumination light are generated in the captured image.

[0139] On the other hand, for example, as Figure 4 shown in B, the frame rate of the imaging element is shortened, and the exposure period of each pixel is extended. Specifically, for example, the exposure of the head row of the imaging element is performed during the period from time t1c to time t3c. This exposure period is set to be longer than Figure 4 the exposure period in A. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element. Then, the exposure of the final row of the imaging element is performed during the period from time t2c to time t4c. Note that the exposure of the final row starts at time t2c before time t3c when the exposure of the head row ends.

[0140] On the other hand, for example, during the period from time t2c to time t3c, the imaging range of the camera is irradiated with illumination light. That is, the illumination light is irradiated during the exposure of all pixels of the imaging element. As a result, the imaging range of all pixels of the imaging element is irradiated with the illumination light substantially uniformly.

[0141] In this example, since the irradiation time of the illumination light is shortened, the power consumption of the illumination light is reduced.

[0142] On the other hand, the effect of the illumination light is reduced. That is, since the ratio of the irradiation period of the illumination light during the exposure period of each pixel becomes low, for example, in a scene irradiated with sunlight, the effect of the illumination light is hardly exerted.

[0143] Figure 5 Another example of the relationship between the exposure timing of the imaging element and the light emission timing of the illumination light during one frame period in the case where the camera includes an imaging element of the rolling shutter method is shown. In the figure, the horizontal axis represents time and the vertical axis represents the rows of the imaging element.

[0144] For example, during the period from time t1d to time t2d, the exposure of the head row of the imaging element is performed. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element. Then, during the period from time t3d to time t4d, the exposure of the final row of the imaging element is performed.

[0145] On the other hand, for example, the imaging range of the camera is continuously irradiated with illumination light. As a result, the imaging range of all pixels of the imaging element is irradiated with the illumination light substantially uniformly.

[0146] In this example, the effect of the illumination light is improved. That is, since the ratio of the irradiation period of the illumination light during the exposure period of each pixel increases, for example, even in a scene irradiated with sunlight, the effect of the illumination light is fully exerted.

[0147] On the other hand, since the irradiation time of the illumination light becomes long, the power consumption of the illumination light becomes large.

[0148] On the other hand, the present technology improves the utilization efficiency of the light emitting unit in the case of using an image sensor of the rolling shutter method. For example, the present technology improves the effect of the light emitting unit while suppressing the power consumption of the light emitting unit.

[0149] <<3. First Embodiment>>

[0150] Next, with reference to Figures 6 to 9 the first embodiment of the present technology will be described.

[0151] <Configuration Example of Sensing System 201>

[0152] Figure 6 Shows a configuration example of a sensing system 201 to which the present technology is applicable.

[0153] The sensing system 201 can be applied to the vehicle 1. The sensing system 201 includes a sensing unit 211 and a light emitting unit 212.

[0154] The sensing unit 211 includes an image sensor 221 and a control unit 222.

[0155] The image sensor 221 includes an imaging element 221A of, for example, a rolling shutter method, in which pixels including light receiving elements such as photodiodes (PDs) are two-dimensionally arranged.

[0156] The control unit 222 comprehensively controls the image sensor 221 and the light emitting unit 212. For example, the control unit 222 comprehensively controls the imaging timing of the image sensor 221 and the light emitting timing of the light emitting unit 212.

[0157] In the light emitting unit 212, for example, light emitting elements (light sources) are two-dimensionally arranged in the row direction (horizontal direction) and the column direction (vertical direction), and light emission can be controlled for each light emitting element. For example, as Figure 7 shown, the light emitting unit 212 includes vertical cavity surface emitting lasers (VCSELs).

[0158] Specifically, the light emitting unit 212 includes a substrate 241 and a plurality of light emitting elements 242. The light emitting elements 242 are two-dimensionally arranged on the substrate 241. Each light emitting element 242 emits laser light (hereinafter referred to as illumination light) as IR light in a direction perpendicular to the substrate 241.

[0159] Therefore, for example, as Figure 8 shown, the light emitting unit 212 can scan the illumination light extending in the row direction (horizontal direction) in the vertical direction.

[0160] The light emitting unit 212 irradiates illumination light in the imaging direction of the image sensor 221. Therefore, the imaging range of the image sensor 221 and the irradiation range of the light emitting unit 212 at least partially overlap. Note that it is desirable that the irradiation range of the light emitting unit 212 is substantially the same as or includes the imaging range of the image sensor 221.

[0161] <Operation example of the sensing system 201>

[0162] Next, an operation example of the sensing system 201 will be described with reference to the timing chart of Figure 9 . In Figure 9 , the horizontal axis represents time, and the vertical axis represents the rows of the imaging element 221A and the light emitting unit 212.

[0163] For example, during the period from time t1e to time t4e, the exposure of the head row of the imaging element 221A is performed. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element 221A. Then, during the period from time t5e to time t8e, the exposure of the final row of the imaging element 221A is performed.

[0164] On the other hand, the light emission timing and the irradiation range of the light emitting unit 212 are controlled according to the exposure timing of each row of the imaging element 221A. That is, the light emission timing of each row of the light emitting unit 212 is controlled according to the exposure timing of each row of the imaging element 221A.

[0165] For example, during the period from time t2e to time t3e, the imaging range of the head row of the imaging element 221A is irradiated with illumination light. That is, during a predetermined length period within the exposure period of the head row of the imaging element 221A, the imaging range of the head row of the imaging element 221A is irradiated with illumination light.

[0166] Thereafter, while the time gradually moves, the illumination light is sequentially applied to the imaging ranges of each row of the imaging element 221A from the imaging range of the second row to the imaging range of the final row of the imaging element 221A. Then, during the period from time t6e to time t7e, the imaging range of the final row of the imaging element 221A is irradiated with illumination light.

[0167] In this way, according to the exposure period of each row of the imaging element 221A, the imaging range of each row is irradiated with illumination light.

[0168] As a result, in the case of the image sensor 221 using the rolling shutter method, the utilization efficiency of the light emitting unit 212 can be improved. For example, although the light emission time of each row of the light emitting unit 212 is shortened, during the exposure period of each row of the imaging element 221A, the time for irradiating the imaging range of each row with illumination light can be extended. As a result, the light amount of the illumination light for each row of the imaging element 221A is sufficiently ensured, and the power consumption of the light emitting unit 212 is reduced.

[0169] Note that, for example, time t1e and time t2e may coincide, and time t5e and time t6e may coincide. That is, the timing at which each row of the imaging element 221A starts exposure can be synchronized with the timing at which the illumination light irradiation of the imaging range of each row of the imaging element 221A starts.

[0170] In addition, for example, time t3e and time t4e may coincide, and time t7e and time t8e may coincide. That is, the timing at which each row of the imaging element 221A ends exposure can be synchronized with the timing at which the illumination light irradiation of the imaging range of each row of the imaging element 221A ends.

[0171] Note that the number of rows of the imaging element 221A generally does not match the number of rows of the light-emitting units 212. Therefore, the imaging ranges of the respective rows of the imaging element 221A generally do not match the irradiation ranges of the respective rows of the light-emitting units 212. In addition, the number of rows of the imaging element 221A is generally greater than the number of rows of the light-emitting units 212. Therefore, for example, it is assumed that the irradiation ranges of the respective rows of the light-emitting units 212 include the imaging ranges of multiple rows of the imaging element 221A.

[0172] In this case, for example, during the exposure periods of the respective rows of the imaging element 221A, the rows of the light-emitting units 212 corresponding to the respective rows of the imaging element 221A are controlled to emit light for a predetermined time length. Here, the rows of the light-emitting units 212 corresponding to the respective rows of the imaging element 221A are, for example, the rows whose irradiation ranges include the imaging ranges of the respective rows of the imaging element 221A. For example, in the case where the irradiation range of the head row of the light-emitting units 212 includes the imaging ranges of rows 1 to 10 of the imaging element 221A, the rows of the light-emitting units 212 corresponding to rows 1 to 10 of the imaging element 221A are the head row of the light-emitting units 12.

[0173] Specifically, for example, the irradiation timings of the respective rows of the light-emitting units 212 are controlled such that the irradiation periods of the illumination light become substantially the same with respect to the imaging ranges of the respective rows of the imaging element 221A. For example, the irradiation timings of the respective rows of the light-emitting units 212 are controlled such that the irradiation periods of the respective rows of the light-emitting units 212 are included in the exposure periods of multiple rows of the imaging element 221A corresponding to the respective rows. For example, in the case where the head row of the light-emitting units 212 corresponds to rows 1 to 10 of the imaging element 221A, the irradiation timing of the head row of the light-emitting units 212 is controlled such that the irradiation period of the head row of the light-emitting units 212 is included in the exposure periods of rows 1 to 10 of the imaging element 221A.

[0174] Note that, for example, the irradiation period of the light-emitting units 212 may include the exposure periods of the respective rows of the imaging element 221A. For example, in Figure 9 the timings at which the respective rows of the imaging element 221A start exposure and the timings at which the respective rows of the light-emitting units 212 start emitting light may be interchanged. In addition, for example, the timings at which the respective rows of the imaging element 221A end exposure and the timings at which the respective rows of the light-emitting units 212 end emitting light may be interchanged.

[0175] Specifically, for example, the exposure of the head row of the imaging element 221A is performed during the period from time t2e to time t3e. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element 221A. Then, during the period from time t6e to time t7e, the exposure of the final row of the imaging element 221A is performed.

[0176] On the other hand, for example, during the period from time t1e to time t4e, the imaging range of the head row of the imaging element 221A is irradiated with illumination light. Thereafter, while the time gradually moves, the illumination light is sequentially applied to the imaging ranges of the respective rows of the imaging element 221A from the imaging range of the second row to the final row of the imaging element 221A. Then, during the period from time t5e to time t8e, the imaging range of the final row of the imaging element 221A is irradiated with illumination light.

[0177] <<4. Second Embodiment>>

[0178] Next, the second embodiment of the present technology will be described with reference to Figures 10 to 14 This will be described.

[0179] Figure 10 A diagram showing an example of the installation positions of the camera 51L and the LiDAR 53L of the vehicle 1.

[0180] The camera 51L is installed near the door mirror on the left side of the vehicle 1. The camera 51L includes, for example, an LED that emits illumination light including IR light, and captures an image while emitting the illumination light. For example, when the driver parks the vehicle 1, the image of the camera 51L is used to check the left side.

[0181] The LiDAR 53L is installed in front of the left side surface of the vehicle 1. The LiDAR 53L is used to monitor the left front and the left side of the vehicle 1. The LiDAR 53L emits measurement light, which is a laser including IR light, and receives the reflected light of the measurement light, thereby sensing the left front and the left side of the vehicle 1. For example, the LiDAR 53L detects the shape, distance, position, etc. of the objects on the left front and the left side of the vehicle 1.

[0182] In this case, when the illumination light from the camera 51L and the measurement light from the LiDAR 53L are emitted simultaneously, they interfere with each other. Therefore, it is necessary to control the camera 51L and the LiDAR 53L so that they do not operate simultaneously. For example, when the vehicle 1 is traveling, the LiDAR 53L is turned on and the camera 51L is turned off. On the other hand, for example, when the vehicle 1 is parked, the camera 51L is turned on and the LiDAR 53L is turned off.

[0183] On the other hand, for example, by integrating the camera and the LiDAR, sharing the light emitting units for the camera and the LiDAR, and controlling the operation timing of each unit, the camera and the LiDAR can be operated simultaneously.

[0184] <Configuration Example of Sensing System 301>

[0185] Figure 11 A diagram showing a configuration example of the sensing system 301 to which the present technology is applied.

[0186] The sensing system 301 is a system that integrates a camera and LiDAR, and can be applied to the vehicle 1. The sensing system 301 includes a sensing unit 311 and a light emitting unit 312.

[0187] The sensing unit 311 includes an image sensor 321, a light detection sensor 322, and a control unit 323.

[0188] The image sensor 321 includes, for example, a rolling shutter type imaging element 321A similar to the imaging element 221A in Figure 6 .

[0189] The light detection sensor 322 includes, for example, a light receiving unit 322A in which light receiving elements such as single photon avalanche diodes (SPADs) are two-dimensionally arranged. That is, in the light receiving unit 322A, a plurality of light receiving elements are arranged in multiple rows. The light receiving unit 322A receives the reflected light of the IR light emitted from the light emitting unit 312. The light detection sensor 322 performs sensing of the surroundings based on the reflected light received by the light receiving unit 322A. For example, the light detection sensor 322 detects the shape, distance, position, etc. of the surrounding objects.

[0190] The control unit 323 comprehensively controls the image sensor 321, the light detection sensor 322, and the light emitting unit 312. For example, the control unit 323 comprehensively controls the imaging timing of the image sensor 321, the sensing timing of the light detection sensor 322, and the light emitting timing of the light emitting unit 312.

[0191] The light emitting unit 312 includes, for example, a VCSEL similar to the light emitting unit 212 in Figure 6 . The light emitting elements are two-dimensionally arranged, and the light emission of each light emitting element can be controlled. The IR light emitted from the light emitting unit 312 is used as illumination light for the image sensor 321 and measurement light for the light detection sensor 322.

[0192] The combination of the image sensor 321 and the light emitting unit 312 constitutes a camera with illumination. The combination of the light detection sensor 322 and the light emitting unit 312 constitutes LiDAR. That is, the sensing system 301 has two functions of a camera and LiDAR.

[0193] The light emitting unit 312 emits IR light in the imaging direction of the image sensor 321 and the sensing direction of the light detection sensor 322. Therefore, the imaging range of the image sensor 321 and the sensing range of the light detection sensor 322 at least partially overlap with the irradiation range of the light emitting unit 312. Note that it is desirable that the irradiation range of the light emitting unit 312 is substantially the same as or includes the range obtained by combining the imaging range of the image sensor 321 and the sensing range of the light detection sensor 322.

[0194] Note that the sensing range of the light detection sensor 322 is, for example, the range in which the light receiving unit 322A can receive the reflected light of the IR light (i.e., the light receiving range of the light receiving unit 322A).

[0195] Figure 12 An example of the hardware configuration of the sensing unit 311 of the sensing system 301 is shown.

[0196] The sensing unit 311 includes a module in which a chip 341 constituting the image sensor 321, a chip 342 constituting the light detection sensor 322, and a chip 343 constituting the control unit 323 are stacked.

[0197] Therefore, by modularizing the sensing unit 311, the size of the sensing system 301 can be reduced.

[0198] <Installation example of the sensing system 301>

[0199] Figure 13 An example of the installation position of the sensing system 301 is shown.

[0200] The sensing system 301 is installed, for example, near the door mirror on the left side of the vehicle 1.

[0201] <Operation example of the sensing system 301>

[0202] Next, an operation example of the sensing system 301 will be described with reference to Figure 14 the timing diagram of. Figure 14 The horizontal axis in represents time. In Figure 14 , the vertical axis represents the rows of the imaging element 321A, the rows of the light receiving unit 322A, and the rows of the light emitting unit 312.

[0203] For example, during the period from time t1f to time t3f, the exposure of the head row of the imaging element 321A is performed. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element 321A. Then, during the period from time t2f to time t6f, the exposure of the final row of the imaging element 321A is performed.

[0204] In addition, during the period from time t3f to time t5f, the head row of the imaging element 321A is in the period of stopped exposure (non-exposure period). The non-exposure period includes the read period for reading the pixel signals. Thereafter, while gradually moving the time from the second row to the final row of the imaging element 321A, the non-exposure period is sequentially set. Then, during the period from time t6f to time t8f, the final row of the imaging element 321A is in the non-exposure period.

[0205] Next, in the period from time t5f to time t9f, exposure of the head row of the next frame of the imaging element 321A is performed. Thereafter, while time gradually moves, exposure of each row is performed sequentially from the second row to the final row of the imaging element 321A. Then, in the period from time t8f to time t12f, exposure of the final row of the imaging element 321A is performed.

[0206] Then, the light emitting unit 312 emits light in a predetermined pattern in each of the exposure period and the non-exposure period of the imaging element 321A.

[0207] For example, in the exposure period of the imaging element 321A, in a bright state around the vehicle 1 , such as during the day, the light emitting unit 312 does not emit light.

[0208] For example, during the exposure period of the imaging element 321A, in a dark state around the vehicle 1, for example, at night, the light emitting unit 312 emits light. Figure 9 Similar to the above method, each row of the light emitting unit 312 emits light according to the exposure period of each row of the imaging element 321A. As a result, during the exposure period of each row of the imaging element 321A, the imaging range of each row is illuminated by the illumination light for a predetermined length of time.

[0209] On the other hand, the LiDAR process is performed during the non-exposure period of the imaging element 321A.

[0210] Specifically, for example, in the period from time t3f to time t4f, light emission of the head row of the light emitting unit 312 for the light detection sensor 322 is performed. Thereafter, while time gradually moves, light emission of each row is sequentially performed from the second row to the final row of the light emitting unit 312 for the light detection sensor 322. Then, in the period from time t6f to time t7f, light emission of the final row of the light emitting unit 312 for the light detection sensor 322 is performed.

[0211] In this way, the light emission timing of the light emitting unit 312 is controlled for each row according to the non-exposure period of each row of the imaging element 321A.

[0212] Furthermore, for example, during the period from time t4f to time t5f, light reception, distance measurement, etc. of the head row of the light receiving unit 322A are performed. Thereafter, while time gradually moves, light reception, distance measurement, etc. of each row are sequentially performed from the second row to the final row of the light receiving unit 322A. Then, during the period from time t7f to time t8f, light reception, distance measurement, etc. of the final row of the light emitting unit 312 are performed.

[0213] Hereinafter, after time t9f, the above-described processing from time t3f to time t8f is repeatedly performed.

[0214] Note that, in the example of Figure 14 Figure 14 , the light emission for the light detection sensor 322 starts immediately after the exposure of the imaging element 321A ends. However, the light emission may start after a slight interval.

[0215] In addition, for example, when the sensing range of the light detection sensor 322 is narrower than the irradiation range of the IR light of the light emitting unit 312, it is not always necessary to cause all the light emitting elements in the light emitting unit 312 to emit light.

[0216] In addition, even when the light receiving elements of the light receiving unit 322A receive light simultaneously, the light detection sensor 322 can perform processing. Therefore, for example, when there is an interval between the end of the exposure of the last row of the imaging element 321A and the start of the exposure of the head row of the imaging element 321A, that is, when there is an interval between the end of the exposure of the previous frame of the imaging element 321A and the start of the exposure of the next frame, multiple rows of the light emitting unit 312 can emit light simultaneously during this interval, and all rows of the light receiving unit 322A receive light simultaneously.

[0217] As described above, in the case of using the image sensor 321 with the rolling shutter method, the utilization efficiency of the light emitting unit 312 can be improved.

[0218] For example, one light emitting unit 312 can be shared in the functions of the camera and LiDAR. Therefore, the size of the sensing system 301 can be reduced. In addition, the camera and LiDAR can be integrated into one system, and the degree of freedom of the installation position is improved.

[0219] For example, the light emission time of each row of the light emitting unit 312 can be suppressed within the range required for the functions of the camera and LiDAR. As a result, the power consumption of the light emitting unit 312 is reduced.

[0220] <<5. Third Embodiment>>

[0221] Next, the third embodiment of the present technology will be described with reference to Figures 15 to 17 Figures 15 to 17 .

[0222] <Configuration Example of Sensing System 401>

[0223] Figure 15 A configuration example of the sensing system 401 to which the present technology is applicable is shown.

[0224] The sensing system 401 can be applied to the vehicle 1. The sensing system 401 includes a front image sensor 411, a front light emitting unit 412, an upper image sensor 413, an upper light emitting unit 414, and a control unit 415.

[0225] Similar to Figure 6Similar to the imaging element 221A, the front image sensor 411 includes an imaging element 411A in a rolling shutter mode. The front image sensor 411 is installed, for example, in the upper front part of the vehicle 1 (e.g., near the rearview mirror). For example, the front image sensor 411 images Figure 16 the imaging range A1 in. That is, the front image sensor 411 images the vicinity of the passengers on the driver's seat and the passenger seat of the vehicle 1. The images captured by the front image sensor 411 are used for, for example, DMS, the occupant monitoring system (OMS), and video chat.

[0226] The front light emitting unit 412 is installed at substantially the same position as the front image sensor 411. Similar to Figure 6 the light emitting unit 212 in, the front light emitting unit 412 includes a VCSEL. The irradiation range of the illumination light of the front light emitting unit 412 overlaps at least partially with the imaging range A1. Note that it is desirable that the irradiation range of the front light emitting unit 412 be substantially the same as or include the imaging range A1.

[0227] Similar to Figure 6 the imaging element 221A, the upper image sensor 413 includes an imaging element 413A in a rolling shutter mode. The upper image sensor 413 is installed, for example, at the upper center of the vehicle 1 (e.g., near the center of the ceiling of the vehicle). For example, in Figure 16 the upper image sensor 413 images the imaging range A2. That is, the upper image sensor 413 images the entire interior of the vehicle from above.

[0228] For example, the images captured by the upper image sensor 413 are used to detect the departure of children or the like inside the vehicle.

[0229] The upper light emitting unit 414 is installed at substantially the same position as the upper image sensor 413. Similar to Figure 6 the light emitting unit 212 in, the upper light emitting unit 414 includes a VCSEL. The irradiation range of the illumination light of the upper light emitting unit 414 overlaps at least partially with the imaging range A2. Note that it is desirable that the irradiation range of the upper light emitting unit 414 be substantially the same as or include the imaging range A2.

[0230] The control unit 415 comprehensively controls the front image sensor 411, the front light emitting unit 412, the upper image sensor 413, and the upper light emitting unit 414. For example, the control unit 415 comprehensively controls the exposure timing of the front image sensor 411, the light emission timing of the front light emitting unit 412, the exposure timing of the upper image sensor 413, and the light emission timing of the upper light emitting unit 414.

[0231] <Operation example of the sensing system 401>

[0232] Next, the operation of the sensing system 401 will be described with reference to Figure 17 the timing diagram of Figure 17 The upper part of Figure 17 shows the operation of the front image sensor 411, and the lower part shows the operation of the upper image sensor 413. Figure 17 The horizontal axis in Figure 17 represents time.

[0233] For example, during the period from time t1g to time t2g, the exposure of the head row of the imaging element 413A of the upper image sensor 413 is performed. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element 413A. Then, during the period from time t4g to time t5g, the exposure of the final row of the imaging element 413A is performed.

[0234] At this time, similar to the above method described with reference to Figure 9 each row of the upper light emitting unit 414 emits light according to the exposure period of each row of the imaging element 413A. As a result, during the exposure period of each row of the imaging element 413A, the imaging range of each row is irradiated with illumination light for a predetermined time length.

[0235] In this way, according to the non-exposure period of each row of the front image sensor 411, each row of the upper light emitting unit 414 emits light, and each row of the upper image sensor 413 is exposed.

[0236] Next, during the period from time t2g to time t3g, the exposure of the head row of the imaging element 411A of the front image sensor 411 for the first DMS and OMS is performed. Thereafter, while the time gradually moves, the exposure of each row is sequentially performed from the second row to the final row of the imaging element 411A for the first DMS and OMS. Then, during the period from time t5g to time t6g, the exposure of the final row of the imaging element 411A for the first DMS and OMS is performed.

[0237] At this time, similar to the above method described with reference to Figure 9 each row of the front light emitting unit 412 emits light according to the exposure period of each row of the imaging element 411A. As a result, during the exposure period of each row of the imaging element 411A, the imaging range of each row is irradiated with illumination light for a predetermined time length.

[0238] Next, in the period from time t3g to time t5g, exposure of the head row of the imaging element 411A of the front image sensor 411 for video chatting is performed. Thereafter, while time gradually moves, exposure of each row is sequentially performed from the second row to the final row of the imaging element 411A for video chatting. Then, in the period from time t6g to time t8g, exposure of the final row of the imaging element 411A for video chatting is performed.

[0239] Note that, for example, in the exposure period for video chatting of the imaging element 411A, in a bright state around the vehicle 1 , such as during the day, the front light emitting unit 412 does not emit light.

[0240] On the other hand, for example, during the exposure period for video chat of the imaging element 411A, in a dark state around the vehicle 1, for example, at night, the front light emitting unit 412 emits light. Figure 9 Similar to the above method, each row of the front light emitting unit 412 emits light according to the exposure period of each row of the imaging element 411A. As a result, during the exposure period of each row of the imaging element 411A, the imaging range of each row is illuminated by the illumination light for a predetermined length of time.

[0241] Note that, although not shown, a predetermined interval is provided for a readout period or the like between an exposure period of each row of the imaging element 411A for the first DMS and OMS and an exposure period of each row of the imaging element 411A for video chatting.

[0242] Next, in the period from time t5g to time t6g, exposure of the head row of the imaging element 411A of the front image sensor 411 for the second DMS and OMS is performed. Thereafter, while time gradually moves, exposure of each row is sequentially performed from the second row to the final row of the imaging element 411A for the second DMS and OMS. Then, in the period from time t8g to time t9g, exposure of the final row of the imaging element 411A for the second DMS and OMS is performed.

[0243] Note that during the exposure period for the second DMS and OMS, the front light emitting unit 412 does not emit light.

[0244] Note that, although not shown, a predetermined interval is provided for a readout period or the like between the exposure period of each row of the imaging element 411A for video chatting and the exposure period of each row of the imaging element 411A for the second DMS and OMS.

[0245] Then, obtain the difference between the images for the first DMS and OMS captured by the front image sensor 411 in the state of being irradiated with illumination light and the images for the second DMS and OMS captured by the front image sensor 411 in the state of not being irradiated with illumination light. Thus, the influence of ambient light is eliminated from the images for DMS and OMS.

[0246] In this way, according to the non-exposure periods of the respective rows of the upper image sensor 413, the respective rows of the front light-emitting unit 412 emit light, and the respective rows of the front image sensor 411 are exposed.

[0247] Next, during the period from time t8g to time t9g, the exposure of the head row of the next frame of the imaging element 413A of the upper image sensor 413 is performed.

[0248] Thereafter, after time t8g, the above-described processing from time t1g to time t9g is repeatedly executed.

[0249] Note that an interval can be set between the exposure periods of the respective rows of the front image sensor 411 and the exposure periods of the respective rows of the upper image sensor 413.

[0250] As described above, similar to the above-described first embodiment, in the case of using the front image sensor 411 and the upper image sensor 413 in the rolling shutter method, the utilization efficiency of the front light-emitting unit 412 and the upper light-emitting unit 414 can be improved.

[0251] In addition, the exposure timing of the front image sensor 411, the light-emitting timing of the front light-emitting unit 412, the exposure timing of the upper image sensor 413, and the light-emitting timing of the upper light-emitting unit 414 are appropriately controlled. For example, the light-emitting timing of the front light-emitting unit 412 is controlled so that the illumination light of the front light-emitting unit 412 does not affect the exposure (imaging) of the respective rows of the upper image sensor 413. For example, the light-emitting timing of the upper light-emitting unit 414 is controlled so that the illumination light of the upper light-emitting unit 414 does not affect the exposure (imaging) of the respective rows of the front image sensor 411.

[0252] Note that in DMS, OMS, and video chat, since the movement of the subject is very small, almost no adverse effects due to rolling shutter distortion occur.

[0253] <<6. Modification Example>>

[0254] Hereinafter, a modification example of the above-described embodiment of the present technology will be described.

[0255] For example, it is not necessary for the light-emitting elements in each row of the light-emitting unit to all emit light simultaneously. For example, the light-emitting elements in each row can be made sparser as needed to emit light. For example, in the case of light emission for LiDAR, the light-emitting elements in each row can also be scanned in the horizontal direction.

[0256] For example, in the present technology, a light-emitting unit that mechanically scans light in the vertical direction (column direction) using a mirror or the like can also be used.

[0257] For example, in the present technology, a light-emitting unit in which the light-emitting elements are arranged one-dimensionally can also be used. For example, a light-emitting unit in which one light-emitting element is provided in each row can also be used. For example, a light-emitting unit in which a plurality of light-emitting elements are arranged in a row and the light emitted from that row is mechanically scanned in the vertical direction can be used.

[0258] In the above description, an example in which the light-emitting unit emits IR light has been described, but the wavelength of the light emitted by the light-emitting unit is not particularly limited. For example, visible light or the like can be used as needed.

[0259] For example, in the present technology, the rolling shutter type imaging element used not only includes an imaging element that performs exposure (scanning) for each row, but also includes an imaging element that performs exposure (scanning) for each row of multiple rows.

[0260] In the above description, an example in which the light-emitting timing of each row of the light-emitting unit is controlled according to the exposure timing of each row of the image sensor has been described. However, the exposure timing of each row of the image sensor can be controlled according to the light-emitting timing of each row of the light-emitting unit.

[0261] In addition to the above LiDAR, the present technology can also be applied to the case where an imaging element is combined with a sensor that can share the imaging element and the light-emitting unit.

[0262] For example, in addition to vehicles, the present technology can also be applied to a system or device including an image sensor having a rolling shutter type imaging element. For example, the present technology can also be applied to a moving body other than a vehicle. For example, the present technology can be applied to a monitoring system that monitors a predetermined area such as a building. For example, the present technology can be applied to an information processing device such as a smartphone.

[0263] <<7. Others>>

[0264] In this specification, a system refers to a group of multiple constituent elements (devices, modules (components), etc.), and it is not important whether all the constituent elements are located in the same housing. Therefore, a plurality of devices housed in one housing and connected via a network, and a single device having a plurality of modules housed in one housing are both systems.

[0265] In addition, the embodiments of the present technology are not limited to the above embodiments, and various changes can be made without departing from the gist of the present technology.

[0266] For example, the present technology can be configured as cloud computing, in which one function is shared by multiple devices via a network for joint processing.

[0267] <Example of Composition>

[0268] The present technology can also adopt the following composition:

[0269] (1) A sensing system, comprising:

[0270] A first imaging sensor including an array of light receiving elements;

[0271] A first light emitter including an array of light emitting elements; and

[0272] A control circuit configured to control the first imaging sensor and the first light emitter such that the imaging range of a subset of the array of light receiving elements overlaps with the irradiation range of a subset of the array of light emitting elements.

[0273] (2) The sensing system according to (1), wherein the subset of the array of light receiving elements is multiple rows in the array of light receiving elements, and the subset of the array of light emitting elements is the same number of rows in the array of light emitting elements.

[0274] (3) The sensing system according to (2), wherein the number of rows is 1.

[0275] (4) The sensing system according to (1), wherein

[0276] the control circuit controls the exposure timing of the first imaging sensor based on the light emission timing of the first light emitter.

[0277] (5) The sensing system according to (2), wherein

[0278] the control circuit controls the light emission timing of the first light emitter based on the exposure timing of the first imaging sensor.

[0279] (6) The sensing system according to (5), wherein

[0280] the control circuit controls the light emission timing of each row of the array of light emitting elements to be related to the exposure timing of each row of the array of light receiving elements.

[0281] (7) The sensing system according to (6), wherein

[0282] The control circuit causes each row of the array of light-emitting elements to emit light for a predetermined period during the exposure period of each row of the array of light-receiving elements.

[0283] (8) The sensing system according to (1), further comprising:

[0284] A light detection sensor including an array of light-receiving elements,

[0285] wherein the control circuit is configured to control a first light emitter to emit light in a first mode during a non-exposure period, the non-exposure period being a period other than the exposure period of a first imaging sensor.

[0286] (9) The sensing system according to (8), wherein

[0287] the control circuit controls the light emission timing of each row of the array of light-emitting elements according to the non-exposure period of each row of the array of light-receiving elements in the first imaging sensor.

[0288] (10) The sensing system according to (8), wherein

[0289] the control circuit causes multiple rows of the array of light-emitting elements to emit light simultaneously during the non-exposure period of the first imaging sensor.

[0290] (11) The sensing system according to (8), wherein

[0291] the control circuit causes the first light emitter to emit light in a second mode during the exposure period of the first imaging sensor.

[0292] (12) The sensing system according to (8), wherein

[0293] Another array of light-receiving elements of the light detection sensor receives the reflected light of the light emitted by the first light emitter.

[0294] (13) The sensing system according to (8), wherein

[0295] The first imaging sensor, the light detection sensor, and the control circuit are provided in the same chip assembly.

[0296] (14) The sensing system according to (1), further comprising:

[0297] A second imaging sensor in a rolling shutter mode, including an array of light-receiving elements; and

[0298] A second light emitter, including an array of light-emitting elements,

[0299] The control circuit is configured to control the second imaging sensor and the second light emitter such that an imaging range of a subset of the array of light receiving elements of the second imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the second light emitter.

[0300] (15) The sensing system according to (14), wherein

[0301] the control circuit exposes the second imaging sensor and causes the second light emitter to emit light during a non-exposure period, which is a period other than the exposure period of the first imaging sensor.

[0302] (16) The sensing system according to (15), wherein

[0303] the control circuit exposes each row of the array of light receiving elements of the second imaging sensor and causes each row of the array of light emitting elements of the second light emitter to emit light according to the non-exposure period of each row of the first imaging sensor.

[0304] (17) The sensing system according to (16), wherein

[0305] the control circuit exposes the first imaging sensor and causes the first light emitter to emit light during a non-exposure period of the second imaging sensor.

[0306] (18) The sensing system according to (17), wherein

[0307] the control circuit exposes each row of the array of light receiving elements of the first imaging sensor and causes each row of the array of light emitting elements of the first light emitter to emit light according to the non-exposure period of each row of the second imaging sensor.

[0308] (19) The sensing system according to (1), wherein

[0309] the first imaging sensor, the first light emitter, and the control circuit are provided in a vehicle.

[0310] (20) A sensing control device, comprising:

[0311] a control circuit configured to control a first imaging sensor and a first light emitter, the first imaging sensor including an array of light receiving elements, the first light emitter including an array of light emitting elements, wherein

[0312] an imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the first light emitter.

[0313] (21) A sensing method, comprising:

[0314] Control a first imaging sensor, the first imaging sensor including an array of light receiving elements; and

[0315] Control a first light emitter, the first light emitter including an array of light emitting elements, wherein

[0316] The imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with the irradiation range of a subset of the array of light emitting elements of the first light emitter.

[0317] (22) A non-transitory computer-readable medium storing program code, the program code being executable to perform operations including the following:

[0318] Control a first imaging sensor, the first imaging sensor including an array of light receiving elements; and

[0319] Control a first light emitter, the first light emitter including an array of light emitting elements, wherein

[0320] The imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with the irradiation range of a subset of the array of light emitting elements of the first light emitter.

[0321] (B1) A sensing system, comprising:

[0322] A first image sensor that controls exposure for each row;

[0323] A first light emitting unit whose irradiation range overlaps with at least a part of the imaging range of the first image sensor and is capable of controlling its light emission timing for each row; and

[0324] A control unit that comprehensively controls the first image sensor and the first light emitting unit.

[0325] (B2) The sensing system according to (B1), wherein

[0326] The control unit controls one of the exposure timing of the first image sensor and the light emission timing of the first light emitting unit based on the other.

[0327] (B3) The sensing system according to (B2), wherein

[0328] The control unit controls one of the exposure timing of each row of the first image sensor and the light emission timing of each row of the first light emitting unit based on the other.

[0329] (B4) The sensing system according to (B3), wherein

[0330] During the exposure of each row of the first image sensor, the control unit causes the rows of the first light emitting unit corresponding to each row of the first image sensor to emit light for a predetermined length of time.

[0331] (B5) The sensing system according to (B1) further includes

[0332] a light detection sensor, wherein a plurality of light receiving elements are arranged in multiple rows,

[0333] wherein the control unit causes the first light emitting unit to emit light in a first mode for the light detection sensor during a non-exposure period, which is a period other than the exposure period of the first image sensor.

[0334] (B6) The sensing system according to (B5), wherein

[0335] the control unit controls the light emission timing of the first light emitting unit for each row according to the non-exposure period of each row of the first image sensor.

[0336] (B7) The sensing system according to (B5), wherein

[0337] the control unit causes multiple rows of the first light emitting unit to emit light simultaneously during the non-exposure period of the first image sensor.

[0338] (B8) The sensing system according to any one of (B5) to (B7), wherein

[0339] the control unit causes the first light emitting unit to emit light in a second light emission mode during the exposure period of the first image sensor.

[0340] (B9) The sensing system according to any one of (B5) to (B8), wherein

[0341] each of the light receiving elements of the light detection sensor receives the reflected light of the light emitted by the first light emitting unit.

[0342] (B10) The sensing system according to any one of (5) to (9), wherein

[0343] the first image sensor, the light detection sensor, and the control unit are provided in one module.

[0344] (B11) The sensing system according to (B1) further includes:

[0345] a second image sensor that controls exposure for each row; and

[0346] a second light emitting unit whose irradiation range overlaps at least a part of the imaging range of the second image sensor and can control its light emission timing for each row,

[0347] The control unit comprehensively controls the first image sensor, the first light-emitting unit, the second image sensor, and the second light-emitting unit.

[0348] (B12) The sensing system according to (B11), wherein

[0349] the control unit exposes the second image sensor and causes the second light-emitting unit to emit light during a non-exposure period, which is a period other than the exposure period of the first image sensor.

[0350] (B13) The sensing system according to (B12), wherein

[0351] the control unit exposes each row of the second image sensor and causes each row of the second light-emitting unit to emit light according to the non-exposure period of each row of the first image sensor.

[0352] (B14) The sensing system according to (B13), wherein

[0353] the control unit exposes the first image sensor and causes the first light-emitting unit to emit light during the non-exposure period of the second image sensor.

[0354] (B15) The sensing system according to (B14), wherein

[0355] the control unit exposes each row of the first image sensor and causes each row of the first light-emitting unit to emit light according to the non-exposure period of each row of the second image sensor.

[0356] (B16) The sensing system according to any one of (B1) to (B15), wherein

[0357] the first image sensor, the first light-emitting unit, and the control unit are provided in a vehicle.

[0358] (B17) A sensing control device, comprising

[0359] a control unit that comprehensively controls a first image sensor and a light-emitting unit, the first image sensor controls exposure for each row, and the irradiation range of the light-emitting unit overlaps at least a part of the imaging range of the image sensor and can control its light-emitting timing for each row.

[0360] (B18) A sensing method, comprising

[0361] comprehensively controlling a first image sensor and a light-emitting unit, the first image sensor controls exposure for each row, and the irradiation range of the light-emitting unit overlaps at least a part of the imaging range of the image sensor and can control its light-emitting timing for each row.

[0362] Note that the effects described in this specification are merely examples and are not limiting, and other effects may be provided.

[0363] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0364] [reference numerals list]

[0365] 1 Vehicle

[0366] 11Vehicle Control System

[0367] 51 cameras

[0368] 53LiDAR

[0369] 201 Sensing System

[0370] 211 Sensing Unit

[0371] 212 light emitting units

[0372] 221 Image Sensor

[0373] 221A imaging element

[0374] 222 Control Unit

[0375] 301 Sensing System

[0376] 311 Sensing Unit

[0377] 312 light emitting units

[0378] 321 Image Sensor

[0379] 321A imaging element

[0380] 322 light detection sensor

[0381] 322A optical receiving unit

[0382] 323 Control Unit

[0383] 341~343 chips

[0384] 401 Sensing System

[0385] 411 front image sensor

[0386] 411A imaging element

[0387] 412 front light unit

[0388] 413 Upper Image Sensor

[0389] 413A imaging element

[0390] 414 upper light-emitting unit

[0391] 415 control unit

Claims

1. A sensing system, comprising: a first imaging sensor including an array of light receiving elements; a first light emitter including an array of light emitting elements; and a control circuit configured to control the first imaging sensor and the first light emitter such that an imaging range of a subset of the array of light receiving elements overlaps with an illumination range of a subset of the array of light emitting elements.

2. The sensing system according to claim 1, wherein the subset of the array of light receiving elements is multiple rows in the array of light receiving elements, and the subset of the array of light emitting elements is the same number of rows in the array of light emitting elements.

3. The sensing system according to claim 2, wherein the number of rows is 1.

4. The sensing system according to claim 1, wherein the control circuit controls an exposure timing of the first imaging sensor based on a light emission timing of the first light emitter.

5. The sensing system according to claim 2, wherein the control circuit controls a light emission timing of the first light emitter based on an exposure timing of the first imaging sensor.

6. The sensing system according to claim 5, wherein the control circuit controls light emission timings of respective rows of the array of light emitting elements to be related to exposure timings of respective rows of the array of light receiving elements.

7. The sensing system according to claim 6, wherein the control circuit causes respective rows of the array of light emitting elements to emit light for a predetermined period during an exposure of respective rows of the array of light receiving elements.

8. The sensing system according to claim 1, further comprising: a light detection sensor including an array of light receiving elements, wherein the control circuit is configured to control the first light emitter to emit light in a first mode for the light detection sensor during a non-exposure period, the non-exposure period being a period other than an exposure period of the first imaging sensor.

9. The sensing system according to claim 8, wherein the control circuit controls light emission timings of respective rows of the array of light emitting elements according to a non-exposure period of each row of the array of light receiving elements in the first imaging sensor.

10. The sensing system according to claim 8, wherein the control circuit causes multiple rows of the array of light emitting elements to emit light simultaneously during a non-exposure period of the first imaging sensor.

11. The sensing system according to claim 8, wherein the control circuit causes the first light emitter to emit light in a second mode during an exposure period of the first imaging sensor.

12. The sensing system according to claim 8, wherein another array of light receiving elements of the light detection sensor receives reflected light of light emitted by the first light emitter.

13. The sensing system according to claim 8, wherein the first imaging sensor, the light detection sensor, and the control circuit are provided in the same chip component.

14. The sensing system according to claim 1, further comprising: a second imaging sensor including an array of light receiving elements; and a second light emitter including an array of light emitting elements, The control circuit is configured to control the second imaging sensor and the second light emitter such that an imaging range of a subset of the array of light receiving elements of the second imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the second light emitter.

15. The sensing system according to claim 14, wherein the control circuit exposes the second imaging sensor and causes the second light emitter to emit light during a non-exposure period, which is a period other than the exposure period of the first imaging sensor.

16. The sensing system according to claim 15, wherein the control circuit exposes each row of the array of light receiving elements of the second imaging sensor and causes each row of the array of light emitting elements of the second light emitter to emit light according to the non-exposure period of each row of the first imaging sensor.

17. The sensing system according to claim 16, wherein the control circuit exposes the first imaging sensor and causes the first light emitter to emit light during the non-exposure period of the second imaging sensor.

18. The sensing system according to claim 17, wherein the control circuit exposes each row of the array of light receiving elements of the first imaging sensor and causes each row of the array of light emitting elements of the first light emitter to emit light according to the non-exposure period of each row of the second imaging sensor.

19. The sensing system according to claim 1, wherein the first imaging sensor, the first light emitter, and the control circuit are provided in a vehicle.

20. A sensing control device, comprising: a control circuit configured to control a first imaging sensor and a first light emitter, the first imaging sensor including an array of light receiving elements, the first light emitter including an array of light emitting elements, wherein an imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the first light emitter.

21. A sensing method, comprising: controlling a first imaging sensor, the first imaging sensor including an array of light receiving elements; and and controlling a first light emitter, the first light emitter including an array of light emitting elements, wherein an imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the first light emitter.

22. A non-transitory computer-readable medium storing program code executable to perform operations including: controlling a first imaging sensor, the first imaging sensor including an array of light receiving elements; and controlling a first light emitter, the first light emitter including an array of light emitting elements, wherein an imaging range of a subset of the array of light receiving elements of the first imaging sensor overlaps with an illumination range of a subset of the array of light emitting elements of the first light emitter.

Images