Solid-state imaging device, control method for solid-state imaging device, and control program for solid-state imaging device

Abstract

The present invention inhibits a deterioration in image quality. This solid-state imaging device according to an embodiment comprises: a pixel array section (101) provided with a plurality of pixels; a control unit (105) that reads first image data from the pixel array section during each of a plurality of cycles within one frame; and a processing unit (108) that generates second image data of the one frame on the basis of a plurality of pieces of the first image data read within the one frame.

Description

Technical Field

[0004] , ,

[0006] , ,

[0005] , , ,

[0001] The present disclosure relates to a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device.

Background Art

[0002] In recent years, with the spread of the automation of moving objects such as automobiles and robots and the Internet of Things (IoT), etc., speeding up and improving the accuracy of image recognition have been strongly desired.

Prior Art Documents

Patent Documents

[0003] ​​​​​​​​​​​​​​​​​​​​​The reduced detection accuracy of subjects with flicker components and the occurrence of blur in images not only degrade image quality but also reduce recognition accuracy in image recognition. However, as mentioned above, there is a trade-off relationship between the detection accuracy of subjects with flicker components and image quality. Therefore, conventional technologies had the problem that the image quality of image data acquired in a space containing subjects with flicker components could be reduced.

[0007] Therefore, this disclosure proposes a solid-state imaging device capable of suppressing image quality degradation, a control method for the solid-state imaging device, and a control program for the solid-state imaging device. [Means for solving the problem]

[0008] To solve the above problems, one embodiment of a solid-state imaging device according to the present disclosure includes a pixel array unit having a plurality of pixels, a control unit that reads first image data from the pixel array unit in each of a plurality of cycles within a frame, and a processing unit that generates second image data for a frame based on a plurality of first image data read within a frame. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example configuration of a vehicle control system. [Figure 2] This figure shows an example of a sensing area. [Figure 3] This is a block diagram showing a schematic configuration example of an image sensor according to one embodiment. [Figure 4] This figure illustrates an example of a read operation performed by an image sensor during one frame period according to one embodiment. [Figure 5] This figure illustrates an example of a read operation performed by an image sensor during one cycle period according to one embodiment. [Figure 6] This figure illustrates an example of the configuration and operation of the frame data generation unit in an image sensor according to one embodiment. [Figure 7]This diagram illustrates the principle behind the occurrence of SNR drop. [Figure 8] This figure illustrates an example of another readout operation performed by the image sensor during one frame period according to one embodiment. [Figure 9] This figure illustrates an example of another readout operation performed by the image sensor during one frame period according to one embodiment. [Figure 10] This figure illustrates an example of vibration correction according to one embodiment. [Figure 11] This block diagram shows an example configuration for realizing vibration correction according to one embodiment. [Figure 12] This block diagram shows another configuration example for realizing vibration compensation according to one embodiment. [Figure 13] This is a schematic diagram illustrating the case where vibration correction is applied to the basic operation according to one embodiment. [Figure 14] This is a diagram (part 1) illustrating a motion vector search operation according to one embodiment. [Figure 15] This is a diagram (part 2) illustrating the motion vector search operation according to one embodiment. [Figure 16] This is a diagram (part 3) illustrating the motion vector search operation according to one embodiment. [Figure 17] This is a block diagram showing an example configuration for realizing motion vector search according to one embodiment. [Figure 18] This figure illustrates an example of a read operation performed by an image sensor during a single frame period, according to a modified version of one embodiment. [Figure 19] This is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device related to this disclosure. [Modes for carrying out the invention]

[0010] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0011] Also, the present disclosure will be described in accordance with the item order shown below. 0. Introduction 1. One Embodiment 1.1 Configuration Example of Vehicle Control System 1.2 Configuration Example of Solid-State Imaging Device (Image Sensor) 1.3 Basic Operation Example (Generation of Frame Data by 4-Cycle Sequence and DOL Synthesis) 1.4 Other Countermeasure Examples for SNR Drop 关于1.5 闪烁的检测和去除 关于1.6 抖动校正 关于1.7 运动矢量搜索(ME) 1.8 Summary 1.9 Variation 2. Hardware Configuration

[0012] 0. Introduction The image sensors according to the prior art have the following technical problems.

[0013] (Flickering Light Source) In the prior art, due to the relationship between the bright-dark cycle of the subject and the frame rate of the image sensor, there may occur a problem that the subject is detected as being in a state where it remains extinguished for a long time or remains lit for a long time. Such a problem can be solved by setting the exposure time of the image sensor to a long exposure time longer than the bright-dark cycle of the subject, but it may induce problems such as blur described later.

[0014] (Blur) When long exposures are used to improve the detection accuracy of flicker light sources such as traffic lights and road signs, increasing the exposure time increases the distance the subject and background move within the image sensor's field of view, which can cause blurring in the acquired image. While this problem can be reduced by shortening the exposure time, it leads to a dilemma where the detection accuracy of flicker light sources decreases. Furthermore, short exposure times can result in reduced visibility of images acquired in low-light environments such as at night or in dark areas.

[0015] (Low light performance) While the visibility of images acquired in low-light environments can be improved by increasing the exposure time, this creates a dilemma: the image acquired by the image sensor becomes blurred. Furthermore, in recent years, the trend towards smaller pixel pitches has led to decreased pixel sensitivity, making it increasingly difficult to ensure visibility.

[0016] (Dynamic range) In addition to these, recent image sensors have seen improvements in dynamic range through advancements in pixel structure and techniques such as binning. However, there remains a challenge: SNR drop can occur due to heat generated by the sensor chip itself, which degrades the signal-to-noise ratio (SNR).

[0017] (Balancing detection and visual inspection applications) Furthermore, detection frames often use frame rates different from those used for general viewing (e.g., 30fps (frames per second) or 60fps) to suppress the effects of flicker, which can make it difficult to achieve compatibility between detection and viewing applications.

[0018] (Effects of shaking) Furthermore, when image sensors mounted on moving objects such as vehicles detect distant subjects at high resolution, the detection frame (also called the field of view) moves in accordance with the shaking due to vibrations of the image sensor itself, making it difficult to determine the movement of the detected object. In addition, if the detection frame moves significantly, the correlation of the images in the time axis direction is lost, resulting in a situation where image recognition must be performed on a single image, which makes it difficult to improve recognition accuracy.

[0019] Therefore, in the following embodiments, a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device will be described with examples, which can simultaneously achieve the generation of images unaffected by the flicker of a flicker light source and the detection of a flicker light source. Furthermore, in the following embodiments, a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device will be described with examples, which can suppress blur to an extent that does not affect image recognition. Furthermore, in the following embodiments, a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device will be described with examples, which can suppress the deterioration of image quality in low light conditions. Furthermore, in the following embodiments, a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device will be described with examples, which can enable imaging with a high dynamic range. Furthermore, in the following embodiments, a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device will be described with examples, which can be used for both detection and viewing purposes. Furthermore, in the following embodiments, an improvement in the accuracy of image recognition can be expected, which will be described with examples, which will be described.

[0020] 1. One Embodiment Next, one embodiment relating to this disclosure will be described in detail with reference to the drawings.

[0021] 1.1 Example of a vehicle control system configuration First, the mobile device control system according to this embodiment will be described. Figure 1 is a block diagram showing an example configuration of a vehicle control system 11, which is an example of the mobile device control system according to this embodiment.

[0022] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to driving assistance and autonomous driving of the vehicle 1.

[0023] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiver unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a driver monitoring system (DMS) 30, a human-machine interface (HMI) 31, and a vehicle control unit 32.

[0024] The vehicle control ECU 21, communication unit 22, map information storage unit 23, GNSS receiver unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driving support / automatic driving control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with one another via a communication network 41. The communication network 41 consists of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 41 may be used depending on the type of data to be communicated; for example, CAN is applied to data related to vehicle control, and Ethernet is applied to large-capacity data. In addition, each part of the vehicle control system 11 may be directly connected using wireless communication intended for relatively short-range communication, such as near-field communication (NFC) or Bluetooth (registered trademark), without going through the communication network 41.

[0025] In the following, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the processor 21 and the communication unit 22 communicating.

[0026] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0027] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 22 can communicate using multiple communication methods.

[0028] A brief explanation will be given regarding the external communication capabilities of the communication unit 22. The communication unit 22 communicates with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication methods such as 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 to communicate with the external network is not particularly limited, as long as it is a wireless communication method that enables digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0029] Furthermore, for example, the communication unit 22 can communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by mobile objects that move at relatively low speeds, such as pedestrians and cyclists, terminals that are fixedly installed in places such as stores, or MTC (Machine Type Communication) terminals. In addition, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and other entities, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with a house, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0030] The communication unit 22 can, for example, receive programs from an external source (over the air) to update the software that controls the operation of the vehicle control system 11. The communication unit 22 can also receive map information, traffic information, information about the vehicle 1's surroundings, etc., from an external source. Furthermore, the communication unit 22 can transmit information about the vehicle 1 and information about the vehicle 1's surroundings to an external source. Information about the vehicle 1 that the communication unit 22 transmits to an external source includes, for example, data indicating the status of the vehicle 1 and recognition results from the recognition unit 73. Furthermore, the communication unit 22 can also perform communications corresponding to vehicle emergency notification systems such as e-Call.

[0031] A brief overview of the communication capabilities of the communication unit 22 with the vehicle interior will be provided. The communication unit 22 can communicate with various devices in the vehicle, for example, using wireless communication. The communication unit 22 can communicate wirelessly with devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wireless communication, such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 can also communicate with various devices in the vehicle using wired communication. For example, the communication unit 22 can communicate with various devices in the vehicle via wired communication through a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with various devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wired communication, such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), and MHL (Mobile High-definition Link).

[0032] Here, "devices inside the vehicle" refers to, for example, devices inside the vehicle that are not connected to the communication network 41. Examples of devices inside the vehicle include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0033] For example, the communication unit 22 receives electromagnetic waves transmitted by road traffic information communication systems (VICS® (Vehicle Information and Communication System)) such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0034] The map information storage unit 23 stores either or both maps acquired from external sources and maps created by the vehicle 1. For example, the map information storage unit 23 stores high-precision 3D maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0035] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to vehicle 1 from an external server. A point cloud map is a map composed of point clouds (point cloud data). Here, a vector map refers to a map adapted for ADAS (Advanced Driver Assistance System) that maps traffic information such as the location of lanes and traffic lights to a point cloud map.

[0036] The point cloud map and vector map may be provided from, for example, an external server, or they may be created in the vehicle 1 as maps for matching with the local map described later, based on sensing results from radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred square meters relating to the planned route that the vehicle 1 will travel will be acquired from the external server.

[0037] The GNSS receiver 24 receives GNSS signals from GNSS satellites and acquires the position information of the vehicle 1. The received GNSS signals are supplied to the driving assistance / automatic driving control unit 29. The GNSS receiver 24 is not limited to using GNSS signals; for example, it may acquire position information using beacons.

[0038] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the external recognition sensor 25 are arbitrary.

[0039] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0040] The shooting method of camera 51 is not particularly limited as long as it is a shooting method capable of distance measurement. For example, camera 51 can be a camera of various shooting methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, as needed. In addition, camera 51 may simply be for acquiring images, regardless of distance measurement.

[0041] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment relative to the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a raindrop sensor, fog sensor, sunshine sensor, snow sensor, and illuminance sensor.

[0042] Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0043] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the in-vehicle sensor 26 are not particularly limited as long as the number can realistically be installed in the vehicle 1.

[0044] For example, the in-vehicle sensor 26 can be equipped with one or more sensors from among a camera, radar, seat sensor, steering wheel sensor, microphone, and biosensor. The camera equipped in the in-vehicle sensor 26 can be a camera of various imaging types capable of distance measurement, such as a ToF camera, stereo camera, monocular camera, or infrared camera. However, it is not limited to these, and the camera equipped in the in-vehicle sensor 26 may be one that simply acquires images, regardless of distance measurement. The biosensor equipped in the in-vehicle sensor 26 is installed, for example, on the seat or steering wheel, and detects various biometric information of the driver or other passengers.

[0045] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the vehicle sensor 27 are not particularly limited as long as the number can realistically be installed on the vehicle 1.

[0046] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0047] The recording unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 28 can be used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium can be a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 1 before and after an event such as an accident, as well as biometric information acquired by the in-vehicle sensors 26.

[0048] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0049] The analysis unit 61 performs analysis processing on the vehicle 1 and its surroundings. The analysis unit 61 comprises a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0050] The self-position estimation unit 71 estimates the vehicle's position based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on sensor data from the external recognition sensor 25 and estimates the vehicle's position by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel relative to the axle.

[0051] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around vehicle 1 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of vehicle 1 by the recognition unit 73.

[0052] The self-position estimation unit 71 may estimate the vehicle 1's own position based on the GNSS signal and sensor data from the vehicle sensor 27.

[0053] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and union.

[0054] The recognition unit 73 performs a detection process to detect the external conditions of the vehicle 1, and a recognition process to recognize the external conditions of the vehicle 1.

[0055] For example, the recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0056] Specifically, for example, the recognition unit 73 performs detection and recognition processing of objects around the vehicle 1. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0057] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering, which classifies point clouds based on sensor data from LiDAR 53 or radar 52 into clusters of point clouds. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0058] For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the velocity and direction of travel (movement vector) of objects around the vehicle 1 to be detected.

[0059] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., from the image data supplied from the camera 51. Alternatively, it may recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0060] For example, the recognition unit 73 can perform traffic rule recognition processing around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and the recognition result of objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the location and status of traffic signals, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on.

[0061] For example, the recognition unit 73 can perform recognition processing of the environment surrounding the vehicle 1. The surrounding environment that the recognition unit 73 is intended to recognize may include weather, temperature, humidity, brightness, and road surface conditions.

[0062] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route following processes.

[0063] Global path planning is the process of planning a rough route from the start to the goal. This path planning also includes a process called local path planning, which involves generating a path that allows vehicle 1 to travel safely and smoothly in its vicinity, taking into account the motion characteristics of vehicle 1 along the planned path. Path planning may be distinguished as long-term path planning, and starting point generation as short-term path planning or local path planning. Safety-prioritized routes represent a similar concept to starting point generation, short-term path planning, or local path planning.

[0064] Route following is the process of planning actions to safely and accurately travel the route planned by route planning within a planned time. The action planning unit 62 can, for example, calculate the target speed and target angular velocity of vehicle 1 based on the results of this route following process.

[0065] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0066] For example, the motion control unit 63 controls the steering control unit 81, brake control unit 82, and drive control unit 83, which are included in the vehicle control unit 32 described later, to perform acceleration / deceleration control and direction control so that the vehicle 1 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow driving, vehicle speed maintenance, collision warning for the vehicle, and lane departure warning for the vehicle. For example, the motion control unit 63 performs cooperative control for the purpose of autonomous driving, such as driving autonomously without driver operation.

[0067] The DMS30 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensors 26 and input data input to the HMI31, which will be described later. In this case, the driver status to be recognized by the DMS30 is expected to include, for example, physical condition, level of alertness, level of concentration, level of fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0068] Furthermore, the DMS30 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0069] HMI31 handles the input of various data and instructions, and presents various data to the driver and other users.

[0070] A brief explanation of data input by HMI31 is provided. HMI31 is equipped with an input device for human data input. HMI31 generates input signals based on data and instructions input by the input device and supplies them to each part of the vehicle control system 11. HMI31 is equipped with operators such as a touch panel, buttons, switches, and levers as input devices. However, HMI31 may also be equipped with input devices that allow information to be input by methods other than manual operation, such as voice or gestures. Furthermore, HMI31 may use external connected devices such as a remote control device using infrared or radio waves, or a mobile device or wearable device that corresponds to the operation of the vehicle control system 11, as input devices.

[0071] This section provides a brief overview of how HMI31 presents data. HMI31 generates visual, auditory, and tactile information for the occupant or those outside the vehicle. HMI31 also performs output control, managing the output, content, timing, and method of each of these generated pieces of information. As visual information, HMI31 generates and outputs information indicated by images and light, such as operation screens, vehicle status displays, warning displays, and monitor images showing the surroundings of vehicle 1. As auditory information, HMI31 generates and outputs information indicated by sound, such as voice guidance, warning sounds, and warning messages. Furthermore, as tactile information, HMI31 generates and outputs information that is perceived by the occupant's sense of touch through force, vibration, movement, etc.

[0072] As output devices for visual information output by HMI31, for example, a display device that presents visual information by displaying images itself, or a projector device that presents visual information by projecting images, can be applied. In addition to display devices with ordinary displays, the display device may also be a device that displays visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality. Furthermore, HMI31 can also use display devices such as the navigation system, instrument panel, CMS (Camera Monitoring System), electronic mirrors, and lamps installed in the vehicle 1 as output devices for visual information output.

[0073] For HMI31, output devices that output auditory information can include, for example, audio speakers, headphones, and earphones.

[0074] As an output device for HMI31 to output tactile information, for example, a haptic element using haptic technology can be applied. The haptic element is installed in parts of the vehicle 1 that are in contact with by the occupant, such as the steering wheel and the seat.

[0075] The vehicle control unit 32 controls various parts of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0076] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0077] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 also includes, for example, a control unit such as an ECU that controls the brake system.

[0078] The drive control unit 83 detects and controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a control unit such as an ECU that controls the drive system.

[0079] The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system.

[0080] The light control unit 85 detects and controls the status of various lights on the vehicle 1. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights.

[0081] The horn control unit 86 detects and controls the status of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn.

[0082] Figure 2 shows examples of sensing areas using the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 of the external recognition sensor 25 shown in Figure 1. In Figure 2, the vehicle 1 is schematically shown as viewed from above, with the left end being the front end of the vehicle 1 and the right end being the rear end of the vehicle 1.

[0083] Sensing regions 91F and 91B show examples of sensing regions of the ultrasonic sensor 54. Sensing region 91F covers the area around the front end of the vehicle 1 by multiple ultrasonic sensors 54. Sensing region 91B covers the area around the rear end of the vehicle 1 by multiple ultrasonic sensors 54.

[0084] The sensing results in sensing area 91F and sensing area 91B are used, for example, for parking assistance for vehicle 1.

[0085] Sensing areas 92F to 92B show examples of sensing areas for short-range or medium-range radar 52. Sensing area 92F covers a position further in front of vehicle 1 than sensing area 91F. Sensing area 92B covers a position further in rear of vehicle 1 than sensing area 91B. Sensing area 92L covers the rear periphery of the left side of vehicle 1. Sensing area 92R covers the rear periphery of the right side of vehicle 1.

[0086] The sensing results in sensing region 92F are used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region 92B are used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing regions 92L and 92R are used, for example, to detect objects in blind spots to the sides of vehicle 1.

[0087] Sensing areas 93F to 93B show examples of sensing areas by camera 51. Sensing area 93F covers a position further in front of vehicle 1 than sensing area 92F. Sensing area 93B covers a position further in rear of vehicle 1 than sensing area 92B. Sensing area 93L covers the periphery of the left side of vehicle 1. Sensing area 93R covers the periphery of the right side of vehicle 1.

[0088] The sensing results in sensing region 93F can be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 93B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing regions 93L and 93R can be used, for example, for surround view systems.

[0089] Sensing area 94 shows an example of the sensing area of ​​LiDAR 53. Sensing area 94 covers a position further in front of vehicle 1 than sensing area 93F. On the other hand, sensing area 94 has a narrower range in the left-right direction than sensing area 93F.

[0090] The sensing results in the sensing region 94 are used, for example, to detect objects such as surrounding vehicles.

[0091] Sensing region 95 shows an example of the sensing region of the long-range radar 52. The sensing area 95 covers a position further in front of the vehicle 1 than the sensing area 94. On the other hand, the range of the sensing area 95 is narrower in the left-right direction than that of the sensing area 94.

[0092] The sensing results in sensing area 95 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0093] Furthermore, the sensing areas of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 included in the external recognition sensor 25 may take various configurations other than those shown in Figure 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1. Also, the installation positions of each sensor are not limited to the examples described above. In addition, there may be one or more sensors.

[0094] 1.2 Example Configuration of a Solid-State Imaging Device (Image Sensor) Next, an example of the configuration of the solid-state imaging device (hereinafter referred to as the image sensor) that constitutes the camera 51 of the external recognition sensor 25 will be described. Figure 3 is a block diagram showing a schematic example of the image sensor according to this embodiment. Here, the image sensor may be an image sensor created, for example, by applying or partially using a CMOS process.

[0095] The image sensor 100 according to this embodiment has a stacked structure in which, for example, a semiconductor chip on which a pixel array section 101 is formed and a semiconductor chip on which peripheral circuits are formed are stacked. The peripheral circuits may include, for example, a vertical drive circuit 102, a column processing circuit 103, a horizontal drive circuit 104, and a system control unit 105.

[0096] The image sensor 100 further includes a signal processing unit 108 and a data storage unit 109. The signal processing unit 108 and the data storage unit 109 may be provided on the same semiconductor chip as the peripheral circuitry, or on a separate semiconductor chip.

[0097] The pixel array section 101 has a configuration in which pixels 110, each having a photoelectric conversion element that generates and stores an electric charge corresponding to the amount of light received, are arranged in a two-dimensional grid in the row and column directions, i.e., in a matrix. Here, the row direction refers to the arrangement direction of pixels in a pixel row (horizontal direction in the drawing), and the column direction refers to the arrangement direction of pixels in a pixel column (vertical direction in the drawing).

[0098] In the pixel array section 101, pixel drive lines LD are wired along the row direction for each pixel row in the matrix-like pixel arrangement, and vertical signal lines VSL are wired along the column direction for each pixel column. The pixel drive lines LD transmit drive signals for driving when reading signals from pixels. In Figure 3, the pixel drive lines LD are shown as individual wires, but they are not limited to one wire each. One end of the pixel drive line LD is connected to the output terminal corresponding to each row of the vertical drive circuit 102.

[0099] The vertical drive circuit 102 is composed of a shift register, an address decoder, and the like, and drives each pixel of the pixel array section 101 simultaneously or row by row. In other words, the vertical drive circuit 102, together with the system control unit 105 that controls the vertical drive circuit 102, constitutes a drive unit that controls the operation of each pixel of the pixel array section 101. The specific configuration of this vertical drive circuit 102 is not shown in the diagram, but generally it has two scanning systems: a read scanning system and a sweep scanning system.

[0100] The readout scanning system sequentially selects and scans the pixels 110 of the pixel array 101 row by row in order to read signals from the pixels. The signals read from the pixels 110 are analog signals. The sweep scanning system performs a sweep scan ahead of the readout scan performed by the readout scanning system by the exposure time.

[0101] This sweep scanning system resets the photoelectric converter by sweeping away unwanted charge from the photoelectric converter of pixel 110 in the readout row. By sweeping away (resetting) the unwanted charge with this sweep scanning system, a so-called electronic shutter operation is performed. Here, electronic shutter operation refers to the operation of discarding the charge of the photoelectric converter and starting a new exposure (starting charge accumulation).

[0102] The signal read out by the readout scanning system corresponds to the amount of light received since the previous readout operation or electronic shutter operation. The period from the readout timing of the previous readout operation or the sweep timing of the electronic shutter operation to the readout timing of the current readout operation is the charge accumulation period (also called the exposure period) at pixel 110.

[0103] The signals output from each pixel 110 of the pixel row selected and scanned by the vertical drive circuit 102 are input to the column processing circuit 103 through each of the vertical signal lines VSL for each pixel column. The column processing circuit 103 performs predetermined signal processing on the signals output from each pixel of the selected row through the vertical signal lines VSL for each pixel column of the pixel array section 101, and temporarily holds the pixel signals after signal processing.

[0104] Specifically, the column processing circuit 103 performs at least noise reduction processing as signal processing, such as CDS (Correlated Double Sampling) processing or DDS (Double Data Sampling) processing. For example, CDS processing removes pixel-specific fixed pattern noise such as reset noise and threshold variations of amplification transistors within pixels. In addition, the column processing circuit 103 also has an AD (analog-to-digital) conversion function, which converts the analog pixel signal read from the photoelectric conversion element into a digital signal and outputs it.

[0105] The horizontal drive circuit 104 consists of a shift register and an address decoder, and sequentially selects the readout circuits (hereinafter referred to as pixel circuits) corresponding to the pixel rows of the column processing circuit 103. Through this selective scanning by the horizontal drive circuit 104, the pixel signals processed for each pixel circuit in the column processing circuit 103 are output sequentially.

[0106] The system control unit 105 is composed of a timing generator that generates various timing signals, and controls the vertical drive circuit 102, column processing circuit 103, and horizontal drive circuit 104 based on the various timings generated by the timing generator.

[0107] The signal processing unit 108 has at least an arithmetic processing function and performs various signal processing, such as arithmetic processing, on the pixel signals output from the column processing circuit 103. The data storage unit 109 temporarily stores the data necessary for the signal processing performed by the signal processing unit 108.

[0108] Furthermore, the image data output from the signal processing unit 108 may be subjected to predetermined processing in, for example, the driving assistance / automatic driving control unit 29 of the vehicle control system 11 equipped with the image sensor 100, or transmitted to an external source via the communication unit 22.

[0109] 1.3 Basic Operation Example (Generation of Frame Data by 4-Cycle Sequence and DOL Synthesis) Next, a basic operation example of the image sensor 100 according to this embodiment will be described. Figure 4 is a diagram illustrating an example of a read operation performed by the image sensor during one frame period according to this embodiment, and Figure 5 is a diagram illustrating an example of a read operation performed by the image sensor during one cycle period according to this embodiment. In the following description, the operation of reading image data (cycle data, etc.) from the pixel array unit 101, including the read operation exemplified in Figure 4, is performed by the vertical drive circuit 102, column processing circuit 103, and horizontal drive circuit 104 operating according to instructions from the system control unit 105 in Figure 3, for example. Therefore, in this description, these will be collectively referred to as the control unit.

[0110] As shown in Figure 4, in this embodiment, the period Tf for outputting one image data (hereinafter referred to as frame data) is divided into multiple cycles (in this example, four cycles C11 to C14). In this embodiment, each of the four equal periods obtained by dividing the 1 frame period Tf is set as 1 cycle period Tc. In each cycle period Tc, for example, image data of the same size as the frame data (hereinafter referred to as cycle data) may be read out by an operation similar to the normal read operation for reading frame data.

[0111] As shown in Figure 5, each cycle period Tc includes an accumulation exposure period Tp that outputs image data based on the charge generated by the photoelectric conversion of each pixel 110 photoelectric conversion element, and a non-accumulation exposure period Tq that outputs image data without exposure, which is not based on the charge generated by photoelectric conversion. Although cycle C14 is shown in Figure 5, the other cycles C11 to C13 may have a similar configuration.

[0112] The non-accumulation exposure period Tq may be shorter than the accumulation exposure period Tp (for example, 1 / 16th of the period). For example, if the frame rate of the image sensor 100 is 60fps, the frame period can be 16.67ms (milliseconds), the cycle period Tc can be 4.167ms, the accumulation exposure period Tp can be 3.9ms, and the non-accumulation exposure period Tq can be 0.24ms. However, these are merely examples and the values ​​are not limited to these.

[0113] Image data Q1 output during the non-accumulation exposure period Tq can be used to remove noise components such as dark current and fixed pattern noise from image data P1 output during the accumulation exposure period Tp. That is, image data P1 output during the accumulation exposure period Tp may contain signal components based on charge generated by photoelectric conversion and noise components caused by dark current. Therefore, by subtracting image data Q1, which is the noise component, from image data P1, it is possible to generate image data (cycle data) with a high purity of signal components. In this case, in order to match the noise levels of image data P1 and image data Q1, the pixel values ​​of image data Q1 may be amplified, for example, based on each exposure period. In this example, for example, the pixel values ​​of image data Q1 may be multiplied by Tp / Tq. Such removal of noise components may be performed, for example, in the column processing circuit 103 in Figure 3, or in the signal processing unit 108.

[0114] Furthermore, as shown in Figures 4 and 5, at least one of the multiple cycle periods Tc obtained by dividing one frame period Tf (in this example, cycle C14) may include an ultra-high brightness accumulation exposure period Tr. For example, if the charge of pixel 110 (specifically, the charge accumulation area such as the floating diffusion region in pixel 110) becomes saturated due to exposure at a higher illumination than expected during the accumulation exposure period Tp, then for saturated pixels 110 or cycle data or frame data containing saturated pixels, frame data may be generated using image data generated during the ultra-high brightness accumulation exposure period Tr instead of image data P1 generated during the accumulation exposure period Tp. The ultra-high brightness accumulation exposure period Tr may be, for example, even shorter than the non-accumulation exposure period Tq. For example, the ultra-high brightness accumulation exposure period Tr may be on the order of microseconds. Also, noise components of image data R1 generated during the ultra-high brightness accumulation exposure period Tr may be removed using image data Q1 generated during the non-accumulation exposure period Tq. However, if the ultra-high brightness storage exposure period Tr is short enough that the noise component can be ignored, the noise component of the image data R1 does not need to be removed.

[0115] Next, we will explain the operation of generating a single frame data from the cycle data read in each cycle C11 to C14. In this explanation, we will illustrate the case where frame data is generated using DOL (Digital Overlap) synthesis, but we are not limited to this, and various methods may be used. Figure 6 is a diagram illustrating an example of the configuration and operation of the frame data generation unit in the image sensor according to this embodiment. Note that this generation unit may be implemented, for example, in the column processing circuit 103 or the signal processing unit 108 in Figure 3.

[0116] As shown in Figure 6, the frame data generation unit according to this embodiment may be implemented as, for example, an averaging circuit. The cycle data S1 read in the first cycle C11 during the frame period Tf is input to the data compression unit 111 in the generation unit. The data compression unit 111 and the data compression units 124 and 134, described later, may reduce the amount of cycle data using a data compression method such as PWL (PieceWise Linear), which reduces the number of bits by making the higher bits sparse. However, the data compression method employed in these data compression units is not limited to PWL, and may be any lossless or lossy compression method capable of reducing the amount of cycle data. Furthermore, if PWL is employed, a method using a LUT (Lookup Table) for data compression may be employed, or a method without using a LUT may be employed.

[0117] The cycle data S1 compressed by the data compression unit 111 is temporarily stored in memory 112. Memory 112, as well as memories 121, 125, 131, 135, 141, 144, and 145 described later, are, for example, memories placed at boundaries where calculations occur between cycles, and can be realized, for example, by the data storage unit 109 in Figure 3. In this case, if the alignment between the cycle data to be added has been completed in advance, that is, if motion vector search (Motion Estimation: ME) for each of the cycle data to be added has been completed in advance, and the alignment between the cycle data has been completed in advance based on the amount of image motion estimated by this search, it is possible to directly add the cycle data together, thereby reducing the capacity of each memory. Furthermore, as described later, it is also possible to reduce the capacity of each memory by pre-setting the addition ratio of each cycle data.

[0118] The cycle data S1 temporarily stored in memory 112 is decompressed by the data decompression unit 113 and then input to the divider 114, which divides the cycle data S1 based on a preset addition ratio. The data decompression unit 113 and the data decompression units 126 and 136 described later may employ a data decompression method corresponding to the data compression method used in the data compression units 111, 124, and 134. The divider 114 and the divider 122 described later may employ, for example, a bit shift arithmetic unit that divides a value by shifting bits. In this case, to suppress the reduction in accuracy of pixels with low brightness, the lower bits (for example, the lower 4 bits) may be excluded from the bits to be divided. In this example, the divider 114 divides the cycle data S1 by a preset addition ratio (=1 / 2). The cycle data S1 divided by the divider 114 is input to the adder 123.

[0119] Furthermore, the adder 123 also receives the cycle data S2 read in the next cycle C12. Specifically, the cycle data S2 read in the next cycle C12 is temporarily stored in memory 121, then input to the divider 122, where it is divided based on a preset addition ratio (=1 / 2), and then input to the adder 123. The cycle data S1 and S2, each divided based on the addition ratio (=1 / 2), are added (DOL combined) in the adder 123. The combined cycle data S5 generated by this addition is compressed by the data compression unit 124 and then temporarily stored in memory 125. The addition ratios set in the dividers 114 and 122 may be set, for example, so that they sum to '1' and the component ratios of cycle data S1 and S2 in the combined cycle data S5 are 1:1.

[0120] The composite cycle data S5 temporarily stored in memory 125 is decompressed by the data decompression unit 126 and then input to the divider 127. The divider 127 divides the cycle data S5 by a preset addition ratio (=2 / 3) and inputs the resulting composite cycle data S5 to the adder 133.

[0121] Furthermore, the cycle data S3 read in the next cycle C13 is also input to the adder 133. Specifically, the cycle data S3 read in the next cycle C13 is temporarily stored in memory 131, then input to the divider 132, where it is divided based on a preset addition ratio (=1 / 3), and then input to the adder 133. Note that bitwise arithmetic units may be used for the divider 132 and the dividers 137 and 142 described later. The composite cycle data S5 and cycle data S3, divided based on the respective addition ratios (=2 / 3 and 1 / 3), are added (DOL composite) in the adder 133. The composite cycle data S6 generated by this addition is compressed by the data compression unit 134 and then temporarily stored in memory 135. The addition ratios set in the dividers 127 and 132 may be set, for example, so that they add up to '1', and the component ratios of cycle data S1, S2, and S3 are 1:1:1 in the composite cycle data S6.

[0122] The composite cycle data S6 temporarily stored in memory 135 is decompressed by the data decompression unit 136 and then input to the divider 137. The divider 137 divides the cycle data S6 by a preset addition ratio (=3 / 4) and inputs the resulting composite cycle data S6 to the adder 143.

[0123] Furthermore, the cycle data S4 read in the next cycle C14 is also input to the adder 143. Specifically, the cycle data S4 read in the next cycle C14 is temporarily stored in memory 141, then input to the divider 142, where it is divided based on a pre-set addition ratio (=1 / 4), and then input to the adder 143. The combined cycle data S6 and cycle data S4, divided based on the respective addition ratios (=3 / 4, 1 / 4), are added together (DOL combination) in the adder 143. The combined cycle data S7 generated by this addition is temporarily stored in memory 144. The addition ratios set in the dividers 137 and 142 may be set, for example, so that they sum to '1', and the component ratios of cycle data S1, S2, S3, and S4 are 1:1:1:1 in the combined cycle data S7.

[0124] The composite cycle data S7 temporarily stored in memory 144 is input to the HDR (High Dynamic Range) composite unit 146. The HDR composite unit 146 also receives image data S8 read out during the ultra-high brightness storage exposure period Tr. For example, if there are pixels in the composite cycle data S7 whose values ​​are saturated, the HDR composite unit 146 performs HDR synthesis by replacing the pixel values ​​of these pixels with the pixel values ​​of the corresponding pixels in the image data S8.

[0125] The image data output from the HDR synthesis unit 146 is compressed by the data compression unit 147 and then output externally as frame data. The data compression method used in the data compression unit 147 is not limited to PWL, but may be any of the various data compression methods.

[0126] As described above, in this embodiment, one frame period Tf is divided into multiple cycle periods Tc, and cycle data S1 to S4 are read out in each cycle period Tc. This shortens the exposure time (accumulated exposure period Tp) when reading each cycle data S1 to S4, making it possible to reduce the blur of each cycle data S1 to S4. Then, since the blur-reduced cycle data S1 to S4 are added together (DOL synthesis) to generate frame data, the blur of the final output frame data is also reduced. As a result, it is possible to generate frame data in which blur is suppressed to the extent that it does not affect image recognition.

[0127] Furthermore, in this embodiment, since image data (cycle data) is read out at a faster cycle rate than the frame rate, it becomes possible to detect flicker light sources that blink at a predetermined period, such as LEDs (Light Emitting Diodes). This makes it possible to simultaneously generate images unaffected by the blinking of flicker light sources and detect flicker light sources.

[0128] Furthermore, in this embodiment, image data Q1 consisting of noise components is acquired, and the noise components are removed from image data P1 containing both signal and noise components using image data Q1. As a result, even in situations where the signal-to-noise ratio (SNR) deteriorates due to heat generation of the sensor chip itself, it becomes possible to generate image data (cycle data and frame data) with higher signal component purity, thereby suppressing SNR drops and enabling the generation of higher-quality image data.

[0129] Furthermore, as in this embodiment, by dividing the accumulation exposure period within one frame period Tf into multiple cycles, it is possible to suppress the reduction in the effective accumulation exposure time when generating one frame data. This makes it possible to generate frame data with reduced blur while suppressing the decrease in visibility of images acquired in low-light environments.

[0130] Furthermore, as in this embodiment, by dividing the accumulation exposure period within a single frame period Tf into multiple cycles, it becomes possible to distribute the amount of charge accumulated in one frame across multiple cycles, thereby expanding the effective dynamic range when generating a single frame data.

[0131] Furthermore, as in this embodiment, when adding cycle data (DOL synthesis), the motion vector for each cycle data to be added is searched in advance, and the alignment between the cycle data is completed in advance based on the amount of image motion estimated by this search, thereby reducing the capacity of each memory.

[0132] Furthermore, generally speaking, the influence of global motion caused by the movement of the image sensor 100 is overwhelmingly greater than the influence of local motion caused by the movement of individual subjects when blurring image data. Therefore, as in this embodiment, when adding cycle data (DOL synthesis), the motion vector for each cycle data to be added is searched in advance, and the alignment between cycle data is completed in advance based on the amount of image motion estimated by this search. This makes it possible to accurately suppress the influence of global motion that may occur when adding cycle data. As a result, it becomes possible to generate clearer image data with less blur.

[0133] In this explanation, the case where one frame period Tf is equally divided to set up multiple cycles C11 to C14 is given as an example, but the explanation is not limited to this, and the time length of at least one cycle period Tc among the multiple cycles C11 to C14 may differ from the time length of the cycle period Tc of the other cycles. Similarly, in this explanation, the case where the time length of the accumulation exposure period Tp and the time length of the non-accumulation exposure period Tq are the same for each cycle C11 to C14 is given as an example, but the explanation is not limited to this, and the time length of the accumulation exposure period Tp and / or the time length of the non-accumulation exposure period Tq in at least one cycle may differ from those in other cycles.

[0134] Furthermore, in this embodiment, changing the frame duration Tf, or in other words, changing the frame rate, may be achieved, for example, by adjusting the length of the blanking period at the end of each frame duration Tf. This eliminates the need to change the operation of the image sensor 100 in each cycle C11 to C14, making it possible to change the frame rate using a simpler method.

[0135] 1.4 Other examples of countermeasures for SNR drops In the basic operation example described above, we illustrated a case where image data Q1 consisting of noise components is acquired, and the noise components are removed from image data P1 containing both signal and noise components using image data Q1, thereby increasing the purity of the signal components and suppressing the occurrence of SNR drop. However, countermeasures against SNR drop are not limited to this. First, we will explain the principle of SNR drop occurrence. Figure 7 is a diagram illustrating the principle of SNR drop occurrence.

[0136] As shown in Figure 7, during the non-accumulation exposure period Tq of each cycle period Tc, image data Q1 consisting of noise components is read out. This noise component is also included in the image data P1 read out during the accumulation exposure period Tp, superimposed on the actual signal component P0.

[0137] The main components of noise are noise caused by dark current and fixed pattern noise. Of these, fixed pattern noise is a qualitative noise component determined during the design and manufacturing stages of the image sensor 100. In addition, noise caused by dark current is a qualitative noise component that correlates with the chip temperature of the image sensor 100 and varies with temperature. Therefore, noise components mainly consisting of fixed pattern noise and noise caused by dark current (i.e., image data Q1) can be acquired before the image sensor 100 is shipped.

[0138] Therefore, in this embodiment, image data Q1 under no exposure conditions may be acquired in advance for each predetermined temperature range at predetermined timings before and / or after shipment of the image sensor 100, and the acquired image data Q1 for each temperature range may be stored in the data storage unit 109 or the memory of the column processing circuit 103. Note that "under no exposure conditions" may refer to a state in which the image sensor 100 is placed in a dark area or a state in which the charge generated by the photoelectric conversion of the pixels 110 is discarded. When reading cycle data S1 to S4, instead of reading the image data Q1, the image data Q1 stored in the data storage unit 109 or the like may be read. In this case, the image data Q1 to be read from the data storage unit 109 or the like may be selected based on the temperature detected by a temperature sensor (not shown). Such measures to address SNR drop may be implemented, for example, in the column processing circuit 103 in Figure 3, or in the signal processing unit 108.

[0139] In this way, by acquiring the noise component, image data Q1, in advance, it becomes possible to omit the operation of reading out image data Q1 during each cycle period Tc, which simplifies the operation when generating frame data and reduces power consumption.

[0140] 1.5 Detection and Removal of Flicker As illustrated using Figure 6 above, if the cycle period Tc is defined as a period obtained by dividing the frame period Tf into, for example, four equal parts, there are cases where the flickering of the flicker light source cannot be detected depending on the relationship between the frame rate and the flickering period of the flicker light source. For example, if the cycle frequency of the image sensor 100 is approximately 240 Hz, a flicker light source around 120 Hz will be detected as an object that is continuously lit, off, or slowly flickers. This suggests that detection may be difficult for flicker light sources that operate at general power supply frequencies, such as traffic signals using LEDs.

[0141] Therefore, in this embodiment, LFM (LED Flicker Mitigation) is implemented to display a flicker light source without flickering, while also implementing a function to determine that it is a flicker light source.

[0142] Specifically, in this embodiment, for example, as illustrated in Figure 8, the cycle frequency is set to a frequency different from a power of two of the frame frequency. For example, if the frame frequency is 60 Hz (i.e., the frame rate is 60 fps), and the cycle period is defined as a period obtained by dividing the frame period into four parts, the cycle frequency becomes 240 Hz, which is 2 squared times 60 Hz, making it difficult to detect flicker light sources around 120 Hz. Therefore, in this embodiment, for example, the cycle frequency is set to 300 Hz, which is different from a power of two of 60 Hz. This makes it possible to shift the cycle period of the image sensor 100 from the flicker period of the flicker light source, thereby enabling the detection of the flicker light source.

[0143] In this embodiment, the number of cycles during one frame period Tf (four in this example) may be fixed. Therefore, if the frame period Tf is not equally divided by the number of cycles, a remaining period may occur. In that case, as shown in Figure 8, the remaining period may be used as a blanking period Tb during which no exposure is performed.

[0144] In this case, it is preferable to set the cycle period Tc and blanking period Tb such that the sum of the accumulated exposure periods Tp in one frame period Tf is 50% or more of the frame period Tf (i.e., a duty cycle of 50% or more). This helps to suppress the decrease in image visibility due to underexposure. For example, if the one frame period Tf is 16.67 ms (corresponding to 60 Hz) and the one cycle period Tc is 3.3 ms (corresponding to approximately 300 Hz), the period from the start of exposure of the first cycle C11 to the end of exposure of the last cycle C14 in one frame (hereinafter referred to as the LFM period Tlfm) may be set to 11 ms or more and 16.67 ms or less, taking into account the non-accumulated exposure period Tq in each cycle and the blanking period between each cycle.

[0145] If the cycle period Tc is set for each frame period Tf according to the rules described above, it is possible to generate frame data in which the flicker of the light source is suppressed by adding the cycle data S1 to S4 read out in each cycle. For example, DOL synthesis as illustrated in Figure 5 may be used to add the cycle data S1 to S4.

[0146] Furthermore, if a cycle period Tc is set for each frame period Tf according to the rules described above, it is possible to determine whether or not a flicker light source is captured by comparing the brightness of the whole or each region between each cycle. In other words, if there is a region where the brightness changes between cycle data, that region can be determined to be a region where a flicker light source is captured (hereinafter referred to as a flicker region). In this case, the following equation (1) can be used to detect the flicker region. In equation (1), cy1 may be the pixel value of each pixel in the cycle data S1 or statistical data information such as the average value using surrounding pixels. The same applies to cy2 to cy4.

number

[0147] For pixels that do not flicker during a single frame, the value of equation (1) will be zero or close to zero. Therefore, by calculating equation (1) for each pixel, it is possible to identify the flicker region in the frame data. This identification of the flicker region may be performed, for example, in the column processing circuit 103 in Figure 3, or in the signal processing unit 108.

[0148] Furthermore, as shown in Figure 9, during each frame period Tf, in addition to the output of cycle data S1 to S4, image data for detecting flicker regions (hereinafter referred to as flicker detection frames) FDF1 to FDF4 may be read out. The read flicker detection frames FDF1 to FDF4 may also be output to the outside in parallel with the frame data.

[0149] Flicker detection frames FDF1 to FDF4 may use cycle data read at a specific period (in the example shown in Figure 9, a period with 6 cycles in between), without being bound by the frame period Tf. Furthermore, flicker detection frames FDF1 to FDF4 may use cycle data or reduced image data thereof (see flicker detection frames FDF1 and FDF3), or image data generated by averaging two or more cycle data sets or reduced image data thereof (see flicker detection frames FDF2 and FDF4). In addition, flicker detection frames FDF1 to FDF4 may be image data with the same field of view as the cycle data, or ROI (Region of Interest) image data of the region detected as a flicker area.

[0150] The flicker detection frames FDF1 to FDF4 output in this manner may be used as frame data to reproduce the flickering of a flicker light source. For example, by combining the frame data and the flicker detection frames so that the areas identified as flicker regions in the frame data are replaced with the corresponding areas in the flicker detection frames, it is possible to generate frame data that reproduces the flickering of a flicker light source. However, this is not limited to this, and the areas in the frame data that are replaced by the flicker detection frames may be areas identified by image recognition of the luminance values ​​of each area in the frame data or flicker detection frames, or the frame data, etc. Furthermore, the combination of frame data and flicker detection frames may be performed, for example, in the signal processing unit 108 in Figure 3, or in other parts such as the driving assistance / automatic driving control unit 29 (see Figure 1).

[0151] However, as mentioned above, when detecting flicker light sources based on blinking, in addition to actual flicker light sources such as LED light sources, objects that reflect blinking light or reflective objects that vibrate rapidly may also be detected as flicker light sources. Therefore, in this embodiment, it is possible to determine whether or not an object is an actual flicker light source based on the brightness, color, and position within the field of view of the blinking subject. For example, a threshold may be set based on a predetermined brightness value, color information, and position information within the field of view, or a brightness value, color information, and position information within the field of view obtained from frame data (for example, the average value of each value), or the brightness value and / or color information of an area where the brightness is significantly higher than the surrounding area or the color is different. An area with a brightness value higher than this threshold and located outside the outer edge of the field of view may be identified as a flicker area. In this case, the amount of computation can be reduced by truncating the lower bits of the pixel value (for example, bits corresponding to values ​​below the threshold) and performing the calculation using only the higher bits.

[0152] Furthermore, to identify the flicker region, not only the cycle data S1 to S4 read out during each frame period Tf, but also one or more frame data and / or cycle data from before and after the current frame may be used. In that case, by making the region identified as the flicker region include the area surrounding the area that actually flickers, it becomes possible to reduce the chance of missing flicker light sources due to changes in the field of view or subject between frames.

[0153] 1.6 About Image Stabilization The camera 51 mounted on vehicle 1 experiences shaking due to the movement of vehicle 1, causing the position of subjects in the image to change in accordance with the vehicle's movement. This change in position in the image tends to be more pronounced for distant subjects. Therefore, when performing prediction processing on images acquired by camera 51 mounted on vehicle 1 using methods such as RNN (Recurrent Neural Network), the position of subjects in the image (i.e., the position of the bounding box) is unstable. This can lead to the system misinterpreting the movement of subjects due to the vehicle's movement as the movement of the subjects themselves, potentially resulting in incorrect prediction results.

[0154] Furthermore, if the subject is a flicker light source such as a traffic light, and its position in the image is not stable, it becomes difficult to determine whether the subject is a flicker light source or a moving object. This can lead to the problem of not being able to properly identify the flicker region.

[0155] Furthermore, when generating a single frame data by adding multiple cycle data S1 to S4 (DOL synthesis), as in this embodiment, there is a problem that if focal plane distortion occurs due to fluctuations in the imaging range during the frame period Tf, the frame data generated by the addition may suffer from image quality degradation such as distortion and blurring.

[0156] Furthermore, when applying motion-compensated temporal filtering (MCTF) to frame data, there is a challenge in that the accuracy of motion compensation may decrease because the subject may unintentionally move within the field of view due to shaking of the imaging area.

[0157] Therefore, in this embodiment, information such as the direction, speed, and / or angular velocity of the camera 51's shaking may be detected using a vehicle sensor 27 (e.g., an IMU) mounted on the vehicle 1, and based on the detected information, shaking correction may be performed on the frame data within the image sensor 100. This reduces the positional shift of the same subject between preceding and succeeding frames due to shaking in the imaging range, and thus makes it possible to expect improvements in the performance of post-processing performed by the driving assistance / automatic driving control unit 29, etc. (for example, improved recognition rate, improved MCTF correction performance, reduced computational load, improved robustness, etc.).

[0158] Furthermore, performing shake correction on the frame data within the image sensor 100 simplifies external processing, which can lead to reduced total costs and smaller module sizes. Additionally, performing shake correction on the frame data within the image sensor 100 can reduce the amount of data output from the image sensor 100 to the external chip. However, performing shake correction on the frame data within the image sensor 100 is not mandatory; it may be performed externally by the processor 21 or the driving assistance / autonomous driving control unit 29.

[0159] In this embodiment, techniques such as electronic image stabilization (EIS) may be applied to correct the shake of frame data. Figure 10 is a diagram illustrating an example of shake correction according to this embodiment. In Figure 10, a case is illustrated in which frame data U_t-1 is output during the frame period at time t-1, and frame data U_t is output during the frame period at the next time t.

[0160] As shown in Figure 10, in this embodiment, for example, based on information regarding the shaking of the vehicle 1, i.e., the shaking of the camera 51 (hereinafter referred to as shaking information), input from the vehicle sensor 27, the region to be extracted from the frame data U_t is determined, and the extracted image data U1_t is output as frame data. When determining the region to be extracted, the position and range of the extraction region determined for previous frame data may be taken into consideration. Furthermore, the shaking information may include information commonly used in EIS, such as information regarding the acceleration and angular velocity of the camera 51 detected by the IMU of the vehicle sensor 27.

[0161] Figure 11 is a block diagram showing an example configuration for realizing vibration correction according to this embodiment, and Figure 12 is a block diagram showing another example configuration for realizing vibration correction according to this embodiment.

[0162] First, as shown in Figure 11, the image sensor 100, the IMU 271 in the vehicle sensor 27, and the vibration correction unit 201 may be mounted on a chip integrated as a camera module 511, for example. The image sensor 100 and the vibration correction unit 201 in the camera module 511 may have a configuration corresponding to the camera 51 in Figure 1, or they may be integrated into a single chip such as a stacked chip. The vibration correction unit 201 may also be implemented in a configuration such as the signal processing unit 108 in Figure 3.

[0163] In this configuration, the vibration correction unit 201 determines the region to be extracted from the frame data based on vibration information input from the IMU 271 at the same or approximately the same timing as the frame data input from the image sensor 100. Therefore, the frame data output from the image sensor 100 may be the frame data before EIS. The generated frame data produced by this vibration correction is input to the recognition unit 202 incorporated in the SoC (System on Chip) or ECU (Electronic Control Unit) (hereinafter referred to as SoC / ECU 291) and used for image recognition and other processing in the recognition unit 202. At that time, the frame data transmitted from the camera module 511 to the SoC / ECU 291 may have vibration information used to generate this frame data added to it. Also, the recognition unit 202 may be, for example, the recognition unit 73 included in the driving assistance / automatic driving control unit 29 in Figure 1.

[0164] Furthermore, in the example shown in Figure 12, the vibration correction unit 201 is located on the SoC / ECU 291 in a configuration similar to that shown in Figure 11. The IMU 271 also inputs the detected vibration information to the image sensor 100. In this configuration, the image sensor 100 adds the vibration information input from the IMU 271 to the frame data at the same or approximately the same timing as the frame data generation and transmits it to the SoC / ECU 291. At that time, the image sensor 100 (for example, the signal processing unit 108) may convert the vibration information input from the IMU 271 into information that is easy for the SoC / ECU 291 to process and add it to the frame data.

[0165] The shake correction unit 201, which receives frame data with added shake information from the camera module 511, generates shake-corrected frame data by performing EIS based on the shake information added to each frame data. The generated frame data is input to the recognition unit 202 and used for image recognition and other processing in the recognition unit 202.

[0166] The vibration correction unit 201 may also perform processing such as MCTF in addition to vibration correction. Furthermore, the vibration correction unit 201 may also perform processing to correct focal plane distortion in addition to vibration correction.

[0167] Next, we will explain the case where the above-described vibration correction is applied to the basic operation (4-cycle sequence) according to this embodiment. Figure 13 is a schematic diagram for explaining the case where vibration correction is applied to the basic operation according to this embodiment, where (a) is a diagram showing vibration correction when the 4-cycle sequence is not applied, and (b) is a diagram showing vibration correction when the 4-cycle sequence is applied. In Figure 13, the case in which the 1-frame period Tf from time t to time t+1 is divided into four equal periods, and each of these periods is defined as 1 cycle period Tc.

[0168] As shown in Figure 13(a), when the 4-cycle sequence according to this embodiment is not applied, blur occurs in both the frame data at time t and the frame data at the next time t+1. Therefore, the shake-corrected frame data output as the output frame at time t+1 retains the blur that occurred in both the frame data at time t and the frame data at the next time t+1.

[0169] In contrast, as shown in Figure 13(b), when the blur in the frame data at each time point is reduced by applying the 4-cycle sequence according to this embodiment, the blur in the shake-corrected frame data output as the output frame at time t+1 is also reduced. This makes it possible to improve the accuracy of image recognition, etc. Furthermore, the signal-to-noise ratio of each frame data may be improved by performing MCTF on each frame data.

[0170] 1.7 About Motion Vector Search (ME) Normally, the position of a vanishing point in an image is determined based on motion vectors obtained for each block into which the image is divided into small sections. On the other hand, in the case of frame data acquired by a camera mounted on a vehicle, such as the camera 51 in this embodiment, the direction of travel of the vehicle is generally constant, so the position of the vanishing point does not change much between preceding and succeeding frame data. Also, the motion vector within each frame data is a vector along the radial direction centered on the vanishing point. Therefore, in this embodiment, the search range for motion vectors may be limited based on the position of the vanishing point. This makes it possible to significantly reduce the search time for motion vectors and improve their accuracy. The motion vector search (ME) in this embodiment may be performed, for example, in the signal processing unit 108 in Figure 3, or in other units such as the driving assistance / automatic driving control unit 29 (see Figure 1).

[0171] Figures 14 to 16 illustrate the motion vector search operation according to this embodiment. Figure 14 shows the frame data at time t-1 (referred to as the previous frame), and Figure 15 shows the frame data at time t (referred to as the current frame). Figure 16 illustrates the search range for the motion vector relative to the current frame.

[0172] As shown in Figure 14, for the frame before time t-1, if two or more regions of interest (for example, regions of the subject in the image (corresponding to bounding boxes or regions used for block matching when calculating optical flow)) R01 and R02 have motion vectors V01 and V02 respectively, the position of the vanishing point LP0 in the previous frame is determined based on the intersection of the extensions L01 and L02 of the respective motion vectors V01 and V02. If there are three or more motion vectors, the centroid of the intersection or the center of the distribution range may be determined as the vanishing point.

[0173] The position of the vanishing point LP0 of the previous frame, as identified in this way, is set, for example, as the position of the vanishing point LP1 of the next frame, the current frame (time t). In this embodiment, lines L11 and L12 are set connecting the vanishing point LP1 to the center points O11 and O12 of the regions of interest R11 and R12, respectively, in the current frame.

[0174] In this way, by setting lines L11 and L12 connecting the vanishing point LP1 to the center points O11 and O12 respectively, the search range for the motion vectors of each of the target regions R11 and R12 is determined based on lines L11 and L12, as shown in Figure 22. Specifically, the search range is set based on lines L01 and L02, which are obtained by projecting lines L11 and L12 onto the same coordinates in the previous frame, and the motion vectors V11 and V12 of regions R11 and R12 are calculated by searching for the regions corresponding to each of the target regions R11 and R12 within that range. The width of the search range may be determined based on, for example, the maximum value of the predicted motion vector (e.g., the maximum scalar value of the motion vector).

[0175] In addition, the width of the search range may be set based on, for example, the predicted computational load (for example, by setting constraints on the amount of computation that may occur and narrowing the search range so as not to exceed that amount of computation), the camera's installation position (for example, the front or side of vehicle 1), the distance to objects captured in the area of ​​interest (for example, the further away the object is, the narrower the search range can be), or information such as vehicle speed (for example, the slower the vehicle speed, the narrower the search range can be). However, the width of the search range may be determined based on various parameters, but is not limited to these.

[0176] However, the position of vanishing point LP1 in the current frame is not limited to the position of vanishing point LP0 in the frame immediately preceding the current frame (corresponding to the previous frame mentioned above), but may be determined based on the positions of vanishing points in two or more frames prior to the current frame. For example, the position of the vanishing point in the current frame may be determined or estimated based on the amount and direction of movement of the vanishing points in two or more frames prior to the current frame.

[0177] Furthermore, the position of the vanishing point changes depending on the direction of travel and steering conditions of vehicle 1. Therefore, the position of the vanishing point LP1 in the current frame may be estimated by taking into consideration the direction of travel and steering conditions of vehicle 1. For example, if vehicle 1 is turning to the left, the position of the vanishing point in the frame data shifts to the left in the image. Therefore, the position of the vanishing point in the current frame may be estimated based on the position of the vanishing point determined in the previous frame and the turning direction and turning radius of vehicle 1.

[0178] Figure 17 is a block diagram showing an example configuration for realizing motion vector search according to this embodiment. As shown in Figure 17, the configuration for realizing motion vector search according to this embodiment includes, for example, a vanishing point position estimation unit 211, an optical flow calculation unit 212, and a vanishing point calculation unit 213. Note that the configuration for realizing motion vector search may be realized, for example, in the signal processing unit 108 in Figure 3, or in another unit such as the driving support / automatic driving control unit 29 (see Figure 1).

[0179] The vanishing point position estimation unit 211 estimates the position of the vanishing point in the current frame based on the input vehicle information of vehicle 1 and vanishing point information identified in one or more previous frames (hereinafter referred to as past vanishing point information). The vehicle information may include at least one of the following: odometry information such as steering angle (angle of steering wheel rotation, etc.), IMU detection results, vehicle body information (overall length, width, height, etc.), camera installation position and its attitude, vehicle speed, and turn signal operation information. Past vanishing point information may also be stored in a memory (not shown) of the vanishing point position estimation unit 211 (for example, a data storage unit 109).

[0180] Information regarding the vanishing point position estimated by the vanishing point position estimation unit 211 is input to the optical flow calculation unit 212 along with the frame data of the current frame. The optical flow calculation unit 212 calculates the motion vector (optical flow) for the current frame, for example, using the method described above with reference to Figures 14 to 16. The optical flow calculation unit 212 then inputs the calculated motion vector to the vanishing point calculation unit 213. The calculated motion vector may be input to, for example, the signal processing unit 108 of the image sensor 100 and used for alignment between frame data or cycle data, or for adding cycle data within a single frame period (DOL synthesis) in the signal processing unit 108.

[0181] With the above configuration, when performing cycle data addition (DOL synthesis), it becomes possible to use motion vectors calculated based on the previous frame and the current frame, each with the flicker component removed, in the motion vector, thereby improving the alignment accuracy of the flicker region. However, since the frame interval is longer than the cycle interval (for example, four times longer if the 1-frame period is divided into four equal parts as described above), the motion vector used in cycle data addition (DOL synthesis) may be a reduced vector of the motion vector calculated by the optical flow calculation unit 212 (for example, a vector obtained by multiplying the motion vector calculated by the optical flow calculation unit 212 by Tc / Tf). Furthermore, the motion vector may be calculated not only between frames but also between each cycle.

[0182] However, with this method, motion vectors are calculated between frames or cycles. Therefore, if the direction of travel of vehicle 1 changes, for example, in areas where the subject periodically flickers, such as the flicker region, the motion vector in that region cannot be corrected according to the change in the direction of travel of vehicle 1, and errors may accumulate.

[0183] Therefore, in this embodiment, the flicker region, including its surrounding region, may be excluded from the motion vector search. In this case, the surrounding region may be determined based on the maximum movement of the subject between cycles (e.g., the maximum scalar value of the motion vector). In this case, the motion vector of the region excluded from the motion vector search may be interpolated, for example, using the motion vector of a neighboring region. Furthermore, the correction applied to the motion vector interpolated using the motion vector of a neighboring region may be, for example, a correction applied to the motion vector in a previous frame.

[0184] The vanishing point calculation unit 213 determines the position of the vanishing point in the current frame based on the motion vector input from the optical flow calculation unit 212. For example, as described above, the vanishing point calculation unit 213 determines the position of the vanishing point in the current frame based on the intersection of the extensions of each of the motion vectors calculated for the current frame. The vanishing point position determined in this way is input to the vanishing point position estimation unit 211 and used to estimate the vanishing point position in the next frame. The determined vanishing point position may also be output to an external device, such as the driving support / automatic driving control unit 29, along with the frame data of the current frame.

[0185] The region of interest targeted for motion vector search may be determined by region determination on the frame data (for example, region determination performed during noise reduction in the signal processing unit 108, or other region determination such as object recognition), but is not limited to this. For example, if the camera 51 employs image plane phase-detection autofocus (ZAF), the frame data may be segmented based on the circle of confusion observed by ZAF, and the region determined in this way may be designated as the region of interest.

[0186] Furthermore, in the search for motion vectors for a region of interest, if distance information to the subject can be obtained using ZAF, radar 52, LiDAR 53, ultrasonic sensor 54, etc., the motion vectors may be estimated based on the distance information obtained using these devices.

[0187] 1.8 Summary As described above, this embodiment makes it possible to simultaneously achieve the generation of images unaffected by the flicker of a flicker light source and the detection of the flicker light source. Furthermore, this embodiment makes it possible to suppress blur to an extent that does not affect image recognition. Moreover, this embodiment makes it possible to suppress the deterioration of image quality in low light conditions. Furthermore, this embodiment makes it possible to realize a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device that are capable of capturing images with a high dynamic range. Furthermore, this embodiment makes it possible to realize a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device that can be used in combination for detection and viewing purposes. Furthermore, this embodiment makes it possible to realize a solid-state imaging device, a control method for the solid-state imaging device, and a control program for the solid-state imaging device that can be expected to improve the accuracy of image recognition.

[0188] 1.9 Variations In the embodiments described above, the exposure time for generating one cycle data is shortened by dividing one frame period into multiple cycle periods, and the cycle data read out in each cycle period is added together (DOL synthesis) to generate frame data with reduced blur and improved signal-to-noise ratio. However, the method is not limited to this. Therefore, the following describes embodiments that can achieve blur reduction and improved signal-to-noise ratio using methods different from those described above. In the following description, redundant explanations of configurations and operations similar to those in the embodiments described above will be omitted by referencing them.

[0189] Typically, areas in frame data where significant blurring occurs are those where the distance from the image sensor 100 to the subject is short. In such areas, sufficient brightness can be ensured, thus enabling a good signal-to-noise ratio even with short exposure times.

[0190] On the other hand, in areas where the distance to the subject is long, a long exposure time is required to ensure a sufficient signal-to-noise ratio, while blur is less likely to occur because the subject's movement speed within the field of view is slow.

[0191] Therefore, in this modified example, as shown in Figure 18, the frame period Ts is divided into a cycle period Tc1 with a long exposure time and a cycle period Tc2 with a short exposure time. For regions where the distance to the subject is short, the image data P3 read out in cycle period Tc2 is used, and for other regions (for example, regions where the distance to the subject is long), the image data P2 read out in cycle period Tc1 is used to generate frame data. As a result, blur is suppressed with a good signal-to-noise ratio in regions where the distance to the subject is short, and frame data with a small blur and a sufficient signal-to-noise ratio can be generated in other regions (for example, regions where the distance to the subject is long).

[0192] Furthermore, the separation of areas close to the subject from other areas may be performed, for example, based on the motion vector calculated in the previous frame. That is, subjects close to the image sensor 100 move faster within the field of view, so their motion vectors are larger, while the motion vectors of subjects farther from the image sensor 100 are smaller. Therefore, by separating the areas based on the magnitude (scalar quantity) of the motion vectors, it is possible to separate areas close to the subject from other areas.

[0193] In this modified example, the case where one frame period Ts is divided into two cycle periods, Tc1 with a long exposure time and Tc2 with a short exposure time, is illustrated. However, the example is not limited to this, and one frame period Ts may be divided into three or more cycle periods. In that case, the image data for areas close to the subject may be generated using image data read in one cycle period, while the image data for areas far from the subject may be generated by adding (DOL synthesis) image data read in two or more cycle periods.

[0194] Other configurations, operations, and effects may be the same as those of the embodiments described above, so a detailed explanation is omitted here.

[0195] 2. Hardware Configuration The recognition unit 120 according to the embodiments, modifications thereof, and application examples described above can be realized by a computer 1000 having a configuration such as that shown in Figure 19. Figure 19 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device constituting the recognition unit 120. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0196] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0197] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0198] HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by such programs. Specifically, HDD1400 is a recording medium that records a projection control program according to this disclosure, which is an example of program data 1450.

[0199] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0200] The input / output interface 1600 includes the I / F unit 18 described above and is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as keyboards and mice via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as displays, speakers, and printers via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0201] For example, the CPU 1100 of computer 1000 functions as the recognition unit 120 according to the above embodiment by executing a program loaded onto RAM 1200. The HDD 1400 stores the program and other data related to this disclosure. While the CPU 1100 reads and executes program data 1450 from HDD 1400, as an alternative, these programs may be obtained from other devices via an external network 1550.

[0202] While embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0203] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0204] Furthermore, this technology can also be configured as follows. (1) A pixel array section comprising multiple pixels, A control unit that reads first image data from the pixel array unit in each of multiple cycles within one frame, A processing unit that generates a second image data for one frame based on a plurality of first image data read within the one frame, A solid-state imaging device equipped with the following features. (2) Each of the cycles includes a first period for accumulating the charge generated in the pixel and a second period for discarding the charge generated in the pixel. The control unit generates the first image data based on the image data read from the pixel array unit during the first period and the image data read from the pixel array unit during the second period. The solid-state imaging device described in (1) above. (3) The control unit generates the first image data based on the image data read from the pixel array unit and the previously stored image data from when the image was not exposed. The solid-state imaging device described in (1) above. (4) The processing unit generates the second image data by adding the plurality of first image data. A solid-state imaging device as described in any one of (1) to (3) above. (5) The processing unit generates the second image data by adding the plurality of first image data at a predetermined ratio. The solid-state imaging device described in (4) above. (6) The processing unit generates the second image data by adding the plurality of first image data so that the ratio of each of the plurality of first image data is equal. The solid-state imaging device described in (4) or (5) above. (7) The processing unit generates the second image data by compressing and decompressing each of the plurality of first image data and then adding them together. A solid-state imaging device as described in any one of (4) to (6) above. (8) The compression is PWL compression. The solid-state imaging device described in (7) above. (9) The duration of each of the aforementioned multiple cycles is the same. A solid-state imaging device as described in any one of (1) to (8) above. (10) Within the aforementioned frame, the duration of at least one of the plurality of cycles is different from the duration of the other cycles. A solid-state imaging device as described in any one of (1) to (8) above. (11) The control unit reads out the third image data from the pixel array unit at a period different from the period of the frame. The processing unit detects a flicker region that periodically flashes based on the third image data. A solid-state imaging device as described in any one of (1) to (10) above. (12) The processing unit corrects the position of the subject region included in the plurality of first image data, and generates the second image data based on the plurality of first image data after the correction. A solid-state imaging device as described in any one of (1) to (11) above. (13) The processing unit corrects the position of the subject in the plurality of first image data based on the motion vector of the subject's region. The solid-state imaging device described in (12) above. (14) The motion vector of the subject region is a vector based on motion vectors calculated based on second image data from one or more frames prior to the current frame. The solid-state imaging device described in (13) above. (15) The processing unit identifies a vanishing point in the second image data of the previous frame and calculates a motion vector of the region of the subject included in the second image data of the current frame based on the identified vanishing point. The solid-state imaging device described in (13) or (14) above. (16) The processing unit limits the range in which the motion vector is searched based on a straight line connecting the vanishing point and the center of the subject's region, and calculates the motion vector by searching for the motion vector within the limited range. The solid-state imaging device described in (15) above. (17) The processing unit corrects the position of the region of the subject included in the plurality of first image data based on acceleration and / or angular velocity information input from an external source. A solid-state imaging device as described in any one of (12) to (16) above. (18) The processing unit sets the region of the subject based on at least one of the image plane phase difference when the subject is imaged and the distance to the subject detected by an external sensor when the subject is imaged. A solid-state imaging device as described in any one of (12) to (17) above. (19) The aforementioned multiple cycles are 4 cycles. A solid-state imaging device as described in any one of (1) to (18) above. (20) At least one of the plurality of cycles includes a third period that accumulates the charge generated in the pixel and is shorter than the first and second periods. A solid-state imaging device as described in any one of (1) to (19) above. (twenty one) In each of the multiple cycles within a single frame, the first image data is read from a pixel array unit having multiple pixels. Based on the multiple first image data read within the aforementioned frame, a second image data of the aforementioned frame is generated. A control method for a solid-state imaging device, including the following. (twenty two) The process involves reading out first image data from a pixel array unit having multiple pixels in each of the multiple cycles within a single frame, A step of generating a second image data for one frame based on a plurality of first image data read within the first frame, A control program for a solid-state imaging device that causes a computer to execute a command. [Explanation of symbols]

[0205] 100 Solid-state imaging device 101 Pixel Array Section 102 Vertical drive circuit 103 Column Processing Circuit 104 Horizontal drive circuit 105 System Control Unit 108 Signal Processing Unit 109 Data Storage Unit 111, 124, 134, 147 Data compression section 112, 121, 125, 131, 135, 141, 144, 145 memory 113, 126, 136 Data Expansion Section 114, 122, 127, 132, 137, 142 dividers 123, 133, 143 adder 146 HDR composite section 201 Vibration Correction Unit 202 Recognition part 211 Vanishing point position estimation unit 212 Optical Flow Calculation Unit 213 Vanishing point calculation section 271 IMU 291 SoC / ECU 511 Camera Module

Claims

1. A pixel array section comprising multiple pixels, A control unit that reads first image data from the pixel array unit in each of multiple cycles within one frame, A processing unit that generates a second image data for one frame based on a plurality of first image data read within the first frame, Equipped with, The processing unit corrects the position of the subject in the plurality of first image data based on the motion vector of the subject's region included in the plurality of first image data, and generates the second image data by adding the corrected plurality of first image data so that the ratio of each of the plurality of first image data is equal. Each of the cycles includes a first period for accumulating the charge generated in the pixel and a second period for discarding the charge generated in the pixel. The frequency of the cycle is set to a frequency that is different from a power of two of the frame frequency, and the sum of the first periods of the plurality of cycles within one frame is set to be more than half of the frame period. Solid-state imaging device.

2. The control unit generates the first image data based on the image data read from the pixel array unit during the first period and the image data read from the pixel array unit during the second period. The solid-state imaging apparatus according to claim 1.

3. The duration of each of the aforementioned multiple cycles is the same. The solid-state imaging apparatus according to claim 1.

4. Solid-state imaging device, In each of the multiple cycles within a single frame, the first image data is read from a pixel array unit having multiple pixels. Based on the multiple first image data read within the first frame, a second image data of the first frame is generated. This includes, In generating the second image data, Based on the motion vector of the subject's region included in the plurality of first image data, the position of the subject in the plurality of first image data is corrected, and the second image data is generated by adding the corrected plurality of first image data so that the ratio of each of the plurality of first image data is equal. Each of the cycles includes a first period for accumulating the charge generated in the pixel and a second period for discarding the charge generated in the pixel. The frequency of the cycle is set to a frequency that is different from a power of two of the frame frequency, and the sum of the first periods of the plurality of cycles within one frame is set to be more than half of the frame period. A method for controlling a solid-state imaging device.

5. The process involves reading first image data from a pixel array unit having multiple pixels in each of the multiple cycles within a single frame, A step of generating a second image data for one frame based on a plurality of first image data read within the first frame, A control program for a solid-state imaging device to be executed by a computer, In the process of generating the second image data, Based on the motion vector of the subject's region included in the plurality of first image data, the position of the subject in the plurality of first image data is corrected, and the second image data is generated by adding the corrected plurality of first image data so that the ratio of each of the plurality of first image data is equal. Each of the cycles includes a first period for accumulating the charge generated in the pixel and a second period for discarding the charge generated in the pixel. The frequency of the cycle is set to a frequency that is different from a power of two of the frame frequency, and the sum of the first periods of the plurality of cycles within one frame is set to be more than half of the frame period. Control program for a solid-state imaging device.

Images