Imaging device, image processing method, and image processing program

Abstract

The present invention suppresses delays in transmission and recognition processing and suppresses processing costs. An imaging device according to an embodiment comprises: a determination unit (108) which determines, on the basis of the focused degrees of images acquired by imaging units (101, 106), two or more regions including first and second regions in the imaging regions of the imaging units; and a processing unit (108) which generates a first image based on the first region and a second image based on the second region, wherein the first image has higher resolution than the second image.

Description

Technical Field

[0001] The present disclosure relates to an imaging device, an image processing method, and an image processing program.

Background Art

[0002] In recent years, with the spread of the automation of moving objects such as automobiles and robots and the Internet of Things (IoT), there has been a strong demand for faster and more accurate image recognition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the increase in the resolution of imaging devices, the amount of image data has increased dramatically. As a result, the amount of data to be processed in image recognition and the amount of data transferred from the imaging device to a recognizer or the like have increased, resulting in problems such as an increase in cost and a lengthening of processing time.

[0005] Therefore, the present disclosure proposes an imaging device, an image processing method, and an image processing program capable of reducing the amount of data.

Means for Solving the Problems

[0006] In order to solve the above problems, an imaging device according to one aspect of the present disclosure includes a determination unit that determines two or more regions including a first region and a second region with respect to an imaging region of the imaging unit based on the degree of focus of an image acquired by the imaging unit, and a processing unit that generates a first image based on the first region and a second image based on the second region, wherein the first image has a higher resolution than the second image.

Brief Description of the Drawings

[0007] [Figure 1] This is a block diagram showing an example configuration of a vehicle control system. [Figure 2] This figure shows an example of a sensing area. [Figure 3] This is a block diagram showing a schematic configuration example of an image sensor according to one embodiment. [Figure 4] This is a diagram (part 1) illustrating general recognition processes. [Figure 5] This is a diagram (part 2) illustrating general recognition processes. [Figure 6] This is a block diagram illustrating an image processing apparatus that determines the ROI outside the chip according to one embodiment. [Figure 7] This block diagram shows an overview of an image sensor when determining the ROI within the chip according to one embodiment. [Figure 8] This figure illustrates a method of binning pixels according to one embodiment. [Figure 9] This figure illustrates a method of binning using an operation according to one embodiment. [Figure 10] This is a schematic diagram (part 1) showing an example of the focusing position of an imaging lens according to one embodiment. [Figure 11] This is a schematic diagram (part 2) showing an example of the focusing position of an imaging lens according to one embodiment. [Figure 12] This is a schematic diagram (part 3) showing an example of the focusing position of an imaging lens according to one embodiment. [Figure 13] This is a diagram (part 1) illustrating the operation of determining the ROI using image plane phase difference according to one embodiment. [Figure 14] This is a diagram (part 2) illustrating the operation of determining the ROI using the image plane phase difference according to one embodiment. [Figure 15] This is a diagram (part 3) illustrating the operation of determining the ROI using the image plane phase difference according to one embodiment. [Figure 16]This is a diagram for explaining other operations when determining an ROI using image plane phase difference according to an embodiment. [Figure 17] This is a diagram for explaining other operations when determining an ROI using a vanishing point according to an embodiment. [Figure 18] This is a diagram for explaining an ROI determination method according to a second modification example of an embodiment. <0OO0074 [Figure 19] This is a diagram for explaining the operation of the area detection unit shown in FIG. 18. [Figure 20] This is a diagram (Part 1) for explaining a motion vector search operation according to an embodiment. [Figure 21] This is a diagram (Part 2) for explaining a motion vector search operation according to an embodiment. [Figure 22] This is a diagram (Part 3) for explaining a motion vector search operation according to an embodiment. [Figure 23] This is a block diagram showing a configuration example for realizing estimation of a vanishing point and search of a motion vector according to an embodiment. [Figure 24] This is a hardware configuration diagram showing an example of a computer for realizing the functions of an information processing apparatus according to the present disclosure.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] Also, the present disclosure will be described in accordance with the following item order. 0. First 1. One embodiment I.I Configuration example of vehicle control system 1.2 Configuration example of solid-state imaging device (image sensor) 1.3 Regarding reduction of data amount 1.4 Schematic configuration example of image processing apparatus 1.4.1 When determining an ROI outside the chip 1.4.2 When determining ROI within the chip 1.5 Determination of ROI using image plane phase difference 1.6 Generation of frame data when using image plane phase difference 1.6.1 Method for binning on pixels 1.6.2 How to perform binning using calculations 1.7 Focusing position when using image plane phase difference 1.8 Method for determining ROI using image plane phase difference 1.8.1 First Variation of ROI Determination Method 1.8.2 Second Variation of ROI Determination Method 1.9 Estimation of Vanishing Points and Motion Vector Search (ME) 1.10 Summary 2. Hardware Configuration

[0010] 0. Introduction Image sensors mounted on mobile devices such as in-vehicle cameras are increasingly using multiple pixels to improve recognition accuracy for objects at greater distances. However, the increased data volume resulting from higher pixel counts prolongs the time from image capture to recognition. Furthermore, in recognition processing, objects at close range do not require the same level of resolution as objects at greater distances. In addition, some recognition systems have a fixed image size as the processing unit, resulting in redundant processing such as downsampling the image data to reduce its size when processing images larger than that size. Thus, acquiring the entire image at the same high resolution required for distant objects is redundant for nearby objects, leading to increased costs and longer processing times.

[0011] Therefore, in the following embodiment, we propose an imaging device, an image processing method, and an image processing program that can reduce the amount of data.

[0012] 1. One Embodiment Hereinafter, one embodiment relating to this disclosure will be described in detail with reference to the drawings.

[0013] 1.1 Example of a vehicle control system configuration First, the mobile device control system according to this embodiment will be described. Figure 1 is a block diagram showing an example configuration of a vehicle control system 11, which is an example of the mobile device control system according to this embodiment.

[0014] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to driving assistance and autonomous driving of the vehicle 1.

[0015] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiver unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a driver monitoring system (DMS) 30, a human-machine interface (HMI) 31, and a vehicle control unit 32.

[0016] The vehicle control ECU 21, communication unit 22, map information storage unit 23, GNSS receiver unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driving support / automatic driving control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with one another via a communication network 41. The communication network 41 consists of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 41 may be used depending on the type of data to be communicated; for example, CAN is applied to data related to vehicle control, and Ethernet is applied to large-capacity data. In addition, each part of the vehicle control system 11 may be directly connected using wireless communication intended for relatively short-range communication, such as near-field communication (NFC) or Bluetooth (registered trademark), without going through the communication network 41.

[0017] In the following, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the processor 21 and the communication unit 22 communicating.

[0018] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0019] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 22 can communicate using multiple communication methods.

[0020] A brief explanation will be given regarding the external communication capabilities of the communication unit 22. The communication unit 22 communicates with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication methods such as 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 to communicate with the external network is not particularly limited, as long as it is a wireless communication method that enables digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0021] Furthermore, for example, the communication unit 22 can communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by mobile objects that move at relatively low speeds, such as pedestrians and cyclists, terminals that are fixedly installed in places such as stores, or MTC (Machine Type Communication) terminals. In addition, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and other entities, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with a house, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0022] The communication unit 22 can, for example, receive programs from an external source (over the air) to update the software that controls the operation of the vehicle control system 11. The communication unit 22 can also receive map information, traffic information, information about the vehicle 1's surroundings, etc., from an external source. Furthermore, the communication unit 22 can transmit information about the vehicle 1 and information about the vehicle 1's surroundings to an external source. Information about the vehicle 1 that the communication unit 22 transmits to an external source includes, for example, data indicating the status of the vehicle 1 and recognition results from the recognition unit 73. Furthermore, the communication unit 22 can also perform communications corresponding to vehicle emergency notification systems such as e-Call.

[0023] A brief overview of the communication capabilities of the communication unit 22 with the vehicle interior will be provided. The communication unit 22 can communicate with various devices in the vehicle, for example, using wireless communication. The communication unit 22 can communicate wirelessly with devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wireless communication, such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 can also communicate with various devices in the vehicle using wired communication. For example, the communication unit 22 can communicate with various devices in the vehicle via wired communication through a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with various devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link).

[0024] Here, "devices inside the vehicle" refers to, for example, devices inside the vehicle that are not connected to the communication network 41. Examples of devices inside the vehicle include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0025] For example, the communication unit 22 receives electromagnetic waves transmitted by road traffic information communication systems (VICS (Vehicle Information and Communication System) (registered trademark)) such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0026] The map information storage unit 23 stores either or both maps acquired from external sources and maps created by the vehicle 1. For example, the map information storage unit 23 stores high-precision 3D maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0027] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to vehicle 1 from an external server. A point cloud map is a map composed of point clouds (point cloud data). Here, a vector map refers to a map adapted for ADAS (Advanced Driver Assistance System) that maps traffic information such as the location of lanes and traffic lights to a point cloud map.

[0028] The point cloud map and vector map may be provided from, for example, an external server, or they may be created in the vehicle 1 as maps for matching with the local map described later, based on sensing results from radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred square meters relating to the planned route that the vehicle 1 will travel will be acquired from the external server.

[0029] The GNSS receiver 24 receives GNSS signals from GNSS satellites and acquires the position information of the vehicle 1. The received GNSS signals are supplied to the driving assistance / automatic driving control unit 29. The GNSS receiver 24 is not limited to using GNSS signals; for example, it may acquire position information using beacons.

[0030] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the external recognition sensor 25 are arbitrary.

[0031] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0032] The shooting method of camera 51 is not particularly limited as long as it is a shooting method capable of distance measurement. For example, camera 51 can be a camera of various shooting methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, as needed. In addition, camera 51 may simply be for acquiring images, regardless of distance measurement.

[0033] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment relative to the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a raindrop sensor, fog sensor, sunshine sensor, snow sensor, and illuminance sensor.

[0034] Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0035] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the in-vehicle sensor 26 are not particularly limited as long as the number can realistically be installed in the vehicle 1.

[0036] For example, the in-vehicle sensor 26 can be equipped with one or more sensors from among a camera, radar, seat sensor, steering wheel sensor, microphone, and biosensor. The camera equipped in the in-vehicle sensor 26 can be a camera of various imaging types capable of distance measurement, such as a ToF camera, stereo camera, monocular camera, or infrared camera. However, it is not limited to these, and the camera equipped in the in-vehicle sensor 26 may be one that simply acquires images, regardless of distance measurement. The biosensor equipped in the in-vehicle sensor 26 is installed, for example, on the seat or steering wheel, and detects various biometric information of the driver or other passengers.

[0037] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the vehicle sensor 27 are not particularly limited as long as the number can realistically be installed on the vehicle 1.

[0038] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0039] The recording unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 28 can be used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium can be a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 1 before and after an event such as an accident, as well as biometric information acquired by the in-vehicle sensors 26.

[0040] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0041] The analysis unit 61 performs analysis processing on the vehicle 1 and its surroundings. The analysis unit 61 comprises a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0042] The self-position estimation unit 71 estimates the vehicle's position based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on sensor data from the external recognition sensor 25 and estimates the vehicle's position by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel relative to the axle.

[0043] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around vehicle 1 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of vehicle 1 by the recognition unit 73.

[0044] The self-position estimation unit 71 may estimate the vehicle 1's own position based on the GNSS signal and sensor data from the vehicle sensor 27.

[0045] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and union.

[0046] The recognition unit 73 performs a detection process to detect the external conditions of the vehicle 1, and a recognition process to recognize the external conditions of the vehicle 1.

[0047] For example, the recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0048] Specifically, for example, the recognition unit 73 performs detection and recognition processing of objects around the vehicle 1. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0049] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering, which classifies point clouds based on sensor data from LiDAR 53 or radar 52 into clusters of point clouds. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0050] For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the velocity and direction of travel (movement vector) of objects around the vehicle 1 to be detected.

[0051] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., from the image data supplied from the camera 51. Alternatively, it may recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0052] For example, the recognition unit 73 can perform traffic rule recognition processing around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and the recognition result of objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the location and status of traffic signals, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on.

[0053] For example, the recognition unit 73 can perform recognition processing of the environment surrounding the vehicle 1. The surrounding environment that the recognition unit 73 is intended to recognize may include weather, temperature, humidity, brightness, and road surface conditions.

[0054] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route following processes.

[0055] Global path planning is the process of planning a rough route from the start to the goal. This path planning also includes a process called local path planning, which involves generating a path that allows vehicle 1 to travel safely and smoothly in its vicinity, taking into account the motion characteristics of vehicle 1 along the planned path. Path planning may be distinguished as long-term path planning, and starting point generation as short-term path planning or local path planning. Safety-prioritized routes represent a similar concept to starting point generation, short-term path planning, or local path planning.

[0056] Route following is the process of planning actions to safely and accurately travel the route planned by route planning within a planned time. The action planning unit 62 can, for example, calculate the target speed and target angular velocity of vehicle 1 based on the results of this route following process.

[0057] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0058] For example, the motion control unit 63 controls the steering control unit 81, brake control unit 82, and drive control unit 83, which are included in the vehicle control unit 32 described later, to perform acceleration / deceleration control and direction control so that the vehicle 1 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow driving, vehicle speed maintenance, collision warning for the vehicle, and lane departure warning for the vehicle. For example, the motion control unit 63 performs cooperative control for the purpose of autonomous driving, such as driving autonomously without driver operation.

[0059] The DMS30 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensors 26 and input data input to the HMI31, which will be described later. In this case, the driver status to be recognized by the DMS30 is expected to include, for example, physical condition, level of alertness, level of concentration, level of fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0060] Furthermore, the DMS30 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0061] HMI31 handles the input of various data and instructions, and presents various data to the driver and other users.

[0062] A brief explanation of data input by HMI31 is provided. HMI31 is equipped with an input device for human data input. HMI31 generates input signals based on data and instructions input by the input device and supplies them to each part of the vehicle control system 11. HMI31 is equipped with operators such as a touch panel, buttons, switches, and levers as input devices. However, HMI31 may also be equipped with input devices that allow information to be input by methods other than manual operation, such as voice or gestures. Furthermore, HMI31 may use external connected devices such as a remote control device using infrared or radio waves, or a mobile device or wearable device that corresponds to the operation of the vehicle control system 11, as input devices.

[0063] This section provides a brief overview of how HMI31 presents data. HMI31 generates visual, auditory, and tactile information for the occupant or those outside the vehicle. HMI31 also performs output control, managing the output, content, timing, and method of each of these generated pieces of information. As visual information, HMI31 generates and outputs information indicated by images and light, such as operation screens, vehicle status displays, warning displays, and monitor images showing the surroundings of vehicle 1. As auditory information, HMI31 generates and outputs information indicated by sound, such as voice guidance, warning sounds, and warning messages. Furthermore, as tactile information, HMI31 generates and outputs information that is perceived by the occupant's sense of touch through force, vibration, movement, etc.

[0064] As output devices for visual information output by HMI31, for example, a display device that presents visual information by displaying images itself, or a projector device that presents visual information by projecting images, can be applied. In addition to display devices with ordinary displays, the display device may also be a device that displays visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality. Furthermore, HMI31 can also use display devices such as the navigation system, instrument panel, CMS (Camera Monitoring System), electronic mirrors, and lamps installed in the vehicle 1 as output devices for visual information output.

[0065] For HMI31, output devices that output auditory information can include, for example, audio speakers, headphones, and earphones.

[0066] As an output device for HMI31 to output tactile information, for example, a haptic element using haptic technology can be applied. The haptic element is installed in parts of the vehicle 1 that are in contact with by the occupant, such as the steering wheel and the seat.

[0067] The vehicle control unit 32 controls various parts of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0068] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0069] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 also includes, for example, a control unit such as an ECU that controls the brake system.

[0070] The drive control unit 83 detects and controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a control unit such as an ECU that controls the drive system.

[0071] The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system.

[0072] The light control unit 85 detects and controls the status of various lights on the vehicle 1. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights.

[0073] The horn control unit 86 detects and controls the status of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn.

[0074] Figure 2 shows examples of sensing areas using the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 of the external recognition sensor 25 shown in Figure 1. In Figure 2, the vehicle 1 is schematically shown as viewed from above, with the left end being the front end of the vehicle 1 and the right end being the rear end of the vehicle 1.

[0075] Sensing regions 91F and 91B show examples of sensing regions of the ultrasonic sensor 54. Sensing region 91F covers the area around the front end of the vehicle 1 by multiple ultrasonic sensors 54. Sensing region 91B covers the area around the rear end of the vehicle 1 by multiple ultrasonic sensors 54.

[0076] The sensing results in sensing area 91F and sensing area 91B are used, for example, for parking assistance for vehicle 1.

[0077] Sensing areas 92F to 92B show examples of sensing areas for short-range or medium-range radar 52. Sensing area 92F covers a position further in front of vehicle 1 than sensing area 91F. Sensing area 92B covers a position further in rear of vehicle 1 than sensing area 91B. Sensing area 92L covers the rear periphery of the left side of vehicle 1. Sensing area 92R covers the rear periphery of the right side of vehicle 1.

[0078] The sensing results in sensing region 92F are used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region 92B are used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing regions 92L and 92R are used, for example, to detect objects in blind spots to the sides of vehicle 1.

[0079] Sensing areas 93F to 93B show examples of sensing areas by camera 51. Sensing area 93F covers a position further in front of vehicle 1 than sensing area 92F. Sensing area 93B covers a position further in rear of vehicle 1 than sensing area 92B. Sensing area 93L covers the periphery of the left side of vehicle 1. Sensing area 93R covers the periphery of the right side of vehicle 1.

[0080] The sensing results in sensing region 93F can be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 93B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing regions 93L and 93R can be used, for example, for surround view systems.

[0081] Sensing area 94 shows an example of the sensing area of ​​LiDAR 53. Sensing area 94 covers a position further in front of vehicle 1 than sensing area 93F. On the other hand, sensing area 94 has a narrower range in the left-right direction than sensing area 93F.

[0082] The sensing results in the sensing region 94 are used, for example, to detect objects such as surrounding vehicles.

[0083] Sensing area 95 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 95 covers a position further in front of vehicle 1 than sensing area 94. On the other hand, sensing area 95 has a narrower range in the left-right direction than sensing area 94.

[0084] The sensing results in sensing area 95 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0085] Furthermore, the sensing areas of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 included in the external recognition sensor 25 may take various configurations other than those shown in Figure 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1. Also, the installation positions of each sensor are not limited to the examples described above. In addition, there may be one or more sensors.

[0086] 1.2 Example Configuration of a Solid-State Imaging Device (Image Sensor) Next, an example of the configuration of a solid-state imaging device (hereinafter referred to as an image sensor) that constitutes the camera 51 of the external recognition sensor 25 will be described. The camera 51 may include an image sensor 100, which will be described later, and an optical system arranged with respect to the light-receiving surface (also called the imaging area) of the image sensor 100. Figure 3 is a block diagram showing a schematic configuration example of the image sensor according to this embodiment. Here, the image sensor may be an image sensor created, for example, by applying or partially using a CMOS process.

[0087] The image sensor 100 according to this embodiment has a stacked structure in which, for example, a semiconductor chip on which a pixel array section 101 is formed and a semiconductor chip on which peripheral circuits are formed are stacked. The peripheral circuits may include, for example, a vertical drive circuit 102, a column processing circuit 103, a horizontal drive circuit 104, and a system control unit 105.

[0088] The image sensor 100 further includes a signal processing unit 108 and a data storage unit 109. The signal processing unit 108 and the data storage unit 109 may be provided on the same semiconductor chip as the peripheral circuitry, or on a separate semiconductor chip.

[0089] The pixel array section 101 has a configuration in which pixels 110, each having a photoelectric conversion element that generates and stores an electric charge corresponding to the amount of light received, are arranged in a two-dimensional grid in the row and column directions, i.e., in a matrix. Here, the row direction refers to the arrangement direction of pixels in a pixel row (horizontal direction in the drawing), and the column direction refers to the arrangement direction of pixels in a pixel column (vertical direction in the drawing).

[0090] In the pixel array section 101, pixel drive lines LD are wired along the row direction for each pixel row in the matrix-like pixel arrangement, and vertical signal lines VSL are wired along the column direction for each pixel column. The pixel drive lines LD transmit drive signals for driving when reading signals from pixels. In Figure 3, the pixel drive lines LD are shown as individual wires, but they are not limited to one wire each. One end of the pixel drive line LD is connected to the output terminal corresponding to each row of the vertical drive circuit 102.

[0091] The vertical drive circuit 102 is composed of a shift register, an address decoder, and the like, and drives each pixel of the pixel array section 101 simultaneously or row by row. In other words, the vertical drive circuit 102, together with the system control unit 105 that controls the vertical drive circuit 102, constitutes a drive unit that controls the operation of each pixel of the pixel array section 101. The specific configuration of this vertical drive circuit 102 is not shown in the diagram, but generally it has two scanning systems: a read scanning system and a sweep scanning system.

[0092] The readout scanning system sequentially selects and scans the pixels 110 of the pixel array 101 row by row in order to read signals from the pixels. The signals read from the pixels 110 are analog signals. The sweep scanning system performs a sweep scan ahead of the readout scan performed by the readout scanning system by the exposure time.

[0093] This sweep scanning system resets the photoelectric converter by sweeping away unwanted charge from the photoelectric converter of pixel 110 in the readout row. By sweeping away (resetting) the unwanted charge with this sweep scanning system, a so-called electronic shutter operation is performed. Here, electronic shutter operation refers to the operation of discarding the charge of the photoelectric converter and starting a new exposure (starting charge accumulation).

[0094] The signal read out by the readout scanning system corresponds to the amount of light received since the previous readout operation or electronic shutter operation. The period from the readout timing of the previous readout operation or the sweep timing of the electronic shutter operation to the readout timing of the current readout operation is the charge accumulation period (also called the exposure period) at pixel 110.

[0095] The signals output from each pixel 110 of the pixel row selected and scanned by the vertical drive circuit 102 are input to the column processing circuit 103 through each of the vertical signal lines VSL for each pixel column. The column processing circuit 103 performs predetermined signal processing on the signals output from each pixel of the selected row through the vertical signal lines VSL for each pixel column of the pixel array section 101, and temporarily holds the pixel signals after signal processing.

[0096] Specifically, the column processing circuit 103 performs at least noise reduction processing as signal processing, such as CDS (Correlated Double Sampling) processing or DDS (Double Data Sampling) processing. For example, CDS processing removes pixel-specific fixed pattern noise such as reset noise and threshold variations of amplification transistors within pixels. In addition, the column processing circuit 103 also has an AD (analog-to-digital) conversion function, which converts the analog pixel signal read from the photoelectric conversion element into a digital signal and outputs it.

[0097] The horizontal drive circuit 104 consists of a shift register and an address decoder, and sequentially selects the readout circuits (hereinafter referred to as pixel circuits) corresponding to the pixel rows of the column processing circuit 103. Through this selective scanning by the horizontal drive circuit 104, the pixel signals processed for each pixel circuit in the column processing circuit 103 are output sequentially.

[0098] The system control unit 105 is composed of a timing generator that generates various timing signals, and controls the vertical drive circuit 102, column processing circuit 103, and horizontal drive circuit 104 based on the various timings generated by the timing generator.

[0099] The signal processing unit 108 has at least an arithmetic processing function and performs various signal processing, such as arithmetic processing, on the pixel signals output from the column processing circuit 103. The data storage unit 109 temporarily stores the data necessary for the signal processing performed by the signal processing unit 108.

[0100] Furthermore, the image data output from the signal processing unit 108 may be subjected to predetermined processing in, for example, the driving assistance / automatic driving control unit 29 of the vehicle control system 11 equipped with the image sensor 100, or transmitted to an external source via the communication unit 22.

[0101] 1.3 Regarding the reduction of data volume Next, we will explain the reduction of data volume according to this embodiment. Figures 4 and 5 are diagrams illustrating a general recognition process.

[0102] In typical recognition processes, processing is performed on image data read at a uniform resolution. Alternatively, the image data may be divided into regions, and processing may be performed on each of these regions.

[0103] As shown in Figure 4, in a normal readout operation, the region R1 that captures distant objects and the region R2 that captures nearby objects are read out at the same resolution. Therefore, for example, the region R2 that captures nearby objects will be read out at a finer resolution than the resolution required for recognition processing.

[0104] In such cases, typical recognition processes reduce the resolution of the entire image to an appropriate resolution. Alternatively, if region partitioning is performed on the image data, as shown in Figure 5, the image data G21 read from the partitioned region R2 is reduced to an appropriate resolution image data G22 or G23. This means that in data transfer for recognition purposes, unnecessary traffic is generated due to the difference between the amount of data in the RAW image data G21 and the amount of data in the image data G22 or G23 with a resolution suitable for recognition. It also means that redundant processing, specifically resolution reduction, occurs during the recognition process.

[0105] Therefore, in this embodiment, the region R1 that captures distant objects is read out at high resolution, and the other regions are read out at low resolution. This makes it possible to reduce the amount of data in the image data (frame data) output from the image sensor 100, and to omit the redundant process of reducing resolution, thereby suppressing increased costs and longer processing times.

[0106] 1.4 Example of a schematic configuration of an image processing device Next, an image processing apparatus according to this embodiment, which enables the reduction of the amount of data described above, will be explained. In this embodiment, for example, image data of the entire field of view (however, if the frame data is limited to a part of the field of view, then only that part of the field of view) is first read from the image sensor 100. Then, if there is a distant subject in the field of view, the area in which that distant object is captured (corresponding to the ROI (Region of Interest) described later) is read at high resolution. Subsequently, by integrating the low-resolution image and the high-resolution image (for example, by incorporating the high-resolution image into the ROI in the low-resolution image), image data (frame data) is generated in which the ROI in which the distant object is captured is represented at high resolution, and the other areas are represented at low resolution. Alternatively, the low-resolution image and the high-resolution image may not be integrated, but output as separate image data.

[0107] In this embodiment, the ROI may be determined outside the image sensor 100 (hereinafter also referred to as "outside the chip") or inside the image sensor 100 (hereinafter also referred to as "inside the chip"). The configurations for each case will be described below with examples.

[0108] 1.4.1 When ROI is determined outside the chip Figure 6 is a block diagram illustrating the overview of an image processing device when determining the ROI outside the chip. As shown in Figure 6, the image processing device comprises an image sensor 100 and a recognition unit 120.

[0109] The image sensor 100 corresponds to, for example, the camera 51 or in-vehicle sensor 26 described above using Figure 1, and generates and outputs image data. The output image data is input to the driving assistance / automatic driving control unit 29, etc., via a predetermined network, such as the communication network 41 described above using Figure 1.

[0110] The recognition unit 120 corresponds, for example, to the recognition unit 73 described above using Figure 1, and detects objects, backgrounds, etc., contained in the image by performing recognition processing on the image data input from the image sensor 100. Objects may include not only moving objects such as cars, bicycles, and pedestrians, but also fixed objects such as buildings, houses, and trees. The background, on the other hand, may be a wide area located in the distance, such as the sky, mountains, plains, or the sea.

[0111] Furthermore, the recognition unit 120 determines the regions of objects and backgrounds obtained as a result of the recognition processing of the image data as ROIs, which are regions of interest (ROIs) that are part of the effective pixel area in the pixel array unit 101. In addition, the recognition unit 120 determines the resolution of each ROI. The recognition unit 120 then notifies the image sensor 100 of the determined ROI and resolution information (hereinafter referred to as ROI-resolution information), thereby setting the ROIs to be read out and the resolution to be used when reading image data from each ROI in the image sensor 100.

[0112] The ROI information may, for example, be information regarding the address of the pixel that serves as the starting point of the ROI and its size in the vertical and horizontal directions. In this case, each ROI will be a rectangular region. However, it is not limited to this, and an ROI may be circular, elliptical, or polygonal, or it may be an irregularly shaped region identified by information specifying its boundary (contour). Furthermore, if the recognition unit 120 determines multiple ROIs, it may determine a different resolution for each ROI.

[0113] 1.4.2 When determining ROI within the chip Figure 7 is a block diagram illustrating the overview of an image sensor when the ROI is determined within the chip. In the configuration shown in Figure 7, the vertical drive circuit 102, column processing circuit 103, horizontal drive circuit 104, and system control unit 105 of the image sensor 100 illustrated in Figure 3 are combined into a control unit 106. In this explanation, the pixel array unit 101 and the control unit 106 are also referred to as the imaging unit.

[0114] In this example, the signal processing unit 108 identifies one or more regions within the field of view of a distant object based on image data read from the pixel array unit 101 and distance information input from external distance measuring sensors such as radar 52, LiDAR 53, and ultrasonic sensor 54, and determines each of the identified regions as an ROI. That is, the signal processing unit 108 functions as a determination unit that determines two or more regions, including one or more ROIs (also called a first region) for the effective pixel area of ​​the image sensor 100 and other regions (also called a second region). At this time, the signal processing unit 108 may also determine the resolution of each ROI. The signal processing unit 108 then sets the determined ROI and resolution information (ROI-resolution information) in the control unit 106. The ROI-resolution information may be the same as when the ROI is determined outside the chip.

[0115] However, the operation may be simplified when determining the ROI within the chip or when determining the ROI outside the chip. That is, if an ROI has already been set, the ROI may be read out at high resolution, then the area other than the ROI may be read out at low resolution, and the frame data may be generated by incorporating the previously read high-resolution image into the low-resolution image from which the ROI has been removed. Alternatively, the area other than the ROI may be read out at low resolution, then the ROI may be read out at high resolution, and the read high-resolution image may be incorporated into the low-resolution image from which the ROI has been removed to generate the frame data. Or, in a single readout operation, the ROI may be read out at high resolution and the other area (also called the second area) may be read out at low resolution to generate the frame data.

[0116] 1.5 Determination of ROI using image plane phase difference As described above, determining the ROI (Region of Interest) where a distant object is captured in image data can be performed based on image recognition of the frame data of the previous or current frame, or on distance information input from an external distance measuring sensor. However, if, for example, the pixel array unit 101 is equipped with pixels capable of detecting image plane phase difference, it is also possible to determine the ROI based on the image plane phase difference.

[0117] Here, a pixel capable of detecting image plane phase difference (hereinafter also referred to as an image plane phase difference pixel) is, for example, a pixel whose light-receiving surface is divided into left and right or top and bottom halves depending on its positional relationship with the optical center, with one area being shielded from light. By reading out a pair of pixels—one of which is shielded from light (top / bottom or left / right) and another adjacent pixel whose other is shielded from light (top / bottom or left / right)—and comparing their brightness, it is possible to detect the image plane phase difference. That is, when the image is in focus, the brightness values ​​detected by the two image plane phase difference pixels in a pair will be equal, and when the image is out of focus, there will be a difference in the brightness values ​​detected by the two image plane phase difference pixels. Image plane phase difference autofocus is a technique that uses this ratio of brightness values ​​(corresponding to the image plane phase difference) to focus on a specific subject. Furthermore, an image plane phase difference pixel may have the same configuration as a normal pixel 110, except that half of its light-receiving surface is shielded from light.

[0118] By arranging such image plane phase difference pixels in part or all of the pixel array 101, it becomes possible to calculate the amount of defocus in each region of the pixel array 101 based on the image plane phase difference, thereby making it possible to divide the field of view of the image sensor 100 into regions where the distance to the object is long and regions where it is not. When arranging image plane phase difference pixels in part of the pixel array 101, it is preferable that the image plane phase difference pixels be evenly distributed throughout the entire pixel array 101.

[0119] 1.6 Generation of frame data when using image plane phase difference This section describes the generation of frame data when using image plane phase difference imaging. When using image plane phase difference imaging, there are two possible methods for generating frame data that includes both high-resolution and low-resolution regions: one is to perform pixel binning before reading the image data (hereinafter referred to as the pixel binning method), and the other is to perform computational binning on the read image data (hereinafter referred to as the computational binning method). The following sections describe each generation method.

[0120] 1.6.1 Method for binning on pixels Figure 8 is a diagram illustrating a method of binning on pixels according to this embodiment. In Figure 8, the image plane phase difference pixels 111a and 111b are evenly distributed throughout the entire pixel array 101. Also in Figure 8, the case in which a quad-Bayer array color filter is used as the color filter, in which each color filter of the Bayer array is further divided into four parts. In this case, the image plane phase difference pixels 111a and 111b may be placed in two diagonally opposite 2x2 color filters that selectively transmit green light.

[0121] As shown in Figure 8, in the pixel binning method, first, reading is performed on the image plane phase difference pixels 111a and 111b in the pixel array unit 101. The pixel signals read from each image plane phase difference pixel 111a and 111b are input to the defocus amount calculation unit 181.

[0122] The defocus amount calculation unit 181 may be implemented, for example, in the signal processing unit 108. This defocus amount calculation unit 181 calculates the defocus amount of the region (which may include the surrounding region) where each pair of image plane phase difference pixels 111a and 111b is located by calculating the ratio of the brightness values ​​indicated by the pixel signals read from the paired image plane phase difference pixels 111a and 111b. Then, based on the calculated defocus amount, the defocus amount calculation unit 181 identifies the region in the pixel array unit 101 where a distant object is displayed and determines the identified region as the ROI. Then, the defocus amount calculation unit 181 sets the determined ROI (ROI information) in the control unit 106. As a result, a high-resolution image is read from the region where the ROI is set, and a low-resolution image is read from the region other than the ROI. Note that the order in which the high-resolution image and the low-resolution image are read does not matter. Furthermore, the high-resolution image read from the ROI may then be remosaiced, for example, in the signal processing unit 108, to convert it into, for example, Bayer array image data.

[0123] Furthermore, in this example, since the resolution of the images read from the ROI and the non-ROI regions is fixed, the defocus amount calculation unit 181 does not determine the resolution for the ROI and the non-ROI regions. However, if the resolution is not fixed, or if there are three or more resolution options, the defocus amount calculation unit 181 may determine the resolution for the ROI and the non-ROI regions together with the ROI.

[0124] As described above, the pixel-based binning method makes it possible to reduce the number of pixels to be driven and the number of AD conversion circuits in the column processing circuit 103 for low-resolution readout areas, thereby reducing driving power. Furthermore, for low-resolution images, it is possible to reduce the remosaicing process, which can lead to reduced processing load and reduced required memory capacity.

[0125] 1.6.2 How to perform binning using calculations Figure 9 is a diagram illustrating the binning method in the calculation according to this embodiment. In Figure 9, as in Figure 8, the image plane phase difference pixels 111a and 111b are evenly arranged throughout the pixel array 101, and a case in which a quad Bayer array color filter is used as the color filter is illustrated.

[0126] As shown in Figure 9, in the calculation-based binning method, first, a high-resolution image is read out targeting all pixels (however, within the effective pixel area) of the pixel array unit 101. The read-out image data is input to the defocus amount calculation unit 181.

[0127] For example, the defocus amount calculation unit 181 implemented in the signal processing unit 108 calculates the defocus amount of the region (which may include the surrounding region) where each pair of image plane phase difference pixels 111a and 111b is located by calculating the ratio of the pixel values ​​(corresponding to brightness values) of the pixels read from the input image data, which are paired image plane phase difference pixels 111a and 111b. Then, based on the calculated defocus amount, the defocus amount calculation unit 181 identifies the region in the pixel array unit 101 where a distant object is displayed and determines the identified region as the ROI. Then, the defocus amount calculation unit 181 sets the determined ROI (ROI information) in the signal processing unit 108. This signal processing unit 108 may be a processing unit other than the part of the signal processing unit 108 that functions as the defocus amount calculation unit 181.

[0128] The signal processing unit 108, which has an ROI set, performs remosaicing on the ROI in the high-resolution image read from the pixel array unit 101. The signal processing unit 108 also performs binning to reduce the resolution of areas other than the ROI in the high-resolution image read from the pixel array unit 101. For example, it converts a 4x4 quad-Bayer array to a 2x2 Bayer array.

[0129] As described above, the computational binning method allows the read operation on the pixel array 101 to be combined into a single read operation for high-resolution images, thereby simplifying the read sequence. Furthermore, for low-resolution images, the remosaic process can be reduced, leading to benefits such as reduced processing load and reduced memory requirements. In addition, since no blank areas (corresponding to ROI areas) occur in the low-resolution images when reading low-resolution and high-resolution images separately, image data processing can also be simplified.

[0130] 1.7 Focusing position when using image plane phase difference Next, we will explain the focus position of the optical system (hereinafter referred to as the imaging lens for simplicity of explanation) arranged on the light-receiving surface of the image sensor 100. Figures 10 to 12 are schematic diagrams showing examples of the focus position of the imaging lens. Figure 10 shows the case where the focus position is set at a predetermined distance (however, less than infinity) along the optical axis from the light-receiving surface of the image sensor 100. Figure 11 shows the case where the focus position is set at infinity. Figure 12 shows the case where the focus position is set at a distance greater than infinity (hereinafter also referred to as Over-INF).

[0131] As shown in Figure 10, in a typical image sensor, the focus position of the imaging lens is set so that the image is in focus from a predetermined distance (5 m in this example) along the optical axis from the imaging lens to infinity. The Near end of the acceptable circle of confusion is the lower limit of the focusable range, which is the position at a predetermined distance (5 m in this example) along the optical axis from the imaging lens, and the Far end is the upper limit of the focusable range, which is infinity. In this explanation, it is assumed that at the focus position, the size of the circle of confusion is smaller than the size of pixel 110, and in the range closer to the Near end of the acceptable circle of confusion and further from the Far end, the size of the circle of confusion is larger than the size of pixel 110.

[0132] Furthermore, as illustrated in Figure 11, for example, if the focus position of the imaging lens is set to a more distant position (infinity in Figure 11), the focusing range of the imaging lens can be shifted to a more distant position.

[0133] Therefore, in this embodiment, the focus position of the imaging lens is adjusted to define the boundary between distant objects and other objects. For example, as illustrated in Figure 12, if an object that is 20 m or more away from the light-receiving surface of the image sensor 100 along the optical axis is considered a distant object, the focus position of the imaging lens is set so that the lower limit of the focusable range is 20 m away from the light-receiving surface of the image sensor 100 along the optical axis. In this case, the focus position of the imaging lens may be further than infinity (Over-INF).

[0134] In this way, by setting the lower limit of the focusing range of the imaging lens to a position defined as far away, it becomes possible to determine whether an object is far away or closer than far away based on whether or not it is in focus, that is, whether or not an image plane phase difference occurs. In this case, an object that is in focus may be judged as an object located far away, and an object that is out of focus may be judged as an object located closer than far away. In this example, the position of the imaging lens relative to the light-receiving surface of the image sensor 100 is assumed to be fixed.

[0135] 1.8 Method for determining ROI using image plane phase difference Next, we will explain how to determine the ROI using image plane phase difference. In this explanation, we will illustrate the case where multiple image plane phase difference detection regions are evenly distributed across the entire pixel array 101. Note that an image plane phase difference detection region is a region in which one or more image plane phase difference pixels 111a and 111b are arranged, and may be a unit region for determining the distance to an object based on the image plane phase difference.

[0136] Figures 13-15 and 16 illustrate the operation of determining an ROI using image plane phase difference. For example, as shown in Figure 13, when multiple image plane phase difference detection regions 112 are evenly distributed across the entire pixel array 101, and as shown in Figure 14, when regions 112a where an image plane phase difference below a predetermined threshold is detected are distributed together, a rectangular region 113a encompassing all of the distributed regions 112a may be set as the ROI, as shown in Figure 15. Alternatively, as shown in Figure 16, a region 113b of a fixed or arbitrary size with the center 114 being the center or centroid of the distributed regions 112a may be set as the ROI. If the distribution of regions 112a is divided into two or more, an ROI may be set for each region 113a / 113b. If region 113b is of an arbitrary size, for example, the size of region 113b may be determined based on the distribution size of region 112a.

[0137] 1.8.1 First Variation of ROI Determination Method Furthermore, the ROI may be determined not by the image plane phase difference, but, for example, by a vanishing point. For example, as shown in Figure 17, a vanishing point 115 may be identified in the image data read from the pixel array unit 101, and a region 113c of a fixed or arbitrary size centered on the identified vanishing point may be set as the ROI.

[0138] 1.8.2 Second Variation of ROI Determination Method Furthermore, while the above example illustrates the determination of ROI based on image plane phase difference within the chip, it is also possible to determine ROI within the chip using different methods.

[0139] For example, when the Far end of the acceptable circle of confusion is set to infinity or beyond, a circle of confusion will normally occur in the image data read from the image sensor 100 in the range closer to the Near end of the acceptable circle of confusion. The size of this circle of confusion will increase the closer it is to the image sensor 100. In this embodiment, the edges of objects captured in the image data may be detected, for example, based on contrast. If the detected edges are clear, they may be determined to be distant objects, and if they are blurred, they may be determined to be nearby objects, and an ROI may be set accordingly.

[0140] Figure 18 is a diagram illustrating the ROI determination method according to the second modified example. Figure 19 is a diagram illustrating the operation of the area detection unit shown in Figure 18. In Figures 18 and 19, as with Figure 8, an example is shown in which a quad Bayer array color filter is used as the color filter.

[0141] As shown in Figure 18, the image data (RAW data) read from the pixel array unit 101 is input to, for example, the area detection unit 182 implemented in the signal processing unit 108. In the area detection unit 182, remosaic is performed for high-resolution imaging of the ROI, and binning is performed by calculation for lower resolution of the other areas.

[0142] As shown in Figure 19, the area detection unit 182 includes a conversion unit 1821, a buffer unit 1822, a smoothing unit 1824, edge extraction units 1823 and 1825, and a comparison unit 1826.

[0143] The conversion unit 1821 converts the RAW data read from the pixel array unit 101 into monochrome luminance image data. For example, the conversion unit 1821 interpolates the R and B pixels of the RAW data, which is composed of the RGB primary colors, with the values ​​of the surrounding G pixels to generate luminance image data in which all pixels are G pixels. The generated luminance image data is input to the buffer unit 1822 and the smoothing unit 1824, respectively.

[0144] The buffer unit 1822 temporarily holds the input luminance image data and then inputs it to the edge extraction unit 1823. The edge extraction unit 1823 extracts edges contained in the luminance image data, for example, by using a Laplacian filter, and inputs the results to the comparison unit 1826.

[0145] Meanwhile, the smoothing unit 1824 smooths the input luminance image data using filters such as moving averages or Gaussian filters, and inputs the smoothed luminance image data to the edge extraction unit 1825. The edge extraction unit 1825 extracts edges contained in the smoothed luminance image data using filters such as Laplacian filters, and inputs the results to the comparison unit 1826.

[0146] The comparison unit 1826 identifies out-of-focus areas by comparing edges extracted from unsmoothed luminance image data with edges extracted from smoothed luminance image data. In other words, in areas that are in focus in the RAW data (in-focus areas), smoothing makes the edges in the image smoother, while in out-of-focus areas (de-focus areas), the edges in the RAW data are already smooth, so smoothing does not change the edges in the image much. Therefore, when comparing edges extracted from unsmoothed luminance image data with edges extracted from smoothed luminance image data, in in-focus areas the ratio of edges extracted from unsmoothed luminance image data to edges extracted from smoothed luminance image data is greater than 1, while in de-focus areas this ratio is close to 1. The comparison unit 1826 then identifies areas where the ratio of edges extracted from unsmoothed luminance image data to edges extracted from smoothed luminance image data is close to 1 as de-focus areas and sets these identified areas as ROIs.

[0147] 1.9 Estimation of Vanishing Points and Motion Vector Search (ME) Here, we will explain the estimation of vanishing points and the search for motion vectors in image data. Normally, the position of a vanishing point in an image is determined based on line segments extracted from the image. On the other hand, in the case of frame data acquired by a camera mounted on a vehicle, such as the camera 51 in this embodiment, the direction of travel of the vehicle is generally constant, so the position of the vanishing point does not change much between preceding and succeeding frame data. Also, the motion vector in each frame data is a vector along the radial direction centered on the vanishing point. Therefore, in this embodiment, the position of the vanishing point in the current frame may be estimated based on the motion vector detected in the previous frame, and the search range for the motion vector may be limited based on the estimated position of the vanishing point. This makes it possible to significantly shorten the time for estimating the vanishing point and the time for searching for the motion vector, and to improve the accuracy. The motion vector search (ME) in this embodiment may be performed, for example, in the signal processing unit 108 in Figure 3, or in other units such as the driving assistance / automatic driving control unit 29 (see Figure 1).

[0148] Figures 20 to 22 are diagrams illustrating the motion vector search operation according to this embodiment. Figure 20 shows the frame data at time t-1 (referred to as the previous frame), and Figure 21 shows the frame data at time t (referred to as the current frame). Figure 22 is a diagram illustrating the search range for the motion vector relative to the current frame.

[0149] As shown in Figure 20, for the previous frame at time t-1, if two or more regions of interest (for example, regions of the subject in the image (corresponding to bounding boxes or regions used for block matching when calculating optical flow)) R01 and R02 have motion vectors V01 and V02 respectively, the position of the vanishing point LP0 in the previous frame is determined based on the intersection of the extensions L01 and L02 of the respective motion vectors V01 and V02. If there are three or more motion vectors, the centroid of the intersection or the center of the distribution range may be determined as the vanishing point.

[0150] The position of the vanishing point LP0 of the previous frame, as identified in this way, is set, for example, as the position of the vanishing point LP1 of the next frame, the current frame (time t). In this embodiment, lines L11 and L12 are set connecting the vanishing point LP1 to the center points O11 and O12 of the regions of interest R11 and R12, respectively, in the current frame.

[0151] In this way, by setting lines L11 and L12 connecting the vanishing point LP1 to the center points O11 and O12 respectively, the search range for the motion vectors of each of the target regions R11 and R12 is determined based on lines L11 and L12, as shown in Figure 22. Specifically, the search range is set based on lines L01 and L02, which are obtained by projecting lines L11 and L12 onto the same coordinates in the previous frame, and the motion vectors V11 and V12 of regions R11 and R12 are calculated by searching for the regions corresponding to each of the target regions R11 and R12 within that range. The width of the search range may be determined based on, for example, the maximum value of the predicted motion vector (e.g., the maximum scalar value of the motion vector).

[0152] In addition, the width of the search range may be set based on, for example, the predicted computational load (for example, by setting constraints on the amount of computation that may occur and narrowing the search range so as not to exceed that amount of computation), the camera's installation position (for example, the front or side of vehicle 1), the distance to objects captured in the area of ​​interest (for example, the further away the object is, the narrower the search range can be), or information such as vehicle speed (for example, the slower the vehicle speed, the narrower the search range can be). However, the width of the search range may be determined based on various parameters, but is not limited to these.

[0153] However, the position of vanishing point LP1 in the current frame is not limited to the position of vanishing point LP0 in the frame immediately preceding the current frame (corresponding to the previous frame mentioned above), but may be determined based on the positions of vanishing points in two or more frames prior to the current frame. For example, the position of the vanishing point in the current frame may be determined or estimated based on the amount and direction of movement of the vanishing points in two or more frames prior to the current frame.

[0154] Furthermore, the position of the vanishing point changes depending on the direction of travel and steering conditions of vehicle 1. Therefore, the position of the vanishing point LP1 in the current frame may be estimated by taking into consideration the direction of travel and steering conditions of vehicle 1. For example, if vehicle 1 is turning to the left, the position of the vanishing point in the frame data shifts to the left in the image. Therefore, the position of the vanishing point in the current frame may be estimated based on the position of the vanishing point determined in the previous frame and the turning direction and turning radius of vehicle 1.

[0155] Figure 23 is a block diagram showing an example configuration for realizing vanishing point estimation and motion vector search according to this embodiment. As shown in Figure 23, the configuration for realizing vanishing point estimation and motion vector search according to this embodiment includes, for example, a vanishing point position estimation unit 211, an optical flow calculation unit 212, and a vanishing point calculation unit 213. The configuration for realizing motion vector search may be realized, for example, in the signal processing unit 108 in Figure 3, or in another unit such as the driving support / automatic driving control unit 29 (see Figure 1).

[0156] The vanishing point position estimation unit 211 estimates the position of the vanishing point in the current frame based on the input vehicle information of vehicle 1 and vanishing point information identified in one or more previous frames (hereinafter referred to as past vanishing point information). The vehicle information may include at least one of the following: odometry information such as steering angle (angle of steering wheel rotation, etc.), IMU detection results, vehicle body information (overall length, width, height, etc.), camera installation position and its attitude, vehicle speed, and turn signal operation information. Past vanishing point information may also be stored in a memory (not shown) of the vanishing point position estimation unit 211 (for example, a data storage unit 109).

[0157] The vanishing point position information estimated by the vanishing point position estimation unit 211 is input to the optical flow calculation unit 212 along with the frame data of the current frame. The optical flow calculation unit 212 calculates the motion vector (optical flow) for the current frame, for example, using the method described above with reference to Figures 20 to 22. The optical flow calculation unit 212 then inputs the calculated motion vector to the vanishing point calculation unit 213.

[0158] The vanishing point calculation unit 213 determines the position of the vanishing point in the current frame based on the motion vector input from the optical flow calculation unit 212. For example, as described above, the vanishing point calculation unit 213 determines the position of the vanishing point in the current frame based on the intersection of the extensions of each of the motion vectors calculated for the current frame. The vanishing point position determined in this way is input to the vanishing point position estimation unit 211 and used to estimate the vanishing point position in the next frame. The determined vanishing point position may also be output to an external device, such as the driving support / automatic driving control unit 29, along with the frame data of the current frame.

[0159] The region of interest targeted for motion vector search may be determined by region determination of the frame data (for example, region determination performed during noise reduction in the signal processing unit 108, or other region determination such as object recognition), but is not limited to this. For example, if the camera 51 is equipped with an image sensor that has an image plane phase difference detection region, the frame data may be segmented based on the image plane phase difference, and the region determined by this segmentation may be designated as the region of interest.

[0160] Furthermore, in the search for motion vectors for a region of interest, if distance information to the subject can be obtained using ZAF, radar 52, LiDAR 53, ultrasonic sensor 54, etc., the motion vectors may be estimated based on the distance information obtained using these devices.

[0161] 1.10 Summary As described above, according to this embodiment, high-resolution images are acquired for areas of interest (ROI) where high resolution is required in processing such as image authentication, and images with a sufficiently high resolution (referred to as low resolution in this example) are acquired for other areas. This makes it possible to effectively reduce the amount of data in the image data. As a result, it is possible to suppress increased costs and prolonged processing times.

[0162] 2. Hardware Configuration The recognition unit 120 according to the embodiments, modifications thereof, and application examples described above can be realized by a computer 1000 having a configuration such as that shown in Figure 24. Figure 24 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device constituting the recognition unit 120. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0163] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0164] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0165] HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by such programs. Specifically, HDD1400 is a recording medium that records a projection control program according to this disclosure, which is an example of program data 1450.

[0166] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0167] The input / output interface 1600 includes the I / F unit 18 described above and is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as keyboards and mice via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as displays, speakers, and printers via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0168] For example, the CPU 1100 of computer 1000 functions as the recognition unit 120 according to the above embodiment by executing a program loaded onto RAM 1200. The HDD 1400 stores the program and other data related to this disclosure. While the CPU 1100 reads and executes program data 1450 from HDD 1400, as an alternative, these programs may be obtained from other devices via an external network 1550.

[0169] While embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0170] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0171] Furthermore, this technology can also be configured as follows. (1) A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. Imaging device. (2) The determination unit determines the degree of focus based on the size of the pixels in the imaging area and the size of the circle of confusion formed in the imaging area. The imaging device described in (1) above. (3) The imaging unit includes a plurality of image plane phase difference pixels capable of detecting the image plane phase difference of the image formed in the imaging region, The determination unit determines the two or more regions, including the first and second regions, based on the image plane phase difference detected by the plurality of image plane phase difference pixels. The imaging device described in (1) above. (4) The determination unit determines the two or more regions, including the first and second regions, by calculating the amount of defocus of the image based on the image plane phase difference. The imaging device described in (3) above. (5) The determination unit determines the two or more regions, including the first and second regions, by comparing the image with an image obtained by smoothing the image. The imaging device described in (1) above. (6) The first region is a region in which an object located at a greater distance from the imaging unit than the object captured in the second region is captured. An imaging device as described in any one of (1) to (5) above. (7) The imaging unit further comprises an optical system arranged with respect to the imaging area, The focusing position of the optical system is set to be at a predetermined distance or greater from the imaging unit. An imaging device as described in any one of (1) to (6) above. (8) The aforementioned focusing position is near infinity of the optical system. The imaging device described in (7) above. (9) The aforementioned focusing position is a position farther than infinity in the optical system. The imaging device described in (7) above. (10) The position of the optical system with respect to the imaging region is adjusted such that the near edge of the permissible circle of confusion, which is imaged onto the imaging region via the optical system, is located at the boundary that switches between the first region and the second region. The imaging device described in any one of (7) to (9) above. (11) The determination unit determines the region in which the circle of confusion formed by the optical system on the imaging region is smaller than the size of the pixels in the imaging region to be the first region. An imaging device as described in any one of (7) to (10) above. (12) The processing unit generates a third image data by integrating the first image and the second image. An imaging device as described in any one of (1) to (11) above. (13) The processing unit generates the first image by remosaicing the image data read from the first region in the imaging region. An imaging device as described in any one of (1) to (12) above. (14) The processing unit generates the second image by binning the pixels of the image data read from the second region in the imaging region. An imaging device as described in any one of (1) to (13) above. (15) The system further includes a control unit that performs a readout operation on the imaging unit, The control unit reads out the second image by binning the pixels included in the second region. An imaging device as described in any one of (1) to (13) above. (16) Based on the degree of focus of the image acquired by the imaging unit, two or more regions, including a first and a second region, are determined within the imaging area of ​​the imaging unit. A first image based on the first region and a second image based on the second region are generated. This includes, The first image has a higher resolution than the second image. Image processing methods. (17) A program for making the processor installed in an image processing device function, A step of determining two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A step of generating a first image based on the first region and a second image based on the second region, The processor is made to execute the above, The first image has a higher resolution than the second image. Image processing program. [Explanation of Symbols]

[0172] 100 Image Sensors 101 Pixel Array Section 102 Vertical drive circuit 103 Column Processing Circuit 104 Horizontal drive circuit 105 System Control Unit 106 Control Unit 108 Signal Processing Unit 109 Data Storage Unit 110 pixels 111a, 111b Image plane phase difference pixel 112 Image plane phase difference detection region 112a, 113a, 113b, 113c areas 114 center 120 Recognition part 181 Defocus Amount Calculation Unit 182 Area detection unit 211 Vanishing point position estimation unit 212 Optical Flow Calculation Unit 213 Vanishing point calculation section 1821 Conversion Unit 1822 Buffer section 1823, 1825 Edge extraction section 1824 Smoothing section 1826 Comparison Section

Claims

1. A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. The processing unit generates the first image by remosaicing the image data read from the first region in the imaging region. Imaging device.

2. The determination unit determines the degree of focus based on the size of the pixels in the imaging area and the size of the circle of confusion formed in the imaging area. The imaging apparatus according to claim 1.

3. The imaging unit includes a plurality of image plane phase difference pixels capable of detecting the image plane phase difference of the image formed in the imaging region, The determination unit determines the two or more regions, including the first and second regions, based on the image plane phase difference detected by the plurality of image plane phase difference pixels. The imaging apparatus according to claim 1.

4. The determination unit determines the two or more regions, including the first and second regions, by calculating the amount of defocus of the image based on the image plane phase difference. The imaging device according to claim 3.

5. A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. The determination unit determines the two or more regions, including the first and second regions, by comparing the image with an image obtained by smoothing the image. Imaging device.

6. The first region is a region in which an object located at a greater distance from the imaging unit than the object captured in the second region is captured. The imaging apparatus according to claim 1.

7. The imaging unit further comprises an optical system arranged with respect to the imaging area, The focusing position of the optical system is set to be at a predetermined distance or greater from the imaging unit. The imaging apparatus according to claim 1.

8. A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. The imaging unit further comprises an optical system arranged with respect to the imaging area, The focusing position of the optical system is set to a distance greater than a predetermined distance from the imaging unit. The aforementioned focusing position is near infinity of the optical system. Imaging device.

9. A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. The imaging unit further comprises an optical system arranged with respect to the imaging area, The focusing position of the optical system is set to a distance greater than a predetermined distance from the imaging unit. The aforementioned focusing position is a position farther than infinity in the optical system. Imaging device.

10. A determination unit that determines two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit based on the degree of focus of the image acquired by the imaging unit, A processing unit that generates a first image based on the first region and a second image based on the second region, Equipped with, The first image has a higher resolution than the second image. The imaging unit further comprises an optical system arranged with respect to the imaging area, The focusing position of the optical system is set to a distance greater than a predetermined distance from the imaging unit. The position of the optical system with respect to the imaging region is adjusted such that the near edge of the permissible circle of confusion, which is imaged onto the imaging region via the optical system, is located at the boundary that switches between the first region and the second region. Imaging device.

11. The determination unit determines the region in which the circle of confusion formed by the optical system on the imaging region is smaller than the size of the pixels in the imaging region to be the first region. The imaging apparatus according to claim 7.

12. The processing unit generates a third image data by integrating the first image and the second image. The imaging apparatus according to claim 1.

13. The processing unit generates the second image by binning the pixels of the image data read from the second region in the imaging region. The imaging apparatus according to claim 1.

14. The system further includes a control unit that performs a readout operation on the imaging unit, The control unit reads out the second image by binning the pixels included in the second region. The imaging apparatus according to claim 1.

15. Based on the degree of focus of the image acquired by the imaging unit, two or more regions, including a first and a second region, are determined within the imaging area of ​​the imaging unit. A first image based on the first region and a second image based on the second region are generated. This includes, The first image has a higher resolution than the second image. Furthermore, the method includes generating the first image by remosaicing the image data read from the first region in the imaging region, Image processing methods.

16. A program for making the processor installed in an image processing device function, A step of determining two or more regions, including a first and a second region, within the imaging area of ​​the imaging unit, based on the degree of focus of the image acquired by the imaging unit, A step of generating a first image based on the first region and a second image based on the second region, The processor is made to execute the above, The first image has a higher resolution than the second image. Furthermore, the processor is instructed to perform the step of generating a first image by remosaicing the image data read from the first region in the imaging region. Image processing program.

Images