Information processing system and information processing method

Abstract

[Problem] To provide an information processing system and an information processing method for generating an image effective for image recognition of a vehicle's moving environment. [Solution] An information processing system according to the present disclosure comprises: a visible light irradiation unit that is mounted on a vehicle and emits visible light toward a predetermined direction of the vehicle; an infrared light irradiation unit that is mounted on the vehicle and emits infrared light toward the predetermined direction of the vehicle; a control unit that controls the irradiation range of at least one of the visible light irradiation unit and the infrared light irradiation unit; an imaging unit that captures an image in the predetermined direction of the vehicle on the basis of light entering an imaging region and generates image data; and a processing unit that, on the basis of first information on the position of a first region of the imaging region which the visible light enters and second information on the position of a second region of the imaging region which the infrared light enters, executes separate image signal processing on the image data for each of the first region and the second region.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] Regulations require drivers of oncoming vehicles to use headlights that reduce glare during nighttime driving. One automotive lighting technology that provides high brightness while reducing glare for oncoming drivers is the adaptive front lighting system (AFS), which dynamically adjusts light distribution to reduce glare. AFS dynamically controls the light emitted by the vehicle to prevent glare from reaching the eyes of oncoming drivers during nighttime driving, thereby preventing and reducing accidents. More specifically, AFS is a type of dynamically adjustable structure installed in vehicle headlights. For example, when cornering, the headlight beam is aligned in the direction of steering, improving visibility by shining light in the direction of travel. Depending on the light distribution, headlight light can be dazzling to oncoming drivers, potentially blurring their vision and preventing them from maintaining the visibility necessary for normal driving. Therefore, in recent years, AFS has become popular that not only simply changes the light distribution dynamically to match the direction (direction) of the road curve, but also performs more complex dynamic light distribution to prevent headlight light from entering the eyes of drivers of oncoming vehicles or the driver's eyes through the rearview mirror of the vehicle in front. This makes it possible to provide a bright field of view to the driver of one's own vehicle while taking care to prevent light from entering the eyes of oncoming vehicles and other drivers of vehicles in front.

[0003] While such AFSs offer significant benefits, such as improving the driver's visual visibility while reducing glare for oncoming vehicles, the selective illumination described above results in a bias toward low beam headlights. As a result, the far-away area ahead of the vehicle may not be illuminated brightly enough with visible light. In particular, when AFSs are used in conjunction with autonomous driving and image recognition is performed using a camera, the lack of illumination in the far-away area results in dark images, resulting in a loss of image quality. For example, when driving at high speeds at night, the far-away area ahead of the vehicle may not be illuminated, resulting in dark images being acquired, making it difficult to accurately predict the vehicle's course.

[0004] On the other hand, imaging technology using cameras is limited by the dynamic range when capturing images of subjects with large differences in brightness, and is therefore limited in its ability to handle large differences in brightness from low to high. Typical examples of this are the limited transmission bandwidth and the brightness reproduction bandwidth of display devices. One of the technologies being introduced to address this issue is local tone mapping, an automatic image gradation adjustment technique.

[0005] Instead of the usual uniform input / output gain applied to the entire image, this technique performs scene recognition evaluation of the image and dynamically adjusts input / output gain for regions of different brightness using variable adaptive adjustment. Gain differences are absorbed at inconspicuous region boundaries, constraining the entire image to a limited bandwidth. However, simply applying local tone mapping results in an image dominated by noise in dark, distant roads. To compensate for this, a method for supplementing environmental recognition by providing brightness on distant, dark roads is to use active infrared (including near-infrared) illumination, a so-called night vision-equivalent supplemental lighting technique, which does not cause glare to the eyes. Illuminating with infrared light ensures brightness on distant, dark roads without dazzling oncoming drivers. However, in a situation where infrared light is dominant and illumination light is mixed, demosaic development of the detected signal image (raw image) of the image sensor, which is essential for distinguishing area and color information such as segmentation in image recognition, cannot be performed correctly due to sampling, resulting in side effects such as false colors and moire, which results in a problem of deterioration in the detection accuracy of the recognition process.

[0006] International Publication No. 2019 / 225349

[0007] The present disclosure provides an information processing system and an information processing method for generating images that are effective for image recognition of a vehicle's moving environment.

[0008] The information processing system of the present disclosure includes a visible light irradiation unit that is mounted on a vehicle and irradiates visible light toward the predetermined direction of the vehicle, an infrared light irradiation unit that is mounted on the vehicle and irradiates infrared light toward the predetermined direction of the vehicle, a control unit that controls the irradiation range of at least one of the visible light irradiation unit and the infrared light irradiation unit, an imaging unit that images the predetermined direction of the vehicle based on light incident on an imaging area and generates image data, and a processing unit that performs separate image signal processing on the image data for each of the first area and the second area based on first information regarding the position of a first area in the imaging area into which the visible light is incident and second information regarding the position of a second area into which the infrared light is incident.

[0009] 1 is a block diagram showing an example configuration of a vehicle control system according to the present disclosure. FIG. 2 is a diagram showing an example of a sensing area and an installation position of an external recognition sensor. FIG. 3 is a diagram showing a schematic functional configuration of an information processing system according to the present disclosure. FIG. 4 is a diagram showing an example of an imaging area. FIG. 5 is a diagram showing an example of a region illuminated according to a headlight illumination map and a region illuminated by an infrared light source. FIG. 6 is a diagram showing an example of mapping an infrared light incidence region and a headlight incidence region onto a coordinate system (camera view angle frame) of image data of a camera. FIG. 7 is a diagram showing an example of extending the periphery of an infrared light incidence region. FIG. 8 is a flowchart of an example operation of a calibration process controlled by a calibration processing unit. FIG. 9 is a diagram showing an example of an illuminated check pattern. FIG. 10 is a diagram showing a cross check in image data acquired from a camera. FIG. 11 is a diagram showing an example of calculation of a difference Δφ in the infinity direction. FIG. 12 is an explanatory diagram of a specific example of a calibration process.

[0010] Hereinafter, an embodiment of the present technology will be described.

[0011] <<1. Configuration Example of Vehicle Control System>> FIG. 1 is a block diagram showing a configuration example of a vehicle control system 11, which is a non-limiting example of a mobility device control system to which the present technology is applied.

[0012] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes levels 1 to 5 of automated driving, as well as remote driving and / or remote assistance of the vehicle 1 by a remote driver. The level of automated driving may refer to the Society of Automotive Engineers (SAE) J3016™ APL2021 Levels of Driving Automation, where SAE Level 0 denotes the lowest level of automated driving and SAE Level 5 denotes the highest level of automated driving. For example, SAE Level 1 automated driving may be composed of driver assistance functions that provide steering or braking / acceleration support to the driver, and SAE Level 5 automated driving may be composed of automated driving functions that can drive the vehicle under all conditions.

[0013] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0014] Two or more (or in some cases, all) of the vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are communicatively connected to each other via a communication network 41. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital bidirectional communication standard such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). In some embodiments, the communication network 41 may include two or more types of communication networks, and different types of communication networks may be used depending on the type of data being transmitted. For example, a CAN may be used for data related to vehicle control, and an Ethernet may be used for large-volume data. In some embodiments, two or more (or in some cases, all) units of the vehicle control system 11 may be directly connected using wireless communication (e.g., communication at a relatively short distance) without using the communication network 41. In some embodiments, the wireless communication may use a short-range wireless communication technology. Non-limiting examples of short-range wireless communication technologies include near field communication (NFC) and Bluetooth (registered trademark). In some embodiments, two or more (or in some cases, all) units of the vehicle control system 11 may be connected using the communication network 41 and a wireless communication technology (e.g., a short-range wireless communication technology).

[0015] Hereinafter, in an embodiment in which two or more units of the vehicle control system 11 communicate with each other via the communication network 41, the description of the communication network 41 will be omitted. For example, in an embodiment in which the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0016] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0017] The communication unit 22 communicates with various devices inside the vehicle 1 (hereinafter referred to as in-vehicle devices), various devices outside the vehicle 1 (hereinafter referred to as out-vehicle devices), other vehicles, base stations, etc., and transmits and receives various types of data. In some embodiments, the communication unit 22 may communicate using multiple communication technologies.

[0018] A non-limiting example of communication between the communication unit 22 and an external device will now be briefly described. In some embodiments, the communication unit 22 may communicate with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using wireless communication technology. Non-limiting examples of wireless communication technology include 5G (5th Generation Mobile Communication System), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. The external network with which the communication unit 22 can communicate is, for example, the Internet, a cloud network, or a network specific to an operator. The communication technology used by the communication unit 22 to communicate with the external network is not particularly limited as long as it is a wireless communication technology that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0019] In some embodiments, the communication unit 22 may use P2P (Peer to Peer) technology to communicate with a terminal located near the vehicle. The terminal located near the vehicle may be, for example, a terminal attached to a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, and / or an MTC (Machine Type Communication) terminal. In some embodiments, the communication unit 22 may perform V2X (Vehicle to Everything) communication. V2X communication generally refers to communication between the vehicle and another entity. Non-limiting examples of V2X communication include vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried or worn by a pedestrian.

[0020] In some embodiments, the communication unit 22 may receive a program for updating software that controls the operation of the vehicle control system 11 from outside the vehicle 1 (e.g., over the air). In some embodiments, the communication unit 22 may receive map information, traffic information, information about the surroundings of the vehicle 1, etc. from outside the vehicle 1. In some embodiments, the communication unit 22 may transmit information about the vehicle 1 or information about the surroundings of the vehicle 1, etc. to an external device or an external network. Non-limiting examples of information about the vehicle 1 that the communication unit 22 transmits to an external device or an external network include data indicating the status of the vehicle 1, recognition results by the recognition processing unit 61, etc. In some embodiments, the communication unit 22 may communicate with a vehicle emergency notification system. Non-limiting examples of a vehicle emergency notification system include eCall, etc.

[0021] In some embodiments, the communication unit 22 may receive electromagnetic waves transmitted by a road traffic information communication system. In some embodiments, the electromagnetic waves may be transmitted using a radio beacon, an optical beacon, FM multiplex broadcasting, or the like.

[0022] Non-limiting examples of communication with in-vehicle devices that can be performed by the communication unit 22 will now be briefly described. In some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wireless communication. For example, in some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wireless communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wireless communication technology include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). Alternatively, the communication unit 22 may communicate with the in-vehicle devices using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wired communication via a cable connected to a connection terminal (not shown). In some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wired communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wired communication technologies include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI)®, and Mobile High-definition Link (MHL).

[0023] Here, the in-vehicle devices refer to, for example, devices inside the vehicle 1 that are not connected to the communication network 41. The in-vehicle devices are divided into devices that constitute the vehicle control system 11 and devices that do not constitute the vehicle control system 11. Non-limiting examples of in-vehicle devices that do not constitute the vehicle control system 11 include mobile devices and wearable devices carried by users of the vehicle 1 (for example, the driver or passengers), and information devices that are temporarily installed inside the vehicle 1. These devices can, for example, be moved outside the vehicle 1 and become external devices.

[0024] The map information storage unit 23 stores maps acquired from an external device or an external network and / or maps created by the vehicle 1. For example, the map information storage unit 23 may store a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, or the like.

[0025] The high-precision map may be, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map may be, for example, a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and may be provided to the vehicle 1 from an external server or the like. The point cloud map may be a map composed of a point cloud (point cloud data). The vector map may be, for example, a map adapted to automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0026] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, the LiDAR 52, the radar 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters regarding the planned route along which the vehicle 1 will travel may be acquired from the external server or the like in order to reduce communication capacity.

[0027] The position information acquisition unit 24 acquires position information of the vehicle 1. The acquired position information may be supplied to the driving automation control unit 29. In some embodiments, the position information acquisition unit 24 may receive GNSS (Global Navigation Satellite System) signals from GNSS satellites. In some embodiments, the position information acquisition unit 24 may receive signals from beacons or the like.

[0028] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0029] In some embodiments, the external recognition sensor 25 may include a camera 51, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 52, a radar 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the LiDAR 52, the radar 53, and the ultrasonic sensor 54. The number of cameras 51, the LiDAR 52, the radar 53, and the ultrasonic sensors 54 is not particularly limited as long as the number is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0030] The camera 51 may use any suitable imaging method. In some embodiments, the camera 51 may use an imaging method capable of distance measurement. Non-limiting examples of cameras using imaging methods capable of distance measurement include a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. However, the camera 51 may simply acquire an image without distance measurement.

[0031] In some embodiments, the external recognition sensor 25 may include an environmental sensor for detecting characteristics of the environment around the vehicle 1. Non-limiting examples of environmental characteristics that may be detected include weather, climate, brightness, etc. In some embodiments, the environmental sensor may include various sensors such as a rain sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0032] The interior sensor 26 includes various sensors for detecting information about the interior of the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of the types and number that can be realistically installed in the vehicle 1.

[0033] In some embodiments, the interior sensor 26 may include one or more sensors selected from the group consisting of a camera, radar, a seat sensor, a microphone, and a biometric sensor. In some embodiments, the camera included in the interior sensor 26 may use an imaging method capable of measuring distances. Non-limiting examples of cameras using imaging methods capable of measuring distances include a Time of Flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. The camera included in the interior sensor 26 may also be a camera simply used to acquire captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat or a steering wheel, and may detect various types of biometric information of the user.

[0034] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of the types and number that can be realistically installed on the vehicle 1.

[0035] In some embodiments, the vehicle sensor 27 may include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU) that integrates these. In some embodiments, the vehicle sensor 27 may include a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation (e.g., pedal force, pedal stroke), and / or a brake sensor that detects the amount of brake pedal operation (e.g., pedal force, pedal stroke). In some embodiments, the vehicle sensor 27 may include a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and / or a wheel speed sensor that detects the rotation speed of the wheels. In some embodiments, the vehicle sensor 27 may include a battery sensor that detects the remaining battery level and temperature, and / or an impact sensor that can detect external impacts.

[0036] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of storage media include magnetic storage devices such as electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), and / or hard disk drives (HDDs), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The storage unit 28 stores various programs and data used by one or more (or in some cases, all) units of the vehicle control system 11. In some embodiments, the storage unit 28 may include an event data recorder (EDR) or a data storage system for automated driving (DSSAD), and may store information about the vehicle 1 before and after an event such as an accident, as well as information acquired by the in-vehicle sensors 26.

[0037] The driving automation control unit 29 controls the driving automation function of the vehicle 1. In some embodiments, the driving automation control unit 29 includes a recognition processing unit 61, an action planning unit 62, and an operation control unit 63.

[0038] The recognition processing unit 61 analyzes the vehicle 1 and / or the surrounding situation, for example, performs detection processing and recognition processing of objects present around the vehicle 1.

[0039] In some embodiments, the recognition processing unit 61 may estimate the self-position of the vehicle 1 based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the recognition processing unit 61 may generate a local map based on the sensor data from the external recognition sensor 25 and estimate the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 may be based on, for example, the center of the rear wheel pair axle.

[0040] In some embodiments, the local map may be a three-dimensional high-precision map, an occupancy grid map, or the like created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map may be, for example, the point cloud map described above. The occupancy grid map may be a map obtained by dividing a three-dimensional or two-dimensional space around the vehicle 1 into grids of a predetermined size and indicating the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects may be indicated, for example, by the presence or absence of an object or a probability of its presence. In some embodiments, the local map may also be used, for example, for detection and / or recognition of the situation outside the vehicle 1.

[0041] In some embodiments, the recognition processing unit 61 may estimate the own position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and / or sensor data from the vehicle sensor 27 .

[0042] The recognition processing unit 61 performs sensor fusion processing to obtain information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 53). Methods for combining different types of sensor data include, but are not limited to, compounding, integration, fusion, and association.

[0043] The recognition processing unit 61 may perform detection processing and / or recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information on the self-position estimation result, information obtained through sensor fusion processing, etc. Specifically, the recognition processing unit 61 may perform detection processing of objects present around the vehicle 1 and / or recognition processing of the objects. The object detection processing may be, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing may be, for example, processing to recognize attributes such as the type of object or to recognize a specific object. For example, if the object is a vehicle, the type of object may be recognized as either a general vehicle or an emergency vehicle. The detection processing and the recognition processing are not necessarily clearly separated and may overlap at least partially.

[0044] In some embodiments, the recognition processing unit 61 may detect objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the LiDAR 52 and / or the radar 53, etc. into clusters of points. This makes it possible to detect the presence, size, shape, and position of objects around the vehicle 1.

[0045] In some embodiments, the recognition processing unit 61 may detect the movement of objects around the vehicle 1 by tracking the movement of clusters of point clouds classified by clustering. This makes it possible to detect the speed and / or traveling direction (movement vector) of objects around the vehicle 1.

[0046] In some embodiments, the recognition processing unit 61 may detect and / or recognize vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. In some embodiments, the recognition processing unit 61 may recognize the type of object around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0047] In some embodiments, the recognition processing unit 61 may perform a process of recognizing traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the vehicle's own position, and / or the recognition result of objects around the vehicle 1. Through this process, the recognition processing unit 61 may recognize the positions and / or states of traffic lights, the contents of traffic signs and / or road markings, the contents of traffic regulations, and / or lanes that can be driven, etc.

[0048] In some embodiments, the recognition processing unit 61 may perform recognition processing of the environment around the vehicle 1. In some embodiments, the recognition processing unit 61 may recognize weather characteristics (temperature, humidity, brightness) and / or road surface conditions, etc.

[0049] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 may create a behavior plan for the vehicle 1 by performing route planning and route tracking based on the recognition result of the recognition processing unit 61.

[0050] In some embodiments, route planning may include global path planning and local path planning. Global path planning may include a process for planning a rough route from a start to a goal. Local path planning, also referred to as trajectory planning, may include generating a trajectory that allows the vehicle 1 to proceed safely and smoothly along a planned route in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1, the presence of any obstacles, and the like. When a nearby emergency vehicle is detected as a recognition result, an operation to avoid the emergency vehicle may be planned as an operation of the vehicle 1 so that the emergency vehicle can travel preferentially. For example, examples of an operation to avoid the emergency vehicle include moving the vehicle 1 to the edge of a lane or on a shoulder strip and stopping it, or flashing its hazard lights and temporarily stopping it.

[0051] In some embodiments, the path following may be a planning of an operation for safely and accurately traveling along a route planned by the route planner within a planned time. The behavior planning unit 62 may, for example, calculate a target speed and / or a target angular velocity of the vehicle 1 based on the result of the path following process.

[0052] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .

[0053] For example, in some embodiments, the operation control unit 63 may control the steering control unit 81, the brake control unit 82, and / or the drive control unit 83 included in the vehicle control unit 32 (described later) to perform lateral vehicle motion control and / or longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 may perform control (e.g., lateral vehicle motion control, longitudinal vehicle motion control) for one or more driver assistance functions and / or driving automation. Non-limiting examples of driver assistance functions include collision avoidance or impact mitigation, following distance control (e.g., control to maintain a specific distance from a vehicle traveling in front of the vehicle 1), vehicle speed control (e.g., control to maintain a specific speed), vehicle collision warning, and lane departure warning. Non-limiting examples of driving automation include driving without operation by a driver or a remote driver.

[0054] In some embodiments, the DMS 30 may perform a driver authentication process and / or a driver state recognition process based on sensor data from the in-vehicle sensors 26 and / or input data input to the HMI 31 (described later), etc. Non-limiting examples of the driver state that may be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, posture, etc.

[0055] In some embodiments, the DMS 30 may perform authentication processing of a user other than the driver (e.g., a passenger) and / or recognition processing of the state of the user. In some embodiments, the DMS 30 may perform recognition processing of the interior situation of the vehicle 1 based on sensor data from the interior sensors 26. Non-limiting examples of characteristics of the interior situation of the vehicle 1 that can be recognized include temperature, humidity, brightness, odor, etc.

[0056] The HMI 31 receives various data, instructions, etc. as input, and presents the various data to the user.

[0057] The input of data to the HMI 31 will be briefly described. The HMI 31 includes an input device through which a person inputs data, instructions, etc. The HMI 31 generates an input signal based on the data, instructions, etc. input via the input device and supplies the signal to one or more (or in some cases, all) units of the vehicle control system 11. In some embodiments, the HMI 31 may include a touch panel, buttons, switches, and / or levers as input devices. Without being limited thereto, the HMI 31 may also include an input device that allows information to be input by a method other than manual operation, such as voice or gestures. In some embodiments, the HMI 31 may include an input device such as a remote control device using infrared and / or radio waves, or an externally connected device that can operate the vehicle control system 11. Non-limiting examples of externally connected devices include a mobile device (e.g., a smartphone) and a wearable device (e.g., a smart watch).

[0058] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and / or tactile information for the user and / or a person outside the vehicle 1. The HMI 31 may also perform output control, controlling the output, output content, output timing, and / or output method of each piece of generated information. Non-limiting examples of visual information that can be generated and output by the HMI 31 include information displayed by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. Non-limiting examples of auditory information that can be generated and output by the HMI 31 include voice guidance, warning sounds, warning messages, etc. Non-limiting examples of tactile information that can be generated and output by the HMI 31 include information imparted to the user's sense of touch by force, vibration, movement, etc.

[0059] In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image itself or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be, in addition to or instead of a typical display device, a device that displays visual information within the user's field of view, such as a head-up display, a see-through display, or a wearable device with an augmented reality (AR) function. In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1.

[0060] In some embodiments, the HMI 31 may include an audio speaker, headphones, or earphones as output devices capable of outputting auditory information.

[0061] In some embodiments, the HMI 31 may include a haptic element using haptic technology as an output device capable of outputting tactile information. The haptic element may be provided on a part of the vehicle 1 that the user comes into contact with, such as the steering wheel or the seat.

[0062] The vehicle control unit 32 controls one or more (or in some cases, all) units of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0063] The steering control unit 81 detects and / or controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0064] The brake control unit 82 detects and / or controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0065] The drive control unit 83 detects and / or controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0066] The body system control unit 84 detects and / or controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0067] The light control unit 85 detects and / or controls the status of various lights of the vehicle 1. Non-limiting examples of lights that can be controlled by the light control unit 85 include headlights, infrared light sources (including near-infrared light sources), backlights, fog lights, turn signals, brake lights, projector lights, and bumper indicators. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like. The light control unit 85 includes an AFS (Adaptive Front Lighting System) system that dynamically controls the light irradiation pattern.

[0068] The horn control unit 86 detects and / or controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0069] Fig. 2 is a diagram showing an example of a sensing area by the camera 51, the LiDAR 52, the radar 53, the ultrasonic sensor 54, etc. of the external recognition sensor 25 in Fig. 1. Fig. 2 schematically shows the vehicle 1 as viewed from above.

[0070] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F (e.g., sensing area of ​​the multiple ultrasonic sensors 54) covers the periphery of the front end of the vehicle 1. Sensing area 101B (e.g., sensing area of ​​the multiple ultrasonic sensors 54) covers the periphery of the rear end of the vehicle 1.

[0071] The sensing results in sensing area 101F and / or sensing area 101B may be used, for example, for parking assistance for vehicle 1.

[0072] Sensing area 102F, sensing area 102B, sensing area 102L, and sensing area 102R show examples of sensing areas of a short-range or medium-range radar 53. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the surrounding area behind the left side of the vehicle 1. Sensing area 102R covers the surrounding area behind the right side of the vehicle 1.

[0073] The sensing results in the sensing area 102F may be used, for example, to detect vehicles, pedestrians, etc. present in front of the vehicle 1. The sensing results in the sensing area 102B may be used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing area 102L and / or the sensing area 102R may be used, for example, to detect one or more objects in blind spots on the left and / or right sides of the vehicle 1.

[0074] Sensing area 103F, sensing area 103B, sensing area 103L, and sensing area 103R show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther behind vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery on the left side of vehicle 1. Sensing area 103R covers the periphery on the right side of vehicle 1.

[0075] The sensing results in sensing area 103F may be used, for example, for recognizing traffic lights and traffic signs, a lane departure prevention assistance system, or an automatic headlight control system. The sensing results in sensing area 103B may be used, for example, for parking assistance and / or a surround view system. The sensing results in sensing area 103L and / or sensing area 103R may be used, for example, for a surround view system.

[0076] Sensing area 104 shows an example of the sensing area of ​​LiDAR 52. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction of vehicle 1 than sensing area 103F.

[0077] The sensing results in the sensing area 104 may be used to detect objects such as surrounding vehicles, for example.

[0078] Sensing area 105 shows an example of the sensing area of ​​the long-distance radar 53. Sensing area 105 covers a position further ahead of the vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction of the vehicle 1 than sensing area 104.

[0079] The sensing results in the sensing area 105 may be used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0080] In some embodiments, the sensing area of ​​each of the external recognition sensors 25 (e.g., the camera 51, the LiDAR 52, the radar 53, and the ultrasonic sensor 54) may have various configurations other than the configuration shown in Fig. 2. Specifically, in some embodiments, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 52 may sense the rear of the vehicle 1. Furthermore, the installation position of each sensor is not limited to the above-mentioned examples. Furthermore, the number of each sensor may be one or more.

[0081] <<2. Configuration Example of Information Processing System>> In the present disclosure, in a vehicle that performs dynamic light distribution lighting using headlights equipped with an AFS (Adaptive Front Lighting System) and performs autonomous driving using image recognition by a camera 51 (see FIG. 1), infrared light (including near-infrared light) is emitted from an infrared light source in a complementary manner to ensure brightness in dark areas. In this case, by detecting the infrared light and visible light areas from an image detected by the camera 51 in which infrared light and visible light are mixed, and performing appropriate image signal processing for each area, an image that can provide highly accurate recognition results is generated, thereby realizing highly accurate image recognition.

[0082] 3 is a schematic diagram illustrating the functional configuration of an information processing system 1000 according to the present disclosure. The information processing system 1000 corresponds to the portion of the vehicle control system 11 in FIG. 1 that is primarily related to image processing and recognition processing. The information processing system 1000 includes a map information storage unit 23, a vehicle sensor 27, an external sensor 50, a processing unit 100, an AFS lamp light distribution control unit 210, a headlight 210A, an infrared light irradiation control unit 220, an infrared light source 220A, a camera 51, a transmission unit 230, and a display 240.

[0083] The headlight 210A corresponds to a visible light irradiating unit mounted on the vehicle that irradiates visible light in a predetermined direction of the vehicle. The infrared light source 220A corresponds to an infrared light irradiating unit mounted on the vehicle that irradiates infrared light in a predetermined direction of the vehicle. The camera 51 corresponds to an imaging unit that captures an image of a predetermined direction of the vehicle based on light incident on an imaging area and generates image data. The predetermined direction is, for example, the front or rear of the vehicle, but may also be a side direction. The AFS lamp light distribution control unit 210 and the infrared light irradiation control unit 220 correspond to control units that control the irradiation range of at least one of the headlight 210A and the infrared light source 220A.

[0084] The processing unit 100 performs separate image signal processing for the first region and the second region on the image data captured by the camera 51, or performs image signal processing with variable weighting during signal processing, based on first information regarding the position of a first region in the imaging region of the imaging unit (camera 51) onto which visible light (e.g., reflected visible light) from the visible light irradiator (headlight 210A) is incident, and second information regarding the position of a second region onto which infrared light (e.g., reflected infrared light) from the infrared light source (infrared light source 220A) is incident. The first information is, for example, information indicating the position of the first region or information for identifying the position of the first region. The second information is, for example, information indicating the position of the second region or information for identifying the position of the second region. The information for identifying the position of the first region may include, for example, information indicating the visible light irradiated region in an illumination map described below, information about the headlight 210A (installation position, installation direction), etc. The information for identifying the position of the second region may include, for example, information indicating the infrared light irradiation region in the infrared light illumination map described below (when the irradiation range of the infrared light source is variable), information about the infrared light source 220A (installation position, installation direction, light source characteristic information), etc.

[0085] The processing unit 100 includes an illumination map generation unit 110, an illumination / camera space mapping unit 120, a local demosaicing unit 130, an LTM unit 140, a region specification LTM unit 150, a recognition processing unit 160, a calibration processing unit 170, a behavior planning unit 62, and an operation control unit 63.

[0086] All or part of the processing unit 100 can be included in the driving automation control unit 29 in the vehicle control system 11 of Fig. 1. The AFS lamp light distribution control unit 210 and the infrared light irradiation control unit 220 correspond to the light control unit 85, and the external sensors 50 correspond to the LiDAR 52, radar 53, ultrasonic sensor 54, etc. Furthermore, the transmission unit 230 corresponds to the bus 41, and the display 240 corresponds to the HMI 31. The camera 51, map information storage unit 23, and vehicle sensor 27 each correspond to the components of the same name shown in Fig. 1.

[0087] An information processing device may be configured to include the processing unit 100. In this case, the information processing device may include a first acquisition unit that acquires sensing data (detection signals) from the external recognition sensor 25, a second acquisition unit that acquires steering information of the driver from the vehicle sensor 27, and a third acquisition unit that acquires map information from the map information storage unit 23.

[0088] The map information storage unit 23 stores map information of the environment in which the vehicle 1 can travel. The map information may be a three-dimensional high-precision map, or a global map that has lower precision than a high-precision map and covers a wider area. Details of the map information storage unit 23 are as described above.

[0089] The vehicle sensor 27 includes the various sensors described above for detecting the state of the vehicle 1, and may include, for example, at least one of a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), an inertial measurement unit (IMU), a steering angle sensor, a yaw rate sensor, an accelerator sensor, and a brake sensor. Details of the vehicle sensor 27 have been described above and will not be repeated here.

[0090] The external sensor 50 is a sensor that detects the situation outside the vehicle 1, and corresponds to, for example, at least one of the LiDAR 52, radar 53, and ultrasonic sensor 54 described above. Details of these sensors have been described above and will not be repeated here. The external sensor 50 may include sensors other than those listed here as long as it can detect the situation outside the vehicle 1. For example, the external sensor 50 may include a microphone that detects surrounding sounds.

[0091] The camera 51 is a camera capable of capturing both visible light and infrared light. Visible light includes reflected visible light emitted from the headlight 210A, and light (ambient light) such as sunlight and streetlights.

[0092] The camera 51 is a camera (RGB-IR camera) having an imaging area in which four types of pixels, including pixels for R (red) (hereinafter referred to as R pixels), pixels for G (green) (hereinafter referred to as G pixels), pixels for B (blue) (hereinafter referred to as B pixels), and pixels for infrared light (hereinafter referred to as IR pixels), are arranged in an array in a predetermined arrangement.

[0093] Figure 4 shows an example of a portion of the imaging area. Four types of pixels are arranged: R pixels, G pixels, B pixels, and IR pixels. They are arranged in an array in a predetermined pattern.

[0094] However, the camera 51 is not limited to an RGB-IR camera, and may be, for example, a camera in which R, G, and B pixels are arranged in the imaging area, with at least one of the R, G, and B pixels being sensitive to infrared light. For example, the B pixel may be sensitive to infrared light. In this case, in image processing, a B signal and an infrared light signal can be separated from the imaging signal of that pixel using a filter. Although color filters of different combinations, such as complementary colors, other than RGB, can also be used, the following description will use RGB filters as a representative example of processing.

[0095] As another example of the camera 51, it is not excluded to use two cameras, one for RGB visible light and the other for infrared light.

[0096] The illumination map generation unit 110 generates an illumination map (first map) that indicates the area (or direction) to be illuminated within the range that can be illuminated by the headlight 210A of the vehicle 1. The illumination map can be expressed, for example, in a two-dimensional coordinate system. The illumination map corresponds to the first map that indicates the area to be illuminated within the range that can be illuminated by the headlight 210A (visible light illumination unit). For example, the illumination map determines the illumination area of ​​the headlight 210A that reduces illumination to oncoming vehicles and vehicles ahead, so as to avoid dazzling the eyes of drivers of oncoming vehicles due to light entering the eyes of oncoming vehicles and drivers of vehicles ahead due to light reflected by the rearview mirror or side mirror. The illumination map is then generated or updated to indicate the determined illumination area. The illumination map can be formed, for example, by dividing the entire range of illumination directions into a grid and using a grid map that indicates whether or not illumination is to be provided for each grid as the coordinate system.

[0097] For example, the illumination map generation unit 110 detects roads, oncoming vehicles, preceding vehicles, road signs, etc., using map information from the map information storage unit 23, steering information from the vehicle sensor 27, etc., and sensing data from the external sensor 50, to grasp the driving environment of the vehicle 1 and determine the traveling direction of the vehicle 1. The illumination map generation unit 110 determines the illumination area of ​​the headlights 210A from the driving environment and traveling direction, etc. The illumination map generation unit 110 may further use a recognition result from a recognition processing unit 160, which will be described later, in determining the illumination area. The illumination map generation unit 110 sends the generated illumination map (grid map) to the AFS lamp light distribution control unit 210 and also to the lighting / camera space mapping unit 120.

[0098] The AFS lamp light distribution control unit 210 controls the headlights 210A based on the illumination map to emit visible light, thereby illuminating (projecting) the light onto the area indicated in the illumination map.

[0099] The infrared light irradiation control unit 220 controls the infrared light source 220A to emit infrared light. In the present embodiment, as an example, the direction and range of infrared light irradiation (light distribution of infrared light illumination) are predetermined. The infrared light irradiation control unit 220 controls the infrared light irradiation to constantly emit the infrared light in the predetermined direction. For example, when the AFS lamp light distribution control unit 210 selectively irradiates the headlights 210A, the headlights 210A are usually irradiated downward toward the front of the vehicle 1 or toward a region close to the vehicle 1. For this reason, the infrared light irradiation direction is usually determined to be a forward direction or a distant region. However, instead of constantly irradiating the infrared light, the infrared light may be irradiated only when conditions for infrared light irradiation are met. For example, the irradiation map generation unit 110 may detect the brightness of the environment surrounding the vehicle 1 using an illuminance sensor (external sensor 50) and send instruction data to the infrared light irradiation control unit 220 to irradiate infrared light only when the brightness is below a threshold. Alternatively, the irradiation map generation unit 110 may send instruction data to the infrared light irradiation control unit 220 to irradiate infrared light from the infrared light source 220A only during the time period when the AFS lamp light distribution control unit 210 is controlling the headlight 210A to irradiate.

[0100] As another example, the irradiation range or direction of the infrared light source 220A may be controllable. The irradiation map generation unit 110 determines an irradiation area of ​​infrared light within the range, generates or updates an infrared irradiation map (second map) to indicate the determined area, and sends it to the infrared light irradiation control unit 220. The infrared light irradiation control unit 220 controls the infrared light source 220A based on the infrared irradiation map to irradiate the specified area with infrared light. In this case, the infrared light irradiation control unit 220 may control the irradiation range of the infrared light source 220A based on the received instruction data. The infrared irradiation map corresponds to the second map having a coordinate system indicating the area to be irradiated within the irradiable range of the infrared light source 220A (infrared light irradiator). The following mainly assumes that the irradiation range (irradiation area) of the infrared light source 220A is fixed.

[0101] 5 shows an example in which an area 320 illuminated according to the illumination map of headlight 210A and an area 310 illuminated by infrared light source 220A are shown, and a camera viewing angle 300 is also shown in FIG.

[0102] The illumination / camera space mapping unit 120 maps, in the coordinate system of the camera 51 (the coordinate system of the image data), the area onto which visible light from the headlight 210A is incident and the area onto which infrared light from the infrared light source 220A is incident, based on information about the camera 51, information about the infrared light source 220A, information about the headlight 210A, and the area of ​​visible light irradiation indicated in the visible light irradiation map (first map). At this time, the mapping may also be performed using the distance to the object illuminated by the headlight 210A. The illumination / camera space mapping unit 120 sends the mapping information to the local demosaicing unit 130 and the LTM unit 140.

[0103] The information about the camera 51 includes, for example, external parameters and internal parameters of the camera 51, such as the installation position and installation direction of the camera 51. The information about the infrared light source 220A includes, for example, the irradiation range, infrared light distribution information, installation position and installation direction of the infrared light source 220A. The information about the headlight 210A includes, for example, the irradiation range, visible light distribution information, installation position and installation direction of the headlight 210A. The coordinate system of the camera 51 is also called the coordinate system of image data or the camera view angle frame.

[0104] In the imaging area of ​​camera 51, the area into which visible light from headlight 210A is incident (mapped) is referred to as the headlight incident area, and the area into which infrared light from infrared light source 220A is incident (mapped) is referred to as the infrared light incident area. The infrared light incident area and the headlight incident area may partially overlap.

[0105] The areas other than the headlight incident area and the infrared light incident area correspond to areas where direct or reflected ambient light is incident. For example, these are areas where light from headlights of other vehicles or streetlights is incident. This area is referred to as the ambient light incident area. The processing unit 100 may detect objects present in the headlight illumination area and the infrared light illumination area using sensor data from the external sensor 50, and estimate any one of the headlight incident area, infrared light incident area, and ambient light incident area using information such as the shape of the object and the distance to the object.

[0106] 6 shows an example in which an infrared light incidence region 311 and a headlight incidence region 321 are mapped onto a camera coordinate system 301 (camera view angle frame). The region within the frame other than the infrared light incidence region 311 and the headlight incidence region 321 corresponds to an ambient light incidence region 331.

[0107] The camera 51 has an imaging area (sensor area) in which four types of pixels, R pixels, G pixels, B pixels, and IR pixels, are arranged in a predetermined array. The camera 51 acquires detection signals from these pixels in the imaging area at regular time intervals and outputs the detection signal data to the local demosaic unit 130 as a captured image (RAW data).

[0108] The local demosaic unit 130 divides the captured image into regions using information (first information) about the positions of the headlight incidence regions, information (second information) about the positions of the infrared light incidence regions, and information (third information) about the positions of the ambient light incidence regions, and performs demosaic processing for each region. The first to third information may be obtained from the mapping information described above. As a result, for each region, the R component value (R value), G component value (G value), B component value (B value), and IR component value (IR value) are determined or estimated for each of all pixels (R pixels, G pixels, B pixels, and IR pixels) within the region.

[0109] For example, in an infrared light incidence region, for an arbitrary pixel of interest, pixel values ​​(component values) of colors other than the color of the pixel itself are estimated using pixel values ​​of surrounding pixels. For example, if the pixel of interest is an R pixel, the G value, B value, and IR value of the pixel of interest are calculated by interpolation using the pixel values ​​of the surrounding pixels (G pixels, B pixels, IR pixels). Similarly, if the pixel of interest is an IR pixel, G pixel, or B pixel, component values ​​of colors other than the color of the pixel of interest are determined by interpolation using pixel values ​​of pixels of other colors (including IR) surrounding the pixel of interest. Note that, although the color component values ​​of the pixel of interest are directly used here, the color component values ​​of the pixel of interest may also be calculated by interpolation using pixel values ​​of surrounding pixels of the same color.

[0110] Similarly, for the headlight incidence region, local demosaicing is performed on a region-by-region basis using the pixel values ​​of the pixel group within the region.Furthermore, for the ambient light incidence region, local demosaicing is performed on a region-by-region basis using the pixel values ​​of the pixel group within the region.However, the headlight incidence region and the ambient light incidence region may be treated as regions of the same type (regions where visible light is dominant), and these regions may be combined and demosaiced as a single region.

[0111] By performing demosaicing for each region, the pixel values ​​of pixels in other regions are suppressed or prevented from being used in the interpolation process, which can, for example, improve false colors that are emphasized at the boundaries of regions due to infrared illumination light and suppress jagged edges that occur on the contours of objects. If the pixel values ​​of the headlight incident region were used in the interpolation process for a pixel (target pixel) near the boundary of the infrared light incident region, the pixel values ​​(IR components) of the IR pixels in the headlight incident region would be expected to be low, and the IR value of the target pixel would also be calculated low by the interpolation process. This could result in a degradation of image quality near the boundary. By performing demosaicing for the infrared light incident region independently from other regions, this degradation of image quality can be suppressed.

[0112] If there is an overlapping area between the infrared light incident area and the headlight incident area, a weighted sum of the calculation results (pixel values) of the infrared light incident area and the headlight incident area may be calculated for the pixels in the overlapping area. The weight ratio for each area may be determined in advance. Furthermore, the weight may be different for each color (R, G, B, IR).

[0113] For example, the weighted sum may be calculated for the IR values ​​of each pixel in the overlapping region by increasing the weight of the infrared light incident region and decreasing the weight of the headlight incident region. For example, with respect to the ratio of the weight α1 of the infrared light incident region to the weight α2 of the headlight incident region, α1 may be set to a value of 1 or close to 1, and α2 may be set to a value of 0 or close to 0, or an equal ratio such as 0.5:0.5 may be used. Furthermore, the weighted sum may be calculated for the R values, G values, and B values ​​of each pixel in the overlapping region by decreasing the weight of the infrared light incident region and increasing the weight of the headlight incident region. α2 may be set to a value of 1 or close to 1, and α1 may be set to a value of 0 or close to 0, or an equal ratio such as 0.5:0.5 may be used.

[0114] Here, an example has been shown in which the weights for each region are determined in advance, but the weights for each region may be updated as appropriate while the vehicle is traveling using feedback processing to obtain highly accurate (high accuracy rate) recognition results.

[0115] Furthermore, as another example, it is not excluded to change the weight between the regions depending on the pixel position within the overlapping region.

[0116] The demosaicing may use different algorithms for the headlight incident region, infrared light incident region, and ambient light incident region, or the same algorithm may be used for all of these regions. As long as the demosaicing is performed independently for each region, it does not matter whether the algorithm is the same or not.

[0117] Before demosaicing, white balance processing may be performed for each region using information on the headlight incident region, infrared light incident region, and ambient light incident region. Performing white balance processing for each region can further improve false colors accentuated by infrared illumination, even at the boundary or overlapping regions of the regions, and suppress jagged edges that occur on the contours of objects. If white balance processing is performed uniformly within the frame without dividing a subject illuminated by light from multiple different light sources (e.g., infrared light source, headlights), it may be impossible to cancel out the influence of each light source, resulting in regions with a misaligned white balance. Therefore, by performing white balance processing for each region as in the present disclosure, white can be adjusted across all regions, which is expected to improve the recognition rate of white lines, signs, and the like.

[0118] Considering the possibility of errors in mapping the infrared light incidence region to the camera frame during the demosaicing process described above, the boundary of the infrared light incidence region in the image data may be extended outward by a certain width, and the extended infrared light incidence region may be subjected to region-by-region demosaicing. While performing optimal processing for each light source-irradiated region from a limited pixel arrangement makes it possible to obtain high-quality images within each region, significant changes at these boundaries may lead to false recognition as object boundaries by observers or image signal processing such as semantic segmentation, or may attract attention to the observation due to an attention-grabbing effect. To avoid this, at the boundaries of the estimated light source region generated by adaptive image processing after camera image processing, the blending ratio of the image of the region of interest for each region is changed from 100% to 0% to ensure smooth image continuity at the boundaries of each region, and this is then carried over to the images of the surrounding regions, thereby reducing unnaturalness that may occur at the boundary regions. It is to be noted that even if optimal development processing is performed individually for each region, it is possible that the detection accuracy of the automatic recognition processing at a later stage will decrease due to the light source estimation differing for each region due to the image recognition processing performed at a later stage after the camera output. Therefore, depending on the intended use, by outputting blending application rate information or light source estimation information for each region together with the image data, it is possible to accurately determine judgments that comply with the laws and regulations of each country, such as white lines, yellow lines, blue lines, and green regions, and thereby improve detection accuracy.

[0119] 7 shows an example in which the periphery of the infrared light entrance region 311 is expanded to form a region 311A. Demosaicing is performed on region 311A. With this method, if there is no error in the position of the infrared light entrance region, the image quality near the boundary of the infrared light entrance region may be reduced. However, by expanding the region, it is possible to maintain a constant image quality even if there is some error overall.

[0120] The LTM unit 140 performs local tone mapping (LTM) on the image data after region-by-region demosaicing as an input image. LTM is a technology that expands the dynamic range of the entire image by locally adjusting contrast, as opposed to global tone mapping, which performs uniform luminance conversion on the entire input image. LTM makes it possible to set an appropriate dynamic range for each local area, allowing luminance conversion to be performed while maintaining information such as the texture and contrast of the image.

[0121] The LTM unit 140 generates image data for display (color image data) by performing LTM on an image containing RGB values ​​for all pixels in the input image. At this time, by using the mapping information from the illumination / camera space mapping unit 120 to distinguish between headlight incident areas, infrared light incident areas, and ambient light incident areas, it is possible to correctly identify the scene in the input image and obtain high-quality image data. The LTM 140 sends the generated image data to the transmission unit 230.

[0122] Alternatively, the LTM unit 140 may perform LTM on an input image containing IR values ​​for all pixels to generate a grayscale image. In this case, the LTM may be performed by identifying the headlight incident area, infrared light incident area, and ambient light incident area using the mapping information from the illumination / camera space mapping unit 120. The LTM unit 140 may generate image data for display by combining the post-LTM image data of the image containing IR values ​​with the post-LTM image data of the image containing RGB values. For example, the combination may use post-LTM image data of IR in the infrared light incident area, and post-LTM image data of the image containing RGB values ​​in other areas. This results in a grayscale image in the infrared light incident area, but a high-contrast image in the infrared light incident area, even at night, and a high-contrast display image overall. The LTM 140 sends the generated image data to the transmission unit 230. Note that in areas where the headlight incident area and the infrared light incident area overlap, either image data may be selected. For example, if the external environment of vehicle 1 is bright (daytime), image data after LTM of an image containing RGB values ​​may be selected, and if the external environment is dark (nighttime), image data after LTM of an image containing IR values ​​may be selected.

[0123] The transmitter 230 transmits the image data received from the LTM 140 to the display 240, which then displays the image represented by the image data on the screen. As described above, the image displayed has been corrected for false colors and suppressed jagged edges around the object contours.

[0124] The region-specifying LTM unit 150 performs local tone mapping (LTM) independently for each of the headlight incident region, infrared light incident region, and ambient light incident region based on the image data after local demosaicing.

[0125] For example, LTM is performed on an image containing the IR values ​​of all pixels in the infrared light incident region. For example, LTM is performed on an image containing the RGB values ​​of all pixels in the infrared light incident region.

[0126] Additionally, LTM is performed on an image containing the RGB values ​​of all pixels in the headlight incident region. LTM is performed on an image containing the IR values ​​of all pixels in the headlight incident region. LTM is also performed on an image containing the RGB values ​​of all pixels in the ambient light incident region. LTM is also performed on an image containing the IR values ​​of all pixels in the ambient light incident region.

[0127] Although an example of performing LTM for each IR value and RGB value for each region has been shown, it is not necessary to perform all of these. For example, it is possible to perform only LTM for the IR value for the infrared light incident region, and only LTM for the RGB value for the headlight incident region and the ambient light incident region.

[0128] By performing LTM on each region in this way, high-contrast images can be obtained for each region, enabling more accurate image recognition.

[0129] LTM may be performed on an image containing the RGB values ​​of all pixels in a single region that combines the headlight incidence region and the ambient light incidence region. LTM may be performed on an image containing the IR values ​​of all pixels in a single region that combines the headlight incidence region and the ambient light incidence region.

[0130] The region-designated LTM unit 150 generates image data for recognition processing by synthesizing the post-LTM images for each region. As an example, for the infrared light incident region, image data is generated in which each pixel contains the IR value and RGB value of the infrared light incident region after LTM, for the headlight incident region, image data in which each pixel contains the RGB value and IR value of the headlight incident region after LTM, and for the ambient light incident region, image data in which each pixel contains the IR value and RGB value of the ambient light incident region after LTM. For regions where the headlight incident region and the infrared light incident region overlap, a weighted sum of the pixel values ​​(IR value, RGB value) of each region may be calculated. The details of the processing may be the same as those of the demosaic processing described above.

[0131] The area designation LTM unit 150 sends the generated image data for recognition processing to the recognition processing unit 160 .

[0132] The region-designating LTM unit 150 may generate image data for display based on the images after LTM for each region. For example, the image data for display may be image data in which an image including IR values ​​after LTM for the infrared light incident region is arranged in the infrared light incident region, an image including RGB values ​​after LTM for the headlight incident region is arranged in the headlight incident region, and an image including RGB values ​​after LTM for the ambient light incident region is arranged in the ambient light incident region. Other methods of composition may also be used to generate image data for display.

[0133] The region-specific LTM unit 150 sends the generated image data to the display 240 via the transmission unit 230, and the display 240 displays the image represented by the image data on the screen. The display image data generated by the region-specific LTM unit 150 may be less smooth (natural) overall than the image data generated by the LTM unit 140, but an image with a wide dynamic range is displayed for each region. While the processing described above describes independent signal processing for each region, similar to adaptive local tone mapping performed as a conventional well-known technique, even when abrupt changes in luminance occur at boundaries between different illuminations, smooth transition processing is performed to eliminate abrupt changes in gradation, thereby enabling the display of an image with high visibility for each region. This also results in an image with improved false colors caused by incorrect spectral estimation of the light source for each region and suppressed jaggedness that occurs around the contours of objects.

[0134] Either the image data from the LTM 140 or the image data from the region designation LTM unit 150 may be selectively displayed in response to a user instruction, or both may be displayed.

[0135] The recognition processing unit 160 performs recognition processing (image recognition) based on the image data for image recognition generated by the region designation LTM unit 150. At this time, recognition processing is performed for each region based on the mapping information of the illumination / camera space mapping unit 120. As an example, recognition processing may be performed based on an IR value image in the infrared light incident region (recognition processing using a grayscale image), and recognition processing may be performed based on an RGB value image in the headlight incident region and the ambient light incident region (recognition processing using a color image). Object detection and recognition are performed through recognition processing.

[0136] As another example, when the environment outside the vehicle is bright, recognition may be performed using a color image using an image of RGB values ​​in the infrared light incident region.

[0137] In the area where the infrared light incident area and the headlight incident area overlap, the recognition results of each area may be integrated to obtain a recognition result. For example, the recognition result with the higher reliability value may be adopted. The recognition processing unit 160 sends the recognition result to the action planning unit 62.

[0138] The region-specifying LTM 150 may send image data of the LTM result for each region to the recognition processing unit 160, and the recognition processing unit 160 may perform recognition processing on a region-by-region basis based on the image data of the LTM result for each region.

[0139] The operations of the action planning unit 62 and the operation control unit 63 have been described in the explanation of FIG. 1 and will not be described here.

[0140] The recognition processing unit 160 may send the recognition result to the illumination / camera space mapping unit 120, and the calibration processing unit 170 in the illumination / camera space mapping unit 120 may update the mapping operation based on the recognition result. This is expected to improve the accuracy of mapping the headlight incident area and the infrared light incident area in the camera coordinate system. In other words, the mapping operation in the illumination / camera space mapping unit 120 may be updated by feedback control.

[0141] For example, if there are areas within the infrared light incident area where the reliability of the recognition result is low and areas where it is high, and the reliability of the recognition result in the low area is lower than a threshold, adjustment may be made to shift the infrared light incident area toward the high area. This is because it is possible that processing (demosaicing, area-specific LTM, etc.) was performed as if the low area were an infrared light incident area, even though infrared light was not actually incident on that area. If the average reliability of the recognition result in the infrared light incident area is equal to or greater than a certain value, it may be determined that updating is not necessary. Updates may also be performed using other methods.

[0142] The following describes a method for improving the accuracy of estimating the headlight incident area and the infrared light incident area in the above-described information processing system 1000. The following mainly describes the headlight incident area as an example, but a similar method can also be applied to the infrared light irradiation area.

[0143] The headlight 210A and the camera 51 (assumed to be a camera installed in front of the vehicle) are usually installed at different positions on the vehicle 1. If we consider a stereo camera, with the camera 51 as one camera and the headlight 210A as the other camera, a deviation occurs that corresponds to the parallax that corresponds to the distance between the cameras in a stereo camera. The magnitude of this deviation is determined depending on the installation positions of the headlight 210A and the camera 51 (more specifically, the distance between the installation positions) and the distance to the object (the distance from the camera 51 to the object and the distance from the headlight 210A to the object). It may also depend on the installation direction of the headlight 210A and the camera 51 relative to the object.

[0144] That is, similar to the parallax according to distance in a stereo camera, the displacement corresponding to the parallax is determined according to the distance to the object illuminated by headlight 210A and captured by the camera. Therefore, it is not possible to uniformly determine and correct the amount of displacement of the object in the image captured by camera 51.

[0145] Here, if precise correction of the amount of deviation is required, it is possible to calculate the distance to each detected object contained in the image and calculate and correct the amount of deviation using triangulation. However, this requires accurate calculation of the distance to all objects in the image, and requires the use of LiDAR point cloud data or other distance calculation techniques to generate a separate distance map. For image recognition applications required for autonomous driving, precise correction associated with the parallax of the headlight 210A's illumination position is not required, and considering the processing load required to calculate the distance, there is little benefit to calculating precise and detailed parallax (amount of deviation).

[0146] However, if the orientations of the camera 51 and the headlight 210A are left unmanaged, the misalignment correction becomes uncertain, and the estimation of the headlight incident area of ​​the headlight 210A in image signal processing, i.e., the processing based on mapping the headlight incident area onto the camera coordinate system, becomes meaningless, which is undesirable. Therefore, it is desirable to perform a so-called calibration process to calibrate the orientations of the camera 51 and the headlight 210A periodically or upon request from another device. This calibration process may be performed not only for the headlight 210A but also for other light sources, such as the infrared light source 220A.

[0147] Factors that determine the amount of deviation equivalent to parallax can be broadly divided into the installation positional spacing between the headlight 210A and the camera 51 and the installation orientation of each of the headlight 210A and the camera 51. The installation positional spacing typically does not change significantly unless the headlight 210A and the camera 51 are reattached. On the other hand, in addition to changes due to reattachment, there are many factors that can cause changes in the installation orientation, such as distortion of the vehicle body due to temperature and progressive changes over time due to vibration, etc. Typically, the basic reference direction (orientation) of a front camera (front sensing camera) that controls forward recognition control related to autonomous driving is the direction of a distant infinity point ahead, toward which the vehicle moves when moving straight ahead. This basic reference direction is an important factor in various recognition processes, such as lane departure warning, lane keeping control, and the screen reference position for estimating obstacle distances, and therefore calibration, such as periodic management of its orientation, is essential.

[0148] As described above, the calibration of the camera 51 is typically performed periodically, regardless of whether or not an AFS is used. Therefore, it is preferable to perform calibration to align the infinity reference point in the illumination map of the headlight 210A with the orientation of the camera 51 as a reference. Unlike the camera calibration process of a stereo camera, the headlight 210A cannot capture images. However, the AFS that controls the light distribution of the headlight 210A can project and illuminate specific patterns. In particular, the AFS of the headlight 210A, which can generate programmable matrix patterns, can generate and illuminate illumination patterns with specific drawing shapes, such as a circle mark, a square mark, or a check mark. Therefore, the calibration process can be performed by having the camera 51 capture an illumination pattern in synchronization with the AFS.

[0149] The irradiation pattern is preferably one that can determine one or more directions and whose characteristic coordinates can be accurately read even if there is boundary blur in the image captured by the camera 51. Examples of characteristic coordinates include the center coordinates of a circle, the intersection of a check cross mark, etc., and circles, check cross marks, etc. can also be used as patterns.

[0150] The following describes the alignment of the reference point at infinity using the projection irradiation of a specific pattern using the AFS.

[0151] In a typical road space, there are various objects that exist in the direction of the headlight 210A. There may be a plane that is close to the headlight 210A, rising from the road at a short distance, and is quasi-perpendicular to the headlight 210A. On the other hand, there may be a situation where only a flat road spreads out ahead and there are no objects that obstruct the vehicle's progress. Under conditions determined during the vehicle manufacturing process, etc., it is possible to determine the illumination target in advance, but it is difficult to perform periodic calibration (for example, during maintenance) on a vehicle that has already been released (shipped) to the market unless a system is in place.

[0152] Even if a full set of general vehicle maintenance equipment is not available at a vehicle dealership or service garage, situations where it is easy to determine environmental conditions for calibration include highway toll booths, gas stations, charging stations (for EVs (electric vehicles)), highway parking lots, and service areas. It is also effective to use a calibration target (e.g., with a calibration pattern drawn on it) installed at a specified position and distance from the reference position to easily perform a calibration sequence at these locations while traveling. In particular, to automatically perform the calibration process without the driver's knowledge, the vehicle may be temporarily made to pass through a dedicated lane on which a calibration pattern is drawn.

[0153] FIG. 8 is a flowchart showing an example of the operation of the calibration process controlled by the calibration processing unit 170.

[0154] The calibration processing unit 170 detects a calibration irradiation target based on at least one of the LiDAR 52, the radar 53, the camera 51, etc. while the vehicle is running or stopped. Alternatively, information indicating the position and direction of the irradiation target may be acquired from another device (S101). When using an image from the camera 51, the image from the camera 51 may be subjected to conventional demosaicing and conventional LTM processing without using mapping information from the illumination / camera space mapping unit 120. However, the image data may be acquired from the camera 51 via processing by the local demosaicing unit 130 and the LTM unit 140, or may be acquired from the camera 51 via processing by the local demosaicing unit 130 and the region-designating LTM unit 150.

[0155] The calibration processing unit 170 sets up linkage with the camera 51 and generates a check pattern for irradiation (S102). The check pattern is associated with the irradiation map (grid coordinates) of the AFS lamp light distribution control unit 210. In other words, the calibration processing unit 170 determines an area to be irradiated with the check pattern (specific pattern) within the range defined by the illumination map, updates the illumination map to indicate the determined area, and instructs the AFS lamp light distribution control unit 210 to irradiate the check pattern onto the irradiation target based on this illumination map.

[0156] Figure 9 shows an example of an illuminated check pattern (cross check pattern) in the grid coordinate system (illumination map) of the AFS lamp light distribution control unit 210. Point 401 (intersection of the solid cross) is the origin (∞) of the AFS grid reference coordinates recognized by the AFS lamp light distribution control unit 210, i.e., the point (second reference position) recognized as corresponding to the infinity direction recognized by the AFS lamp light distribution control unit 210.

[0157] Point 402 (the intersection of the dashed cross lines) is a point (position) corresponding to the true infinity direction (first reference direction). Points 401 and 402 are shifted by Δφ in the vertical axis direction of the illumination grid coordinate system (the vertical direction parallel to the paper surface). There is a similar shift in the horizontal axis direction. In other words, the reference position (second reference position) of the illumination map actually corresponds to a direction (second reference direction) different from the true infinity direction.

[0158] The projected cross-check pattern consists of two cross-checks, each consisting of a pair of rectangles arranged diagonally from each other and connected at their vertices, and these are referred to as cross-checks L1 and L2. In each cross-check, the point where the vertices of the two rectangles meet is also called the intersection.

[0159] The cross check L1 is a distance φ １ The cross check L2 is located at a distance φ ２ The cross checks L1 and L2 are positioned at the same distance to the left of the point 401 on the horizontal axis.

[0160] The calibration processing unit 170 acquires image data from the camera 51 representing an image including an illumination target from the reflected light of a check pattern illuminated from the headlight 210A in front of the vehicle 1 under the control of the AFS lamp light distribution control unit 210 (S103).

[0161] FIG. 10 shows cross checks P1 and P2 in image data acquired from camera 51. Cross check P1 is an image corresponding to cross check L1 illuminated from headlight 210A under the control of AFS lamp light distribution control unit 210, and cross check P2 is an image corresponding to cross check L2 illuminated from headlight 210A. Point 403 (the intersection of the solid crosses) is a point (first reference position) corresponding to the infinity direction (first reference direction) in the image coordinate system. The camera 51 is calibrated for infinity by periodic calibration, and the position of point 403 is assumed to be correct. Cross check P1 is separated from point 403 by a distance x1 downward along the vertical axis, and cross check P2 is separated from point 403 by a distance x2 downward along the vertical axis. Furthermore, cross checks P1 and P2 are each separated from point 403 by the same distance to the right along the horizontal axis.

[0162] The calibration processing unit 170 identifies feature points of the check patterns from the image data (in the example of FIG. 10, the intersections (cross points) of cross check P1 and cross check P2) (S104), and reads out information on the identified feature points as information on the positional relationship of each check pattern with respect to the first reference position. In other words, the positional relationship information includes information on the positional relationship between point 403 (first reference position) and each check pattern (specific pattern). The calibration processing unit 170 also identifies the positional relationship between a second reference position in the grid coordinate system (illumination map) (a position corresponding to a direction (second reference direction) different from the true infinity direction) and the irradiation area of ​​the check pattern. Based on the identified positional relationship and the above positional relationship information, the calibration processing unit 170 calculates the difference Δφ in the infinity direction (see FIG. 9) (S105).

[0163] FIG. 11 shows an example of calculating the difference Δφ in the infinity direction. Points (φ1, x1) and (φ2, x2) are mapped onto a coordinate system with φ on the horizontal axis and x on the vertical axis, and a straight line passing through these points is calculated. The distance between the origin and point J1 where the line intersects with the horizontal axis is calculated as the difference Δφ. In this example, the difference Δφ is calculated by extrapolating from a linear change. The explanation is based on the assumption that the camera 51 is a central projection lens, and that the coordinates of the projection instruction of the AFS lamp light distribution control unit 210 are backprojected from the central projection, just like the camera 51. However, the lens projection method is not limited to central projection.

[0164] Here, an example of calculating the difference in the vertical axis direction has been shown, but the difference in the horizontal axis direction toward infinity can also be calculated in the same way.

[0165] It is desirable to perform the above-mentioned calibration process whenever an event occurs that changes the orientation of the camera 51, etc. The distance between the camera 51 and the headlight 210A generally does not change unless the mounting arrangement is mechanically changed. On the other hand, it is expected that the mounting angle (direction) of the camera 51 and the headlight 210A can easily shift due to various conditions such as distortion of the vehicle body, impact, and mounting method. It is desirable to first perform this calibration process when manufacturing is completed prior to shipping the vehicle. Furthermore, because the mounting orientation can change due to various factors such as twisting of the vehicle body, mounting misalignment, and temperature distortion, it is particularly desirable to perform the calibration process for the reference infinity angle periodically.

[0166] Once the infinity direction (infinite point) of the camera 51 and the infinity direction (infinite point) of the AFS lamp light distribution control unit 210 are determined in this manner, the lighting / camera space mapping unit 120 can more accurately estimate the area corresponding to the above-mentioned illumination area in the image captured by the camera 51 by referring to the data in the illumination grid map (illumination map) of the AFS lamp light distribution control unit 210 for the illumination area of ​​a distant object (i.e., estimate which area in the camera coordinate system corresponds to the area into which reflected light of light irradiated from the headlight 210A enters).

[0167] In an environment where there is ambient light that is a mixture of natural light, such as that from sunrise or sunset, and street light, the spectral characteristics of each area captured by camera 51 (estimation of the headlight incidence area and natural light incidence area in the image of camera 51) may be estimated by performing weighting processing for each light source (headlight 210A, light source of ambient light) based on the brightness of the ambient light estimated by AFS from areas in the image where light from headlight 210A is not incident, and the ambient light detected by the vehicle itself using an external light receiving element, etc.

[0168] Other than the calibration of the camera 51 and the AFS in the infinity direction, as described above, a directional shift corresponding to parallax occurs between the headlight illumination (light source illumination) and the image captured by the camera 51, depending on the installation interval between the headlight 210A and the camera 51 and the distance from each to the object. The distance to the object may be recognized by the camera 51 itself, or the distance may be acquired from another device, and the amount of shift may be estimated based on trigonometry to correct the illumination map, which is a grid map of the area illuminated by the AFS.

[0169] However, calculating all light source deviations (direction deviations) on a pixel-by-pixel basis is not effective because it imposes a heavy processing load.

[0170] Here, light from the headlight 210A by the AFS is irradiated as a closed collective area as a mass, and the area irradiated by the same light source (headlight 210A) is distributed widely and shifts in a predetermined direction by the parallax of the area of ​​the object depending on the distance to the object. Also, various techniques for extracting areas from images have been developed, and for example, a technique is widely known in which a person participating in a telephone conference, etc. from their own room in front of the camera, separates the background and face area and replaces the background image with another image.

[0171] Therefore, in the present disclosure, similarly, the headlight incident region (visible light incident region of the AFS) in the captured image is specified using a region extraction technique based on the AFS illumination grid map (illumination map), segmentation is performed to compensate for the region movement caused by the parallax, and information on the boundary movement directionality obtained by the segmentation (the direction of movement and the amount of movement according to the parallax) is set as a condition in the illumination / camera space mapping unit 120, thereby improving the accuracy of region estimation (mapping accuracy). Such processing for improving the accuracy of region specification (setting of mapping information) is performed under the control of the calibration processing unit 170.

[0172] It is anticipated that future advances in AFS technology will enable more limited and precise illumination, such as illumination of road signs. It is also anticipated that future advances will allow for control of the illumination map or AFS illumination control, taking into account parallax with the image captured by the camera 51. In cases where such control is possible, instead of performing simple estimation using the shift movement estimation described above, correction data may be passed to the illumination / camera space mapping unit 120 along with the illumination map (AFS illumination direction information) to update the area estimate of the headlight illumination area (AFS light source application area estimate) within the image captured by the camera 51. However, since there are few benefits, such strict correction processing is not required for image recognition applications for sensing. When illuminating an object with the headlight 210A, the boundary of the illuminated area is usually gradually attenuated, and strict matching control of illumination to the object is not required. However, if the effect of parallax is not taken into account at all, significant parallax deviations can lead to area determination and unnatural image processing in subsequent stages, and therefore it is desirable to avoid this.

[0173] Mounting the headlight 210A and the camera 51 at separate locations results in a significant offset in parallax. That is, as shown in FIG. 12 , the origins of the field-of-view angles of the headlight 210A and the camera 51 are installed at positions separated by a base line length D. Therefore, even if the direction of the field-of-view angle of the camera 51 and the direction of the headlight 210A of the AFS are aligned, a shift in the incident area of ​​the headlight 210A (particularly at its boundary) occurs in the image captured by the camera 51, including the object OB1 (which is not located at infinity). Similarly, a shift in the incident area of ​​the headlight 210A occurs in the image captured by the camera 51 for the object OB2. This shift depends on the distances L and 2L (2L is twice L) to the illuminated objects OB1 and OB2 (i.e., objects that block and reflect light). As shown in FIG. 13 , the amount of shift between the illuminated area 411 at distance L and the illuminated area 412 at distance 2L in the image captured by the camera 51 is inversely proportional to the distances L and 2L. In other words, the deviation amount of the distance L is larger than the deviation amount of the distance 2L.

[0174] If the distance to the object is considered to be infinite, and calibration processing has been performed, the parallax will be 0 (zero), and the direction of the camera 51 will match the direction of the illumination grid map (illumination map) of the headlight 210A (AFS). On the other hand, if the object is at a finite distance and is illuminated by the headlight 210A, the illuminated area will be shifted by an amount of shift 410 (see FIG. 12 ) that is inversely proportional to the direction in which the camera 51 and the headlight 210A (AFS) are separated from each other (disposition separation direction) depending on the distance.

[0175] In other words, the amount of deviation in the designated direction (orientation) expected depending on the distance (X) to the area hit by the illumination light changes by a factor of 1 / X. Figure 12 shows the deviation amounts h(L) and h(2L) in the designated direction (downward along the diagram) expected depending on the distance to the objects OB1 and OB2. Therefore, at a distance (X → ∞ (infinity)), the deviation becomes infinitely small (close to "0") as 1 / X, and the direction of the camera 51 essentially coincides with the grid map direction of the AFS (direction of the illumination map).

[0176] On the other hand, as the vehicle gradually approaches the illuminated object while driving, the AFS illumination range (illumination range of headlight 210A) illuminated on an object that was previously visible in the same distant direction, such as a pedestrian on the side of the road or a road sign, expands to 1 / X times the distance X to the object.

[0177] Therefore, when the illumination direction of the headlight 210A by the AFS is illuminating an object at a finite distance, the illumination grid map (illumination map) of the headlight 210A needs to be corrected by shifting it toward the baseline length side (upward along the diagram) by the amount of the effect of this parallax.

[0178] Because drivers tend to neglect their attention when staring at a screen without looking ahead, drivers typically do not drive while visually monitoring the screen displaying the image captured by the forward-facing camera. While forward-facing cameras are sometimes used for indirect visual monitoring, such as in dashcams, they are primarily used for recognition processing (machine recognition). Calculating and correcting the deviation of the boundary contours of the AFS headlight illumination area precisely on a pixel-by-pixel basis offers little cost-to-noise benefit, so precise pixel-by-pixel correction is not necessary. Instead, for the recognition of road signs and pedestrians, proper demosaicing and color reproduction adjustment, such as facial expressions, are desirable. An effective solution is to segment approaching objects that cannot be considered infinitely distant as the vehicle moves, separating them from the cluster area, and then shifting (correcting) the illumination grid map (illumination map) so that the separated objects gradually move away from the vanishing point and toward the AFS lights. (A cluster of obstacles, road signs, etc. is recognized as a single recognition target, and if they are not close to each other, their unevenness can be ignored compared to the distance to the object.) This allows adaptive image development demosaicing and color correction processing to be performed according to the AFS headlight illumination.

[0179] In addition to controlling the direction of visible light illumination using the AFS, similar correction processing may also be performed on infrared light illumination. Furthermore, since this change is a smooth transition that occurs with movement, smoothing processing and continuous estimation processing may be performed smoothly along the image frames captured while driving to estimate and correct the area.

[0180] As mentioned above, a collection of obstacles, road signs, etc. is perceived as a single recognition target, and if they are not close together, their unevenness can be ignored compared to the distance to the object, so the distance from the vehicle is considered to be roughly constant and is used in the shift (correction) process described above. In other words, the range of segmentation of vehicles, pedestrians, road signs, etc., will not be affected by shifting them uniformly as a single collection, even if the distance from the vehicle to the object varies slightly due to unevenness on the surface.

[0181] The displacement amount corresponding to the parallax described above is applied to the entire (segmented) area of ​​the captured object, and the area in the grid map originally specified at an infinite angle is shifted and applied. Here, the offset of the detected object from the optical axis of the vehicle-mounted camera 51 in the traveling direction is not known in advance, and the area estimated from the area detected in the previous frame of the video as the vehicle progresses is applied as the shifted area. For example, the method described above, i.e., the area is estimated to expand at a rate of "1 / X" as the distance X shrinks as the vehicle progresses, is applied. Note that information on the distance to the object is obtained using an external sensor 50 (such as LiDAR, RADAR, or stereo camera) or a single-unit distance estimation using AI. Performing this process enables area correction, including shifts in the AFS illumination area boundary due to differences in the placement positions of the camera 51 and the headlights 210A.

[0182] (Another Operation Example 1) Although the above mainly shows an example of estimating the headlight incident area, similar processing can be performed for infrared light source 220A with a fixed irradiation range. Furthermore, as a simple method, for example, at night, the center of gravity (e.g., the center) of the infrared light pixel values ​​in image data may be calculated, and a certain range from the center of gravity determined by the infrared light illumination light distribution information, size, etc. of infrared light source 22 may be set as the infrared light incident area (second area). This is because infrared light source 220A with a fixed irradiation range is basically directed into the distance and its range is not large, and therefore it is considered that the infrared light incident area will not vary much depending on the driving environment.

[0183] (Other Operation Example 2) In the above-described embodiment, it is mainly assumed that the irradiation range of the infrared light source 220A is fixed, but as described above, the irradiation range may be adjustable similarly to the headlight 210A. In this case, the processing related to the headlight and AFS in the processing of the above-described embodiment can be similarly applied to the infrared light source 220A.

[0184] For example, the illumination map generation unit 110 of the processing unit 100 determines an area to be illuminated within the range that can be illuminated by the infrared light source 220A. For example, the illumination map generation unit 110 determines an area to be illuminated from an area that is not illuminated by the headlight 210A. The illumination map generation unit 110 updates a second map having a coordinate system that indicates an area to be illuminated within the range that can be illuminated by the infrared light source 220A, so that the second map indicates the determined area. If illumination from the headlight 210A is also to be performed, the second map may be updated to further indicate an area to be illuminated by the headlight 210A.

[0185] The illumination map generation unit 110 sends the second map to the infrared light illumination control unit 220 and the illumination / camera space mapping unit 120. If illumination from the headlight 210A is also to be performed, the second map is also sent to the AFS lamp light distribution control unit 210. The infrared light illumination control unit 220 (control unit) controls the infrared light source 220A to emit infrared light based on the second map. The AFS lamp light distribution control unit 210 (control unit) controls the headlight 210A to emit visible light based on the second map.

[0186] Furthermore, calibration processing may be performed on the infrared light source 220A in the same manner as for the headlight 210A. For example, consider a case where the reference position (first reference position) in the coordinate system of the camera 51 corresponds to the direction of infinity (first reference direction) of the camera 51, and the reference position (third reference position) in the second map corresponds to the reference direction (third reference direction) of the infrared light source 220A. Using a method similar to that of the above-described embodiment, an area onto which the image of a specific pattern is irradiated by the infrared light source 220A is determined, the second map is updated to indicate the area of ​​the image of the determined specific pattern, and the infrared light irradiation control unit 220 controls the infrared light source 220A to irradiate infrared light based on the second map. The processing unit 100 (calibration processing unit 170) detects the specific pattern from the image data of the camera 51 and calculates the positional relationship between the position of the specific pattern in the image data and the first reference position, and the positional relationship between the position of the irradiation area of ​​the specific pattern in the second map and the third reference position. Based on the calculated positional relationship, a position corresponding to the first reference direction is identified in the second map, and the third reference direction is updated to the direction corresponding to the identified position, thereby matching the third reference direction of the infrared light source 220A with the first reference direction of the camera 51. Matching the reference directions in this manner can improve the accuracy of estimating the infrared light incident region (second region) in the coordinate system (or image data) of the camera 51.

[0187] As with the estimation of the headlight incident area (first area) described in the above embodiment, various methods can be used to estimate the second area. For example, the illumination map generation unit 110 updates the second map to indicate the area to be illuminated with infrared light, and the infrared light illumination control unit 220 controls the infrared light source 220A to illuminate the infrared light based on the second map. The illumination / camera space mapping unit 120 determines (estimates) the second area based on the distance to the object to be illuminated with infrared light, the position of the illuminated area in the second map, the installation position / installation direction of the infrared light source 220A, and the installation position / installation direction of the camera 51. As another estimation method, the second area may be determined based on the distribution of pixel values ​​of infrared light in the image data of the camera 51.

[0188] (Other Operation Example 3) When the recognition processing unit 160 detects a predetermined object based on image recognition of the second area where infrared light is incident, for example, at night, the AFS lamp light distribution control unit 210 may control the headlights 210A to irradiate the area including the predetermined object with visible light. The predetermined object may be, for example, a road sign or text guidance information. By displaying the image irradiated with visible light in color, color reproducibility is improved, allowing the driver to check road signs that they may have overlooked.

[0189] (Other Operation Example 4) In the above examples, the illumination area is calculated and determined based on the fact that the distribution of the illumination area of ​​visible light or infrared light by the AFS in the captured image is determined by geometric constraints, but the method of specifying the illumination area does not need to be limited to calculation from geometric constraints. For example, just as a person can visually examine a captured image and accurately distinguish the area illuminated by the AFS from the surrounding area and draw its outline to a certain extent, the illumination area may be determined by learning to extract and determine the illumination area of ​​the AFS or infrared using artificial intelligence. Alternatively, the accuracy of area detection may be improved by combining the method of calculation from geometric constraints and the method of extraction using artificial intelligence.

[0190] (Application Example) If the camera 51 is installed at the rear of the vehicle and the vehicle's brake lights are a potential light source (a light source emitting red light), white balance processing may be performed to prevent the white balance from being affected by the light emitted by the light source. White balance processing is a correction process performed by estimating the color temperature of the light source to make a white subject appear white. However, if the rear camera 51 estimates the color temperature using the vehicle's brake lights as the light source, an unnatural white balance may be applied. Therefore, the image data from the camera 51 may be divided into a brake light incident area (an area where reflected light from the brake lights is incident) where the brake lights are illuminated and other areas, and white balance processing may be performed separately for each area. In the brake light incident area, the color temperature is estimated using the brake lights as a light source, while in other areas, the color temperature is estimated using natural light as a light source. This allows the red color of the brake lights to be preserved in the image data while avoiding an unnatural white balance of the surroundings.

[0191] Here, we have described the case where the brake lights of the vehicle in question are the light source, but even if the light emitted from the brake lights of the vehicle in front is incident on the camera 51 in front of the vehicle as a light source, if the area where the light is incident can be identified, white balance processing by area division can be performed in the same manner as above.

[0192] As described above, according to this embodiment, image processing is performed that is locally optimized for each region of the image data based on information about the region in the image data where reflected light of visible light irradiation by the AFS is incident and the region in the image data where reflected light of infrared light irradiation is incident. This makes it possible to reduce the occurrence of false colors and jagged contours, and to quickly recognize information by appropriately displaying road signs and the like in their original colors, thereby improving the nighttime recognition performance required for autonomous driving.

[0193] Furthermore, the various road signs that vehicles using autonomous driving encounter do not only provide immediate information, but also many signs that warn drivers of upcoming roads. When a driver is required to take over (handing over driving from autonomous to manual driving), it is also necessary to enhance situational awareness after the handover by displaying and informing the following driver of the relevant sign after passing it. For this reason, it is expected to be effective to present visual road sign images with more accurate color reproduction to drivers, even in vehicles equipped with autonomous driving functions, and to encourage early situational awareness.

[0194] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0195] Furthermore, the effects of the present disclosure described in this specification are merely examples, and other effects may also be present.

[0196] The present disclosure may also be configured as follows: [Item 1] An information processing system including: a visible light irradiating unit that is mounted on a vehicle and irradiates visible light in a predetermined direction of the vehicle, an infrared light irradiating unit that is mounted on the vehicle and irradiates infrared light in the predetermined direction of the vehicle, a control unit that controls an irradiation range of at least one of the visible light irradiating unit and the infrared light irradiating unit, an imaging unit that captures an image of the vehicle in the predetermined direction based on light incident on an imaging area and generates image data, and a processing unit that performs separate image signal processing on the image data for each of the first area and the second area based on first information on a position of a first area in the imaging area onto which the visible light is incident and second information on a position of a second area onto which the infrared light is incident. [Item 2] The information processing system according to Item 1, wherein the processing unit determines an area to be irradiated within an irradiatable range of the visible light irradiation unit, and updates a first map indicating an area to be irradiated within an irradiatable range of the visible light irradiation unit to indicate the determined area; the control unit controls the visible light irradiation unit to irradiate visible light based on the first map; and the processing unit determines the first area based on a distance to an object to be irradiated with the visible light, a position of the area to be irradiated in the first map, an installation position of the visible light irradiation unit, and an installation position of the imaging unit.the first reference direction of the visible light irradiator; the second reference direction of the visible light irradiator; the processing unit; a first reference position in a coordinate system of the imaging unit; a second reference position of the first map; a second reference direction of the visible light irradiator; the processing unit; a region where an image of a specific pattern is irradiated within an irradiable range of the visible light irradiator; and an update of the first map to indicate the determined region of the image of the specific pattern; the control unit; a control unit; a control unit; a control unit; a processing unit; a first reference position in a coordinate system of the imaging unit; a first reference direction of the imaging unit; a second reference direction of the visible light irradiator; [Item 4] The information processing system according to any one of items 1 to 3, wherein the processing unit determines an area to be irradiated within an irradiatable range of the infrared light irradiating unit, and updates a second map indicating an area to be irradiated within an irradiatable range of the infrared light irradiating unit to indicate the determined area; the control unit controls the infrared light irradiating unit to irradiate infrared light based on the second map; and the processing unit determines the second area based on a distance to an object to be irradiated with the infrared light, a position of the area to be irradiated in the second map, an installation position of the infrared light irradiating unit, and an installation position of the imaging unit.Item 5. The information processing system according to Item 4, wherein a first reference position in a coordinate system of the imaging unit corresponds to a first reference direction of the imaging unit; a third reference position of the second map corresponds to a third reference direction of the infrared light irradiating unit; the processing unit determines an area to be irradiated with an image of a specific pattern within an irradiatable range of the infrared light irradiating unit, and updates the second map to indicate the determined area of ​​the image of the specific pattern; the control unit controls the infrared light irradiating unit to irradiate infrared light based on the second map; and the processing unit detects the specific pattern from the image data and identifies a position in the second map corresponding to the first reference direction based on a positional relationship between a position of the specific pattern in the image data and the first reference position and a positional relationship between a position of the area to be irradiated in the second map and the third reference position, and updates the third reference direction to a direction corresponding to the identified position, thereby aligning the third reference direction of the infrared light irradiating unit with the first reference direction of the imaging unit. [Item 6] The information processing system according to any one of items 1 to 5, wherein the processing unit determines the second region based on a distribution of pixel values ​​of the infrared light. [Item 7] The information processing system according to any one of items 1 to 6, wherein an irradiation range of the infrared light irradiation unit is fixed and the irradiation range cannot be controlled by the control unit. [Item 8] The information processing system according to item 7, wherein a center of gravity of pixel values ​​of the infrared light in the image data is calculated and the second region is determined based on the center of gravity. [Item 9] The information processing system according to item 7 or 8, wherein the processing unit extends the boundary of the second region outward by a certain width, and wherein the processing unit performs the image signal processing separately for the second region after the extension and the first region. [Item 10] The information processing system according to any one of items 1 to 9, wherein the processing unit performs demosaic processing separately for the first region and the second region in the image data. [Item 11] The information processing system according to item 10, wherein the processing unit performs local tone mapping processing separately for the first region and the second region based on the image data after the demosaic processing.[Item 12] The information processing system according to Item 11, wherein the processing unit performs image recognition based on the image data after the local tone mapping process. [Item 13] The information processing system according to Item 12, wherein the processing unit performs the image recognition separately for the first region and the second region in the image data after the local tone mapping process. [Item 14] The information processing system according to any one of Items 10 to 13, wherein the processing unit calculates a weighted sum of pixel values ​​resulting from the demosaicing process in an area where the first region and the second region overlap. [Item 15] The information processing system according to Item 13, wherein the processing unit calculates a weighted sum of pixel values ​​resulting from the local tone mapping process in an area where the first region and the second region overlap. [Item 16] The information processing system according to Item 13 or 15, wherein, when a predetermined object is detected based on image recognition of the first region by the control unit and the processing unit, the control unit controls the visible light irradiation unit to irradiate the visible light onto an area including the predetermined object. [Item 17] The information processing system according to item 16, wherein the predetermined object is a road sign. [Item 18] The information processing system according to any one of items 1 to 17, wherein the imaging unit is a camera having an imaging area in which red pixels, green pixels, blue pixels, and infrared light pixels are arranged in an array in a predetermined arrangement, or a camera having an imaging area in which red pixels, green pixels, and blue pixels are arranged in an array in a predetermined arrangement, and at least one of the red pixels, the green pixels, and the blue pixels is also sensitive to infrared light. [Item 19] The information processing system according to any one of items 1 to 18, wherein the predetermined direction is forward or backward of the vehicle.[Item 20] An information processing method comprising: a device mounted on a vehicle, irradiating visible light in a predetermined direction of the vehicle; a device mounted on the vehicle, irradiating infrared light in the predetermined direction of the vehicle; controlling an irradiation range of at least one of the visible light irradiating unit and the infrared light irradiating unit; capturing an image of the predetermined direction of the vehicle based on light incident on an imaging area to generate image data; and performing separate image signal processing on the image data for each of the first area and the second area based on first information on a position of a first area in the imaging area onto which the visible light is incident and second information on a position of a second area onto which the infrared light is incident.

[0197] 1 Vehicle 11 Vehicle control system 21 Vehicle control ECU (Electronic Control Unit) 22 Communication unit 23 Map information storage unit 24 Position information acquisition unit 25 External recognition sensor 26 In-vehicle sensor 27 Vehicle sensor 28 Memory unit 29 Driving automation control unit 30 Driver monitoring system (DMS) 31 Human-machine interface (HMI) 32 Vehicle control unit 41 Communication network 51 Camera 52 LiDAR 53 Radar 54 Ultrasonic sensor 61 Recognition processing unit 62 Action planning unit 63 Operation control unit 81 Steering control unit 82 Brake control unit 83 Drive control unit 84 Body system control unit 85 Light control unit 86 Horn control unit 100 Processing unit 110 Illumination map generation unit 110 Illumination map 120 Camera space mapping unit 130 Local demosaicing unit 140 LTM unit 150 Area designation LTM unit 160 Recognition processing unit 170 Calibration processing unit 210 AFS lamp light distribution control unit 210A Headlight 220 Infrared light irradiation control unit 220A Infrared light source 230 Transmission unit 240 Display 300 Viewing angle 301 Coordinate system 310 Area 311 Infrared light incidence area 311A ​​Area 320 Area 321 Headlight incidence area 331 Ambient light incidence area 401 Point 402 Point 403 Point 410 Deviation amount 411 Irradiation area 412 Irradiation area

Claims

1. An information processing system comprising: a visible light irradiation unit that is mounted on a vehicle and irradiates visible light in a predetermined direction of the vehicle; an infrared light irradiation unit that is mounted on the vehicle and irradiates infrared light in the predetermined direction of the vehicle; a control unit that controls the irradiation range of at least one of the visible light irradiation unit and the infrared light irradiation unit; an imaging unit that images the predetermined direction of the vehicle based on light incident on an imaging area and generates image data; and a processing unit that performs separate image signal processing on the image data for each of the first area and the second area based on first information on the position of a first area in the imaging area into which the visible light is incident and second information on the position of a second area into which the infrared light is incident.

2. The information processing system of claim 1, wherein the processing unit determines an area to be irradiated within an irradiable range of the visible light irradiation unit, and updates a first map indicating an area to be irradiated within an irradiable range of the visible light irradiation unit to indicate the determined area; the control unit controls the visible light irradiation unit to irradiate visible light based on the first map; and the processing unit determines the first area based on the distance to the object to be irradiated with the visible light, the position of the area to be irradiated in the first map, the installation position of the visible light irradiation unit, and the installation position of the imaging unit.

3. The information processing system of claim 2, wherein a first reference position in the coordinate system of the imaging unit corresponds to a first reference direction of the imaging unit; a second reference position of the first map corresponds to a second reference direction of the visible light irradiation unit; the processing unit determines an area within an irradiable range of the visible light irradiation unit onto which an image of a specific pattern is irradiated, and updates the first map to indicate the determined area of ​​the image of the specific pattern; the control unit controls the visible light irradiation unit to irradiate visible light based on the first map; the processing unit detects the specific pattern from the image data, and identifies a third position in the first map corresponding to the first reference direction based on the positional relationship between the position of the specific pattern in the image data and the first reference position and the positional relationship between the position of the area to be irradiated in the first map and the second reference position, and updates the second reference direction to a direction corresponding to the identified third position, thereby aligning the second reference direction of the visible light irradiation unit with the first reference direction of the imaging unit.

4. The information processing system of claim 1, wherein the processing unit determines an area to be irradiated within the irradiable range of the infrared light irradiation unit, and updates a second map indicating the area to be irradiated within the irradiable range of the infrared light irradiation unit to indicate the determined area; the control unit controls the infrared light irradiation unit to irradiate infrared light based on the second map; and the processing unit determines the second area based on the distance to the object to be irradiated with the infrared light, the position of the irradiated area in the second map, the installation position of the infrared light irradiation unit, and the installation position of the imaging unit.

5. The information processing system of claim 4, wherein a first reference position in the coordinate system of the imaging unit corresponds to a first reference direction of the imaging unit, a third reference position of the second map corresponds to a third reference direction of the infrared light irradiation unit, the processing unit determines an area within the irradiatable range of the infrared light irradiation unit onto which an image of a specific pattern is irradiated and updates the second map to indicate the determined area of ​​the image of the specific pattern, the control unit controls the infrared light irradiation unit to irradiate infrared light based on the second map, and the processing unit detects the specific pattern from the image data and identifies a position in the second map corresponding to the first reference direction based on the positional relationship between the position of the specific pattern in the image data and the first reference position and the positional relationship between the position of the area to be irradiated in the second map and the third reference position, and updates the third reference direction to a direction corresponding to the identified position, thereby aligning the third reference direction of the infrared light irradiation unit with the first reference direction of the imaging unit.

6. The information processing system according to claim 1, wherein the processing unit determines the second region based on a distribution of pixel values ​​of the infrared light.

7. The information processing system according to claim 1, wherein the irradiation range of the infrared light irradiation unit is fixed, and the irradiation range cannot be controlled by the control unit.

8. The information processing system according to claim 7, wherein the center of gravity of the pixel values ​​of the infrared light in the image data is calculated, and the second region is determined based on the center of gravity.

9. The information processing system according to claim 7, wherein the processing unit extends the boundary of the second region outward by a fixed width, and the processing unit performs separate image signal processing for each of the expanded second region and the first region.

10. The information processing system according to claim 1, wherein the processing unit performs demosaic processing separately on the first region and the second region in the image data.

11. The information processing system according to claim 10, wherein the processing unit performs local tone mapping processing separately for the first region and the second region based on the image data after the design processing.

12. The information processing system according to claim 11, wherein the processing unit performs image recognition based on the image data after the local tone mapping process.

13. The information processing system according to claim 12, wherein the processing unit performs the image recognition separately for each of the first region and the second region in the image data after the local tone mapping process.

14. The information processing system according to claim 10, wherein the processing unit calculates a weighted sum of pixel values ​​resulting from the demosaic processing in an area where the first area and the second area overlap.

15. The information processing system according to claim 13, wherein the processing unit calculates a weighted sum of pixel values ​​resulting from the local tone mapping process in an area where the first area and the second area overlap.

16. The information processing system of claim 13, wherein when a predetermined object is detected based on image recognition of the first area in the control unit and the processing unit, the control unit controls the visible light irradiation unit to irradiate the visible light onto an area including the predetermined object.

17. The information processing system according to claim 16, wherein the predetermined object is a road sign.

18. The information processing system of claim 11, wherein the imaging unit is a camera having an imaging area in which red pixels, green pixels, blue pixels, and infrared light pixels are arranged in an array in a predetermined arrangement, or a camera having an imaging area in which red pixels, green pixels, and blue pixels are arranged in an array in a predetermined arrangement, and at least one of the red pixels, green pixels, and blue pixels is also sensitive to infrared light.

19. The information processing system according to claim 1, wherein the predetermined direction is the front or rear of the vehicle.

20. An information processing method comprising: a device mounted on a vehicle, irradiating visible light in a predetermined direction of the vehicle; a device mounted on the vehicle, irradiating infrared light in the predetermined direction of the vehicle; controlling the irradiation range of at least one of the visible light irradiating unit and the infrared light irradiating unit; capturing an image of the predetermined direction of the vehicle based on light incident on an imaging area to generate image data; and performing separate image signal processing on the image data for each of the first and second areas based on first information regarding the position of a first area in the imaging area into which the visible light is incident and second information regarding the position of a second area into which the infrared light is incident.

Images