Image recording system, vehicle, program, and image recording method of image recording system

Abstract

To suppress unnecessary switching of a recording object of an image recording part.SOLUTION: An image recording unit configured to record one of an infrared image captured by the infrared camera and a visible light image captured by the visible light camera, wherein in a case where a recording target of the image recording unit is the visible light image, when illuminance around the vehicle detected by an illuminance sensor of the vehicle or luminance of the visible light image is less than an infrared switching threshold value, the recording target of the image recording unit is switched from the visible light image to the infrared image, even if the illuminance around the vehicle detected by the illuminance sensor of the vehicle or the luminance of the visible light image becomes less than the infrared light switching threshold, the recording target of the image recording unit is not switched from the visible light image to the infrared light image when a preset switching restriction condition is satisfied based on at least one of the current time, the position information and the map information of the vehicle, the temporal change of the illuminance, the temporal change of the luminance, and the image recognition result of the visible light image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image recording system mounted on a vehicle, a vehicle, a program, and an image recording method for the image recording system. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2022-35319 is a known technical document relating to an image recording system mounted on a vehicle. This publication describes a driver assistance system having two functions: a function for displaying images with color tones suitable for monitoring the vehicle's surroundings, and a function for recording images with color tones suitable for use in a drive recorder. This driver assistance system determines whether it is night or day based on the illuminance around the vehicle, and switches the recording target so that images with color tones suitable for monitoring the vehicle's surroundings are recorded during the day and images with color tones suitable for use in a drive recorder at night. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-35319 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if the determination of night or day is made based solely on illuminance, as in the conventional system described above, an erroneous determination may occur when the vehicle enters a low-illuminance environment such as a tunnel. As a result, there is a problem that the image is corrected as if it were nighttime even though it is daytime. Furthermore, in the case of a system that uses a visible light camera during the day and an infrared camera at night, there is also a problem that the visible light image and the infrared image are mixed when passing through a tunnel. [Means for solving the problem]

[0005] One aspect of the present invention is an image recording system having an infrared camera mounted on a vehicle that captures images of the vehicle's surrounding environment using infrared rays, a visible light camera mounted on the vehicle that captures images of the vehicle's surrounding environment using visible light, and an image recording unit that records either the infrared image captured by the infrared camera or the visible light image captured by the visible light camera, wherein when the image recording unit is recording a visible light image, if the illuminance around the vehicle detected by the vehicle's illuminance sensor or the luminance of the visible light image falls below an infrared switching threshold, the image recording unit switches the recording object from the visible light image to the infrared image, and even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, the image recording unit does not switch the recording object from the visible light image to the infrared image if a preset switching restriction condition is satisfied based on at least one of the current time, vehicle position information and map information, change in illuminance over time, change in luminance over time, and image recognition results of the visible light image.

[0006] According to an image recording system of one aspect of the present invention, even if the illuminance around the vehicle or the brightness of the visible light image falls below the infrared switching threshold, when the switching restriction condition is met based on at least one of the current time, vehicle position information and map information, change in illuminance over time, change in brightness over time, and image recognition results of the visible light image, unnecessary switching of the recording target of the image recording unit can be suppressed by not switching the recording target of the image recording unit from a visible light image to an infrared image.

[0007] In an image recording system according to one aspect of the present invention, the switching restriction condition may be satisfied if the current time is before a preset sunset time or if the current time is within a preset daytime period. According to this image recording system, when the current time is before sunset or when the current time is within the daytime hours, the switching restriction condition is met, thereby preventing unnecessary switching to infrared images during bright daytime hours.

[0008] In an image recording system according to one aspect of the present invention, the map information may include a low-illuminance section including a tunnel, and if it is determined that the vehicle has entered a low-illuminance section based on the vehicle's position information and the map information, the switching restriction condition may be satisfied. This image recording system can avoid unnecessary image switching by satisfying the switching restriction condition when it is determined from the vehicle's position information and map information that the vehicle has entered a low-illuminance section. This makes it possible to continue recording visible light images even in low-illuminance environments such as inside tunnels and indoor parking lots, and maintain the consistency of the recorded images.

[0009] In the image recording system according to one aspect of the present invention, when it is determined that the vehicle has entered a low illumination section based on a time change in illumination or a time change in brightness, the switching restriction condition may be satisfied. According to this image recording system, when it is determined that the vehicle has entered a low-illuminance section based on the change in illuminance or brightness over time, the switching restriction condition is satisfied, thereby avoiding unnecessary image switching. This allows the recording of visible light images to continue even in low-illuminance sections such as tunnels and indoor parking lots, and maintains the consistency of the recorded images.

[0010] In the image recording system according to one aspect of the present invention, when it is determined that the vehicle has entered a low illumination section based on the image recognition result of the visible light image, the switching restriction condition may be satisfied. According to this image recording system, unnecessary image switching can be avoided by satisfying the switching restriction condition when it is determined that the vehicle has entered a low-illuminance section based on the image recognition results of the visible light image. This makes it possible to continue recording visible light images even in low-illuminance environments such as inside tunnels or indoor parking lots, and maintain the consistency of the recorded images.

[0011] In an image recording system according to one aspect of the present invention, the visible light camera includes a first front camera that captures an image in front of the vehicle and a first rear camera that captures an image in rear of the vehicle, and the switching restriction condition may be satisfied when the brightness of the front visible light image captured by the first front camera is smaller than the brightness of the rear visible light image captured by the first rear camera, and the difference in brightness between the front visible light image and the rear visible light image is equal to or greater than the low-illuminance zone entry threshold. According to this image recording system, by using visible light cameras including a first front camera that captures an image in front of the vehicle and a first rear camera that captures an image behind the vehicle, it is possible to determine that the switching restriction condition is satisfied when the luminance of the front visible light image is lower than the luminance of the rear visible light image and the difference in luminance between the front visible light image and the rear visible light image is equal to or greater than the low-illuminance section entry threshold. With this image recording system, when entering a low-illuminance section such as a tunnel or an indoor parking lot, it is possible to suppress unnecessary switching to an infrared image due to the difference in luminance between the front visible light image and the rear visible light image, and it is possible to continue recording the visible light image.

[0012] Another aspect of the present invention is a vehicle equipped with an infrared camera that captures images of the surrounding environment using infrared rays, a visible light camera that captures images of the surrounding environment using visible light, and an image recording unit that records either the infrared image captured by the infrared camera or the visible light image captured by the visible light camera, wherein when the image recording unit is recording a visible light image, if the illuminance around the vehicle detected by the vehicle's illuminance sensor or the luminance of the visible light image falls below an infrared switching threshold, the image recording unit switches the recording object from the visible light image to the infrared image, and even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, the image recording unit does not switch the recording object from the visible light image to the infrared image if a preset switching restriction condition is satisfied based on at least one of the current time, vehicle position information and map information, change in illuminance over time, change in luminance over time, and image recognition results of the visible light image.

[0013] According to a vehicle of another aspect of the present invention, even if the illuminance around the vehicle or the brightness of the visible light image falls below the infrared switching threshold, when a switching restriction condition is met based on at least one of the current time, vehicle position information and map information, changes in illuminance over time, changes in brightness over time, and image recognition results of the visible light image, unnecessary switching of the recording target of the image recording unit can be suppressed by not switching the recording target of the image recording unit from a visible light image to an infrared image.

[0014] Yet another aspect of the present invention is a program for operating a computer mounted on a vehicle having an infrared camera that captures images of the surrounding environment using infrared rays, a visible light camera that captures images of the surrounding environment using visible light, and an image recording unit that records either the infrared image captured by the infrared camera or the visible light image captured by the visible light camera, wherein when the image recording unit is recording a visible light image, if the illuminance around the vehicle detected by the vehicle's illuminance sensor or the luminance of the visible light image falls below an infrared switching threshold, the program operates the computer so that the image recording unit will not switch the image recording unit's recording target from a visible light image to an infrared image even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, if a preset switching restriction condition is satisfied based on at least one of the current time, vehicle position information and map information, change in illuminance over time, change in luminance over time, and image recognition results of the visible light image.

[0015] According to a program relating to yet another aspect of the present invention, even if the illuminance around the vehicle or the brightness of the visible light image falls below the infrared switching threshold, when a switching restriction condition is met based on at least one of the current time, vehicle position information and map information, changes in illuminance over time, changes in brightness over time, and image recognition results of the visible light image, unnecessary switching of the recording target of the image recording unit can be suppressed by not switching the recording target of the image recording unit from a visible light image to an infrared image.

[0016] Yet another aspect of the present invention is an image recording method for an image recording system having an infrared camera mounted on a vehicle that captures images of the vehicle's surrounding environment using infrared rays, a visible light camera mounted on the vehicle that captures images of the vehicle's surrounding environment using visible light, and an image recording unit that records either the infrared image captured by the infrared camera or the visible light image captured by the visible light camera, wherein when the image recording unit's recording target is a visible light image, the image recording system switches the image recording unit's recording target from the visible light image to the infrared image when the illuminance around the vehicle detected by the vehicle's illuminance sensor or the luminance of the visible light image falls below an infrared switching threshold, and the image recording system does not switch the image recording unit's recording target from the visible light image to the infrared image even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold when a preset switching restriction condition is satisfied based on at least one of the current time, vehicle position information and map information, change in illuminance over time, change in luminance over time, and image recognition results of the visible light image.

[0017] According to an image recording method of an image recording system relating to yet another aspect of the present invention, even if the illuminance around the vehicle or the brightness of the visible light image falls below the infrared switching threshold, when a switching restriction condition is met based on at least one of the current time, vehicle position information and map information, changes in illuminance over time, changes in brightness over time, and image recognition results of the visible light image, unnecessary switching of the recording target of the image recording unit can be suppressed by not switching the recording target of the image recording unit from a visible light image to an infrared image. [Effects of the Invention]

[0018] According to each aspect of the present invention, unnecessary switching of the recording target of the image recording unit can be suppressed. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram illustrating an image recording system according to an embodiment. [Figure 2] 10 is a flowchart illustrating an example of an image switching process. [Figure 3] 10A is a flowchart illustrating a first example of a switching restriction condition determination process, and FIG. 10B is a flowchart illustrating a second example of a switching restriction condition determination process. [Figure 4] 10A and 10B are flowcharts illustrating a third and fourth example of the switching restriction condition determination process; [Figure 5] 10 is a flowchart illustrating a fifth example of a switching restriction condition determination process. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0021] [Image recording system configuration] Fig. 1 is a block diagram showing an image recording system 100 according to one embodiment. The image recording system 100 shown in Fig. 1 is mounted on a vehicle, and captures images of the environment surrounding the vehicle using a visible light camera 1 and an infrared camera 2, and records the images in an image recording unit 6. The image recording system 100 records only either the visible light image captured by the visible light camera 1 or the infrared image captured by the infrared camera 2.

[0022] The image recording system 100 is basically intended to record visible light images during the day and infrared images at night. The image recording unit 6 records, for example, images as video, which is time-series data. The image recording unit 6 may also record visible light images or infrared images as a series of still images. The image recording system 100 may also be a so-called drive recorder system.

[0023] As shown in FIG. 1, the image recording system 100 includes an ECU (Electronic Control Unit) 10 that performs overall system management. The ECU 10 is an electronic control unit (computer) having a CPU (Central Processing Unit) and an internal memory 14. The internal memory 14 is composed of, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory). The ECU 10 realizes various functions by, for example, executing programs stored in a storage unit in the CPU. The ECU 10 may be composed of multiple electronic units.

[0024] The ECU 10 is connected to a visible light camera 1 , an infrared camera 2 , an illuminance sensor 3 , a GNSS (Global Navigation Satellite System) receiver 4 , a map database 5 , and an image recorder 6 .

[0025] The visible light camera 1 is a camera that captures images of the surrounding environment of a vehicle using visible light. As the visible light camera 1, an image sensor such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor) can be used. A visible light image is an image captured using visible light. The visible light image contains color information (such as RGB information). The wavelength range of the visible light image is not particularly limited, but it may be an embodiment that uses only a portion of the wavelength range of visible light.

[0026] The visible light camera 1 has a first front camera 1a and a first rear camera 1b. The first front camera 1a is a camera provided at the front of the vehicle and captures images of the area in front of the vehicle using visible light. The visible light image captured by the first front camera 1a is defined as a front visible light image. The first rear camera 1b is a camera provided at the rear of the vehicle and captures images of the area behind the vehicle using visible light. The visible light image captured by the first rear camera 1b is defined as a rear visible light image. Note that the visible light camera 1 may have only the first front camera 1a.

[0027] The infrared camera 2 is a camera that uses infrared rays to capture images of the vehicle's surrounding environment. The infrared camera 2 is configured with an infrared light emitter and an infrared light receiver, and captures infrared images by receiving the infrared light emitted from the infrared light emitter and reflected off an object. Infrared images are not affected by lighting conditions (brightness or darkness), and can capture clear images even at night or in dark places.

[0028] The infrared camera 2 has a second front camera 2a and a second rear camera 2b. The second front camera 2a is a camera provided at the front of the vehicle and captures images of the area in front of the vehicle using infrared rays. The infrared image captured by the second front camera 2a is a front infrared image. The second rear camera 2b is a camera provided at the rear of the vehicle and captures images of the area behind the vehicle using infrared rays. The infrared image captured by the second rear camera 2b is a rear infrared image. Note that if the visible light camera 1 is composed of only the first front camera 1a, the infrared camera 2 may also be composed of only the second front camera 2a.

[0029] The infrared camera 2 may be an RGB-IR camera integrated with the visible light camera 1. The RGB-IR camera has an RGB sensor and an IR sensor in a single camera unit, and can capture visible light images and infrared images simultaneously.

[0030] The illuminance sensor 3 is a sensor for detecting the illuminance around the vehicle. The illuminance sensor 3 is disposed outside the vehicle and detects the intensity of the surrounding light. The illuminance sensor 3 is configured using a light detection element such as a photodiode or a phototransistor, and outputs an electrical signal to the ECU 10 according to the amount of surrounding light.

[0031] The GNSS receiver 4 is a device for acquiring vehicle position information. The GNSS receiver 4 measures the vehicle position (e.g., the latitude and longitude of the vehicle) by receiving signals from positioning satellites. The GNSS receiver 4 may be part of a GPS (Global Positioning System) system. The GNSS receiver 4 transmits the measured vehicle position information to the ECU 10.

[0032] The map database 5 is a database that stores map information. The map database 5 is formed in a storage device such as a hard disk drive (HDD) mounted on the vehicle. The map database 5 may also be formed in a server that can communicate with the vehicle.

[0033] The map information includes information on low-illuminance sections such as tunnels. A low-illuminance section is a section with low illumination where the illumination around the vehicle may be below the infrared switching threshold. A low-illuminance section may include an indoor parking lot or other indoor section. A low-illuminance section may also include a mountain road with many shaded areas due to trees. The map information may include road position information, road shape information (e.g., curves, type of straight section, curvature of curves, etc.), intersection and branch point position information, and structure position information.

[0034] The image recording unit 6 is a database for recording images of the vehicle's surrounding environment. The image recording unit 6 has a function of saving visible light images acquired from the visible light camera 1 or infrared images acquired from the infrared camera 2 as recording targets. The image recording unit 6 is configured using a recording medium such as a hard disk drive (HDD) or a solid state drive (SSD), and is capable of saving large amounts of data for long periods of time. The image recording unit 6 may be an internal memory of the ECU 10. An eMMC (Embedded Multi Media Card) or the like can be used as the internal memory.

[0035] Next, a description will be given of the functional configuration of the ECU 10. As shown in Fig. 1, the ECU 10 includes a switching determination unit 11, a switching restriction condition determination unit 12, an image processing unit 13, and an internal memory .

[0036] The switching determination unit 11 determines whether the illuminance around the vehicle detected by the vehicle's illuminance sensor 3 is less than an infrared switching threshold. The infrared switching threshold is a threshold used to determine whether to switch the recording target of the image recording unit 6 from visible light images to infrared images. The infrared switching threshold is set so that the recording target is switched from visible light images to infrared images when it becomes night, for example. The value of the infrared switching threshold is not particularly limited.

[0037] The switching determination unit 11 may make the determination using the luminance of the visible light image captured by the visible light camera 1, instead of the illuminance around the vehicle. The luminance of the visible light image may be the luminance of the front visible light image captured by the first front camera 1a, the luminance of the rear visible light image captured by the first rear camera 1b, or the average value of the luminance of the front visible light image and the luminance of the rear visible light image. The luminance of the visible light image may be the average value or median value of the luminance of the entire image, or the average value or median value of the luminance of the central part of the visible light image. Alternatively, a representative luminance value may be obtained from the visible light image by a well-known method.

[0038] The switching determination unit 11 determines whether the luminance of the visible light image is less than the infrared switching threshold. Note that the infrared switching threshold for determining the illuminance and the infrared switching threshold for determining the luminance may be the same value or different values.

[0039] The switching restriction condition determination unit 12 determines whether a preset switching restriction condition is satisfied. The switching restriction condition is a condition for restricting switching from a visible light image to an infrared image when the image recording unit 6 is recording a visible light image. The switching restriction condition determination unit 12 determines whether the switching restriction condition is satisfied based on at least one of the current time, vehicle position information, map information, time change in illuminance, time change in luminance, and image recognition results of the visible light image.

[0040] The switching restriction condition determination unit 12 determines whether the switching restriction condition is satisfied based on, for example, the current time. Specifically, the switching restriction condition determination unit 12 determines whether the current time is before a preset sunset time or whether the current time is within a preset daytime period. The current time may be acquired from the GNSS receiver 4, from a timer in the ECU 10, or from an external server via wireless communication. The sunset time is the predicted time when the sun sets (for example, 6 p.m.). The daytime period is the time when the sun is up (for example, from 5 a.m. to 6 p.m.). The sunset time and daytime period change depending on the day or month.

[0041] The sunset time may be stored in the internal memory 14 or may be obtained from an external server via wireless communication. The sunset time may be obtained from the internal memory 14 or an external server based on the vehicle's location information and is the time corresponding to sunset in the area where the vehicle is located. The same applies to the daytime period. The switching restriction condition determination unit 12 determines that the switching restriction condition is satisfied if the current time is before sunset or if the current time is included in the daytime period. Instead of determining that the current time is included in the daytime period, it may also determine that the current time is not included in the nighttime period.

[0042] The switching restriction condition determination unit 12 may determine whether or not the switching restriction condition is satisfied based on the vehicle position information acquired from the GNSS receiver 4 and the map information stored in the map database 5. The switching restriction condition determination unit 12 determines whether or not the vehicle has entered a low-illuminance area such as a tunnel or indoors, based on the vehicle position information and the map information. Accurate position determination is possible by comparing the information with the map information. If the switching restriction condition determination unit 12 determines that the vehicle has entered a low-illuminance area, it determines that the switching restriction condition is satisfied.

[0043] The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a certain period of time after determining that the vehicle has entered a low-illuminance section.The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a period of time after determining that the vehicle has entered a low-illuminance section, until determining that the vehicle has left the low-illuminance section based on the vehicle position information and map information.

[0044] The switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied based on a change in illuminance over time. For example, when the amount of decrease in illuminance over time, as the change in illuminance over time, is equal to or greater than a section entry threshold, the switching restriction condition determination unit 12 determines that the vehicle has entered a low-illuminance section. The section entry threshold is a threshold that corresponds to the change in illuminance over time when the vehicle enters a low-illuminance section. The section entry threshold may be a threshold whose value changes depending on the vehicle speed. The section entry threshold may be a larger value as the vehicle speed increases. When the switching restriction condition determination unit 12 determines that the vehicle has entered a low-illuminance section based on the change in illuminance over time, the switching restriction condition determination unit 12 determines that the switching restriction condition is satisfied.

[0045] The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a certain period of time after determining that the vehicle has entered a low-illuminance section. The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a period of time after determining that the vehicle has entered a low-illuminance section, until determining that the vehicle has left the low-illuminance section based on a change in illuminance over time. The switching restriction condition determination unit 12 determines that the vehicle has left the low-illuminance section, for example, when the increase in illuminance over time, as a change in illuminance over time, is equal to or greater than a section departure threshold. The section departure threshold is a threshold corresponding to a change in illuminance over time when the vehicle leaves the low-illuminance section. The section departure threshold may be a threshold whose value changes depending on the vehicle speed.

[0046] The switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied using a time change in the luminance of the visible light image instead of a time change in the illuminance. The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied when the amount of decrease in luminance over time, as the time change in the luminance of the visible light image, is equal to or greater than the section entry threshold. When the switching determination unit 11 determines whether to switch to infrared light using the illuminance, the switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied using a time change in the luminance of the visible light image. When the switching determination unit 11 determines whether to switch to infrared light using the luminance, the switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied using a time change in illuminance.

[0047] The switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied based on the image recognition results of the visible light image. The switching restriction condition determination unit 12 determines that the vehicle has entered a low-illuminance section from the visible light image using well-known image recognition methods such as pattern matching, silhouette analysis, optical flow analysis, HOG (Histogram of Oriented Gradients) analysis, and a convolutional neural network (CNN) using deep learning. If the switching restriction condition determination unit 12 determines that the vehicle has entered a low-illuminance section based on the image recognition results of the visible light image, it determines that the switching restriction condition is satisfied.

[0048] The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a certain period of time after determining that the vehicle has entered a low-illuminance section. The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a period of time after determining that the vehicle has entered a low-illuminance section until determining that the vehicle has left the low-illuminance section based on the image recognition results of the visible light image. Alternatively, the image recognition process may be performed by the image processing unit 13, which will be described later. Note that the switching restriction condition determination unit 12 may start determining whether the switching restriction condition is satisfied only when determining that it is daytime based on the image recognition results of the visible light image.

[0049] The switching restriction condition determination unit 12 may determine whether the switching restriction condition is satisfied based on the luminance of the front visible light image captured by the first front camera 1a and the luminance of the rear visible light image captured by the first rear camera 1b. Specifically, the switching restriction condition determination unit 12 determines that the switching restriction condition is satisfied when the luminance of the front visible light image is lower than the luminance of the rear visible light image and the difference in luminance between the front visible light image and the rear visible light image is equal to or greater than a low-illuminance section entry threshold. The low-illuminance section entry threshold is a threshold that is set in response to the vehicle entering a low-illuminance section. The value of the low-illuminance section entry threshold is not particularly limited.

[0050] The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a certain period of time after the luminance of the front visible light image is lower than the luminance of the rear visible light image and the difference in luminance between the front visible light image and the rear visible light image becomes equal to or greater than a low-illuminance section entry threshold. The switching restriction condition determination unit 12 may determine that the switching restriction condition is satisfied for a period of time after determining that the vehicle has entered a low-illuminance section and until determining that the vehicle has left the low-illuminance section based on the luminance of the front visible light image and the luminance of the rear visible light image. The switching restriction condition determination unit 12 determines that the vehicle has left the low-illuminance section, for example, if the luminance of the front visible light image is higher than the luminance of the rear visible light image and the difference in luminance between the front visible light image and the rear visible light image is equal to or greater than a low-illuminance section departure threshold. The low-illuminance section departure threshold is a threshold set in response to the vehicle leaving a low-illuminance section.

[0051] Image processing unit 13 switches the recording target of image recording unit 6 based on the determination results of switching determination unit 11 and switching restriction condition determination unit 12. The initial setting recording target is a visible light image. When switching determination unit 11 determines that the illuminance around the vehicle or the luminance of the visible light image is less than the infrared switching threshold, and switching restriction condition determination unit 12 does not determine that the switching restriction condition is satisfied, image processing unit 13 switches the recording target of image recording unit 6 from a visible light image to an infrared image.

[0052] Alternatively, the image processing unit 13 may select images to be recorded using a known method. For example, if an image does not include a moving object, the image processing unit 13 may exclude the image from being recorded.

[0053] The internal memory 14 is a recording unit within the ECU 10. The internal memory 14 stores, for example, the sunset time and the daytime hours. The internal memory 14 may also temporarily store information on whether or not a switching restriction condition is satisfied.

[0054] [program] The program causes the ECU 10 (computer) to function (operate) as the above-mentioned switching determination unit 11, switching restriction condition determination unit 12, and image processing unit 13. The program is provided by a non-transitory recording medium such as a ROM or a semiconductor memory. Alternatively, the program may be provided from a network via wireless communication.

[0055] [Image recording method for image recording system] Next, an image recording method of the image recording system 100 according to this embodiment will be described with reference to the drawings. The image recording method includes, for example, the image switching process shown in Fig. 2 and the switching restriction condition determination process shown in Figs.

[0056] Fig. 2 is a flowchart showing an example of the image switching process. The image switching process shown in Fig. 2 is executed when the image to be recorded by the image recording unit 6 is a visible light image. As shown in Fig. 2, first, in step S10, the ECU 10 causes the switching determination unit 11 to determine whether the illuminance around the vehicle or the luminance of the visible light image is less than the infrared switching threshold.

[0057] Specifically, the ECU 10 acquires the illuminance from the illuminance sensor 3 or the luminance of the visible light image from the visible light camera 1, and determines whether the illuminance or luminance is less than the infrared switching threshold. This determination enables the determination of whether the surrounding environment is low in illuminance. If the ECU 10 determines that the illuminance or luminance is less than the infrared switching threshold, the ECU 10 proceeds to step S11. If the ECU 10 does not determine that the illuminance or luminance is less than the infrared switching threshold, the ECU 10 ends the image switching process.

[0058] Next, in step S11, the ECU 10 determines whether or not the switching restriction condition is satisfied by the switching restriction condition determination unit 12. The switching restriction condition determination unit 12 determines whether or not the switching restriction condition is satisfied based on the current time, vehicle position information, map information, time-varying illuminance, time-varying luminance, image recognition results of the visible light image, etc. If the switching restriction condition is satisfied, the ECU 10 ends the image switching process. If the switching restriction condition is not satisfied, the ECU 10 proceeds to step S12.

[0059] In step S12, the ECU 10 switches the recording target of the image recording unit 6 from visible light images to infrared images by the image processing unit 13. The image recording unit 6 records the infrared images of the infrared camera 2.

[0060] 3A is a flowchart showing a first example of the switching restriction condition determination process. As shown in FIG. 3A, in step S20, the ECU 10 first determines whether the current time is before sunset using the switching restriction condition determination unit 12. Specifically, the switching restriction condition determination unit 12 compares the current time acquired from an internal timer or the GNSS receiver 4 with a preset sunset time to determine whether the current time is after sunset. Note that daytime hours may be used instead of sunset time.

[0061] If the current time is not before sunset, the ECU 10 ends this switching restriction condition determination process. If the current time is before sunset, the ECU 10 proceeds to step S21. In step S21, the ECU 10 determines that the switching restriction condition is satisfied. Thereafter, the ECU 10 ends this switching restriction condition determination process.

[0062] 3(b) is a flowchart showing a second example of the switching restriction condition determination process. As shown in FIG. 3(b), in step S30, the ECU 10 first determines whether the vehicle has entered a low-illuminance area based on the vehicle's position information and map information using the switching restriction condition determination unit 12. Specifically, the switching restriction condition determination unit 12 compares the vehicle's position information acquired from the GNSS receiver 4 with the map information stored in the map database 5, and determines whether the vehicle has entered a low-illuminance area such as a tunnel or an indoor parking lot.

[0063] If the vehicle has not entered a low-illuminance section, the ECU 10 ends this switching restriction condition determination process. If the vehicle has entered a low-illuminance section, the ECU 10 proceeds to step S31. In step S31, the ECU 10 determines that the switching restriction condition is satisfied. Thereafter, the ECU 10 ends this switching restriction condition determination process. The ECU 10 may determine that the switching restriction condition is satisfied for a certain period of time after the vehicle has entered the low-illuminance section.

[0064] 4(a) is a flowchart showing a third example of the switching restriction condition determination process. First, in step S40, the ECU 10 determines whether the vehicle has entered a low-illuminance area during the daytime based on the change in illuminance or the change in luminance over time using the switching restriction condition determination unit 12. Specifically, the switching restriction condition determination unit 12 analyzes the illuminance detected by the illuminance sensor 3 or the luminance of the visible light image from the visible light camera 1 in time series, and determines that the vehicle has entered a low-illuminance area such as a tunnel or an indoor parking lot if the illuminance or luminance suddenly drops.

[0065] If the change in illuminance or brightness over time does not indicate that the vehicle is entering a low-illuminance section, the ECU 10 ends this switching restriction condition determination process. If the change in illuminance or brightness over time indicates that the vehicle is entering a low-illuminance section, the ECU 10 proceeds to step S41.

[0066] In step S41, the ECU 10 determines that the switching restriction condition is satisfied. Then, the ECU 10 ends the switching restriction condition determination process. The ECU 10 may determine that the switching restriction condition is satisfied for a certain period of time after the vehicle enters the low illumination section.

[0067] 4(b) is a flowchart showing a fourth example of the switching restriction condition determination process. As shown in FIG. 4(b), first, in step S50, the switching restriction condition determination unit 12 determines whether the vehicle has entered a low-illuminance section based on the image recognition results of the visible light image. Specifically, the switching restriction condition determination unit 12 detects features of a tunnel entrance or an indoor parking lot based on the recognition results of the visible light image analyzed by the image processing unit 13, and determines that the vehicle has entered a low-illuminance section.

[0068] If the image recognition result of the visible light image does not indicate that the vehicle has entered a low illuminance section, the ECU 10 ends this process for determining the switching restriction condition. If the image recognition result of the visible light image indicates that the vehicle has entered a low illuminance section, the ECU 10 proceeds to step S51.

[0069] In step S51, the ECU 10 determines that the switching restriction condition is satisfied. Then, the ECU 10 ends the switching restriction condition determination process. The ECU 10 may determine that the switching restriction condition is satisfied for a certain period of time after the vehicle enters the low illumination section.

[0070] Fig. 5 is a flowchart showing a fifth example of the switching restriction condition determination process. As shown in Fig. 5, in step S60, the ECU 10 first determines whether the luminance of the front visible light image is lower than the luminance of the rear visible light image using the switching restriction condition determination unit 12. Specifically, the switching restriction condition determination unit 12 compares the luminance of the front visible light image acquired from the first front camera 1a with the luminance of the rear visible light image acquired from the first rear camera 1b, and determines whether the luminance of the front visible light image is lower than the luminance of the rear visible light image.

[0071] If the luminance of the front visible light image is not lower than the luminance of the rear visible light image, the ECU 10 ends this switching restriction condition determination process. If the luminance of the front visible light image is lower than the luminance of the rear visible light image, the ECU 10 proceeds to step S61.

[0072] In step S61, the ECU 10 determines whether the difference in luminance between the rear visible light image and the front visible light image is equal to or greater than the low illuminance section entry threshold using the switching restriction condition determination unit 12. Specifically, the switching restriction condition determination unit 12 calculates the difference in luminance between the front visible light image and the rear visible light image, and determines whether the difference is equal to or greater than the low illuminance section entry threshold.

[0073] If the difference in luminance between the rear visible light image and the front visible light image is not equal to or greater than the low illuminance section entry threshold, the ECU 10 ends this process for determining the switching restriction condition. If the difference in luminance between the rear visible light image and the front visible light image is equal to or greater than the low illuminance section entry threshold, the ECU 10 proceeds to step S62.

[0074] In step S62, the ECU 10 determines that the switching restriction condition is satisfied. Then, the ECU 10 ends the switching restriction condition determination process. The ECU 10 may determine that the switching restriction condition is satisfied for a certain period of time after determining that the difference in luminance between the rear visible light image and the front visible light image is equal to or greater than the low illuminance section entry threshold.

[0075] According to the image recording system 100 of the present embodiment, even if the illuminance around the vehicle or the brightness of the visible light image falls below the infrared switching threshold, if a switching restriction condition is met based on at least one of the current time, vehicle position information and map information, time-varying illuminance, time-varying luminance, and image recognition results of the visible light image, the image recording unit does not switch the recording target from visible light images to infrared images. This prevents unnecessary switching of the recording target image in low-illuminance areas, which occurs when determining using only an illuminance sensor in the past. This maintains the consistency of recorded images and improves the accuracy of analysis and confirmation, even when the vehicle passes through low-illuminance areas such as tunnels or indoor parking lots. Furthermore, delays and frame drops caused by switching to infrared images can be prevented, reducing the occurrence of problems that make it difficult to identify objects or people.

[0076] Furthermore, according to the image recording system 100, if the current time is before sunset or if the current time is within the daytime hours, the switching restriction condition is met, thereby suppressing unnecessary switching to infrared images during bright daylight hours. This prioritizes recording of visible light images and maintains image consistency, thereby improving the accuracy of analysis and confirmation. Furthermore, by preventing switching to infrared images under certain conditions, even during bright daylight hours, the quality of recorded images can be maintained and useful information can be provided to the user.

[0077] Furthermore, with the image recording system 100, unnecessary image switching can be prevented by satisfying the switching restriction condition when it is determined from the vehicle's position information and map information that the vehicle has entered a low-illuminance section. This makes it possible to continue recording visible light images even in low-illuminance environments such as inside tunnels or indoor parking lots, and maintain the consistency of the recorded images.

[0078] Furthermore, according to the image recording system 100, when it is determined that the vehicle has entered a low-illuminance section based on the change in illuminance or brightness over time, the switching restriction condition is satisfied, thereby avoiding unnecessary image switching. This allows visible light images to continue being recorded even in low-illuminance sections such as tunnels and indoor parking lots, and maintains the consistency of the recorded images.

[0079] Furthermore, with the image recording system 100, unnecessary image switching can be avoided by satisfying the switching restriction condition when it is determined that the vehicle has entered a low-illuminance section based on the image recognition results of the visible light image. This makes it possible to continue recording visible light images even in low-illuminance environments such as tunnels and indoor parking lots, thereby maintaining the consistency of the recorded images. Furthermore, the use of image recognition technology makes it possible to more accurately determine low-illuminance sections even in situations where it is difficult to determine the low-illuminance section using only illuminance sensors and location information, thereby achieving more reliable image recording.

[0080] Furthermore, the image recording system 100 uses a visible light camera 1 including a first front camera 1a that captures an image in front of the vehicle and a first rear camera 1b that captures an image behind the vehicle, and can determine that the switching restriction condition is satisfied when the luminance of the front visible light image is lower than the luminance of the rear visible light image and the difference in luminance between the front visible light image and the rear visible light image is equal to or greater than the low-illuminance section entry threshold. The image recording system 100 can suppress unnecessary switching to an infrared image due to the difference in luminance between the front visible light image and the rear visible light image when entering a low-illuminance section such as a tunnel or an indoor parking lot, thereby making it possible to continue recording the visible light image. Similar effects can be obtained in a vehicle having the image recording system 100 according to this embodiment, a program that causes the vehicle's ECU 10 to operate as the image recording system 100, and an image recording method for the image recording system 100.

[0081] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. The present invention can be embodied in various forms, including the above-described embodiments, with various modifications and improvements made based on the knowledge of those skilled in the art.

[0082] The image recording system 100 may determine whether the switching restriction condition is satisfied based on at least one of the current time, vehicle position information, map information, time variation in illuminance, time variation in luminance, and the image recognition result of the visible light image. The image recording system 100 does not necessarily need to use the current time for the determination. In this case, it is not necessary to acquire the current time, sunset time, daytime period, etc.

[0083] The image recording system 100 does not necessarily need to make a determination using vehicle position information. If the image recording system 100 does not make a determination using vehicle position information, it does not need to have the GNSS receiver 4 and the map database 5. The same applies to the time change in illuminance, the time change in luminance, and the image recognition results of visible light images.

[0084] Note that the image recording system 100 may be configured to recognize that it is daytime using the current time before determining the switching restriction condition based on the time-varying change in illuminance or luminance, the image recognition results of the visible light image, the difference in luminance between the front visible light image and the rear visible light image, etc. Whether it is daytime can be recognized from the current time, the sunset time, and information about the daytime hours. This makes it possible to avoid erroneously determining that the vehicle has entered a low-illuminance area when leaving a brightly lit parking lot or the like at night.

[0085] The image recording system 100 does not necessarily need to perform a process for determining whether the switching restriction condition is satisfied. For example, if the current time is before sunset, the image recording system 100 may consider the switching restriction condition to be satisfied and not switch the recording target of the image recording unit 6 from visible light images to infrared images. The same applies when using vehicle position information and map information, time-dependent changes in illuminance, time-dependent changes in brightness, image recognition results of visible light images, etc. [Explanation of symbols]

[0086] 1...visible light camera, 1a...first front camera, 1b...first rear camera, 2...infrared camera, 2a...second front camera, 2b...second rear camera, 3...illuminance sensor, 4...GNSS receiver, 5...map database, 6...image recording unit, 10...ECU (computer), 11...switching determination unit, 12...switching restriction condition determination unit, 13...image processing unit, 14...internal memory, 100...image recording system.

Claims

1. An image recording system having an infrared camera mounted on a vehicle and capturing an image of the vehicle's surrounding environment using infrared rays, a visible light camera mounted on the vehicle and capturing an image of the vehicle's surrounding environment using visible light, and an image recording unit that records either an infrared image captured by the infrared camera or a visible light image captured by the visible light camera, when the illuminance around the vehicle detected by an illuminance sensor of the vehicle or the luminance of the visible light image becomes less than an infrared switching threshold value, the recording target of the image recording unit is switched from the visible light image to the infrared image; an image recording system in which, even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, the image recording unit does not switch the recording target from the visible light image to the infrared image when a preset switching restriction condition is satisfied based on at least one of the current time, position information and map information of the vehicle, changes in the illuminance over time, changes in the luminance over time, and image recognition results of the visible light image.

2. The image recording system according to claim 1 , wherein the switching restriction condition is satisfied when the current time is before a preset sunset time or when the current time is within a preset daytime period.

3. The map information includes a low-light section including a tunnel, The image recording system according to claim 1 , wherein the switching restriction condition is satisfied when it is determined that the vehicle has entered the low-illuminance section based on the vehicle position information and the map information.

4. The image recording system according to claim 1 , wherein the switching restriction condition is satisfied when it is determined that the vehicle has entered a low illumination section based on the time variation of the illuminance or the time variation of the luminance.

5. The image recording system according to claim 1 , wherein the switching restriction condition is satisfied when it is determined that the vehicle has entered a low-illuminance section based on a result of image recognition of the visible light image.

6. The visible light camera includes a first front camera that captures an image in front of the vehicle and a first rear camera that captures an image in rear of the vehicle, 2. The image recording system according to claim 1, wherein the switching restriction condition is satisfied when a luminance of the forward visible light image captured by the first front camera is lower than a luminance of the rear visible light image captured by the first rear camera, and a difference in luminance between the rear visible light image and the forward visible light image is equal to or greater than a low-illuminance section entry threshold.

7. A vehicle comprising: an infrared camera that captures an image of a surrounding environment using infrared rays; a visible light camera that captures an image of the surrounding environment using visible light; and an image recording unit that records either an infrared image captured by the infrared camera or a visible light image captured by the visible light camera, when the illuminance around the vehicle detected by an illuminance sensor of the vehicle or the luminance of the visible light image becomes less than an infrared switching threshold value, the recording target of the image recording unit is switched from the visible light image to the infrared image; a vehicle in which, even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, the image recording unit does not switch the recording target from the visible light image to the infrared image when a preset switching restriction condition is satisfied based on at least one of the current time, position information and map information of the vehicle, changes in the illuminance over time, changes in the luminance over time, and image recognition results of the visible light image.

8. A program for operating a computer mounted on a vehicle, the computer having an infrared camera that captures an image of the vehicle's surrounding environment using infrared rays, a visible light camera that captures an image of the vehicle's surrounding environment using visible light, and an image recording unit that records either an infrared image captured by the infrared camera or a visible light image captured by the visible light camera, when the illuminance around the vehicle detected by an illuminance sensor of the vehicle or the luminance of the visible light image becomes less than an infrared switching threshold value, the recording target of the image recording unit is switched from the visible light image to the infrared image; a program that causes the computer to operate so as not to switch the recording target of the image recording unit from the visible light image to the infrared image, even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, when a preset switching restriction condition is satisfied based on at least one of the current time, position information and map information of the vehicle, changes in the illuminance over time, changes in the luminance over time, and image recognition results of the visible light image.

9. An image recording method for an image recording system having an infrared camera mounted on a vehicle and capturing an image of the vehicle's surrounding environment using infrared rays, a visible light camera mounted on the vehicle and capturing an image of the vehicle's surrounding environment using visible light, and an image recording unit that records either an infrared image captured by the infrared camera or a visible light image captured by the visible light camera, the image recording system switches the recording target of the image recording unit from the visible light image to the infrared image when the illuminance around the vehicle detected by an illuminance sensor of the vehicle or the luminance of the visible light image becomes less than an infrared switching threshold in a case where the recording target of the image recording unit is the visible light image, an image recording method for an image recording system, wherein even if the illuminance around the vehicle or the luminance of the visible light image falls below the infrared switching threshold, the image recording system does not switch the recording target of the image recording unit from the visible light image to the infrared image when a preset switching restriction condition is satisfied based on at least one of the current time, position information and map information of the vehicle, changes in the illuminance over time, changes in the luminance over time, and image recognition results of the visible light image.

Images