Image generation device, image generation method, and vehicle control system

By flexibly controlling the demosaicing process according to the ambient illuminance, the image generation device avoids resource waste and time increase when generating visible and invisible light images, achieving efficient image generation, which is suitable for applications with high real-time requirements such as vehicle control.

CN115695688BActive Publication Date: 2026-02-13TOYOTA JIDOSHA KK
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210828132.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-07-29
Filing Date
2022-07-13
Publication Date
2026-02-13
Estimated Expiration
2042-07-13

AI Technical Summary

Technical Problem

Existing technologies tend to waste computing and storage resources and increase processing time when generating visible and invisible light images, making them undesirable, especially in applications with high real-time requirements.

Method used

The image generation device flexibly performs de-mosaic processing based on the ambient light around the imaging element, generating visible light or invisible light images only when necessary, avoiding over-processing.

Benefits of technology

By effectively utilizing computing and storage resources, processing time can be reduced, ensuring the generation of suitable images in different environments and meeting real-time requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115695688B_ABST
    Figure CN115695688B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an image generation apparatus, an image generation method, and a vehicle control system. The image generation apparatus acquires first light reception data indicating a light reception result of invisible light and second light reception data indicating a light reception result of visible light from a shooting element that receives the invisible light and the visible light through a filter, respectively. Further, the image generation apparatus performs a demosaicing process on at least one of the first light reception data and the second light reception data based on illumination information indicating an illumination of an environment around the shooting element, thereby generating an image. In a case where the illumination is less than a first threshold value, the image generation apparatus does not perform a second demosaicing process on the second light reception data, but performs a first demosaicing process on the first light reception data, thereby generating an invisible light image.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image generation technique capable of flexibly generating a visible light image and an invisible light image. BACKGROUND

[0002] Japanese Patent Application Publication No. 2007-174277 discloses an image signal processing device provided with a single-plate type imaging element. The single-plate type imaging element is composed of a specific wavelength region signal acquisition element that acquires a visible light region signal such as an RGB signal, and a wide wavelength region signal acquisition element that acquires a light signal including a visible light and an infrared light component. The image signal processing device generates a demosaiced image corresponding to the wide wavelength region signal as a luminance signal. Further, the image signal processing device generates a color difference signal based on a demosaiced image corresponding to the visible light region signal.

[0003] In vehicle control, monitoring of a street, and the like, an image captured by a camera is sometimes used. In a dark environment in which the amount of light of visible light is small, a near-infrared image and the like, which is an invisible light image, is also useful. Thus, it is considered that a camera capable of acquiring both a normal visible light image and a near-infrared image and the like, which is an invisible light image, is useful. However, generating both a visible light image and an invisible light image means that a demosaicing process will be increased accordingly. Excessive demosaicing processing and excessive generation of images can lead to waste of computing resources and storage resources, increase in processing time, and the like. SUMMARY

[0004] An object of the present disclosure is to provide a technique capable of flexibly generating a visible light image and an invisible light image according to a situation without excessive demosaicing processing.

[0005] The first viewpoint is associated with an image generation device that generates an image. The image generation device is provided with one or more processors. The one or more processors receive first light reception data indicating a light reception result of invisible light and second light reception data indicating a light reception result of visible light from an imaging element that respectively transmits a filter for invisible light and a filter for visible light. Further, the one or more processors perform a demosaicing process on at least one of the first light reception data and the second light reception data based on illumination information indicating an illumination of an environment around the imaging element, thereby generating the image. In a case where the illumination is less than a first threshold value, the one or more processors do not perform a second demosaicing process on the second light reception data, but perform a first demosaicing process on the first light reception data, thereby generating an invisible light image as the image.

[0006] A second aspect is directed to an image generation device that generates an image. The image generation device includes one or more processors. The one or more processors acquire first light reception data representing a light reception result of invisible light and second light reception data representing a light reception result of visible light from a photographing element that receives the invisible light and the visible light through a filter, respectively. The one or more processors generate an image by performing demosaicing processing on at least one of the first light reception data and the second light reception data based on illumination information representing an illumination of an environment around the photographing element.

[0007] A third aspect is directed to a vehicle control system. The vehicle control system includes the image generation device of the first aspect or the second aspect, and a control device that controls a vehicle based on the generated image.

[0008] A fourth aspect is directed to an image generation method. The image generation method includes an acquisition process of acquiring first light reception data representing a light reception result of invisible light and second light reception data representing a light reception result of visible light using a photographing element that receives the invisible light and the visible light through a filter, respectively, and a development process of generating an image by performing demosaicing processing on at least one of the first light reception data and the second light reception data based on illumination information representing an illumination of an environment around the photographing element. The development process includes a process of generating an invisible light image as the image by performing first demosaicing processing on the first light reception data and not performing second demosaicing processing on the second light reception data when the illumination is less than a first threshold value.

[0009] A fifth aspect is directed to an image generation method. The image generation method includes an acquisition process of acquiring first light reception data representing a light reception result of invisible light and second light reception data representing a light reception result of visible light using a photographing element that receives the invisible light and the visible light through a filter, respectively, and a development process of generating an image by performing demosaicing processing on at least one of the first light reception data and the second light reception data based on illumination information representing an illumination of an environment around the photographing element. The development process can include a process of generating a visible light image as the image by performing second demosaicing processing on the second light reception data and not performing first demosaicing processing on the first light reception data when the illumination is greater than or equal to a threshold value.

[0010] According to the present disclosure, the necessary processing in the first demosaicing processing and the second demosaicing processing is performed taking into account the illuminance of the environment around the imaging element. That is, the visible light image and the invisible light image can be generated flexibly according to the situation without excessively performing the demosaicing processing. As a result, the waste of the computing resources and the storage resources is prevented, and thus the computing resources and the storage resources are used effectively. Furthermore, the processing time as a whole is also reduced. BRIEF DESCRIPTION OF DRAWINGS

[0011] Hereinafter, features, advantages, and technical and industrial significance of exemplary embodiments of the present application will be described with reference to the accompanying drawings, in which like numerals denote like elements, and wherein:

[0012] Figure 1 is a block diagram showing a configuration example of an image generation apparatus of an embodiment of the present disclosure.

[0013] Figure 2 is a conceptual diagram showing a configuration example of an imaging element of an embodiment of the present disclosure.

[0014] Figure 3 is a conceptual diagram for explaining a first example of demosaicing processing of an embodiment of the present disclosure.

[0015] Figure 4 is a conceptual diagram for explaining a second example of demosaicing processing of an embodiment of the present disclosure.

[0016] Figure 5 is a conceptual diagram for explaining a third example of demosaicing processing of an embodiment of the present disclosure.

[0017] Figure 6 is a conceptual diagram for explaining a fourth example of demosaicing processing of an embodiment of the present disclosure.

[0018] Figure 7 is a flowchart showing an image generation method of an embodiment of the present disclosure.

[0019] Figure 8 is a block diagram showing a configuration example of a vehicle control system of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] An embodiment of the present disclosure will be described with reference to the accompanying drawings.

[0021] 1. Outline of Image Generation Apparatus

[0022] The image generation apparatus 1 of the present embodiment generates an image. For example, the image generation apparatus 1 is a camera mounted on a mobile body such as a vehicle, a robot, or the like. The image generation apparatus 1 (camera) captures a situation around the mobile body and generates an image representing the situation around the mobile body. The generated image is used for control (for example, autonomous travel control, remote control) of the mobile body. As another example, the image generation apparatus 1 is a camera provided on a street of a smart city or the like. The generated image is used for monitoring of the street or the like.

[0023] In the following description, "visible light" is light of a visible region, and "non-visible light" is light of at least a part of a wavelength region other than the visible region (non-visible region). For example, the non-visible light is near-infrared light. The "visible light image IMG_VI" is an image generated by capturing using the visible light. On the other hand, the "non-visible light image IMG_IR" is an image generated by capturing using the non-visible light which is not the visible light. For example, the non-visible light image IMG_IR in the case of near-infrared light is a near-infrared image.

[0024] As described below, the image generation apparatus 1 of the present embodiment is configured to be able to flexibly generate the visible light image IMG_VI and the non-visible light image IMG_IR depending on a situation.

[0025] Figure 1 is a block diagram representing a configuration example of the image generation apparatus 1 of the present embodiment. The image generation apparatus 1 includes a photographing element 10, one or a plurality of processors 20 (hereinafter, simply referred to as "processor 20"), and a storage apparatus 30. Note that the photographing element 10, the processor 20, and the storage apparatus 30 can be integrated in a single apparatus or can be dispersedly configured in a plurality of apparatuses.

[0026] The photographing element 10 receives the non-visible light and the visible light through the filters, respectively. Further, the photographing element 10 acquires "first light reception data RAW_IR" representing a light reception result of the non-visible light and "second light reception data RAW_VI" representing a light reception result of the visible light. The photographing element 10 outputs the acquired first light reception data RAW_IR and second light reception data RAW_VI to the processor 20.

[0027] Figure 2 is a conceptual diagram representing a configuration example of the photographing element 10. The photographing element 10 includes a color filter array (CFA) 11 and a sensor array 12. Light incident to the photographing element 10 is incident to the sensor array 12 through the color filter array 11.

[0028] The color filter array 11 includes a plurality of color filters configured in an array shape. The plurality of color filters are composed of a plurality of kinds of color filters that transmit different wavebands of light. For example, in Figure 2 "R" is a color filter that transmits red light, "G" is a color filter that transmits green light, "B" is a color filter that transmits blue light, and "N" is a color filter that transmits near-infrared light. Red light, green light, and blue light are visible light, and near-infrared light is invisible light.

[0029] The sensor array 12 receives light that has transmitted through the color filter array 11. The sensor array 12 includes a plurality of sensors (cells) configured in an array shape. The plurality of color filters and the plurality of sensors have correspondence established one-to-one. Each sensor includes a photodiode that generates an electric charge corresponding to an amount of light that has transmitted through the corresponding color filter. Light reception data (mosaic data) includes an amount of electric charge generated by each of the plurality of sensors.

[0030] The processor 20 acquires the first light reception data RAW IR and the second light reception data RAW VI from the imaging element 10. Then, the processor 20 performs "development processing" of generating images from the acquired light reception data.

[0031] More specifically, the processor 20 includes a demosaicing processing section 21. The demosaicing processing section 21 performs demosaicing processing on the first light reception data RAW IR, whereby an invisible light image IMG IR can be generated. Hereinafter, the demosaicing processing performed on the first light reception data RAW IR related to invisible light will be referred to as "first demosaicing processing". In addition, the demosaicing processing section 21 performs demosaicing processing on the second light reception data RAW VI, whereby a visible light image IMG VI can be generated. Hereinafter, the demosaicing processing performed on the second light reception data RAW VI related to visible light will be referred to as "second demosaicing processing".

[0032] The processor 20 can also include an image compression section 22 that performs image compression. The image compression section 22 compresses the invisible light image IMG IR and the visible light image IMG VI. The form of compression is not particularly limited.

[0033] The storage device 30 stores the invisible light image IMG IR and the visible light image IMG VI generated by the processor 20. As the storage device 30, a nonvolatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like is exemplified.

[0034] Thus, the image generation device 1 can generate both the visible light image IMG VI and the invisible light image IMG IR at the same time. The visible light image IMG VI is excellent in terms of color reproducibility. On the other hand, the invisible light image IMG IR has an advantage that it can be acquired well even in a dark environment (for example, at night, in a tunnel) where the amount of light of visible light is small.

[0035] However, generating both the visible light image IMG VI and the invisible light image IMG IR means that demosaicing processing will increase accordingly. Excessive demosaicing processing and excessive generation of images can lead to waste of computing resources and storage resources, increase in processing time, and the like. This is not particularly preferable in a control that requires real-time such as vehicle control.

[0036] Therefore, according to the present embodiment, the image generation device 1 is configured to be able to flexibly generate the visible light image IMG VI and the invisible light image IMG IR depending on the situation. In particular, the image generation device 1 is configured to be able to flexibly generate the visible light image IMG VI and the invisible light image IMG IR depending on the illuminance of the surrounding environment.

[0037] Hereinafter, the illuminance-based image generation processing of the present embodiment will be described in detail.

[0038] 2. Illuminance-based image generation processing

[0039] Figure 1 The illustrated illuminance information acquisition device 40 acquires illuminance information ILUM indicating the illuminance of the environment around the imaging element 10 (image generation device 1). That is, the illuminance information acquisition device 40 measures, calculates, or estimates the illuminance of the environment around the imaging element 10. For example, the illuminance information acquisition device 40 is an illuminance sensor that measures the illuminance. As another example, the illuminance information acquisition device 40 can include a laser radar (LIDAR: Laser Imaging Detection and Ranging) and estimate the illuminance from the laser radar reflectance. The illuminance information acquisition device 40 outputs the illuminance information ILUM to the processor 20. Note that the illuminance information acquisition device 40 can be included in the image generation device 1 or can be provided separately from the image generation device 1.

[0040] The processor 20 further includes a demosaicking method decision section 23. The demosaicking method decision section 23 acquires the illumination information ILUM from the illumination information acquisition section 40. Then, the demosaicking method decision section 23 decides the content of the demosaicking processing in the demosaicking processing section 21 based on the illumination indicated by the illumination information ILUM. The demosaicking processing section 21 performs the necessary demosaicking processing in accordance with the decision made by the demosaicking method decision section 23.

[0041] Next, various examples of the demosaicking processing corresponding to the illumination according to the present embodiment will be described.

[0042] 2-1. First Example

[0043] In a dark environment (low-illumination environment) in which the amount of visible light is small, the brightness and quality of the visible light image IMG_VI are low. On the other hand, the invisible light image IMG_IR can be obtained with high sensitivity and high brightness even in a dark environment. Thus, in a dark environment, the invisible light image IMG_IR is useful, and the usefulness of the visible light image IMG_VI is relatively low. It is considered that in such a dark environment, even if the visible light image IMG_VI is not generated, the processing accuracy in the subsequent stage does not change much. In other words, it is considered that it is not necessarily required to generate both the visible light image IMG_VI and the invisible light image IMG_IR, and only the invisible light image IMG_IR is sufficient for the processing in the subsequent stage. From such a viewpoint, the invisible light image IMG_IR is given priority in a dark environment.

[0044] Figure 3 is a conceptual diagram for describing the first example of the demosaicking processing according to the present embodiment. In a case where the illumination is less than the first threshold Thl, the processor 20 performs the first demosaicking processing on the first light reception data RAW_IR but does not perform the second demosaicking processing on the second light reception data RAW_VI. That is, the processor 20 does not generate the visible light image IMG_VI but generates only the invisible light image IMG_IR. Thus, at least in a case where the illumination is less than the first threshold Thl, the demosaicking processing is prevented from being performed excessively. As a result, the waste of the calculation resources and the storage resources is prevented, and in addition, the processing time is reduced.

[0045] Note that in the first example, in a case where the illumination is the first threshold Thl or more, the processor 20 performs the first demosaicking processing and the second demosaicking processing, thereby generating both the invisible light image IMG_IR and the visible light image IMG_VI.

[0046] 2-2. Second Example

[0047] In a bright environment (high-illuminance environment), a good visible light image IMG_VI is generated. In addition, from the viewpoint of color reproducibility, the visible light image IMG_VI is superior to the invisible light image IMG_IR. Thus, in a bright environment, the visible light image IMG_VI is useful, and the usefulness of the invisible light image IMG_IR is low. It is considered that in such a bright environment, even if the invisible light image IMG_IR is not generated, the processing accuracy of the subsequent stage does not change much. In other words, it is considered that it is not necessarily required to generate both the visible light image IMG_VI and the invisible light image IMG_IR, and for the processing of the subsequent stage, only the visible light image IMG_VI is sufficient. From such a viewpoint, the visible light image IMG_VI is given priority in a bright environment.

[0048] Figure 4 is a conceptual diagram for explaining a second example of the demosaicing processing of the present embodiment. In a case where the illuminance is equal to or higher than a second threshold value Th2, the processor 20 performs the second demosaicing processing on the second light reception data RAW_VI, but does not perform the first demosaicing processing on the first light reception data RAW_IR. That is, the processor 20 does not generate the invisible light image IMG_IR, but generates only the visible light image IMG_VI. Thus, at least in a case where the illuminance is equal to or higher than the second threshold value Th2, the demosaicing processing is prevented from being excessively performed. As a result, the waste of the calculation resources and the storage resources is prevented, and in addition, the processing time is also reduced.

[0049] Note that in the second example, in a case where the illuminance is less than the second threshold value Th2, the processor 20 performs the first demosaicing processing and the second demosaicing processing, thereby generating both the invisible light image IMG_IR and the visible light image IMG_VI.

[0050] 2-3. Third Example

[0051] Figure 5 is a conceptual diagram for explaining a third example of the demosaicing processing of the present embodiment. The third example is a combination of the above-described first example and the second example.

[0052] In a case where the illuminance is less than the first threshold value Th1, the processor 20 does not perform the second demosaicing processing, but performs only the first demosaicing processing, does not generate the visible light image IMG_VI, but generates only the invisible light image IMG_IR. On the other hand, in a case where the illuminance is equal to or higher than the second threshold value Th2, the processor 20 does not perform the first demosaicing processing, but performs only the second demosaicing processing, does not generate the invisible light image IMG_IR, but generates only the visible light image IMG_VI. Here, the second threshold value Th2 is higher than the first threshold value Th1. Thus, the effects of both the above-described first example and the second example can be obtained.

[0053] Note that in the third example, in a case where the illuminance is equal to or higher than the first threshold Th1 and lower than the second threshold Th2, the processor 20 performs the first demosaicing process and the second demosaicing process, thereby generating both the invisible light image IMG IR and the visible light image IMG VI.

[0054] 2-4. Fourth example

[0055] Figure 6 is a conceptual diagram for explaining a fourth example of the demosaicing process of the present embodiment. The fourth example is a variation of the above-described third example. Specifically, the first threshold Th1 is equal to the second threshold Th2. In this case, the processor 20 switches according to the illuminance and performs the first demosaicing process and the second demosaicing process.

[0056] 2-5. Processing flow

[0057] Figure 7 is a flowchart that schematically shows the image generation method of the present embodiment.

[0058] In step S10, the image generation apparatus 1 acquires the first light reception data RAW IR and the second light reception data RAW VI using the imaging element 10.

[0059] In step S20, the image generation apparatus 1 performs a development process of generating an image from the acquired light reception data. Specifically, in step S21, the image generation apparatus 1 acquires illuminance information ILUM indicating the illuminance of the environment around the imaging element 10 (the image generation apparatus 1). In step S22, the image generation apparatus 1 decides a demosaicing method according to the illuminance (refer to Figures 3-6 ). Then, in step S23, the image generation apparatus 1 performs a demosaicing process in the method decided in step S22, thereby generating a necessary image.

[0060] In step S30, the image generation apparatus 1 stores the generated image in the storage apparatus 30. Further, the image generation apparatus 1 outputs the generated image to the outside.

[0061] 2-6. Effects

[0062] As explained above, according to the present embodiment, the necessary processing in the first demosaicing processing and the second demosaicing processing is performed in consideration of the illuminance of the environment around the imaging element 10. That is, the visible light image IMG VI and the invisible light image IMG IR can be generated flexibly according to the situation without excessively performing the demosaicing processing. As a result, the waste of the computing resources and the storage resources is prevented, and thus the computing resources and the storage resources are used efficiently. Further, the overall processing time is also reduced. The efficient use of the computing resources and the storage resources and the shortening of the processing time are particularly preferable in a control requiring real-time such as vehicle control.

[0063] 3. Vehicle control system

[0064] The image generation apparatus 1 of the present embodiment is applied to vehicle control (for example, automatic driving control, remote control), for example. Hereinafter, an example in which the image generation apparatus 1 of the present embodiment is applied to vehicle control will be explained.

[0065] Figure 8 is a block diagram showing a configuration example of the vehicle control system 100 of the present embodiment. The vehicle control system 100 includes a sensor group 110, a travel device 120, and a control device 130.

[0066] The sensor group 110 is mounted on a vehicle. The sensor group 110 includes an identification sensor that identifies the situation around the vehicle. The identification sensor includes the image generation apparatus 1 (camera) of the present embodiment. The image generation apparatus 1 photographs the situation around the vehicle and generates an image (invisible light image IMG IR, visible light image IMG VI) representing the situation. The identification sensor can also include a LIDAR, a millimeter wave radar. Further, the identification sensor can also include an illuminance sensor that detects the illuminance of the environment around the vehicle.

[0067] Further, the sensor group 110 includes a vehicle state sensor that detects the state of the vehicle. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a rudder angle sensor, and the like. The sensor group 110 also includes a position sensor that detects the position and the orientation of the vehicle. As the position sensor, a GPS (Global Positioning System) sensor is exemplified.

[0068] The traveling device 120 is mounted on the vehicle. The traveling device 120 includes a steering device, a driving device, and a braking device. The steering device turns the wheels. For example, the steering device includes an electric power steering (EPS) device. The driving device is a power source that generates a driving force. As the driving device, an engine, an electric motor, an in-wheel motor, or the like is exemplified. The braking device generates a braking force.

[0069] The control device 130 controls the vehicle. The control device 130 includes one or a plurality of processors 131 (hereinafter, simply referred to as processor 131) and one or a plurality of storage devices 132 (hereinafter, simply referred to as storage device 132). The processor 131 performs various processes. For example, the processor 131 includes a CPU (Central Processing Unit). The storage device 132 stores various information required for the processes performed by the processor 131. As the storage device 132, a volatile memory, a non-volatile memory, an HDD, an SSD, or the like is exemplified. The control device 130 can also include one or a plurality of ECUs (Electronic Control Units). A part of the control device 130 can also be an information processing device outside the vehicle. In this case, a part of the control device 130 communicates with the vehicle and remotely controls the vehicle.

[0070] The processor 131 performs vehicle traveling control that controls traveling of the vehicle. The vehicle traveling control includes steering control, acceleration control, and deceleration control. The processor 131 performs the vehicle traveling control by controlling the traveling device 120 (the steering device, the driving device, the braking device).

[0071] The processor 131 can also perform automatic driving control. In more detail, the processor 131 recognizes (detects) an object around the vehicle using a recognition sensor. The image generated by the image generation device 1 is used for this object recognition. According to the present embodiment, an image suitable for the surrounding environment (scene) is generated from among the invisible light image IMG IR and the visible light image IMG VI. Thereby, the object recognition is performed with high accuracy. The processor 131 generates a traveling plan, a target trajectory of the vehicle based on the result of the object recognition. Then, the processor 131 performs the vehicle traveling control in such a manner that the vehicle follows the target trajectory.

[0072] The image generated by the image generation device 1 can also be used for remote assistance, remote driving of the vehicle. In this case, the control device 130 communicates with a remote assistance device (not shown) and transmits the image to the remote assistance device. The remote assistance device displays the received image on a display device. A remote operator looks at the displayed image to perform remote instruction or remote operation. The remote assistance device transmits information of the remote instruction or remote operation performed by the remote operator to the vehicle. The control device 130 performs vehicle travel control in accordance with the received remote instruction or remote operation.

[0073] As explained above, the image generated by the image generation device 1 can be used for vehicle control. According to the present embodiment, the image suitable for the surrounding environment (scene) is generated in the invisible light image IMG IR and the visible light image IMG VI. That is, unnecessary demosaicing processing can be suppressed, and the accuracy of vehicle control can also be ensured.

Claims

1.An image generation apparatus that generates an image, wherein the image generation apparatus includes one or more processors, the one or more processors are configured to: obtain first light reception data representing a light reception result of non-visible light and second light reception data representing a light reception result of visible light from a shooting element that respectively receives the non-visible light and the visible light through a filter, and generate the image by performing demosaicing processing on at least one of the first light reception data and the second light reception data based on illumination information representing an illumination of an environment around the shooting element, a second threshold is higher than a first threshold, in a case where the illumination is less than the first threshold, the one or more processors do not perform second demosaicing processing on the second light reception data, but perform first demosaicing processing on the first light reception data, thereby generating a non-visible light image as the image, in a case where the illumination is greater than or equal to the second threshold, the one or more processors do not perform the first demosaicing processing on the first light reception data, but perform the second demosaicing processing on the second light reception data, thereby generating a visible light image as the image, in a case where the illumination is greater than or equal to the first threshold and less than the second threshold, the one or more processors perform the first demosaicing processing and the second demosaicing processing, thereby generating both the non-visible light image and the visible light image. 2.The image generation apparatus according to claim 1, wherein the non-visible light is near-infrared light. 3.A vehicle control system that includes: the image generation apparatus according to claim 1 or 2; and a control apparatus that controls a vehicle based on the image. 4.An image generation method that includes: an obtaining process that obtains first light reception data representing a light reception result of non-visible light and second light reception data representing a light reception result of visible light using a shooting element that respectively receives the non-visible light and the visible light through a filter, and a developing process that generates an image by performing demosaicing processing on at least one of the first light reception data and the second light reception data based on illumination information representing an illumination of an environment around the shooting element, a second threshold is higher than a first threshold, the developing process includes the following processing: in a case where the illumination is less than the first threshold, second demosaicing processing is not performed on the second light reception data, but first demosaicing processing is performed on the first light reception data, thereby generating a non-visible light image as the image, in a case where the illumination is greater than or equal to the second threshold, the first demosaicing processing is not performed on the first light reception data, but the second demosaicing processing is performed on the second light reception data, thereby generating a visible light image as the image, in a case where the illumination is greater than or equal to the first threshold and less than the second threshold, the first demosaicing processing and the second demosaicing processing are performed, thereby generating both the non-visible light image and the visible light image.

Citation Information

Patent Citations

  • Video signal processor, imaging apparatus, video signal processing method, and computer program

    JP2007174277A

  • Imaging device and imaging system

    CN105144688A

  • Image processing apparatus, image pickup device, image pickup apparatus, and image processing method

    US20170280122A1