IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, AND VEHICLE CONTROL SYSTEM

Abstract

To flexibly generate a visible light image and an invisible light image according to the situation without performing demosaicing unnecessarily.SOLUTION: An image generation device acquires first received light data and second received light data indicating a result of receiving invisible light, from an imaging element which receives invisible light and visible light separately through a filter. The image generation device generates an image by performing demosaicing on at least one of the first received light data and the second received light data based on illuminance information indicating illuminance of surrounding environment of the imaging element. When the illuminance is less than a first threshold value, the image generation device generates an invisible light image by performing first demosaicing on the first received light data without performing second demosaicing on the second received light data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image generation technique capable of flexibly generating visible light images and invisible light images. [Background technology]

[0002] Patent Document 1 discloses an image signal processing device equipped with a single-chip imaging element. The single-chip imaging element is composed of a specific wavelength region signal acquisition element that acquires a visible light region signal such as an RGB signal, and a wide wavelength region signal acquisition element that acquires an optical signal including visible light and infrared light components. The image signal processing device generates a demosaic image corresponding to the wide wavelength region signal as a luminance signal. The image signal processing device also generates a color difference signal based on the demosaic image corresponding to the visible light region signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-174277 A Summary of the Invention [Problem to be solved by the invention]

[0004] Images captured by cameras may be used in vehicle control, city monitoring, and the like. In a dark environment with a low amount of visible light, invisible light images such as near-infrared images are also useful. Therefore, a camera capable of acquiring both normal visible light images and invisible light images such as near-infrared images is considered useful. However, generating both visible light images and invisible light images means that the demosaicing process increases accordingly. Performing demosaicing more than necessary and generating more images than necessary leads to waste of computational and storage resources, increased processing time, and the like.

[0005] One object of the present disclosure is to provide a technique capable of flexibly generating visible light images and invisible light images according to the situation, without performing demosaicing more than necessary. [Means for solving the problem]

[0006] The first aspect relates to an image generating device that generates an image. The image generation device comprises one or more processors. The one or more processors acquire first light reception data indicating the result of receiving invisible light and second light reception data indicating the result of receiving visible light from an imaging element that receives invisible light and visible light separately through a filter. Furthermore, the one or more processors generate an image by performing a demosaic process on at least one of the first light reception data and the second light reception data based on illuminance information indicating the illuminance of the environment surrounding the image sensor. When the illuminance is less than the first threshold, the one or more processors generate an invisible light image as an image by performing a first demosaic processing on the first received light data without performing a second demosaic processing on the second received light data.

[0007] A second aspect relates to an image generating device that generates an image. The image generation device comprises one or more processors. The one or more processors acquire first light reception data indicating the result of receiving invisible light and second light reception data indicating the result of receiving visible light from an imaging element that receives invisible light and visible light separately through a filter. Furthermore, the one or more processors generate an image by performing a demosaic process on at least one of the first light reception data and the second light reception data based on illuminance information indicating the illuminance of the environment surrounding the image sensor. When the illuminance is equal to or greater than the threshold, the one or more processors perform a second demosaic process on the second received light data without performing a first demosaic process on the first received light data, thereby generating a visible light image as an image.

[0008] The third aspect relates to a vehicle control system. The vehicle control system An image generating device according to the first or second aspect, A control device that controls the vehicle based on the generated image. Equipped with.

[0009] The fourth aspect relates to an image generating method. The image generation method is A process of acquiring first light reception data indicating a result of receiving invisible light and second light reception data indicating a result of receiving visible light using an image sensor that receives invisible light and visible light separately through a filter; a development process for generating an image by performing a demosaic process on at least one of the first light reception data and the second light reception data based on illuminance information indicating the illuminance of the environment surrounding the image sensor; Includes. The development process includes a process of generating an invisible light image as an image by performing a first demosaic process on the first received light data without performing a second demosaic process on the second received light data when the illuminance is less than a first threshold value.

[0010] The fifth aspect relates to an image generating method. The image generation method is A process of acquiring first light reception data indicating a result of receiving invisible light and second light reception data indicating a result of receiving visible light using an image sensor that receives invisible light and visible light separately through a filter; a development process for generating an image by performing a demosaic process on at least one of the first light reception data and the second light reception data based on illuminance information indicating the illuminance of the environment surrounding the image sensor; Includes. The development process includes a process of generating a visible light image as an image by performing a second demosaic process on the second received light data without performing a first demosaic process on the first received light data when the illuminance is equal to or greater than a threshold value. Effect of the Invention

[0011] According to the present disclosure, the first demosaic process and the second demosaic process are executed as necessary, taking into consideration the illuminance of the environment around the image sensor. In other words, it is possible to flexibly generate visible light images and invisible light images according to the situation without performing demosaic processes more than necessary. As a result, the waste of computational and storage resources is prevented, and the computational and storage resources are used efficiently. In addition, the overall processing time is reduced. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an image generating device according to an embodiment of the present disclosure. [Diagram 2] 1 is a conceptual diagram illustrating a configuration example of an imaging element according to an embodiment of the present disclosure. [Diagram 3] 1 is a conceptual diagram for explaining a first example of demosaic processing according to an embodiment of the present disclosure. FIG. [Figure 4] FIG. 13 is a conceptual diagram for explaining a second example of demosaic processing according to an embodiment of the present disclosure. [Diagram 5] FIG. 13 is a conceptual diagram for explaining a third example of demosaic processing according to an embodiment of the present disclosure. [Figure 6] FIG. 13 is a conceptual diagram for explaining a fourth example of demosaic processing according to an embodiment of the present disclosure. [Figure 7] 1 is a flowchart illustrating an image generating method according to an embodiment of the present disclosure. [Figure 8] 1 is a block diagram illustrating an example configuration of a vehicle control system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0014] 1. Overview of the image generation device The image generating device 1 according to this embodiment generates an image. For example, the image generating device 1 is a camera mounted on a moving object such as a vehicle or a robot. The image generating device 1 (camera) captures the surroundings of the moving object and generates an image showing the surroundings. The generated image is used for controlling the moving object (e.g., autonomous driving control, remote control). As another example, the image generating device 1 is a camera installed in a town such as a smart city. The generated image is used for monitoring the town, etc.

[0015] In the following description, "visible light" refers to light in the visible range, and "invisible light" refers to at least a portion of light in a wavelength range (invisible range) other than the visible range. For example, invisible light is near-infrared light. A "visible light image IMG_VI" is an image generated by imaging using visible light. On the other hand, an "invisible light image IMG_IR" is an image generated by imaging using invisible light that is not visible light. For example, in the case of near-infrared light, the invisible light image IMG_IR is a near-infrared image.

[0016] As will be described below, the image generating device 1 according to this embodiment is configured to be able to flexibly generate a visible light image IMG_VI and an invisible light image IMG_IR depending on the situation.

[0017] 1 is a block diagram showing an example of the configuration of an image generating device 1 according to the present embodiment. The image generating device 1 includes an image sensor 10, one or more processors 20 (hereinafter simply referred to as "processor 20"), and a storage device 30. The image sensor 10, the processor 20, and the storage device 30 may be integrated in a single device, or may be distributed across multiple devices.

[0018] The image sensor 10 receives invisible light and visible light separately through a filter. The image sensor 10 then obtains "first light reception data RAW_IR" indicating the result of receiving invisible light and "second light reception data RAW_VI" indicating the result of receiving visible light. The image sensor 10 outputs the obtained first light reception data RAW_IR and second light reception data RAW_VI to the processor 20.

[0019] 2 is a conceptual diagram showing an example of the configuration of the image sensor 10. The image sensor 10 includes a color filter array (CFA) 11 and a sensor array 12. Light incident on the image sensor 10 passes through the color filter array 11 and enters the sensor array 12.

[0020] The color filter array 11 includes a plurality of color filters arranged in an array. The plurality of color filters are composed of a plurality of types of color filters that transmit light of different wavelength bands. For example, in FIG. 2, "R" is a color filter that transmits red light, "G" is a color filter that transmits green light, "B" is a color filter that transmits blue light, and "N" is a color filter that transmits near-infrared light. Red light, green light, and blue light are visible light, and near-infrared light is invisible light.

[0021] The sensor array 12 receives light transmitted through the color filter array 11. The sensor array 12 includes a plurality of sensors (cells) arranged in an array. The filters and the sensors are in one-to-one correspondence. Each sensor includes a photodiode and generates an electric charge according to the amount of light transmitted through the corresponding filter. The received light data (mosaic data) includes the amount of electric charge generated by each of the plurality of sensors.

[0022] The processor 20 acquires the first received light data RAW_IR and the second received light data RAW_VI from the image sensor 10. Then, the processor 20 performs a "development process" to generate an image from the acquired received light data.

[0023] More specifically, the processor 20 includes a demosaic processing unit 21. The demosaic processing unit 21 can generate an invisible light image IMG_IR by performing a demosaic process on the first received light data RAW_IR. The demosaic process performed on the first received light data RAW_IR related to invisible light is hereinafter referred to as a "first demosaic process." Moreover, the demosaic processing unit 21 can generate a visible light image IMG_VI by performing a demosaic process on the second received light data RAW_VI. The demosaic process performed on the second received light data RAW_VI related to visible light is hereinafter referred to as a "second demosaic process."

[0024] The processor 20 may further include an image compression unit 22 that performs image compression. The image compression unit 22 compresses the invisible light image IMG_IR and the visible light image IMG_VI. The compression format is not particularly limited.

[0025] The storage device 30 stores the invisible light image IMG_IR and the visible light image IMG_VI generated by the processor 20. Examples of the storage device 30 include a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0026] In this way, the image generating device 1 can generate both the visible light image IMG_VI and the invisible light image IMG_IR simultaneously. The visible light image IMG_VI is superior in terms of color reproducibility. On the other hand, the invisible light image IMG_IR has the advantage that it can be obtained well even in a dark environment (e.g., at night, in a tunnel) where the amount of visible light is low.

[0027] However, generating both the visible light image IMG_VI and the invisible light image IMG_IR means that the demosaicing process increases accordingly. Performing demosaicing more than necessary and generating more images than necessary leads to waste of computational and storage resources, increased processing time, etc. These are particularly undesirable in control that requires real-time performance, such as vehicle control.

[0028] Therefore, according to the present embodiment, the image generating device 1 is configured to be able to flexibly generate the visible light image IMG_VI and the invisible light image IMG_IR according to the situation. In particular, the image generating device 1 is configured to be able to flexibly generate the visible light image IMG_VI and the invisible light image IMG_IR according to the illuminance of the surrounding environment.

[0029] The image generation process based on illuminance according to this embodiment will be described in detail below.

[0030] 2. Image generation processing based on illuminance The illuminance information acquisition device 40 shown in FIG. 1 acquires illuminance information ILUM indicating the illuminance of the environment surrounding the image capture element 10 (image generation device 1). That is, the illuminance information acquisition device 40 measures, calculates, or estimates the illuminance of the environment surrounding the image capture element 10. For example, the illuminance information acquisition device 40 is an illuminance sensor that measures illuminance. As another example, the illuminance information acquisition device 40 may include a LIDAR (Laser Imaging Detection and Ranging) and estimate the illuminance from the LIDAR reflectance. The illuminance information acquisition device 40 outputs the illuminance information ILUM to the processor 20. The illuminance information acquisition device 40 may be included in the image generation device 1, or may be provided separately from the image generation device 1.

[0031] The processor 20 further includes a demosaic method determination unit 23. The demosaic method determination unit 23 acquires illuminance information ILUM from the illuminance information acquisition device 40. Then, the demosaic method determination unit 23 determines the content of demosaic processing in the demosaic processing unit 21 based on the illuminance indicated by the illuminance information ILUM. The demosaic processing unit 21 performs necessary demosaic processing in accordance with the determination by the demosaic method determination unit 23.

[0032] Various examples of the demosaic process according to the present embodiment in accordance with the illuminance will be described below.

[0033] 2-1. First Example In a dark environment (low illuminance environment) where the amount of visible light is small, the brightness and quality of the visible light image IMG_VI are low. On the other hand, the invisible light image IMG_IR can be obtained with high sensitivity and high brightness even in a dark environment. Therefore, in a dark environment, the invisible light image IMG_IR is useful, and the usefulness of the visible light image IMG_VI is relatively low. In such a dark environment, even if the visible light image IMG_VI is not generated, it is considered that the accuracy of the subsequent processing does not change significantly. In other words, it is not necessarily necessary to generate both the visible light image IMG_VI and the invisible light image IMG_IR, and it is considered that only the invisible light image IMG_IR is sufficient for the subsequent processing. From this viewpoint, the invisible light image IMG_IR is prioritized in a dark environment.

[0034] FIG. 3 is a conceptual diagram for explaining a first example of demosaic processing according to the present embodiment. When the illuminance is less than the first threshold value Th1, the processor 20 performs the first demosaic processing on the first received light data RAW_IR, but does not perform the second demosaic processing on the second received light data RAW_VI. That is, the processor 20 generates only the invisible light image IMG_IR without generating the visible light image IMG_VI. This prevents the demosaic processing from being performed more than necessary, at least when the illuminance is less than the first threshold value Th1. As a result, waste of computational resources and storage resources is prevented, and processing time is also reduced.

[0035] In the first example, when the illuminance is equal to or greater than the first threshold Th1, the processor 20 performs the first demosaic process and the second demosaic process to generate both the invisible light image IMG_IR and the visible light image IMG_VI.

[0036] 2-2. Second Example In a bright environment (high illuminance environment), a good visible light image IMG_VI is generated. Also, from the viewpoint of color reproducibility, the visible light image IMG_VI is superior to the invisible light image IMG_IR. Therefore, in a bright environment, the visible light image IMG_VI is useful, and the usefulness of the invisible light image IMG_IR is relatively low. In such a bright environment, even if the invisible light image IMG_IR is not generated, it is considered that the accuracy of the subsequent processing does not change significantly. In other words, it is not necessarily necessary to generate both the visible light image IMG_VI and the invisible light image IMG_IR, and it is considered that only the visible light image IMG_VI is sufficient for the subsequent processing. From this viewpoint, the visible light image IMG_VI is prioritized in a bright environment.

[0037] FIG. 4 is a conceptual diagram for explaining a second example of the demosaic processing according to the present embodiment. When the illuminance is equal to or greater than the second threshold value Th2, the processor 20 performs the second demosaic processing on the second received light data RAW_VI, but does not perform the first demosaic processing on the first received light data RAW_IR. That is, the processor 20 generates only the visible light image IMG_VI without generating the invisible light image IMG_IR. This prevents the demosaic processing from being performed more than necessary, at least when the illuminance is equal to or greater than the second threshold value Th2. As a result, waste of computational and storage resources is prevented, and processing time is also reduced.

[0038] In the second example, when the illuminance is less than the second threshold Th2, the processor 20 performs the first demosaic processing and the second demosaic processing to generate both the invisible light image IMG_IR and the visible light image IMG_VI.

[0039] 2-3.Third Example 5 is a conceptual diagram for explaining a third example of the demosaic processing according to the present embodiment. The third example is a combination of the first and second examples described above.

[0040] When the illuminance is less than the first threshold Th1, the processor 20 performs only the first demosaic process without performing the second demosaic process, and generates only the invisible light image IMG_IR without generating the visible light image IMG_VI. On the other hand, when the illuminance is equal to or greater than the second threshold Th2, the processor 20 performs only the second demosaic process without performing the first demosaic process, and generates only the visible light image IMG_VI without generating the invisible light image IMG_IR. Here, the second threshold Th2 is higher than the first threshold Th1. This provides the effects of both the first and second examples described above.

[0041] In addition, in the third example, when the illuminance is greater than or equal to the first threshold Th1 and less than the second threshold Th2, the processor 20 performs the first demosaic processing and the second demosaic processing to generate both the invisible light image IMG_IR and the visible light image IMG_VI.

[0042] 2-4. Fourth Example 6 is a conceptual diagram for explaining a fourth example of the demosaic processing according to the present embodiment. The fourth example is a modified example of the above-mentioned third example. Specifically, the first threshold value Th1 and the second threshold value Th2 are equal. In this case, the processor 20 switches between the first demosaic processing and the second demosaic processing depending on the illuminance.

[0043] 2-5. Processing flow FIG. 7 is a flowchart summarizing the image generating method according to the present embodiment.

[0044] In step S10, the image generating device 1 uses the image sensor 10 to obtain the first received light data RAW_IR and the second received light data RAW_VI.

[0045] In step S20, the image generating device 1 performs a development process to generate an image from the acquired light reception data. Specifically, in step S21, the image generating device 1 acquires illuminance information ILUM indicating the illuminance of the environment surrounding the imaging element 10 (image generating device 1). In step S22, the image generating device 1 determines a demosaic method according to the illuminance (see FIGS. 3 to 6). Then, in step S23, the image generating device 1 executes demosaic processing according to the method determined in step S22 to generate a required image.

[0046] In step S30, the image generating device 1 stores the generated image in the storage device 30. In addition, the image generating device 1 outputs the generated image to the outside.

[0047] 2-6.Effects As described above, according to the present embodiment, the first demosaic process and the second demosaic process are executed as necessary, taking into consideration the illuminance of the environment around the image sensor 10. That is, it is possible to flexibly generate the visible light image IMG_VI and the invisible light image IMG_IR according to the situation without executing the demosaic process more than necessary. As a result, the waste of computational resources and storage resources is prevented, and the computational resources and storage resources are used efficiently. In addition, the overall processing time is also reduced. These are particularly preferable in control that requires real-time performance, such as vehicle control.

[0048] 3.Vehicle Control System The image generating device 1 according to the present embodiment is applied to, for example, vehicle control (e.g., automatic driving control, remote control). An example in which the image generating device 1 according to the present embodiment is applied to vehicle control will be described below.

[0049] 8 is a block diagram showing an example of the configuration of a vehicle control system 100 according to this embodiment. The vehicle control system 100 includes a sensor group 110, a traveling device 120, and a control device .

[0050] The sensor group 110 is mounted on the vehicle. The sensor group 110 includes a recognition sensor that recognizes the surroundings of the vehicle. The recognition sensor includes an image generating device 1 (camera) according to this embodiment. The image generating device 1 captures the surroundings of the vehicle and generates images (invisible light image IMG_IR, visible light image IMG_VI) showing the surroundings. The recognition sensor may further include a LIDAR or a millimeter wave radar. Furthermore, the recognition sensor may include an illuminance sensor that detects the illuminance of the environment around the vehicle.

[0051] The sensor group 110 also includes a vehicle state sensor that detects the state of the vehicle. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The sensor group 110 also includes a position sensor that detects the position and direction of the vehicle. An example of the position sensor is a GPS (Global Positioning System) sensor.

[0052] The traveling device 120 is mounted on a vehicle. The traveling device 120 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0053] The control device 130 controls the vehicle. The control device 130 includes one or more processors 131 (hereinafter simply referred to as processor 131) and one or more storage devices 132 (hereinafter simply referred to as storage devices 132). The processor 131 executes various processes. For example, the processor 131 includes a CPU (Central Processing Unit). The storage device 132 stores various information required for processing by the processor 131. Examples of the storage device 132 include a volatile memory, a non-volatile memory, a HDD, and an SSD. The control device 130 may include one or more ECUs (Electronic Control Units). A part of the control device 130 may be an information processing device outside the vehicle. In this case, a part of the control device 130 communicates with the vehicle and remotely controls the vehicle.

[0054] The processor 131 executes vehicle driving control for controlling the driving of the vehicle. The vehicle driving control includes steering control, acceleration control, and deceleration control. The processor 131 executes the vehicle driving control by controlling the driving device 120 (steering device, driving device, braking device).

[0055] The processor 131 may perform automatic driving control. More specifically, the processor 131 recognizes (detects) objects around the vehicle using a recognition sensor. For this object recognition, an image generated by the image generating device 1 is used. According to this embodiment, one of the invisible light image IMG_IR and the visible light image IMG_VI that is suitable for the surrounding environment (scene) is generated. Thus, object recognition is performed with high accuracy. The processor 131 generates a driving plan and a target trajectory for the vehicle based on the result of the object recognition. Then, the processor 131 performs vehicle driving control so that the vehicle follows the target trajectory.

[0056] The image generated by the image generation device 1 may be used for remote assistance or remote driving of a vehicle. In this case, the control device 130 communicates with a remote assistance device (not shown) and transmits the image to the remote assistance device. The remote assistance device displays the received image on a display device. A remote operator views the displayed image and performs remote instructions or remote operation. The remote assistance device transmits information on the remote instructions or remote operation by the remote operator to the vehicle. The control device 130 controls vehicle driving in accordance with the received remote instructions or remote operation.

[0057] As described above, the image generated by the image generating device 1 can be used for vehicle control. According to this embodiment, one of the invisible light image IMG_IR and the visible light image IMG_VI that is suitable for the surrounding environment (scene) is generated. That is, it is possible to ensure the accuracy of vehicle control while suppressing unnecessary demosaicing. [Explanation of symbols]

[0058] 1 Image generation device 10. Image sensor 11 Color filter array 12 Sensor Array 20 processors 21 Demosaic Processing Unit 22 Image Compression Section 23 Demosaic Method Decision Unit 30 Storage device 40 Illuminance information acquisition device 100 Vehicle Control System 110 Sensor Group 120 Running gear 130 Control device ILUM illuminance information IMG_IR invisible light image IMG_VI Visible light image RAW_IR First light reception data RAW_VI Second light receiving data

Claims

1. An image generating device for generating an image, comprising: One or more processors; The one or more processors: acquiring, from an image sensor that receives invisible light and visible light separately through a filter, first light reception data indicating a result of receiving the invisible light and second light reception data indicating a result of receiving the visible light; The image is generated by performing demosaic processing on at least one of the first light reception data and the second light reception data based on illuminance information indicating illuminance of an environment surrounding the image sensor. It is configured as follows: The second threshold is higher than the first threshold, When the illuminance is less than the first threshold, the one or more processors perform a first demosaic process on the first light reception data without performing a second demosaic process on the second light reception data to generate an invisible light image as the image; When the illuminance is equal to or greater than the second threshold, the one or more processors perform the second demosaic processing on the second light reception data without performing the first demosaic processing on the first light reception data, thereby generating a visible light image as the image; When the illuminance is greater than or equal to the first threshold and less than the second threshold, the one or more processors generate both the invisible light image and the visible light image by performing the first demosaic process and the second demosaic process. Image generating device.

2. 2. The image generating device according to claim 1, The invisible light is near infrared light. Image generating device.

3. The image generating device according to claim 1 ; a control device for controlling a vehicle based on the image; Equipped Vehicle control system.

4. a process of acquiring first light reception data indicating a result of receiving the invisible light and second light reception data indicating a result of receiving the visible light using an image sensor that receives the invisible light and the visible light separately through a filter; a development process for generating an image by performing demosaic processing on at least one of the first light reception data and the second light reception data based on illuminance information indicating the illuminance of the environment surrounding the image sensor; Including, The second threshold is higher than the first threshold, The development process includes: a process of generating an invisible light image as the image by performing a first demosaic process on the first light reception data without performing a second demosaic process on the second light reception data when the illuminance is less than the first threshold value; a process of performing the second demosaic processing on the second light reception data without performing the first demosaic processing on the first light reception data when the illuminance is equal to or greater than the second threshold value, thereby generating a visible light image as the image; generating both the invisible light image and the visible light image by performing the first demosaic processing and the second demosaic processing when the illuminance is equal to or greater than the first threshold value and less than the second threshold value; Includes Image generation method.

Images