Image adjustment device and image adjustment method

Abstract

To provide an image adjustment device and an image adjustment method capable of appropriately adjusting an image.SOLUTION: An imaging system 1 comprises: an object detection unit 43 for detecting objects included in a first image; a selection unit 44 for selecting a target object out of the objects included in the first image, referring to priority information given according to temperatures of the objects; an area identification unit 51 for identifying a determination area in which the target object is included in a second image; a counting unit 52 for counting saturated pixels that are saturated in the determination area; a determination unit 53 for determining whether or not a ratio of the saturated pixels in the determination area is equal to or larger than a threshold value; and a change unit 54 for changing a value of a gain or offset of a second imaging element when the ratio is equal to or larger than the threshold value.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an image adjustment device and an image adjustment method. [Background technology]

[0002] Patent Document 1 discloses a vehicle display control device mounted on a vehicle. This vehicle display control device acquires a visible light image and an infrared image. The vehicle display control device detects an object outside the vehicle based on the infrared image. The control device detects an overlapping area from the infrared image based on the range of the visible light image outside the vehicle. When the object exists within the overlapping area, the control device superimposes information about the object on the visible light image and transmits it as a display image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-191668 A Summary of the Invention [Problem to be solved by the invention]

[0004] Some far-infrared camera image sensors use a bolometer that receives far-infrared rays and generates heat. In far-infrared cameras using a bolometer, even when a uniform temperature surface such as a black body furnace is photographed, the output value of each pixel is not uniform and varies from pixel to pixel. In addition, when the environmental temperature increases, the variation in the output value of each pixel tends to increase.

[0005] In far-infrared cameras, the temperature resolution is determined by setting the gain. However, if the temperature resolution is set high (so that minute temperature differences can be captured), the effect of variations in the output value for each pixel becomes greater, so that in cases where the temperature of the imaging environment is high or a high-temperature object is being imaged, the signal processing range of the far-infrared camera may be exceeded and the pixel output value may become saturated. On the other hand, if the temperature resolution is lowered to prevent saturation, different objects with different temperatures may appear as if they are the same object. On the other hand, the temperature measurement range can be adjusted by offsetting. When the pixel output value becomes saturated, it is possible to adjust it to a non-saturated output value by offsetting.

[0006] In view of the above-mentioned problems, the present disclosure aims to provide an image adjustment device and an image adjustment method that are capable of acquiring a far-infrared image with high temperature resolution and no saturation of a target object that is imaged by a far-infrared camera. [Means for solving the problem]

[0007] An image adjustment device according to one aspect of the present embodiment includes a first image acquisition unit that acquires a first image captured by a first imaging element, a second image acquisition unit that acquires a second image captured by a second imaging element that detects far-infrared light, an object detection unit that detects an object included in the first image, a selection unit that selects a target object from the objects included in the first image by referring to priority information provided according to a temperature of the object, an area identification unit that identifies a judgment area in which the target object is included in the second image, a counting unit that counts saturated pixels that are saturated in the judgment area, a judgment unit that judges whether a ratio of the saturated pixels in the judgment area is equal to or greater than a threshold value, and a modification unit that changes a gain or offset value of the second imaging element if the ratio is equal to or greater than the threshold value.

[0008] An image adjustment method according to one aspect of this embodiment includes the steps of acquiring a first image captured by a first imaging element, acquiring a second image captured by a second imaging element that detects far-infrared light, detecting an object included in the first image, selecting a target object from among the objects included in the first image by referring to priority information given according to the temperature of the object, identifying a judgment area in the second image in which the target object is included, counting saturated pixels in the judgment area, determining whether the ratio of the saturated pixels in the judgment area is equal to or greater than a threshold value, and changing a gain or offset value in the second imaging element if the ratio is equal to or greater than the threshold value. Effect of the Invention

[0009] According to the present embodiment, it is possible to provide an image adjustment device and an image adjustment method that are capable of acquiring a far-infrared image with high temperature resolution and no saturation for a target object that is the image capture target. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of an imaging system. [Diagram 2] FIG. 13 is a diagram showing far-infrared images before and after correction. [Diagram 3] 11A and 11B are diagrams for explaining pixel data when the variation in output characteristics is small. [Figure 4] 11A and 11B are diagrams for explaining pixel data when the output characteristics vary greatly; [Diagram 5] FIG. 2 is a block diagram showing the configuration of a far-infrared camera of the imaging system. [Figure 6] FIG. 2 is a block diagram showing a configuration of a control device of the imaging system. [Figure 7] FIG. 4 is a diagram showing an example of an object detected by an object detection unit. [Figure 8] FIG. 4 is a diagram showing an example of a target object selected by a selection unit; [Figure 9]11 is a diagram showing an example of a determination region identified by a region identifying unit; [Figure 10] 11 is a diagram showing an example of a determination region identified by a region identifying unit based on environmental information; FIG. [Figure 11] 11 is a diagram showing an example of a target object selected by a region specifying unit based on environmental information; [Figure 12] 4 is a flowchart showing an image adjustment method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0012] The imaging system according to the present embodiment can be mounted on a moving object such as a vehicle. For example, the imaging system is used as an in-vehicle device such as a drive recorder. The imaging system may also be used as a surveillance camera or a security camera.

[0013] Fig. 1 is a block diagram showing a schematic system configuration of an imaging system according to this embodiment. The imaging system 1 according to this embodiment includes a visible light camera 2, a far-infrared camera 3, and a control device 4. Although the example of the imaging system 1 shown in Fig. 1 further includes a display 7, the imaging system 1 does not necessarily need to include the display 7.

[0014] The visible light camera 2 detects visible light and captures an image of a subject. The visible light camera 2 may capture moving images or successive still images. The image captured by the visible light camera 2 is referred to as a visible light image or a first image. The visible light camera 2 is connected to the control device 4 via wireless or wired communication. The visible light camera 2 captures an image of a subject at a predetermined angle of view.

[0015] Specifically, the visible light camera 2 includes a lens unit 21 and an imaging element 22. The imaging element 22 is a photodetector such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. The imaging element 22 includes a plurality of pixels (light receiving elements) arranged in the horizontal and vertical directions. The lens unit 21 is disposed on the incident side of the imaging element 22. The lens unit 21 forms an image of a subject on the imaging element 22. The imaging element 22 detects visible light refracted by the lens unit 21. The lens unit 21 includes at least one lens. For example, the lens unit 21 may include a plurality of lenses such as a zoom lens and a focus lens. The imaging element 22 is also referred to as a first imaging element.

[0016] The far-infrared camera 3 detects far-infrared light (also called far-infrared rays) and captures an image of a subject. The far-infrared camera 3 captures moving images. Alternatively, the far-infrared camera 3 captures successive still images. The image captured by the far-infrared camera 3 is a far-infrared image, a thermal image, or a second image. The far-infrared camera 3 is connected to the control device 4 via wireless or wired communication. The far-infrared camera 3 captures an image of a subject at a predetermined angle of view.

[0017] The far-infrared camera 3 includes a lens unit 31 and an imaging element 32. The imaging element 32 is a photodetector such as a microbolometer. The imaging element 32 may be a thermal type (non-cooled type) element or a quantum type (cooled type) element. The imaging element 32 includes a plurality of pixels arranged in the horizontal and vertical directions. The lens unit 31 is disposed on the incident side of the imaging element 32. The lens unit 31 forms an image of a subject on the imaging element 32. The imaging element 32 detects far-infrared light refracted by the lens unit 31. The lens unit 31 may include a plurality of lenses such as a zoom lens and a focus lens. The imaging element 32 is also referred to as a second imaging element. The output bit width of the far-infrared camera 3 is, for example, 10 bits to 14 bits.

[0018] The far-infrared camera 3 and the visible light camera 2 may be installed separately as long as they can capture the same direction and have the same capture range or one captures the other. The relative mounting position and mounting direction of the visible light camera 2 with respect to the far-infrared camera 3 are known.

[0019] The far-infrared camera 3 and the visible light camera 2 may be arranged coaxially. In this case, a beam splitter or the like that splits far-infrared light and visible light may be provided in front of the far-infrared camera 3 and the visible light camera 2.

[0020] The control device 4 has a hardware configuration of a normal computer including, for example, a processor 4a such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), an internal memory 4b such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a storage device 4c such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), an input / output I / F 4d for connecting peripheral devices such as a display 7, and a communication I / F 4e for communicating with devices outside the device. The control device 4 can perform image processing and control, which will be described later, by executing a computer program stored in the storage device 4c.

[0021] The control device 4 is communicatively connected to the display 7. The control device 4 may be connected to the display 7 by wire or wirelessly. The control device 4 performs image processing on the far-infrared image data. The control device 4 may then cause the display 7 to display the far-infrared image. The control device 4 may also cause the display 7 to display the visible light image.

[0022] The display 7 is a display device including, for example, a liquid crystal panel or an organic electroluminescence panel, and is provided at a position where the user can view it. The display 7 displays the image captured by the far-infrared camera 3 via the control device 4.

[0023] With the above-mentioned configuration, the control device 4 displays the far-infrared image captured by the far-infrared camera 3 on the display 7 in a manner that is visible to the user. This allows the control device 4 to allow the user to recognize surrounding objects. If the imaging system 1 does not include the display 7, the control device 4 may store the far-infrared image captured by the far-infrared camera 3 in an external storage device (not shown). Furthermore, the control device 4, the far-infrared camera 3, and the display 7 may be configured as an integrated imaging device.

[0024] Below, we will explain the issues that arise when using a far-infrared camera 3 that captures far-infrared images. The control device 4 performs NUC (Non-Uniformity Correction) on the far-infrared image data. NUC corrects variations in pixel output. Fig. 2 shows an image before correction (before NUC) and an image after correction (after NUC). Fig. 2 is a far-infrared image captured of a uniform surface (blackbody furnace) with a uniform temperature.

[0025] It can be seen that the uncorrected image has unevenness, resulting in variation in pixel output, i.e., even when capturing an image of a subject with a uniform temperature, there is variation in the output characteristics of the bolometer of each pixel.

[0026] The output variation tends to increase as the environmental temperature of the image sensor 32 rises, even after correction. In addition, the output of the bolometer is an AD converted digital signal, and the bit width of the output range is limited. For example, the output bit width of the bolometer is often 10 to 14 bits. Therefore, if the output value of a pixel exceeds this range, saturation occurs, and normal signal processing cannot be performed, resulting in the output of an abnormal image.

[0027] The output characteristics when the gain of the image sensor 32 is low and when it is high will be described with reference to Figs. 3 and 4. Figs. 3 and 4 show pixel output value data of the image sensor 32 in a far-infrared image of a subject with a uniform temperature. The horizontal axis is the ambient temperature, and the vertical axis is the pixel output value of the image sensor 32. Here, the signal processing range of the pixel output value is 14 bits wide. Also, the two lines in each diagram indicate the maximum output value and the minimum output value that occur due to the variation in the pixel output value. Fig. 3 shows the output characteristics when the gain is low, and Fig. 4 shows the output characteristics when the gain is high.

[0028] As the ambient temperature rises, the output value rises. Furthermore, as the ambient temperature rises, the variation between the two pixels also increases. In Figure 3, since the gain is low, the change is not that large and falls within the signal processing range, so there is no problem.

[0029] In Fig. 4, the gain is set higher than in Fig. 3. For this reason, the variation in output value in response to an increase in the environmental temperature becomes larger than in Fig. 3.

[0030] Graph A shown by the dotted line in Figure 4 is an example of characteristics when the gain is high. As the ambient temperature rises, the output value rises rapidly. Therefore, when the ambient temperature becomes high, the 14-bit signal processing range is exceeded and saturation occurs (area C). When this phenomenon occurs, signal processing breaks down and normal images cannot be output. To avoid this state, it is necessary to lower the offset of the sensor output as shown by graph B shown by the solid line in Figure 4. In graph B shown by the solid line in Figure 4, the output value is controlled within the signal processing range. In other words, even in a high ambient temperature, the far-infrared camera 3 can output normal images.

[0031] In this embodiment, the control device 4 detects an object and sets the gain and offset of the image sensor 32. In this way, it is possible to prevent an object in an image from appearing abnormally due to saturation. The control device 4 is able to output a normal image at all times.

[0032] Next, the configuration of the far-infrared camera 3 will be described. Fig. 5 is a block diagram showing a schematic detailed configuration of the far-infrared camera 3. The far-infrared camera 3 includes a lens unit 31, an image sensor 32, a data storage unit 33, an adjustment unit 34, a temperature sensor 35, and a transmission device 36. The data storage unit 33, the adjustment unit 34, etc. may be mounted on a control circuit including an MCU (Micro Controller Unit), or may be mounted on the control device 4. The far-infrared camera 3 may also include a shutter (not shown), etc.

[0033] The lens unit 31 forms an image of far-infrared light from a subject on the light receiving surface of the image sensor 32. The lens unit 31 has at least one lens. For example, the lens unit 31 may have multiple lenses such as a zoom lens and a focus lens.

[0034] The imaging element 32 includes a plurality of pixels. Each pixel of the imaging element 32 receives infrared light from a subject. This allows a far-infrared image of the subject to be captured. For example, the imaging element 32 includes a microbolometer for detecting far-infrared light. The imaging element 32 includes a plurality of pixels arranged in a two-dimensional array. The detection values ​​(detection signals) of each pixel form a far-infrared image of the subject.

[0035] The transmission device 36 serves as an interface for transmitting various signals and data to the control device 4. The transmission device 36 transmits far-infrared image data to the control device 4. The transmission device 36 also receives a control signal from the control device 4. For example, the transmission device 36 receives a control signal for correcting an offset or gain, which will be described later. Furthermore, the transmission device 36 may transmit lens information related to the zoom and focus of the lens unit 31. The transmission device 36 or the adjustment unit 34 may have an A / D converter that converts an analog signal into a digital signal.

[0036] The temperature sensor 35 measures the temperature in the environment in which the far-infrared camera 3 is used. Since the temperature sensor 35 is mounted inside the far-infrared camera 3, the temperature measured by the temperature sensor 35 is regarded as the internal temperature (also referred to as the environmental temperature). The internal temperature is used for adjustment in the adjustment unit 34. Specifically, the adjustment unit 34 uses different tables depending on the internal temperature.

[0037] The data storage unit 33 stores adjustment data for adjusting the detection value of each pixel of the image sensor 32. The adjustment data includes gain and offset values, and is in, for example, a table format. The gain table is a table indicating the gain value set for each pixel. The offset table is a table indicating the offset value set for each pixel.

[0038] The adjustment unit 34 adjusts the output value of the imaging element 32. Specifically, the adjustment unit 34 reads out the offset and gain for adjustment stored in the data storage unit 33, and adjusts the output value of the imaging element 32. Then, the adjustment unit 34 outputs the adjusted output value to the transmission device 36. In other words, a collection of the output values ​​adjusted by the adjustment unit 34 forms far-infrared image data.

[0039] Here, gain is a value that adjusts the temperature resolution that can be captured, and offset is a value that adjusts the temperature measurement range. If the adjusted output value is higher than the signal processing range, saturation occurs. If there are many saturated pixels in the far-infrared image data, the image will be abnormal and it will be difficult to recognize the object. For this reason, far-infrared cameras are usually set with a table of gain and offset values ​​according to the environmental temperature.

[0040] The data storage unit 33 stores, for example, a table for low temperature, a table for normal temperature, and a table for high temperature. When the internal temperature is less than a first threshold temperature, the table for low temperature is used. When the internal temperature is equal to or greater than a second threshold temperature, the table for high temperature is used. The first threshold temperature is a temperature lower than the second threshold temperature. The table for normal temperature is used in a range equal to or greater than the first threshold temperature and less than the second threshold temperature. A gain table and an offset table are set for each temperature range. Of course, the setting range of the temperature for switching the tables is not limited to three stages. For example, the tables may be switched within a two-stage setting range. The adjustment unit 34 switches and uses the tables depending on the internal temperature. That is, the adjustment unit 34 reads out a table according to the internal temperature.

[0041] In this way, the gain and offset are changed depending on the internal temperature. However, when a high-temperature object is captured in the far-infrared image, saturation may occur even if the gain and offset are adjusted according to the internal temperature. The control device 4 changes at least one of the gain and offset settings based on the result of image processing, which will be described later. This prevents the far-infrared image from becoming an abnormal image as described above.

[0042] The occurrence of abnormal images can be avoided by changing the gain and offset settings. Therefore, the control device 4 judges whether or not the gain and offset settings should be changed. Then, based on the judgment result, the gain and offset values ​​are changed. In this way, it becomes possible to prevent the generation of abnormal images. Whether or not saturation of pixel output values ​​has occurred is judged by judging within the range of one object.

[0043] The imaging system 1 is used as an in-vehicle device, for example. In midsummer, the road surface is heated by sunlight during the day, and there are scenes where the temperature does not drop even at night. Also, at night, there are scenes where the road surface is not visible to the naked eye or the visible light camera 2 due to backlighting from other vehicles' headlights, but pedestrians and the like can be imaged by using the far-infrared camera 3. Under such circumstances, the far-infrared camera 3 needs to have an appropriate temperature resolution. In other words, if the output value of the far-infrared camera 3 is set to include high-temperature objects, the output values ​​of the pixels of the objects will be close to each other, and an object may have the same color as another object with a similar temperature, making it impossible to recognize them as different objects. In other words, the temperatures of the road surface and the pedestrians may be close to each other, resulting in an image in which it is difficult to distinguish between the road surface and the pedestrian. Therefore, the control device 4 allows the output value to saturate for objects that are high temperature and do not affect driving. On the other hand, the control device 4 can adjust the image so that objects that affect driving can be distinguished by increasing the temperature resolution of the far-infrared camera.

[0044] 6 is a control block diagram showing the configuration of the control device 4. The control device 4 includes a visible light image acquisition unit 41, a far-infrared image acquisition unit 42, an object detection unit 43, a selection unit 44, an area specification unit 51, a counting unit 52, a determination unit 53, and a change unit 54. The control device 4 may further include an environmental information acquisition unit 56, a NUC unit 61, an AGC (Auto Gain Control) unit 62, and a video output unit 63.

[0045] The visible light camera 2 is connected to the visible light image acquisition unit 41. The visible light image acquisition unit 41 acquires a visible light image captured by the visible light camera 2. The visible light image acquisition unit 41 is also referred to as a first image acquisition unit. The visible light image acquired by the visible light image acquisition unit 41 is also referred to as a first image.

[0046] The far-infrared camera 3 is connected to the far-infrared image acquisition unit 42. The far-infrared image acquisition unit 42 acquires a far-infrared image captured by the far-infrared camera 3. The far-infrared image acquisition unit 42 is also referred to as a second image acquisition unit. The far-infrared image acquired by the far-infrared image acquisition unit 42 is also referred to as a second image.

[0047] The object detection unit 43 detects an object included in the visible light image. For example, the object detection unit 43 detects an object appearing in the visible light image by performing image processing on the visible light image. For example, the object detection unit 43 detects a human (pedestrian), a bicycle, a motorbike, an automobile, an animal such as a dog, a building, a street lamp, the sun, a road, and the like. Note that the object here includes a human and an animal. The object detection unit 43 can identify an object by image processing such as edge detection and pattern matching. The object detection unit 43 may detect an object using a machine learning model such as a CNN (Convolutional Neural Network).

[0048] FIG. 7 is a diagram showing an example of an object detected by the object detection unit 43. The visible light image P1 includes vehicles V1-V4, street lights L1-L2, the sun S, buildings B1-B6, and roads R1-R2. The object detection unit 43 detects each of these objects by image processing. The object detection unit 43 detects pixel addresses according to the position and size of each object. This identifies the range in the visible light image in which each object appears. Then, attribute information indicating the type and name of the object is attached to the object. The object detection unit 43 stores data indicating the detection result in a memory or the like.

[0049] The selection unit 44 selects target objects that affect driving from among the objects included in the visible light image. For example, the selection unit 44 stores priority information indicating the priority of each object for selection as a target object. The priority information is given according to the type of object. Here, the selection unit 44 does not select an object with an extremely high temperature as a target object.

[0050] The object that affects the driving is, for example, an object that needs to be checked for safe driving. Specifically, the objects are vehicles, motorcycles, pedestrians, and bicycles, and the priority order is set in the order of vehicles, motorcycles, pedestrians, and bicycles. On the other hand, an object with a temperature higher than a predetermined temperature is not selected as a target object. Examples of high-temperature objects with a temperature higher than a predetermined temperature include the sun, street lights, lava, and hot springs that are too hot for people to enter. In addition, in midsummer, roads made of asphalt and black buildings may be considered as high-temperature objects. Furthermore, when a vehicle is captured in a visible light image, parts of the vehicle may be identified and each part may be identified as an object. For example, in an image of a vehicle, parts that become high temperature, such as a muffler or a bonnet, may be identified. When a gain and offset set according to an extremely high-temperature object are used, there is a risk that an object with a normal temperature cannot be properly imaged. In other words, other vehicles and pedestrians cannot be imaged with an appropriate temperature resolution. Therefore, the above-mentioned high-temperature objects are removed from the candidates for the target object and are not selected as the target object. However, even if the object is a high-temperature object, a priority order may be assigned as necessary.

[0051] The priority order of the objects may be set in the order of expected temperature, except for objects with extremely high temperatures. For example, it is expected that a vehicle will be hotter than a motorcycle, a human, a bicycle, etc. Thus, a vehicle has a higher priority order than a motorcycle, a human, and a bicycle. For example, priority order information indicating the priority order according to the temperature of the object is stored in a memory or the like. Then, the selection unit 44 selects the target object by referring to the data of the priority order information. That is, the selection unit 44 selects the object with the highest priority order among the objects detected from the visible light image as the target object. Of course, the number of target objects is not limited to one, and two or more target objects may be selected. When selecting two or more target objects, the selection unit 44 sets how many priorities the target objects should have.

[0052] In a far-infrared image, the priority of an object that is to be captured appropriately without saturation may be set to high. For example, if the imaging system 1 is a security system, it is desired to capture an appropriate image of a person, a robot in the building, a device installed in the building, etc. captured by the camera. Therefore, the priority of an object that requires attention may be set to high in order to prevent pixel saturation. Therefore, it is possible to capture an image appropriately by adjusting the gain and offset settings to the object with the high priority.

[0053] Fig. 8 is a diagram for explaining an example of selecting a target object with reference to the priority order. A visible light image P2 shown in Fig. 8 includes a vehicle V1, people H1 to H3, the sun S, and street lights L1 to L2. That is, the object detection unit 43 detects the vehicle V1, people H1 to H3, the sun S, and street lights L1 to L2. Here, the sun S and the street lights L1 to L2 are high-temperature objects, and therefore are not target objects (no priority order is assigned). Furthermore, vehicles have a higher priority order than people. Therefore, the selection unit 44 selects the vehicle V1 as the target object.

[0054] The region specifying unit 51 specifies a judgment region including a target object in the far-infrared image. The region specifying unit 51 converts pixel addresses in the visible light image into pixel addresses in the far-infrared image based on the relative position and direction of the far-infrared camera 3 with respect to the visible light camera 2. In this way, the region specifying unit 51 can specify a judgment region corresponding to the target object. As shown in FIG. 8, the region specifying unit 51 sets a rectangular region including the vehicle V1 as a judgment region RV1. Of course, the shape of the judgment region specified by the region specifying unit 51 is not limited to a rectangular frame. For example, the judgment region may have a shape according to the outer shape of the object. In addition, the judgment region does not have to include the entire target object. In other words, the judgment region only needs to include at least a part of the target object.

[0055] The relative mounting position and mounting direction of the visible light camera 2 with respect to the far-infrared camera 3 are known. The area specifying unit 51 stores data on the relative mounting position and mounting direction. The area specifying unit 51 has a conversion formula and a conversion table for converting pixel addresses of the visible light image into pixel addresses of the image sensor 32. The conversion formula and the conversion table can be created based on design data of the mounting position and mounting direction.

[0056] Furthermore, if the target object includes a high-temperature part that is equal to or higher than a predetermined temperature, the high-temperature part may be excluded from the judgment region. For example, the bonnet or muffler of the vehicle V1 may become extremely hot. The high-temperature parts such as the bonnet or muffler may be excluded from the judgment region RV1. In other words, the region specifying unit 51 may specify, as the judgment region, the part excluding the high-temperature object from the range in which the target object is captured.

[0057] The counting unit 52 counts the number of saturated pixels in the determination region. For example, when the brightness level of each pixel in the imaging data is expressed in 8 bits ranging from zero to 255, a pixel at a coordinate having a brightness level of 255 is called a saturated pixel. Alternatively, the counting unit 52 may count a pixel at a coordinate having a pixel output value equal to or greater than a predetermined value as a saturated pixel.

[0058] The determination unit 53 determines whether the ratio of saturated pixels in the determination region is equal to or greater than a threshold value. For example, the determination unit 53 obtains the ratio of saturated pixels by dividing the number of saturated pixels in the determination region by the total number of pixels in the determination region. The determination unit 53 compares the ratio of saturated pixels in the determination region with a threshold value. When the ratio of the number of saturated pixels in the determination region corresponding to the target object is equal to or greater than a threshold value, the determination unit 53 determines that the target object is not imaged with an appropriate temperature resolution or measurement range. The determination unit 53 determines that correction of the gain or offset is necessary. The threshold value for determining whether correction is necessary may be set in advance. For example, the threshold value may be set according to the type of the target object, or a constant threshold value may be set regardless of the type of the target object. The determination unit 53 may change the threshold value according to the type of the target object or the area on the image.

[0059] When the ratio of saturated pixels is equal to or greater than a threshold, the change unit 54 changes at least one of the gain and offset in the far-infrared camera 3. For example, the change unit 54 outputs a control signal to the far-infrared camera 3 to lower the offset. This causes the offset value stored in the data storage unit 33 to be corrected. For example, the adjustment unit 34 adjusts the pixel output value using a value obtained by correcting the offset table value so that it is reduced by a certain value. In other words, the adjustment unit 34 adjusts the pixel output value using a correction table in which the offset value has been corrected.

[0060] The correction amount of the offset value may be a fixed value set in advance. Alternatively, the correction amount of the offset value may be set according to the saturation ratio, or may be set according to the type of object. Furthermore, the change unit 54 may change both the gain and the offset, or may correct only the offset. Furthermore, the change unit 54 may correct only the gain. When correcting the gain, the correction amount of the gain may be a fixed value set in advance, similar to the correction amount of the offset value, or may be set according to the saturation ratio.

[0061] Graph B in Fig. 4 is an example of preventing pixel saturation by changing the offset. The change unit 54 changes the offset so that the saturated pixel output falls within the signal processing range (14 bits in Fig. 4). Therefore, even if there is variation in the output value, it is possible to prevent the pixels in the determination region corresponding to the target object from becoming saturated, and an appropriate far-infrared image can be captured.

[0062] The NUC unit 61 performs NUC processing on the far-infrared image as described above. The AGC unit 62 performs auto gain control processing on the far-infrared image that has been subjected to the NUC processing. This makes it possible to adjust the brightness of the far-infrared image. The video output unit 63 outputs the far-infrared image that has been subjected to the AGC processing to the display 7. Thus, the display 7 can display the far-infrared image that has been appropriately captured.

[0063] In other words, an object that requires caution when driving is selected as the target object. If pixel saturation is observed in the determination area corresponding to the target object, the change unit 54 changes the gain or offset value. This allows a far-infrared image to be captured with appropriate temperature resolution and measurement range.

[0064] In addition, the selection unit 44 sets a low priority for objects that are unlikely to affect driving. Alternatively, objects that are unlikely to affect driving may not be selected as target objects. For example, the sun, street lights, lava, and other objects that are located far away are not selected as target objects. Since these objects do not affect driving, pixel saturation is allowed. In other words, even if these objects are hot, the driver does not need to pay attention to them, so the objects may be saturated in the far-infrared image. In other words, the gain and offset are appropriately set by removing objects that do not affect driving.

[0065] In this embodiment, the settings of the far-infrared camera 3 can be optimized while taking into consideration the variation in pixel output values. This makes it possible to improve the temperature resolution of the subject. It is possible to obtain a far-infrared image without saturation.

[0066] Next, the process when there are multiple objects with the highest priority will be described with reference to FIG. 9. In the image P3 shown in FIG. 9, multiple vehicles V1 to V4 are included. That is, the object detection unit 43 detects multiple vehicles V1 to V4. Here, since the vehicles V1 to V4 have the same attribute, they have the same highest priority. In this case, the selection unit 44 selects the largest vehicle V1 in the image P3 from among the vehicles V1 to V4 as the target object. Among the vehicles V1 to V4, the vehicle V1 is captured with the largest area. Therefore, the selection unit 44 selects the largest vehicle V1 as the target object. Then, the region identification unit 51 identifies the determination region RV1 corresponding to the vehicle V1. Note that, although the largest vehicle is set as the target object here, an object having a size equal to or larger than a predetermined size may be set as the target object.

[0067] In this way, the determination unit 53 can perform the determination with high accuracy. For example, since the target object is captured over a larger range, the region specification unit 51 can specify a wider determination region RV1. Therefore, the number of pixels used by the determination unit 53 for the determination can be increased, and the determination can be performed with high accuracy.

[0068] Further, a priority order may be set according to the color or material of the object. For example, even if the vehicles are the same, a black vehicle is more likely to become hotter due to light absorption than a white vehicle. Therefore, the priority order of the black vehicle is set high and the priority order of the white vehicle is set low. In this way, the selection unit 44 can select an object with a high temperature as the target object. Therefore, by giving priority to an object that is easily saturated, it is possible to prevent an abnormal image from being generated.

[0069] The control device 4 may include an environmental information acquisition unit 56 that acquires environmental information. The environmental information is information including the position of the sun or weather conditions. The environmental information may include weather, wind direction, wind speed, temperature, etc. The environmental information may include information regarding the position and direction of the sun relative to the vehicle. The environmental information may include information regarding the season. Furthermore, if the imaging system 1 is an on-board device mounted on a vehicle equipped with a GPS or the like, the environmental information acquisition unit 56 may also acquire weather information at the position of the vehicle as environmental information.

[0070] The region identifying unit 51 can identify the judgment region based on the environmental information. An example in which the region identifying unit 51 identifies the judgment region based on the environmental information will be described with reference to FIG. 10. In FIG. 10, a vehicle V1 is selected as a target object. In the far-infrared image P4, it is assumed that the right half of the vehicle V1 is exposed to direct sunlight. Therefore, it is assumed that the right half of the vehicle V1 absorbs more sunlight than the left half and is therefore at a higher temperature. In this case, the region identifying unit 51 identifies the right half of the vehicle V1 as the judgment region RV1.

[0071] Alternatively, the region identifying unit 51 may identify the judgment region based on information such as wind direction. For example, the environmental information acquiring unit 56 is equipped with an anemometer that measures wind direction and wind speed. Alternatively, the environmental information acquiring unit 56 may acquire information indicating wind direction and wind speed from an anemometer mounted on the vehicle. The region identifying unit 51 identifies the judgment region excluding the portion receiving the cold wind. For example, if the vehicle V1 is receiving cold wind from the left side, the region identifying unit 51 identifies the right half of the vehicle V1, which is likely to be saturated, as the judgment region RV1.

[0072] Furthermore, the selection unit 44 may select the target object based on the environmental information. An example of the selection unit 44 selecting the target object based on the environmental information will be described with reference to FIG. 11. In FIG. 11, the vehicle V4 is selected as the target object. In the far-infrared image P4, the road R1 on the left side is assumed to be in the shade due to buildings B2, B3, etc. The road R2 on the right side is in the sun. On the road R2 in the sun, the temperature of the object rises due to the sunlight. That is, the vehicle V4 traveling on the road R2 has a high temperature due to the sunlight. Conversely, the vehicles V1 to V3 traveling on the road R1 are not irradiated with the sunlight, and therefore are relatively low in temperature. Therefore, the selection unit 44 selects the vehicle V4 as the target object. That is, the vehicle V4 with a high temperature becomes the target object, not the vehicle V1 that occupies the largest area on the image. The region identification unit 51 identifies a determination region RV4 including the vehicle V4.

[0073] When information on whether the object is in the sun or in the shade is known from the environmental information, the priority order may be changed depending on whether the object is in the sun or in the shade. In other words, the selection unit 44 gives a high priority to an object in the sun and a low priority to an object in the shade. This allows the selection unit 44 to appropriately select a target object. Alternatively, when the wind direction, wind speed, etc. are known from the environmental information, the selection unit 44 may change the priority order in accordance with these. For example, the priority order may be low for an object that is being cooled by cold wind.

[0074] The image adjustment method will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the image adjustment method. The visible light image acquisition unit 41 acquires a visible light image from the visible light camera 2 (S101). The far-infrared image acquisition unit 42 acquires a far-infrared image from the far-infrared camera 3 (S102). The object detection unit 43 detects an object included in the visible light image (S103). The object detection unit 43 recognizes an object captured in the visible light image by a known image recognition process.

[0075] The selection unit 44 selects a target object from among the objects included in the visible light image (S104). The selection unit 44 selects the target object by referring to priority information indicating the priority order set for each object. Here, one object is selected as the target object from among the objects included in the visible light image.

[0076] The region specifying unit 51 specifies a determination region corresponding to the target object in the far-infrared image (S105). That is, the pixel address of the location where the target object is captured in the visible light image is converted into a pixel address in the far-infrared image. The installation angle and the angle of view of the visible light camera 2 and the far-infrared camera 3 are known. Therefore, the region specifying unit 51 can specify a determination region where the target object is captured in the far-infrared image by using a pixel address conversion formula or conversion table.

[0077] The counting unit 52 counts the number of saturated pixels in the judgment region (S106). The counting unit 52 finds the number of saturated pixels in the judgment region. The judgment unit 53 judges whether the ratio of saturated pixels in the judgment region is equal to or greater than a threshold (S107). That is, the judgment unit 53 finds the ratio of saturated pixels in the judgment region by dividing the number of saturated pixels by the total number of pixels included in the judgment region. The judgment unit 53 compares the ratio of saturated pixels with the threshold.

[0078] If the ratio of saturated pixels is not equal to or greater than the threshold (NO in S107), the process ends. That is, the gain or offset in the far-infrared camera 3 is not changed. If the ratio of saturated pixels is equal to or greater than the threshold (YES in S107), the change unit 54 changes the gain or offset of the far-infrared camera 3 (S108). That is, the change unit 54 outputs a control signal to the far-infrared camera 3 to change the value of the gain or offset. The amount of correction of the gain or offset value may be a predetermined value set in advance, or may be set according to the ratio of saturated pixels, etc. Then, the process ends.

[0079] The imaging system 1 repeats the above process. For example, the imaging system 1 performs the above process for each one or more frames. This allows the control device 4 to adjust the image so that it has an appropriate temperature resolution.

[0080] Furthermore, when a plurality of objects with the same priority are included in the visible light image, the selection unit 44 may select the target object in accordance with the area of ​​the object on the image in S104. That is, the selection unit 44 selects the object with the largest area on the image from among a plurality of objects with the same priority as the target object. As a result, the vehicle V1 is selected as the target object, as shown in FIG. 9.

[0081] Furthermore, in S104, the selection unit 44 may select the target object based on the environmental information acquired by the environmental information acquisition unit 56. For example, when the environmental information includes information indicating shade or sunshine, the selection unit 44 increases the priority of the object in the sun. As a result, as shown in Fig. 11, the vehicle V4 is selected as the target object. Alternatively, when the environmental information includes information indicating wind speed or wind direction, the selection unit 44 decreases the priority of the object being cooled.

[0082] In addition, in S105, the region identification unit 51 may identify the determination region based on the environmental information acquired by the environmental information acquisition unit 56. For example, when the environmental information includes information indicating shade or sunshine, a part of the vehicle V1 is selected as the determination region RV1 as shown in FIG.

[0083] The control device 4 functions as an image adjustment device that adjusts the far-infrared image of the far-infrared camera 3. The control device 4 is not limited to being a single physical device, and may be distributed and arranged in multiple devices. For example, some of the functions of the control device 4 may be installed in the visible light camera 2 or the far-infrared camera 3. The visible light camera 2 may be equipped with an object detection function. The far-infrared camera 3 may be equipped with a processing function such as a determination unit.

[0084] The programs of the control device 4 and the imaging system 1 described above include a set of instructions (or software code) for making the computer perform one or more functions described in the embodiments when the programs are loaded into the computer. The programs may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, and magnetic storage device. The programs may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium may include electrical, optical, acoustic, or other forms of propagating signals.

[0085] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0086] 1. Imaging system 2. Visible light camera 3. Far-infrared camera 4. Control device 7. Display 21 Lens unit 22 Image sensor 31 Lens unit 32 Image sensor 33 Data storage unit 34 Adjustment part 35 Temperature Sensor 36 Transmission Devices 41 Visible light image acquisition unit 42 Far-infrared image acquisition unit 43 Object detection unit 44 Selection section 51 Area identification part 52 Counting Department 53 Judgment section 54 Changes 56 Environmental Information Acquisition Department 61 NUC Department 62 AGC Department 63 Video output section

Claims

1. a first image acquisition unit that acquires a first image captured by the first imaging element; a second image acquisition unit that acquires a second image captured by a second imaging element that detects far-infrared light; an object detection unit that detects an object included in the first image; a selection unit that selects a target object from among the objects included in the first image by referring to priority order information assigned to the objects; an area specifying unit that specifies a determination area in the second image that includes the target object; a counting unit that counts saturated pixels in the determination region; a determination unit that determines whether or not the ratio of saturated pixels in the determination region is equal to or greater than a threshold; and a change unit that changes a gain or offset value of the second imaging element when the gain or offset value is equal to or greater than the threshold value.

2. The selection unit refers to the priority information set according to the temperature of the object or the type of the object. The image adjustment device according to claim 1 .

3. The image adjustment device according to claim 1 , wherein when the first image contains a plurality of objects with the highest priority, the largest object in the first image is selected as the target object from among the plurality of objects with the highest priority.

4. An environmental information acquisition unit that acquires environmental information including the position of the sun or weather conditions, The image adjustment device according to claim 1 , wherein the region specifying unit specifies the determination region in accordance with the environmental information.

5. An environmental information acquisition unit that acquires environmental information including the position of the sun or weather conditions, The image adjustment device according to claim 1 , wherein the selection unit changes the priority information in accordance with the environmental information and selects the target object.

6. The image adjustment device is mounted on a moving body such as a vehicle, the selection unit selects, as the target object, an object that affects travel of the moving object from among objects included in the first image; The image adjustment device according to claim 1 .

7. acquiring a first image captured by a first imaging element; acquiring a second image captured by a second imaging element that detects far-infrared light; detecting an object in the first image; selecting a target object from among the objects included in the first image by referring to priority information assigned to the objects; identifying a determination region in the second image that includes the target object; Counting saturated pixels in the determination region; determining whether the ratio of saturated pixels in the determination region is equal to or greater than a threshold; and changing a gain or offset value in the second imaging element when the value is equal to or greater than the threshold value.

Images