Image processing apparatus, image processing method, and program

Abstract

To provide an image processing apparatus or the like, configured to improve object detection capability while suppressing increase of required processes.SOLUTION: In an image processing apparatus 11, an image data acquisition unit 160 acquires thermal image data from an infrared camera which captures a thermal image of surroundings of a vehicle. An image processing unit 170 sets a plurality of first local regions and second local regions for the thermal image. The image processing apparatus calculates a frequency distribution on pixel values of the first and second local regions, and redistributes excess data that exceeds a preset threshold into the frequency distribution. The image processing apparatus generates first and second local tone curves by accumulating redistribution data. The image processing apparatus synthesizes the first and second local tone curves to generate first and second blend tone curves, and generates far and neighborhood conversion image data by applying the first and second blend tone curves to the local regions.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Many object detection systems using an infrared camera that detects the heat of an object have been developed. The object detection system is expected to accurately detect objects at various positions and distances from a thermal image captured by the infrared camera. Therefore, as a technique for recognizing an object captured by the infrared camera, various image processes such as sharpening, tone correction, and noise removal are known to be performed. In addition, techniques for suppressing a decrease in the detection accuracy of an object have been developed together.

[0003] For example, a technique for generating a stray light map in advance regarding noise due to stray light generated in an infrared camera and performing calibration of a thermal image using the generated stray light map has been published (Patent Document 1).

[0004] Also, for example, a technique for extracting signals in each of a high frequency band, an intermediate frequency band, and a low frequency band from a thermal image generated by an infrared sensor, individually averaging the signals, and then integrating them has been published (Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when performing image processing for detecting an object that is relatively far away from an infrared camera, it results in an effect where a part of an object that is relatively close stands out, leading to a decrease in detection performance. On the other hand, when performing image processing optimized for an object that is relatively close, the image of an object that is relatively far away becomes blurred, the features of the object fade, and the detection performance decreases. Also, if images suitable for detecting a distant object and an image suitable for detecting a nearby object are generated respectively, the detection performance can be improved, but the load of image processing doubles, and when performing image processing for detecting an object at an intermediate distance, the required amount of processing further increases.

[0007] The present invention has been made to solve such problems, and provides an image processing apparatus and the like that improve the object detection ability while suppressing an increase in the amount of processing.

Means for Solving the Problems

[0008] The image processing apparatus according to the present invention includes an image data acquisition unit, a determination unit, an image processing unit, and an image data output unit. The image data acquisition unit acquires thermal image data from an infrared camera that captures a thermal image of the periphery of the vehicle. The image processing unit generates, so as to be outputtable, remote conversion image data and near-field conversion image data from the thermal image data. The image processing unit sets local regions including a plurality of first local regions and a plurality of second local regions that include the first local regions and are larger than the first local regions, with respect to the thermal image. The image processing apparatus calculates a frequency distribution regarding pixel values of pixels constituting the first local regions and the second local regions. The image processing apparatus generates redistribution data in which excess data exceeding a preset threshold value in the frequency distribution is redistributed to the frequency distribution. The image processing apparatus cumulatively adds the redistribution data from a low pixel value toward a high pixel value to generate a first local tone curve regarding the first local regions and a second local tone curve regarding the second local regions. The image processing apparatus synthesizes the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve. The image processing apparatus generates remote conversion image data by applying the first blend tone curve to the local regions, and generates near-field conversion image data by applying the second blend tone curve to the local regions.

[0009] The image processing method according to the present invention is such that a computer executes the following steps. The computer acquires thermal image data from an infrared camera that captures a thermal image of the surroundings of a vehicle. The computer sets local regions for the thermal image, including a plurality of first local regions and a plurality of second local regions that include the first local regions and are larger than the first local regions. The computer calculates a frequency distribution regarding pixel values of pixels constituting the first local regions and the second local regions. The computer generates redistribution data in which excess data exceeding a preset threshold value in the frequency distribution is redistributed to the frequency distribution. The computer cumulatively adds the redistribution data from a low pixel value to a high pixel value to generate a first local tone curve for the first local regions and a second local tone curve for the second local regions. The computer synthesizes the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve. The computer generates, such that remote conversion image data can be output, by applying the first blend tone curve to the local regions, and generates, such that near-neighbor conversion image data can be output, by applying the second blend tone curve to the local regions.

[0010] The program according to the present invention causes a computer to execute the following image processing method. The image processing method includes the computer executing the following steps. The computer acquires thermal image data from an infrared camera that captures a thermal image of the surroundings of a vehicle. The computer sets local regions for the thermal image, including a plurality of first local regions and a plurality of second local regions that include the first local regions and are larger than the first local regions. The computer calculates a frequency distribution regarding the pixel values of the pixels that make up the first local regions and the second local regions. The computer generates redistribution data in which excess data exceeding a preset threshold value in the frequency distribution is redistributed to the frequency distribution. The computer cumulatively adds the redistribution data from a low pixel value to a high pixel value to generate a first local tone curve for the first local region and a second local tone curve for the second local region. The computer synthesizes the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve. The computer generates transform image data for a distance view in a manner that enables output by applying the first blend tone curve to the local regions, and generates transform image data for a near view in a manner that enables output by applying the second blend tone curve to the local regions.

Advantages of the Invention

[0011] According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program that improve the object detection ability while suppressing an increase in the processing amount.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0013] Hereinafter, the present invention will be described through embodiments of the invention. However, the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are essential as means for solving the problems. For the sake of clarity of explanation, the following description and drawings have been appropriately omitted and simplified. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0014] <Embodiment 1> Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram of a vehicle equipped with an image processing apparatus according to Embodiment 1. The image processing apparatus according to the present embodiment receives a thermal image from an infrared camera provided in the vehicle, performs predetermined processing, and then displays the received thermal image on a display.

[0015] At this time, the image processing apparatus 11 generates conversion image data from the thermal image data so as to be outputtable to the display. The conversion image data includes far-field conversion image data and near-field conversion image data. The far-field conversion image data is generated for the purpose of improving the recognition rate of an image of an object existing relatively far among the objects existing in front of the vehicle 90. The near-field conversion image data is generated for the purpose of improving the recognition rate of an image of an object existing relatively near among the objects existing in front of the vehicle 90.

[0016] The image processing apparatus according to the present embodiment also has a function of detecting a predetermined object from the thermal image received from the infrared camera. For example, when the image processing apparatus detects an animal or a person existing in front of the vehicle, it warns the driver. Therefore, the image processing apparatus processes the thermal image for the purpose of detecting a predetermined object. The vehicle 90 shown in FIG. 1 includes an infrared camera 10, an image processing apparatus 11, a display 12, and an ECU 13.

[0017] The infrared camera 10 is provided on the vehicle 90 so as to image the periphery of the vehicle 90. The infrared camera 10 is fixed, for example, in front of the vehicle 90 and images the front of the vehicle 90. The infrared camera 10 is communicably connected to the image processing device 11, receives a predetermined instruction signal from the image processing device 11, and operates according to the received instruction signal. Further, the infrared camera 10 supplies image data (also referred to as thermal image data) related to the image (also referred to as a thermal image) captured by the infrared camera 10 to the image processing device 11.

[0018] The image processing device 11 is fixed at an arbitrary location on the vehicle 90 and is communicably connected to each of the infrared camera 10 and the display 12. The image processing device 11 acquires the image data generated by the infrared camera 10 and outputs and displays the acquired image data on the display 12. Further, the image processing device 11 receives predetermined information (vehicle information) from the ECU 13 and operates according to the received vehicle information. For example, when the image processing device 11 receives vehicle information indicating that the headlight is lit from the ECU 13, it starts imaging by the infrared camera 10 in conjunction with this information. That is, the image processing device 11 according to the present embodiment opens the shutter 103 of the infrared camera 10 and acquires image data from the infrared sensor 104 in a dark place where the vehicle 90 turns on the headlight, such as at night or in a tunnel.

[0019] The display 12 is a display device including, for example, a liquid crystal panel or an organic electroluminescence panel, and is provided at a position visible to the driver in the vehicle 90. The display 12 displays the image captured by the infrared camera 10 via the image processing device 11.

[0020] The ECU13 (Electronic Control Unit) has data related to a predetermined function of the vehicle and controls this function. For example, the ECU13 controls the lighting and extinguishing of the headlights of the vehicle 90. In this case, the ECU13 can supply vehicle information indicating that the headlight is lit to the image processing device 11. Note that instead of the above-mentioned information, the ECU13 may supply, for example, a signal from an illuminance sensor of the vehicle 90 to the image processing device 11.

[0021] Also, the ECU13 may control the operation of the wiper of the vehicle 90. In this case, the ECU13 may provide vehicle information indicating the operation of the wiper to the image processing device 11 as information associated with the contrast of the thermal image captured by the infrared camera 10. The ECU13 may provide, as information associated with the contrast of the thermal image, signals generated by a humidity sensor, a raindrop sensor, etc. of the vehicle 90, or information about rain or fog detected by these sensors as vehicle information to the image processing device 11. In addition to the above-mentioned information, the ECU13 may provide predetermined information acquired from outside the vehicle 90 via a communication device of the vehicle 90 to the image processing device 11 as vehicle information.

[0022] Also, the ECU13 may receive predetermined information from the image processing device 11. For example, when the ECU13 receives a warning signal from the image processing device 11, it may issue a voice or display a warning to the driver using a speaker or a display device of the vehicle 90 according to the received warning signal.

[0023] With the above configuration, the image processing device 11 performs predetermined processing on the image captured by the infrared camera 10 and displays such an image on the display 12 in a manner visible to the driver. Thereby, the image processing device 11 can enable the driver to recognize the objects around the vehicle 90.

[0024] Next, with reference to FIG. 2, the configuration of the infrared camera 10 will be described. FIG. 2 is a configuration diagram of the infrared camera. The infrared camera 10 shown in FIG. 2 mainly includes a housing 101, an objective lens 102, a shutter 103, an infrared sensor 104, and a camera control circuit 105.

[0025] The housing 101 houses each component of the infrared camera 10 and is fixed to the vehicle 90. The objective lens 102 receives infrared rays incident from the range imaged by the infrared camera 10 and projects them onto the infrared sensor 104. The shutter 103 includes a light-shielding plate material and is interposed between the objective lens 102 and the infrared sensor 104 in an openable and closable manner.

[0026] When the shutter 103 is in the closed state, it blocks the light incident from the objective lens 102 to the infrared sensor 104. On the other hand, when the shutter 103 is in the open state, it does not block the light incident from the objective lens 102 to the infrared sensor 104. Also, the shutter 103 is a black body as viewed from the infrared sensor 104, and the calibration of the infrared sensor 104 is performed with the shutter 103 in the closed state.

[0027] The infrared sensor 104 is composed of thermosensitive elements arranged in an array, receives infrared light incident through the objective lens 102, and generates image data based on the change in the resistance value of each thermosensitive element. The infrared sensor 104 is communicably connected to the camera control circuit 105 and operates in response to a predetermined control signal received from the camera control circuit 105. When the infrared sensor 104 generates image data, it supplies the generated image data to the camera control circuit 105.

[0028] Note that when the infrared sensor 104 receives sunlight stronger than a predetermined intensity for a predetermined period or more, the output of the elements in the range receiving the sunlight saturates. Also, since the temperature of the elements in the range receiving the sunlight rises above a specified value, abnormal states such as irreversible deformation of the elements and changes in characteristics occur. Therefore, the infrared camera 10 has a shutter 103 to protect the infrared sensor 104 from direct sunlight.

[0029] In addition, the dynamic range of the infrared sensor 104 is set so that the resolution for detecting objects such as pedestrians is relatively high. Therefore, the infrared sensor 104 is set so that the detection signal saturates even within a temperature range where no abnormality occurs.

[0030] The camera control circuit 105 is a control circuit including an MCU (Micro Controller Unit), and controls the shutter 103 and the infrared sensor 104. The camera control circuit 105 is communicably connected to the image processing device 11, receives a control signal from the image processing device 11, and controls each component of the infrared camera 10 according to the received control signal. When the infrared camera 10 is not performing imaging, the camera control circuit 105 controls the shutter 103 to remain closed. When the infrared camera 10 is performing imaging, the camera control circuit 105 controls the shutter 103 to be open. At this time, the camera control circuit 105 supplies the image data generated by the infrared sensor 104 to the image processing device 11.

[0031] In addition, the camera control circuit 105 temporarily closes the shutter 103 under predetermined conditions. The predetermined conditions are, for example, when the shutter 103 is temporarily closed for the purpose of protecting the infrared sensor 104 or when calibration is performed. In this case, for example, the camera control circuit 105 receives an instruction to temporarily close the shutter 103 from the image processing device 11.

[0032] Next, with reference to FIG. 3, the image processing device 11 will be described. FIG. 3 is a block diagram of the image processing device according to Embodiment 1. The main components of the image processing device 11 include a communication IF 120, a ROM 130, a RAM 140, a system control circuit 150, an image data acquisition unit 160, an image processing unit 170, an image recognition unit 180, and an image data output unit 190. These components are appropriately communicably connected via a bus 110.

[0033] The communication IF 120 is an interface of a communication line for controlling the infrared camera 10. The communication IF 120 supplies a control signal for the image processing apparatus 11 to control the infrared camera 10 to the infrared camera 10.

[0034] Also, the communication IF 120 includes a vehicle information acquisition unit 121. The vehicle information acquisition unit 121 is an interface for acquiring vehicle information from the ECU 13. When the vehicle information acquisition unit 121 included in the communication IF 120 acquires vehicle information from the ECU 13, the acquired vehicle information is supplied to the configuration of the image processing apparatus 11 as appropriate. The vehicle information acquisition unit 121 may acquire information related to the humidity around the vehicle 90 as vehicle information as information associated with the contrast of the thermal image captured by the infrared camera 10. More specifically, the vehicle information may be, for example, information about rain detected by a rain drop sensor included in the vehicle 90 or information about fog detected by a humidity sensor. Also, the vehicle information may be information related to the operation of the wiper. Also, the vehicle information may be a signal indicating the lighting of the fog lamp.

[0035] Also, the vehicle information acquired by the communication IF 120 may include vehicle operation information. The vehicle operation information is information related to the operation of the vehicle 90 and is, for example, the traveling speed or angular velocity of the vehicle 90. When the vehicle information acquisition unit 121 acquires vehicle information as described above, the acquired vehicle information is supplied to the system control circuit 150.

[0036] Also, the communication IF 120 may be an interface for outputting a predetermined signal to the ECU 13. For example, the communication IF 120 may supply a signal for giving a predetermined warning to the driver to the ECU 13.

[0037] The ROM 130 (ROM (Read Only Memory)) is a non-volatile memory that stores preset information or data. The ROM 130 stores, for example, a program for the image processing apparatus 11 to realize the functions according to the present embodiment in advance.

[0038] The RAM 140 (Random Access Memory) is a volatile memory that has a storage area where the image processing apparatus 11 can temporarily expand data. The RAM 140 may be, for example, a DRAM (Dynamic Random Access Memory), or may include registers associated with the system control circuit 150 or the like. The RAM 140 includes an area for expanding and executing the program stored in the ROM 130. Also, the RAM 140 can be used, for example, when processing image data supplied from the infrared camera 10.

[0039] The system control circuit 150 includes arithmetic units such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The system control circuit 150 executes a program for the image processing apparatus 11 to exhibit the functions according to the present embodiment. The program may be configured as software stored in the ROM 130, for example. Also, the program may be configured by a combination of software and hardware. The system control circuit 150 receives a predetermined signal from each component of the image processing apparatus 11, and outputs a predetermined instruction or the like to each component according to the received signal. The system control circuit 150 includes a determination unit 151 and an instruction unit 152 as functional blocks.

[0040] The determination unit 151 makes a predetermined determination on the thermal image related to the image data acquired from the image processing apparatus 11. For example, when the thermal image includes an object such as a pedestrian, the determination unit 151 determines whether to issue a warning. Also, for example, the determination unit 151 may make a determination regarding the opening and closing of the shutter 103 of the infrared camera 10.

[0041] Also, the determination unit 151 receives image analysis data from the image processing unit 170. The determination unit 151 acquires information associated with the contrast of the thermal image included in the received image analysis data, and determines whether to perform processing for suppressing a decrease in the contrast of the thermal image from the acquired information.

[0042] The information associated with the contrast of the thermal image may be, for example, the difference in pixel values (luminance values) of the thermal image calculated by the image processing unit 170. In the following description, the difference in pixel values of the thermal image is referred to as the contrast value. The contrast value of the thermal image may be calculated from the luminance information of the entire thermal image, or may be a value for a partial region extracted from the entire thermal image.

[0043] The process for suppressing contrast reduction is, for example, the sharpening of the thermal image. By suppressing contrast reduction, the image processing apparatus 11 can suppress the blurred tone of the thermal image and improve the detection accuracy of an object. For example, when the vehicle 90 is traveling in a situation where rain or fog is occurring, the thermal image captured by the infrared camera 10 has a blurred tone for the entire image. In such a case, the image processing apparatus 11 generates a converted image obtained by performing a sharpening process on the thermal image data.

[0044] Note that the determination unit 151 may use vehicle information as information associated with the contrast of the thermal image. For example, the determination unit 151 may acquire information regarding the operation of the wiper of the vehicle 90 via the vehicle information acquisition unit 121. Also, for example, the determination unit 151 may acquire information regarding the humidity around the vehicle 90 from the vehicle information acquisition unit 121 and acquire information indicating the occurrence of rain or fog. Further, the determination unit 151 may acquire information regarding the moving speed of the vehicle 90 or the angular velocity of the vehicle 90 from the vehicle information acquisition unit 121. The determination unit 151 supplies the result of the determination made from these pieces of information to the instruction unit 152.

[0045] The instruction unit 152 gives various instructions to each component of the image processing apparatus 11. The instruction unit 152 receives information regarding the result of the determination from the determination unit 151, and gives an instruction to each component according to the received information. For example, when the instruction unit 152 receives a signal for determining to issue a warning from the determination unit 151, it may output a signal indicating to issue a warning to the ECU 13 via the communication IF 120. Further, the instruction unit 152 may output an instruction regarding the opening and closing of the shutter 103 to the infrared camera 10 via the communication IF 120. Further, when the determination unit 151 determines to perform a process for suppressing a contrast decrease, the instruction unit 152 supplies a signal for giving an instruction to generate predetermined image data to the image processing unit 170.

[0046] The image data acquisition unit 160 is an interface for acquiring thermal image data (input thermal image data), which is data related to a thermal image, from the infrared camera 10. The image data acquisition unit 160 periodically acquires image data from the infrared camera 10, for example. For example, the image data acquisition unit 160 receives an image of one frame every 1 / 15 second. When the image data acquisition unit 160 receives the image data, it supplies the received image data to the image processing unit 170.

[0047] The image processing unit 170 is an image processing circuit including, for example, a GPU (Graphics Processing Unit). The image processing unit 170 may be a processing circuit including an FPGA (field-programmable gate array) or a DSP (Digital Signal Processor). The image processing unit 170 cooperates with the RAM 140 to perform a predetermined process on the thermal image data and generate converted image data so as to be outputtable. The converted image data includes converted image data for a distance and converted image data for a vicinity. Note that details of the image processing unit 170 will be described later.

[0048] The image recognition unit 180 receives image data from the image processing unit 170, and detects an object image of a predetermined object such as a vehicle or a person from the received image data. More specifically, for example, the image recognition unit 180 has a recognition dictionary for detecting a predetermined object image, and refers to the data of the recognition dictionary to detect the object image. When the image recognition unit 180 detects a predetermined object image, it generates information regarding the position and size of the detected object image, and supplies the generated information to the system control circuit 150.

[0049] The image data output unit 190 is an interface for outputting the image data (converted image data) processed by the image processing unit 170 to the display 12. The image data output by the image data output unit 190 is output according to a data format corresponding to the specifications of the display 12. This data format is, for example, HDMI (High-Definition Multimedia Interface) (registered trademark), DVI (Digital Visual Interface), or the like.

[0050] Note that the image data output unit 190 outputs the converted image data generated by the image processing unit 170. The converted image data output by the image data output unit 190 may include long-distance converted image data and near-field converted image data. In this case, the converted image data to be output may be the long-distance converted image data, the near-field converted image data, or both. The image data output unit 190 may output image data obtained by further performing gamma correction or the like on the above-described converted image data.

[0051] Next, with reference to FIG. 4, the image processing unit 170 will be described. FIG. 4 is a block diagram of the image processing unit 170 according to the first embodiment. The main components of the image processing unit 170 include a defective pixel correction unit 171, a NUC unit 172, an image conversion processing unit 173, and an image analysis unit 175.

[0052] The defective pixel correction unit 171 pre-stores the defective pixels of the infrared sensor 104, and performs a process (interpolation process) of interpolating the pixel values of the stored defective pixels from the pixel values of the surrounding pixels. The defective pixel correction unit 171 receives the input thermal image data from the infrared camera 10 via the image data acquisition unit 160, and performs the above-described interpolation process on the received image data. The defective pixel correction unit 171 supplies the image data on which the interpolation process has been performed to the NUC unit 172.

[0053] The NUC unit 172 performs NUC (Non-Uniformity Correction), which is a calibration process for suppressing variations in pixel values output for the input light. The NUC unit 172 has pre-set gain and offset values corresponding to the characteristics of each pixel of the infrared sensor 104. The NUC unit 172 calibrates the respective pixel values of the image data according to this pre-set setting for the image data received from the defective pixel correction unit 171.

[0054] The image conversion processing unit 173 receives the calibrated image data from the NUC unit 172, performs a predetermined process on the received image data, and generates converted image data. The converted image data is obtained by improving the contrast or resolution of the image data received from the NUC unit 172. By improving the contrast or resolution of the image data, the image processing apparatus 11 can suppress a decrease in the recognition rate of an object. When the image conversion processing unit 173 generates the converted image data, the generated converted image data is supplied to the image recognition unit 180 and the image data output unit 190.

[0055] The image analysis unit 175 analyzes the image data received from the NUC unit 172, and outputs image analysis data that is the analysis result. More specifically, for example, the image analysis unit 175 calculates the contrast of the image related to the received image data, and supplies the calculated result to the system control circuit 150.

[0056] Note that when the image processing unit 170 transfers image data between the above-described components or performs image processing in each component, it may exchange image data with the RAM 140. Alternatively, the image processing unit 170 may implement functions along the above-described processing flow on the image data stored in the RAM 140.

[0057] Next, with reference to FIG. 5, the process of generating conversion image data executed by the image processing apparatus 11 will be described. FIG. 5 is a second flowchart of the image processing method according to the first embodiment. The flowchart shown in FIG. 5 is started, for example, when imaging using the infrared camera 10 by the image processing apparatus 11 is started. Further, this process is repeatedly executed while imaging using the infrared camera 10 is being performed.

[0058] First, the image data acquisition unit 160 of the image processing apparatus 11 acquires thermal image data from the infrared camera 10 (step S10). When the image data acquisition unit 160 acquires the thermal image data, it supplies the acquired thermal image data to the image processing unit 170.

[0059] Next, when the image processing unit 170 receives the thermal image data, it sets local regions including a plurality of first local regions and a plurality of second local regions that include the first local regions and are larger than the first local regions, for the thermal image (step S11).

[0060] Next, the image processing unit 170 calculates a frequency distribution (histogram) regarding the pixel values of the pixels constituting the first local regions and the second local regions (step S12).

[0061] Next, the image processing unit 170 sets the distribution ratio for each pixel value from the pixel value data of the thermal image (step S13). Specifically, for example, the image processing unit 170 calculates the central value of the thermal image data and sets the distribution ratio from the calculated central value. The central value is, for example, the median in the frequency distribution of the thermal image data. Also, the central value may be the average value calculated from the minimum pixel value and the maximum pixel value. Further, the image processing unit 170 sets a high distribution ratio for the pixel value range for which the contrast is desired to be improved and a low distribution ratio for the pixel value range for which the contrast is desired to be suppressed. Therefore, the distribution ratio is set so that the resolution with respect to the temperature of a human, an animal, etc., which is the detection target in the image processing apparatus 11, is improved, and the resolution with respect to a low temperature like that of air or a high temperature like that of an automobile engine or a muffler is suppressed.

[0062] Next, the image processing unit 170 generates redistribution data in which the excess data exceeding a preset frequency threshold in the frequency distribution is redistributed to the frequency distribution (step S14). Note that the image processing unit 170 independently sets a first threshold value applied to the first local region and a second threshold value applied to the second local region. That is, the first threshold value and the second threshold value may be the same or different.

[0063] Next, the image processing unit 170 generates a local tone curve from the redistribution data (step S15). Specifically, the image processing unit 170 generates a local tone curve by cumulatively adding the frequencies of the redistribution data in order from the lower pixel value to the higher pixel value.

[0064] Next, the image processing unit 170 generates a first blend tone curve and a second blend tone curve (step S16). Specifically, the image processing unit 170 synthesizes two types of blend tone curves for the first local tone curve corresponding to the first local region and the second local tone curve corresponding to the second local region using two different weighting coefficients. Thereby, the image processing unit 170 generates a first blend tone curve used to generate the far-field conversion image data and a second blend tone curve used to generate the near-field conversion image data. Note that the image processing unit 170 normalizes the generated blend tone curve by the width between the maximum value and the minimum value of the pixel values of the thermal image data.

[0065] In the above processing, the image processing unit 170 generates the conversion image data according to, for example, the following expressions (1) and (2).

Equation

Equation

[0066] Also, BT 12(i) is the value of the second blend tone curve. W12 is the weighting coefficient for the first local tone curve when generating the second blend tone curve. W22 is the weighting coefficient for the second local tone curve when generating the second blend tone curve. Also, F12 is the offset value when generating the first blend tone curve.

[0067] In the above formula, the weighting coefficients W11, W12, W21, and W22 are not limited to constant values respectively. The weighting coefficients W11, W12, W21, and W22 can be set according to predetermined conditions. The offset values F11 and F12 may also be set according to predetermined conditions.

[0068] Note that the far - distance conversion image data is set so that the contrast for more distant objects is higher compared to the near - distance conversion image data. That is, the weighting coefficient W11 in the above formula (1) is set to a value larger than the weighting coefficient W12 in formula (2). Note that the weighting coefficient may be set to a negative value. For example, by setting the weighting of the tone curve of the second local area larger than the first local area to a negative value, the image processing apparatus 11 can reduce the blurred tone that occurs over a relatively wide range.

[0069] Next, the image processing unit 170 applies the first and second blend tone curves to the pixels corresponding to each local area. Thereby, the image processing unit 170 generates the far - distance conversion image data and the near - distance conversion image data (step S17) and ends a series of processes.

[0070] Note that the image processing unit 170 generates the above - mentioned conversion image data so that it can be output. For example, the image processing unit 170 supplies the generated conversion image data to the image recognition unit 180. Also, the image processing unit 170 may supply the conversion image data to the image data output unit 190.

[0071] The image processing method executed by the image processing apparatus 11 has been described above. By executing the above-described information, the image processing apparatus 11 can improve the object detection ability while suppressing an increase in the processing amount.

[0072] Next, with reference to FIG. 6, a process of determining a weighting coefficient when generating a blend tone curve will be described. FIG. 6 is a first flowchart showing a weighting setting method in the image processing method. The flowchart shown below is a process executed by the system control circuit 150 when executing the above-described image processing method.

[0073] First, the system control circuit 150 acquires humidity information via the vehicle information acquisition unit 121 (step S51). The humidity information includes information on the humidity acquired from a humidity sensor that detects the humidity around the vehicle 90.

[0074] Next, the determination unit 151 of the system control circuit 150 determines whether the humidity Hmd included in the acquired information is equal to or greater than a preset threshold humidity Hth (step S52). When it is determined that the humidity Hmd is equal to or greater than the threshold humidity Hth (step S52: YES), the system control circuit 150 determines to set the weighting of the first local tone curve to be greater than the weighting of the first local tone curve when the humidity Hmd is less than the threshold humidity Hth (step S53), and ends the process.

[0075] On the other hand, when it is not determined that the humidity Hmd is equal to or greater than the threshold humidity Hth (step S52: NO), the system control circuit 150 determines to set the weighting of the first local tone curve to be smaller than the weighting of the first local tone curve when the humidity Hmd is less than the threshold humidity Hth (step S54), and ends the process.

[0076] By the above-described process, the image processing apparatus 11 can suppress a decrease in the recognition accuracy of distant objects when it is rainy or foggy around the vehicle 90.

[0077] Next, with reference to FIG. 7, another example of the process for determining the weighting coefficient when generating the blend tone curve will be described. FIG. 7 is a second flowchart showing the weighting setting method in the image processing method.

[0078] First, the system control circuit 150 acquires vehicle operation information via the vehicle information acquisition unit 121 (step S61). The vehicle operation information includes the straight-ahead speed of the vehicle 90 measured by a gyro sensor or the like mounted on the vehicle 90.

[0079] Next, the determination unit 151 of the system control circuit 150 determines whether the speed Spd, which is the straight-ahead speed included in the acquired information, is equal to or higher than a preset threshold speed Sth (step S62). When it is determined that the speed Spd is equal to or higher than the threshold speed Sth (step S62: YES), the system control circuit 150 determines to set the weighting of the first local tone curve to be larger than the weighting of the first local tone curve when the speed Spd is less than the threshold speed Sth (step S63), and ends the process.

[0080] On the other hand, when it is not determined that the speed Spd is equal to or higher than the threshold speed Sth (step S62: NO), the system control circuit 150 determines to set the weighting of the first local tone curve to be smaller than the weighting of the first local tone curve when the speed Spd is less than the threshold speed Sth (step S64), and ends the process.

[0081] Through the above process, the image processing apparatus 11 can suppress a decrease in the recognition accuracy of relatively distant objects when the traveling speed of the vehicle 90 is relatively high. Note that in the above process, instead of the speed Spd, the angular velocity of the vehicle 90 may be used. By setting the weighting of the first local tone curve according to the angular velocity, the image processing apparatus 11 can suppress a decrease in the recognition accuracy of relatively distant objects when turning at a relatively high angular velocity.

[0082] Note that the processes in the flowcharts shown in FIGS. 6 and 7 are not exclusive. That is, the image processing apparatus 11 may execute the processes of FIG. 6 and FIG. 7 together.

[0083] In addition to the above-described processes, the image processing apparatus 11 may execute the following processes. The system control circuit 150 acquires information associated with the contrast of the thermal image data. Next, the determination unit 151 of the system control circuit 150 determines whether the contrast has decreased from the acquired information. Specifically, the determination unit 151 compares the contrast value with a preset threshold value, and determines that the contrast has decreased when the contrast value is less than the threshold value. In this case, the determination unit 151 determines to set the weighting of the first local tone curve to be greater than the weighting of the first local tone curve when it is not determined that the contrast has decreased.

[0084] Note that in the process of setting the above-described weighting coefficient, the threshold value is not limited to one. That is, the determination unit 151 may compare the acquired information with a plurality of threshold values and set the weighting coefficient according to the comparison result.

[0085] Next, a specific example of the local area will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the local area according to the first embodiment. A thermal image D11 is shown in FIG. 8. Also, in the thermal image D11, the H direction is set to the right and the V direction is set to the bottom with the upper left as the origin. In the following description, the coordinates in the H direction and the V direction in the thermal image D11 are shown in parentheses as (h, v).

[0086] The image processing unit 170 sets a plurality of first local regions R1 and second local regions R2 for the thermal image D11. For example, the image processing unit 170 sets a first local region R1(0,0) at a position corresponding to the coordinate (0,0) of the image D11. Also, the image processing unit 170 sets first local regions R1(h,v) at arbitrary positions (h,v) so as to be adjacent to each other. That is, as shown by the two-dot chain line in the figure, a first local region R1(h + 1,v) is set adjacent to the right side of the first local region R1(h,v), and a first local region R1(h,v + 1) is set adjacent to the lower side of the first local region R1(h,v). In this way, the image processing unit 170 sets m first local regions R1 in the H direction and n first local regions R1 in the V direction up to the first local region R1(m - 1,n - 1) in the lower right of the thermal image D11.

[0087] Also, the image processing unit 170 sets second local regions R2(h,v) corresponding to each of the above-described first local regions R1. The second local region R2(h,v) corresponding to the coordinates (h, v) is a rectangular region that includes the first local region R1(h,v) and is larger than the first local region R1(h,v). Note that, as shown at the coordinates (0,0) and (m,n) in the figure, the second local region R2 may protrude outside the thermal image D11 at the outer edge of the thermal image D11. In this case, the size of the second local region R2 is cut according to the size of the thermal image D11.

[0088] In this way, the image processing unit 170 sets the first local region R1(h,v) and the second local region R2(h,v) as local regions corresponding to the coordinates (h,v). Then, the image processing unit 170 generates a local tone curve and a blend tone curve in each of the set local regions.

[0089] Next, with reference to FIG. 9, a specific example of the local tone curve will be described. FIG. 9 is a diagram showing an example of a process for generating excess data. In FIG. 9, data D12 is shown on the upper side, and data D13 is shown on the lower side.

[0090] The data D12 shown above FIG. 9 is a frequency distribution in a predetermined local area. In the image processing apparatus 11, the image conversion processing unit 173 of the image processing unit 170 sets a local area from the thermal image data received from the NUC unit 172, and calculates a frequency distribution as shown in the data D12 for the set local area. In the data D12, the horizontal axis represents the pixel value, and the vertical axis represents the frequency corresponding to the pixel value. The frequency distribution shown in the data D12 is represented by a curve having a peak of frequency in the central part.

[0091] The data D13 shown below FIG. 9 is shown by applying hatching to the excess data D131 that exceeds a predetermined threshold with respect to the frequency distribution. The image conversion processing unit 173 extracts the excess data D131 in order to redistribute the excess data D131 that exceeds a preset threshold, and redistributes the frequency of the excess data D131 according to a predetermined distribution ratio with respect to the frequency distribution.

[0092] Next, referring to FIG. 10, the distribution ratio will be described. FIG. 10 is a diagram showing a first example of the distribution ratio. In FIG. 10, a graph E10 showing the relationship between the pixel value and the distribution ratio is shown. In the graph E10, the horizontal axis represents the pixel value, and the vertical axis represents the distribution ratio. Also, a curve E101 is plotted in the graph E10. The curve E101 is a curve that gently descends to the left and right with the position of the pixel value shown on the horizontal axis as the peak of the central value. In this way, the distribution ratio is set so that the central value is the peak and the distribution ratio decreases as the distance from the central value increases.

[0093] More specifically, for example, the first pixel value E121 in FIG. 10 has a first difference E111 from the central value. The second pixel value E122 in FIG. 10 has a second difference E112 that is larger than the first difference from the central value. At this time, the first distribution ratio E131 corresponding to the first pixel value E121 is set to be higher than the second distribution ratio E132 corresponding to the second pixel value E122. Note that the minimum value and the maximum value of the curve E101 are not fixed values, but vary depending on the values in the local area.

[0094] With such a configuration, the image processing apparatus 11 distributes relatively more data to the central value and the pixel values in its vicinity, making the representation of the image at the central value and in its vicinity richer. Thereby, the image processing apparatus 11 improves the contrast of the image at the central value and in its vicinity, and improves the resolution for detecting the target object.

[0095] Next, with reference to FIG. 11, the redistribution data and the local tone curve generated by the image processing unit 170 will be described. FIG. 11 is a diagram showing an example of a process for generating redistribution data and a local tone curve.

[0096] The data D14 shown at the upper side of FIG. 11 is the redistribution data. The image conversion processing unit 173 distributes the frequency corresponding to the excess data D131 to the pixel value data from the minimum value to the maximum value of the thermal image data. In the example shown in the figure, the image conversion processing unit 173 generates the redistribution data D142 by distributing the excess data D131 along a curve with a peak at the central portion like the data D141.

[0097] The data D15 shown at the lower side of FIG. 11 is the data indicating the local tone curve. The data D15 is a cumulative frequency distribution obtained by sequentially accumulating the redistribution data generated in the data D14 from the minimum value to the maximum value of the pixel value. The image conversion processing unit 173 generates the local tone curves in the first local region R1(h, v) and the second local region R2(h, v) by the above-described procedure.

[0098] Next, with reference to FIG. 12, the blend tone curve will be described. FIG. 12 is a diagram showing an example of a blend tone curve. FIG. 12 shows a first local tone curve D16, a second local tone curve D17, a first blend tone curve D181, and a second blend tone curve D182.

[0099] The first local tone curve D16 is the local tone curve of the first local region at a predetermined position of the thermal image data. The second local tone curve D17 is the second local tone curve corresponding to the first local tone curve D16. That is, the second local region corresponding to the second local tone curve D17 includes the first local region corresponding to the first local tone curve D16.

[0100] The first blend tone curve D181 is generated by synthesizing the first local tone curve D16 and the second local tone curve according to the above formula (1). Further, the first blend tone curve D181 is normalized using the maximum pixel value of the thermal image data after synthesizing the first local tone curve D16 and the second local tone curve D17. That is, for the first blend tone curve D181, the horizontal axis is the input pixel value and the vertical axis is the output pixel value. Thereby, the image conversion processing unit 173 generates the converted image data for the distance at each pixel position corresponding to the first blend tone curve D181.

[0101] Similarly, the second blend tone curve D182 is generated by synthesizing the first local tone curve D16 and the second local tone curve according to the above formula (2). Further, the second blend tone curve D182 is normalized using the maximum pixel value of the thermal image data after synthesizing the first local tone curve D16 and the second local tone curve D17. Thereby, the image conversion processing unit 173 generates the converted image data for the vicinity at each pixel position corresponding to the second blend tone curve D182.

[0102] Next, referring to FIG. 13, another example of the distribution ratio will be described. FIG. 13 is a diagram showing a second example of the distribution ratio. In FIG. 13, a graph E20 showing the relationship between the pixel value and the distribution ratio is shown. In the graph E20, the horizontal axis represents the pixel value and the vertical axis represents the distribution ratio. Also, a broken line E201 is plotted in the graph E20. The broken line E201 is a broken line that bends downward while bending left and right with the pixel value near the central value as the peak. Thus, the distribution ratio is set such that the central value is the peak and the distribution ratio decreases as the distance from the central value increases.

[0103] More specifically, for example, the first pixel value E221 in FIG. 13 has a first difference E211 from the central value. The second pixel value E222 in FIG. 13 has a second difference E212 that is larger than the first difference from the central value. At this time, the first distribution ratio E231 corresponding to the first pixel value E221 is set to be higher than the second distribution ratio E232 corresponding to the second pixel value E222.

[0104] The above describes Embodiment 1. As described above, the image processing apparatus 11 generates local tone curves for the first local area and the second local area, sets the weighting thereof, and then synthesizes the converted image data for the distant area and the converted image data for the vicinity area. Thereby, the image processing apparatus 11 generates image data suitable for recognizing an object existing relatively far away and image data suitable for recognizing an object existing relatively nearby, respectively. Also, the image processing apparatus 11 sets the weighting coefficient when generating the image data for the distant area or the vicinity area according to a predetermined situation. Thereby, the image processing apparatus 11 can suppress a decrease in the recognition rate of an object. Therefore, according to Embodiment 1, it is possible to provide an image processing apparatus, an image processing method, and a program that improve the object detection ability while suppressing an increase in the processing amount.

[0105] Incidentally, although the above-described image processing apparatus 11 sets two types of local areas, namely, the first local area and the second local area, the setting of the local area is not limited to this. That is, the image processing apparatus 11 sets a plurality of local areas having different sizes. At this time, when the image processing apparatus 11 sets local areas having three or more different sizes and generates a blend tone curve, local tone curves corresponding to these three or more areas may be synthesized.

[0106] <Embodiment 2> Next, Embodiment 2 will be described. FIG. 14 is a flowchart of the image processing method according to Embodiment 2. The flowchart shown in FIG. 14 shows the processing mainly performed by the image conversion processing unit 173. The flowchart shown in FIG. 14 starts, for example, when thermal image data is supplied from the infrared camera 10 to the image processing apparatus 11 and a signal for setting a weighting coefficient is received from the instruction unit 152.

[0107] First, the image conversion processing unit 173 acquires the calibrated thermal image data from the NUC unit 172 (step S101).

[0108] Next, the image conversion processing unit 173 calculates the frequency distribution of the thermal image data (step S102). Here, when the pixel value of the thermal image data is, for example, about 16 bits, the image conversion processing unit 173 according to the present embodiment calculates the frequency distribution in which about the lower 2 to 4 bits are grouped into one bin (also referred to as a class). Thereby, the image processing apparatus 11 can shorten the processing time required for image conversion while suppressing a decrease in practical calculation accuracy.

[0109] Next, the image conversion processing unit 173 calculates the range of the thermal image data (step S103). In the range calculation, the image conversion processing unit 173 sets the minimum pixel value on the low-temperature side and the maximum pixel value on the high-temperature side in the frequency distribution. Further, the image conversion processing unit 173 sets the center pixel value. Then, the image conversion processing unit 173 calculates a first pixel range that is the difference between the maximum pixel value and the center pixel value. Similarly, the image conversion processing unit 173 calculates a second pixel range that is the difference between the center pixel value and the minimum pixel value. Then, the image conversion processing unit 173 compares the first pixel range and the second pixel range, and applies the larger pixel range to the smaller pixel range to set the pixel range of the entire image.

[0110] With reference to FIG. 15, an example of range calculation will be described. FIG. 15 is a frequency distribution diagram of thermal image data. The data D22 shown in FIG. 15 shows an example of thermal image data. In the data D22, the image conversion processing unit 173 sets the bins after rounding down the upper and lower several percentages (for example, 1%) of the frequency to the minimum pixel value (MIN) and the maximum pixel value (MAX), respectively. By performing such processing, when there is data that is greatly deviated from the image data such as an error, the image processing apparatus 11 can exclude this and process the data. Further, the image conversion processing unit 173 also sets the center value. The center value can be the median or the average value of the frequency distribution. In the case of the example shown in the data D22, the image conversion processing unit 173 sets the median as the center value.

[0111] In the image data shown in the data D22, the first pixel range R11, which is the difference between MAX and MEDIAN, is larger than the second pixel range R12, which is the difference between MEDIAN and MIN. Therefore, the image conversion processing unit 173 sets the range of the pixel values on the low-temperature side to MIN2, which is separated from the MEDIAN by the first pixel range R11. By setting the pixel range in this way, the image processing apparatus 11 improves the contrast after conversion.

[0112] Returning to FIG. 14, the description will continue. The image conversion processing unit 173 performs range limitation (step S104). In the range limitation, the image conversion processing unit 173 changes the setting of data existing outside the range of the set minimum pixel value and maximum pixel value to a bin closer to either the minimum pixel value or the maximum pixel value. In the example shown in FIG. 15, if there is data below MIN2 in the frequency distribution, the image conversion processing unit 173 arranges this data in the bin corresponding to MIN2. Similarly, if there is data above MAX in the frequency distribution, the image conversion processing unit 173 arranges this data in the bin corresponding to MAX.

[0113] Next, the image conversion processing unit 173 divides into unit areas (step S105). A unit area is an area that is an element for constituting a local area for generating a local tone curve. That is, the image conversion processing unit 173 sets each local area as a set of unit areas. By dividing the thermal image into unit areas, the image processing apparatus 11 can efficiently perform processing on a plurality of local areas of different sizes.

[0114] Referring to FIG. 16, the unit areas generated by the image conversion processing unit 173 will be described. FIG. 16 is a diagram showing an example of a unit area. In FIG. 16, a thermal image D23 and a unit area U10 are shown.

[0115] The thermal image D23 is a thermal image related to the input thermal image data received by the image conversion processing unit 173. In the example shown in FIG. 16, the image conversion processing unit 173 divides the thermal image D23 into 64 parts in the horizontal direction and 48 parts in the vertical direction. The unit area U10 is one area when the thermal image D23 is divided as described above. In the thermal image D23 shown in FIG. 16, 64 unit areas U10 are set in the horizontal direction (H-axis direction) with the upper left as the origin, and 48 unit areas U10 are set in the vertical direction (V-axis direction). In FIG. 16, one unit area U10 is indicated by hatching in the upper left of the thermal image D23.

[0116] When the image conversion processing unit 173 sets the upper left as the origin coordinates (0, 0) and the lower right as the coordinates (63, 47), it sets a unit area U10(h, v) at an arbitrary coordinate (h, v). By dividing the thermal image D23 into 64×48 in this way, the image conversion processing unit 173 sets 3072 unit areas U10. For example, when the thermal image D23 is QVGA (Quarter Video Graphics Array) and has a pixel number configuration of 320×240 pixels, the unit area is 5×5 pixels.

[0117] Returning to FIG. 14, the description continues. Next, the image conversion processing unit 173 calculates a unit frequency distribution that is the frequency distribution of the unit areas (step S106). That is, in the case of the example shown in FIG. 16, the image conversion processing unit 173 calculates the frequency distribution for each of the 3072 unit areas. At this time, the image conversion processing unit 173 calculates so that the unit frequency distribution falls within the above-mentioned range limit.

[0118] Next, the image conversion processing unit 173 sets a first local area (step S110). Also, the image conversion processing unit 173 sets a second local area (step S120). Further, the image conversion processing unit 173 sets a third local area (step S130).

[0119] Referring to FIG. 17, the local areas will be described. FIG. 17 is a diagram showing an example of the local areas according to the second embodiment. FIG. 12 shows a 5×5 unit area U10 and, superimposed thereon, a first local area U11, a second local area U12, and a third local area U13.

[0120] The first local area U11 is composed of one unit area U10. The first local area U11 shown in the figure corresponds to the unit area U10 at the coordinates (h, v) located at the center of the unit areas U10 arranged in a 5×5 pattern. The second local area U12 is composed of 3×3 unit areas U10. The second local area U12 corresponds to the unit area U10 (h, v) at the center of the unit areas U10 arranged in a 5×5 pattern and includes the first local area U11 (h, v). Also, the third local area U13 is composed of 5×5 unit areas U10. The third local area U13 includes the first local area U11 (h, v) and the second local area U12 (h, v) at the center. That is, the third local area U13 corresponds to the coordinates (h, v).

[0121] In the process of setting the above-mentioned first local area, second local area, and third local area, the image conversion processing unit 173 calculates the frequency distribution for each local area. At this time, the image conversion processing unit 173 uses the unit frequency distribution. That is, when calculating the frequency distribution for each local area, the image conversion processing unit 173 sums up the unit frequency distributions in the unit areas included in each local area. Thereby, the image conversion processing unit 173 can efficiently calculate the frequency distribution for each local area.

[0122] Returning to FIG. 14, the description will be continued. The image conversion processing unit 173 sets a distribution ratio for each local area (step S111, step S121, and step S131). More specifically, the image conversion processing unit 173 uses the central value set in step S103 to set a distribution rate that decreases as it moves away from the central value, with the central value or the pixel value region including the central value as the peak. Note that the distribution rate set here is set individually for each local area.

[0123] Next, the image conversion processing unit 173 generates redistribution data for each local area (steps S112, S122, and S132). Note that the value of the threshold for excess data set when generating the redistribution data can be set individually for each local area.

[0124] Next, the image conversion processing unit 173 generates a first local tone curve corresponding to the first local area (step S113). Further, the image conversion processing unit 173 generates a second local tone curve corresponding to the second local area (step S123). Similarly, the image conversion processing unit 173 generates a third local tone curve corresponding to the third local area (step S133). Note that the process of generating the above-described redistribution data and the process of generating the local tone curves corresponding to the respective local areas may be performed in parallel or sequentially. Note that the image conversion processing unit 173 normalizes each of the generated local tone curves by dividing by the total frequency in each area.

[0125] Next, the image conversion processing unit 173 synthesizes the generated local tone curves to generate a blend tone curve (step S140). The image conversion processing unit 173 generates a first blend tone curve and a second blend tone curve such that the following equation holds.

Equation

Equation

[0126] In Equation (4), BT 22 (i) represents the second blend tone curve. Wa1 represents the weighting coefficient for the first local tone curve. Wb1 represents the weighting coefficient for the second local tone curve. Wc1 represents the weighting coefficient for the third local tone curve. F21 represents a predetermined offset value.

[0127] Referring to FIG. 18, the concepts of the above Equations (3) and (4) will be described. FIG. 18 is a diagram showing the relationship between the local tone curve and the blend tone curve according to Embodiment 2. FIG. 18 shows each term corresponding to Equations (3) and (4) and the flow of the calculation.

[0128] First, the first local tone curve LTa(i) is multiplied by the weighting coefficient Wa1 (step S211). Next, the second local tone curve LTb(i) is multiplied by the weighting coefficient Wb1 (step S212). Further, the third local tone curve LTc(i) is multiplied by the weighting coefficient Wc1 (step S213). Then, these three terms and the offset value F21 are added together (step S214), and the first blend tone curve BT 21 (i) is calculated.

[0129] Next, the first local tone curve LTa(i) is multiplied by the weighting coefficient Wa2 (step S221). Next, the second local tone curve LTb(i) is multiplied by the weighting coefficient Wb2 (step S222). Further, the third local tone curve LTc(i) is multiplied by the weighting coefficient Wc2 (step S223). Then, these three terms and the offset value F22 are added together (step S224), and the second blend tone curve BT 22 (i) is calculated.

[0130] FIG. 19 shows examples of local tone curves and blend tone curves. FIG. 19 is a diagram showing examples of local tone curves and blend tone curves according to Embodiment 2. In the graph shown in FIG. 19, the first local tone curve, the second local tone curve, the third local tone curve, the first blend tone curve, and the second blend tone curve are superimposed. As shown in the figure, each local tone curve has a respective slope according to the size of the local area. The blend tone curve has weighting coefficients and an offset set so that the image processing apparatus 11 can exhibit a desired function.

[0131] Returning to FIG. 14, the description continues. The image conversion processing unit 173 performs tone conversion of the thermal image data using the generated blend tone curve (step S141). By this processing, the image conversion processing unit 173 generates remote conversion image data and near - field conversion image data from the thermal image data of 1.

[0132] As described above, when calculating the frequency distribution of the thermal image data, the image conversion processing unit 173 calculated the frequency distribution by grouping about 2 to 4 lower - order bits into one bin and performed the processing. In other words, the image conversion processing unit 173 extracted specified pixels so that each pixel in the local area was separated from each other by several bits, and performed the same processing as applying the blend tone curve to the extracted specified pixels. Therefore, in subsequent processing, processing for corresponding to all pixels in the input thermal image data is performed.

[0133] Next, the image conversion processing unit 173 performs image data interpolation processing on each of the far - distance conversion image data and the near - distance conversion image data (step S142). Specifically, the image conversion processing unit 173 interpolates the pixel values of the pixels existing between the specified pixels set in this step S141 by using the converted pixel values of the surrounding specified pixels. The interpolation method is, for example, bilinear interpolation. Thereby, the image conversion processing unit 173 performs tone conversion on all the pixels of the thermal image data. That is, the image conversion processing unit 173 generates conversion image data obtained by converting the input thermal image data.

[0134] Next, the image conversion processing unit 173 outputs the generated conversion image data (step S150). The image processing unit 170 supplies the conversion image data output by the image conversion processing unit 173 to, for example, the image recognition unit 180. Thereby, the image processing apparatus 11 performs object detection using the conversion image data. Further, the image processing unit 170 appropriately performs processing for displaying the conversion image data output by the image conversion processing unit 173 on the display 12. For example, the image processing unit 170 performs gamma correction on the conversion image data and supplies the image data after gamma correction to the display 12 via the image data output unit 190. When the image conversion processing unit 173 outputs the conversion image data, the series of processes is terminated.

[0135] The image processing method according to the second embodiment has been described above. By the above - described method, the image processing apparatus 11 generates far - distance conversion image data and near - distance conversion image data by simple processing and preferably recognizes an object. As described in steps S141 and S142, by extracting and processing the specified pixels, the image processing apparatus 11 can improve the contrast of the image and improve the processing speed of the image processing.

[0136] Next, a modification of Embodiment 2 will be described. FIG. 20 is a flowchart of an image processing method according to a modification of Embodiment 2. The flowchart shown in FIG. 20 differs in the processing between step S140 and step S150 from the flowchart shown in FIG. 14.

[0137] In step S140, after generating the blend tone curve, the image conversion processing unit 173 performs tone curve interpolation (step S143). As described above, when calculating the frequency distribution of the thermal image data, the image conversion processing unit 173 calculates the frequency distribution in which about 2 to 4 lower bits are grouped into one bin and performs the processing. Therefore, the blend tone curve in step S140 does not correspond to all the pixels of the thermal image data. Therefore, the image conversion processing unit 173 generates the tone curve for the pixels existing between the specified pixels by interpolating from the blend tone curve at the surrounding specified pixels. A specific method of interpolation is, for example, bilinear interpolation. By performing tone curve interpolation, the image conversion processing unit 173 generates a tone curve corresponding to all the pixels of the thermal image data.

[0138] Next, the image conversion processing unit 173 performs tone conversion on the thermal image data using the generated tone curve (step S144). The image conversion processing unit 173 performs processing to calculate the output pixel value by converting the input pixel value for all the pixels of the thermal image data. By this processing, the image conversion processing unit 173 generates the converted image data for far away and the converted image data for the vicinity from one thermal image data.

[0139] The above is the description of Embodiment 2. According to Embodiment 2, it is possible to provide an image processing apparatus, an image processing method, and a program that improve the object detection ability while suppressing an increase in the processing amount.

[0140] Note that the present invention is not limited to the above-described embodiments, and can be appropriately changed without departing from the gist thereof.

Description of Reference Numerals

[0141] 10 Infrared camera 11 Image processing device 12 Display 13 ECU 90 Vehicle 101 Housing 102 Objective lens 103 Shutter 104 Infrared sensor 105 Camera control circuit 110 Bus 120 Communication IF 121 Vehicle information acquisition unit 130 ROM 140 RAM 150 System control circuit 151 Judgment unit 152 Instruction unit 160 Image data acquisition unit 170 Image processing unit 171 Defective pixel correction unit 172 NUC unit 173 Image conversion processing unit 175 Image analysis unit 180 Image recognition unit 190 Image data output unit

Claims

1. An image data acquisition unit that acquires thermal image data from an infrared camera that captures a thermal image of the surroundings of a vehicle, and an image processing unit that generates, so as to be outputtable, far-field conversion image data and near-field conversion image data from the thermal image data. The image processing unit includes: a plurality of first local regions each consisting of one unit region set at an arbitrary coordinate composed of a plurality of pixels constituting the thermal image data, and a plurality of second local regions each composed of a plurality of the unit regions surrounding the first local region with the first local region as the center, and sets these for the thermal image; calculates a frequency distribution regarding pixel values of pixels constituting the first local region and the second local regions; generates redistribution data in which excess data exceeding a preset threshold value in the frequency distribution is redistributed to the frequency distribution; cumulatively adds the redistribution data from a low pixel value toward a high pixel value to generate a first local tone curve for the first local region and a second local tone curve for the second local region; combines the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve; generates the far-field conversion image data by applying the first blend tone curve to the first local region and the second local regions, and generates the near-field conversion image data by applying the second blend tone curve to the first local region and the second local regions. An image processing apparatus.

2. The image processing apparatus according to claim 1, further comprising a determination unit that acquires humidity information from a humidity sensor that detects the humidity of the surroundings of the vehicle, and determines to set the weighting of the first local tone curve when the humidity is equal to or higher than a preset threshold humidity to be greater than the weighting of the first local tone curve when the humidity is less than the threshold humidity.

3. The image processing apparatus according to claim 1, further comprising a determination unit that acquires vehicle operation information including at least one of the straight-ahead speed or the angular velocity of the vehicle, and determines to set the weighting of the first local tone curve when the straight-ahead speed or the angular velocity is equal to or higher than a preset threshold speed to be greater than the weighting of the first local tone curve when the straight-ahead speed or the angular velocity is less than the threshold speed. ​ ​ ​ The image processing apparatus according to claim 1 or 2.

4. A computer acquires thermal image data from an infrared camera that captures a thermal image of the surroundings of a vehicle, sets, for the thermal image, a plurality of first local regions each consisting of one unit region set at an arbitrary coordinate composed of a plurality of pixels constituting the thermal image data, and a plurality of second local regions each composed of a plurality of the unit regions surrounding the first local region with the first local region as a center, calculates a frequency distribution regarding pixel values of pixels constituting the first local region and the second local region, generates redistribution data in which excess data exceeding a preset frequency threshold among the frequency distribution is redistributed to the frequency distribution, cumulatively adds the redistribution data from a low pixel value to a high pixel value to generate a first local tone curve for the first local region and a second local tone curve for the second local region, combines the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve, generates transform image data for a distant view so as to be outputtable by applying the first blend tone curve to the first local region and the second local region, and generates transform image data for a near view so as to be outputtable by applying the second blend tone curve to the first local region and the second local region, An image processing method.

5. acquires thermal image data from an infrared camera that captures a thermal image of the surroundings of a vehicle, sets, for the thermal image, a plurality of first local regions each consisting of one unit region set at an arbitrary coordinate composed of a plurality of pixels constituting the thermal image data, and a plurality of second local regions each composed of a plurality of the unit regions surrounding the first local region with the first local region as a center, calculates a frequency distribution regarding pixel values of pixels constituting the first local region and the second local region, generates redistribution data in which excess data exceeding a preset threshold among the frequency distribution is redistributed to the frequency distribution, cumulatively adds the redistribution data from a low pixel value to a high pixel value to generate a first local tone curve for the first local region and a second local tone curve for the second local region, Combine the first local tone curve and the second local tone curve to generate a first blend tone curve and a second blend tone curve, generate, so as to be able to output, distance conversion image data by applying the first blend tone curve to the first local area and the second local area, and generate, so as to be able to output, near vicinity conversion image data by applying the second blend tone curve to the first local area and the second local area, An image processing method, causing a computer to execute Program.

Images