Subject detection device, method performed by the subject detection device, and program

Abstract

This technology provides the ability to properly detect subjects. [Solution] The subject detection device 100 acquires a visible light image including the subject, acquires information about the subject that is different from the visible light image, and corrects the region of the visible light image affected by atmospheric interference. The first detection unit 104 detects the subject from the visible light image. The second detection unit 105 detects the subject from the information about the subject. The reliability calculation unit 106 calculates the reliability of the detection results of the subject detection by the first detection unit and the subject detection by the second detection unit, which were performed on the corrected visible light image. The subject discrimination unit 107 discriminates the subject based on these detection results and the calculated reliability.

Description

Technical Field

[0001] The present disclosure relates to a technique for detecting a subject by means of a camera or the like.

Background Art

[0002] In conventional subject detection devices, subject detection in various environments has been desired. For example, when fog or haze is covering the subject, the contrast is low, the visibility is lacking, and the detection accuracy of the subject decreases.

[0003] Patent Document 1 proposes a method for realizing an image with high contrast. This method calculates the low-luminance part and the high-luminance part in the luminance correction curve according to the luminance histogram of the image, and calculates the intermediate luminance part obtained by connecting the end point on the low-luminance side in the high-luminance part of the calculated luminance correction curve and the end point on the high-luminance side in the low-luminance part. Then, this method corrects the luminance level of the image over the entire tone range using the low-luminance part, the intermediate-luminance part, and the high-luminance part of the calculated luminance correction curve.

[0004] In addition, with respect to the difficulty of subject detection using only visible light images, the technique of Patent Document 2 detects an external environment such as the weather from information obtained from a camera and a radar. Specifically, this technique determines the reliability of the subject detection result based on the weather environment obtained in the external environment detection process.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, Patent Document 1 makes no mention of the possibility that the image brightness correction process may affect the subject detection process. In other words, when image correction processing is performed, new measures are needed to properly detect the subject.

[0007] This disclosure provides a technology that can appropriately detect subjects. [Means for solving the problem]

[0008] A subject detection device according to one embodiment of the present disclosure includes: a first acquisition means for acquiring a visible light image including a subject; a second acquisition means for acquiring information about the subject that is different from the visible light image; a correction means for correcting the visible light image for areas affected by atmospheric obstruction; a first detection means for performing a first detection of the subject from the visible light image; a second detection means for performing a second detection of the subject from the information about the subject; a calculation means for calculating the reliability of the detection results of the first detection and the second detection performed on the visible light image corrected by the correction means; and a discrimination means for determining the subject based on the detection results of the first detection and the second detection and the calculated reliability. [Effects of the Invention]

[0009] According to this disclosure, subject detection processing can be performed appropriately. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the configuration of the imaging system according to the first embodiment. [Figure 2] This is a flowchart showing the operation of the subject detection device. [Figure 3] (A) and (B) show the image acquired by the visible light camera and the subject information acquired by the radar, respectively. [Figure 4] (A) and (B) show the video and histogram of the hazy condition, respectively. [Figure 5] (A) and (B) show the input / output characteristic curves used for haze correction. [Figure 6] (A) and (B) show the image and histogram, respectively, when the haze correction function is enabled and the haze correction strength is set to weak. [Figure 7] (A) and (B) show the image and histogram, respectively, when the haze correction function is enabled and the haze correction intensity is set to "strong". [Figure 8] (A) and (B) show the image before haze correction processing and the image after haze correction processing with a strong haze correction intensity, respectively. [Figure 9] This is a flowchart showing the reliability calculation process in the first embodiment. [Figure 10] (A) and (B) show the video with a low subject detection result and subject information acquired by radar, respectively. [Figure 11] (A) and (B) show images taken in clear or lightly hazy conditions, and images with strong haze correction applied, respectively. [Figure 12] This is a flowchart showing the reliability calculation process in the second embodiment. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following embodiments are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the applicable device and various conditions (usage conditions, usage environment, etc.). The technical scope of this disclosure is determined by the claims and is not limited by the following individual embodiments. Furthermore, some of the embodiments described below may be combined as appropriate.

[0012] One or more of the functional blocks shown in the figures described below may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or may be implemented by a programmable processor such as a CPU or MPU executing software. It may also be implemented by a combination of software and hardware. Therefore, in the following description, even if different functional blocks are described as the operating entity, the same hardware may be implemented as the entity. ASIC is an abbreviation for Application Specific Integrated Circuit. CPU is an abbreviation for Central Processing Unit. MPU is an abbreviation for Micro-Processing Unit. Also, in the system shown below, a memory that provides a working area and a program storage area for the processor is used. The memory includes RAM, ROM, and other storage devices.

[0013] [First Embodiment] FIG. 1 is a diagram illustrating the configuration of an imaging system according to the first embodiment. This imaging system includes a visible light camera 11, a radar 12, and a subject detection device 100.

[0014] The visible light camera 11 captures visible light. The visible light camera 11 includes an imaging optical system including one or more lenses, and a visible light imaging element (visible light sensor) that captures the optical image formed by the imaging optical system and converts it into an electrical signal. The visible light sensor detects visible light in a range of, for example, wavelengths from about 380 nm to about 750 nm. The visible light sensor may have sensitivity in at least a part of the near-infrared wavelength region.

[0015] Radar 12 is a sensing device that measures a subject. Specifically, radar 12 measures the size, distance, and / or orientation, etc. of the subject. Radar 12 emits microwave with a short wavelength and measures the time of ownership from the reflected wave of the subject, thereby, for example, obtaining the size, distance, and orientation of the subject. Also, based on AIS information and GNSS, it is possible to superimpose or associate ship information and nautical chart data with respect to the measured subject. AIS is an abbreviation of Automatic Identification System. GNSS is an abbreviation of (Global Navigation Satellite System). Examples of GNSS include GPS, GLONASS, Galileo, BDS, etc. The size, distance, and / or orientation, etc. of the subject are an example of information regarding the subject. The information regarding the subject may be referred to as "subject information" hereinafter.

[0016] The subject detection device 100 includes a visible light image acquisition unit 101, a radar information acquisition unit 102, a haze correction processing unit 103, a first detection unit 104, a second detection unit 105, a reliability calculation unit 106, and a subject discrimination unit 107.

[0017] The visible light image acquisition unit 101 acquires a visible light image captured by the visible light camera 11. The visible light image acquisition unit 101 is an example of a first acquisition means for acquiring a visible light image including a subject.

[0018] The radar information acquisition unit 102 acquires subject information measured by the radar 12. The subject can be various, such as a human, an animal, a moving object (ship, airplane, vehicle, etc.), a building, a natural object, etc. In the subsequent description of this embodiment, for the convenience of referring to the drawings, the example of the background of the visible light image is the sea, and the examples of the subject are a ship, a seabird, and a rock reef. The radar information acquisition unit 102 is an example of a second acquisition means for acquiring information regarding the subject, which is different from the visible light image including the subject.

[0019] The haze correction processing unit 103 performs correction processing on images with reduced contrast values ​​based on images acquired from the visible light camera 11. The contrast value is the contrast value (contrast ratio) for the entire image or for specific areas of the image. The processing performed by the haze correction processing unit 103 is mainly gradation processing (gradation correction processing), as will be described later. The haze correction processing unit 103 is an example of a correction means that corrects areas of a visible light image that are affected by atmospheric obstruction.

[0020] The first detection unit 104 detects one or more subjects in a visible light image that has undergone haze correction processing. Various methods can be used to detect subjects, such as pattern matching, methods using brightness gradients within local regions, or machine learning-based methods such as deep learning. The first detection unit 104 is an example of a first detection means that performs the first detection of the subjects from a visible light image.

[0021] The second detection unit 105 detects one or more objects based on the radar information obtained from the radar information acquisition unit 102. The method for detecting objects by the second detection unit 105 may be the same as that of the first detection unit 104. The detection methods of the first detection unit 104 and the second detection unit 105 may be different. The second detection unit 105 is an example of a second detection means that performs a second detection of an object based on information about that object.

[0022] The reliability calculation unit 106 calculates the reliability of the subject detection results obtained from the first detection unit 104 and the second detection unit 105, respectively.

[0023] The subject discrimination unit 107 primarily calculates an integrated evaluation value based on the respective detection results (evaluation values) obtained from the first detection unit 104 and the second detection unit 105, and the reliability of one or more subjects, and then discriminates the subject.

[0024] Figure 2 is a flowchart showing an example of the operation of the subject detection device 100. In step S201, the visible light image acquisition unit 101 acquires a visible light image captured by the visible light camera 11.

[0025] Figure 3(A) shows an example of an image acquired by the visible light camera 11. When shooting with the visible light camera 11, under clear weather conditions, it is possible to detect subjects such as flocks of seabirds 301, ships 302, and reefs 303, as shown in Figure 3(A). However, as shown in the figures described later, in situations where fog or haze occurs, the overall contrast of the image decreases, and it may become difficult to detect the aforementioned subjects using only the information from the visible light image that has not undergone haze correction processing.

[0026] In Japanese, the meanings of "kiri" (primarily fog or mist), "kasumi" (primarily haze), and "moi" (primarily haze) are strictly different. However, in this specification, these will not be distinguished and will all be referred to as "kasumi." In this specification, "kasumi" (haze) means moisture (water vapor, rain, snow, etc.), sand, dust, pollen, and / or smoke, and also the phenomenon or state in which visibility is blurred and not clearly seen due to these substances. Examples of atmospheric disturbances other than haze include heat haze and backlighting.

[0027] In step S202, the radar information acquisition unit 102 acquires subject information measured by the radar 12.

[0028] The haze correction processing unit 103, the first detection unit 104, the reliability calculation unit 106, and the subject discrimination unit 107 basically perform processing for each frame of the image that makes up the video (moving image). However, at least one of these functional units may perform processing for two or more predetermined frames.

[0029] Figure 3(B) shows the size, direction, distance, hull information, or chart data of the subject obtained from the subject information measured by the radar 12. The thick-bordered area in Figure 3(B) corresponds to the field of view of the visible light image in Figure 3(A), and it is possible to obtain information on the size, direction, and distance of subjects such as a flock of seabirds 301, a ship 302, and a reef 303, respectively.

[0030] In step S203, the haze correction processing unit 103 performs haze correction (gradation correction) processing on the visible light image. The details of the haze correction processing are described below.

[0031] Figures 4(A) and 4(B) show examples of visible light images and histograms, respectively, under hazy conditions. Under hazy conditions, the distribution of the histogram is concentrated in certain areas, the overall contrast of the image is low, and it becomes difficult to detect subjects. As shown in the example histogram in Figure 4(B), the input luminance levels, which are the luminance values ​​of the visible light image, are particularly concentrated between xl and xh. Input luminance levels where the histogram distribution is concentrated in this way may be calculated using luminance values ​​where the cumulative value of the histogram exceeds a threshold. For example, xl is the value of the input luminance level that exceeds a predetermined threshold when the frequency is sequentially added from the lowest to the highest input luminance levels. Similarly, xh is the value of the input luminance level that exceeds a predetermined threshold when the frequency is sequentially added from the highest to the lowest input luminance levels. The degree of concentration (or dispersion) of such histogram distributions is an example of the strength of the effect of atmospheric pollution.

[0032] Figures 5(A) and (B) show examples of input / output characteristic curves used for haze correction (hereinafter referred to as haze correction curves). In the graphs shown in Figures 5(A) and (B), the horizontal axis represents the input luminance level, and the vertical axis represents the output luminance level. The interval in which the histogram distribution of input luminance is concentrated due to the effect of haze (the interval between xl and xh) is set so that the output luminance changes smoothly from yl to yh. Here, yl and yh are correction parameters that control the effect of haze correction. In other words, the correction parameters are an example of elements for suppressing the effect of atmospheric interference on the visible light image. The smaller the value of yl and the larger the value of yh, the stronger the haze correction effect and the higher the contrast output result. Compared with Figure 5(A), the haze correction curve in Figure 5(B) has a small value of yl and a large value of yh, indicating that it is a haze correction curve with a strong correction effect.

[0033] The above correction parameters may be configured to be set based on user instructions or based on the calculation results of the histogram distribution as described above. For example, the haze correction processing unit 103 may have user-adjustable correction parameters such as "haze correction function" and "haze correction strength". In this embodiment, an example of a configuration in which the user can switch between options such as enabling / disabling the haze correction function or weak / strong haze correction strength is given as a correction parameter.

[0034] Furthermore, if the subject detection device 100 automatically sets the correction intensity based on the calculation result of the histogram distribution, the haze correction processing unit 103 should process as follows. The haze correction processing unit 103 can calculate the degree of concentration in the histogram distribution based, for example, on the difference or ratio between the sum of luminance values ​​in the range where the cumulative value is above a threshold and the sum of luminance values ​​in other ranges. However, other methods are also acceptable. For example, the average value may be used instead of the sum.

[0035] The haze correction processing unit 103 performs haze correction processing by applying the calculated haze correction curve to the visible light image. Figures 6(A) and 6(B) show the image and histogram when the “haze correction function” is “enabled” and the “haze correction intensity” is “weak”. Figures 7(A) and 7(B) show the image and histogram when the “haze correction function” is “enabled” and the “haze correction intensity” is “strong”. The slope of the haze correction curve when the “haze correction intensity” is “strong” (for example, Figure 5(B)) is greater than the slope of the haze correction curve when the “haze correction intensity” is “weak” (for example, Figure 5(A)). In both images, the distribution of the histogram becomes smoother and the image is corrected to have improved contrast between light and dark compared to before haze correction was performed.

[0036] When applying a haze correction curve, a strong S-shaped curve, such as when the haze correction strength is set to "strong," can result in subjects with a bias towards low and high brightness levels suffering from underexposure and overexposure, respectively. However, the contrast of subjects within a specific input brightness range (xl or higher, xh or lower) is significantly corrected. On the other hand, a gentle correction curve, such as when the haze correction strength is set to "weak," limits the range of contrast for subjects, but results in an image with suppressed underexposure and overexposure. Therefore, the effects of haze correction described in this embodiment involve trade-offs depending on the correction parameters, and the parameters are dynamically changed according to the target and area that the user wants to monitor.

[0037] Let's return to the explanation of the flowchart in Figure 2. In step S204, the first detection unit 104 performs subject detection processing on the visible light image from which haze correction processing was performed in step S203. Here, the first detection unit 104 in this embodiment outputs an evaluation value indicating the degree of subject-likeness for each subject to be detected. The evaluation value is output in the range of 0 to 100. Subject-likeness refers to the certainty that an object classified as that subject is present there.

[0038] In step S205, the second detection unit 105 performs subject detection processing on the radar information acquired in step S202. Here, similar to step S204, the second detection unit 105 outputs an evaluation value for each subject indicating the degree of subject-likeness in the range of 0 to 100. As described above, the detection methods of the first detection unit 104 and the second detection unit 105 may be the same or different. In this embodiment, as an example, the first detection unit 104 uses a model that detects seabirds using their shape and color characteristics. The second detection unit 105 uses a model that measures the shape and size of seabirds detected by the radar 12 according to their movement and distance.

[0039] In step S206, the reliability calculation unit 106 calculates the reliability of the subject detection results. Specifically, the reliability calculation unit 106 calculates the reliability of each subject detection result obtained from the first detection unit 104 and the second detection unit 105. The details of how to calculate the reliability of each subject detection result are described below.

[0040] First, we will describe the characteristics of radar images, visible light images, and visible light images with haze correction processing.

[0041] One characteristic of radar is that, because radar 12 measures reflected microwaves from an object, it is less affected by factors such as haze, backlighting, or low light. However, its azimuth resolution is low, making it difficult to determine the size and identify the object.

[0042] On the other hand, a characteristic of visible light images is that, while the accuracy of subject detection decreases due to haze, backlighting, or the generation of contrast and noise during nighttime shooting, they also have the advantage of being able to acquire color information at high resolution. Furthermore, a characteristic of visible light images is that even if some degree of haze occurs, the aforementioned haze correction processing can suppress the decrease in contrast of the subject and thus reduce the decrease in the accuracy of subject detection.

[0043] Next, we will explain the characteristics of visible light images after haze correction processing. A characteristic of haze is that the greater the distance to the subject, the more light scattering occurs, reducing the contrast of the subject. Conversely, when the distance to the subject is short, the reduction in contrast of the subject is limited. Therefore, when the haze correction strength is set to "strong," the effects of haze are removed from distant subjects, but the correction effect on nearby subjects may be excessive, potentially resulting in underexposure or overexposure. Refer to Figures 8(A) and (B) for an explanation.

[0044] Figure 8(A) shows the image before haze correction processing in a hazy situation, and Figure 8(B) shows the image after applying haze correction processing with a "strong" haze correction intensity. The evaluation values ​​shown in Figures 8(A) and (B) represent the degree to which seabirds appear when seabird detection processing is performed on distant seabirds 501 and 504, nearby seabirds 502 and 505, and distant reefs 503 and 506, respectively. As shown in Figure 8(A), in a hazy situation where haze is present and no haze correction is performed, the evaluation values ​​are generally low, and it is difficult to distinguish between seabirds and reefs. In contrast, as shown in Figure 8(B), when strong haze correction is applied, the contrast of distant subjects increases compared to before haze correction processing. In this case, the evaluation value of distant seabird 504 increases, and the evaluation value of distant reef 506 decreases. On the other hand, for nearby subjects, overcorrection causes the sea and seabird 505 to become completely black. Ideally, the evaluation value for seabird 505 should be high, but in this case, the evaluation value becomes low, making detection difficult. Taking into account the characteristics of the visible light image after the haze correction processing described above, the reliability calculation unit 106 calculates the reliability of each subject.

[0045] Figure 9 is a flowchart showing the reliability calculation process in the reliability calculation unit 106 in step S206. Here, the reliability takes a value of, for example, 0.0 to 1.0, and is set so that the reliability of the subject detection results of the visible light image and radar information adds up to 1.0. The reliability calculation unit 106 then weights the evaluation values, which are the two input detection results, and adopts the higher of the weighted result values. For the sake of convenience, in the following explanation of Figure 9, the reliability calculation unit 106 will be referred to as the calculation unit.

[0046] In step S601, the calculation unit determines the histogram distribution of the visible light image. Regarding the method for calculating the histogram distribution, as explained in step S202, the histogram distribution may be determined using luminance values ​​within a range where the cumulative value of the histogram is equal to or greater than a threshold. If the histogram distribution is dispersed, the process proceeds to step S604; if it is concentrated in a certain area, the process proceeds to step S607. Since the histogram distribution has already been calculated in the haze correction process (step 203 in Figure 2), the calculation unit may use that calculation result in step S601.

[0047] In step S607, if the histogram distribution is concentrated, the calculation unit sets a higher confidence level for the subject detection result obtained from the radar information. For example, the confidence level for the subject detection result from the radar information is set to 0.9, and the subject detection result from the visible light image is set to 0.1. Figure 10(A) shows a state where the subject detection result from the visible light image is generally low due to the effect of, for example, dense haze (dense fog). As shown in Figure 10(A), under conditions of dense haze, even if haze correction processing is performed, the contrast is not restored (does not increase), and the subject detection result from the visible light image is generally low, making it impossible to accurately identify the subject. On the other hand, radar information is not significantly affected by weather conditions such as haze, so subject detection results like those in Figure 10(B) can be obtained stably regardless of the weather. Therefore, if the histogram distribution of the visible light image is concentrated, setting a higher confidence level for the subject detection result from the radar information as shown in Figure 10(B) can prevent missing subjects.

[0048] In step S602, the calculation unit determines the subsequent branch according to the correction parameters of the haze correction process obtained from the haze correction processing unit 103. If the "haze correction function" is "disabled" or if the "haze correction function" is "enabled" and the "haze correction strength" is "weak", the process proceeds to step S604. If the "haze correction function" is "enabled" and the "haze correction strength" is "strong", the process proceeds to step S603.

[0049] The process in step S602 is an example of a first determination regarding the setting of the correction parameter in the correction process. In this case, setting the haze correction intensity to "strong" is an example of setting the correction parameter to a value corresponding to a predetermined or greater intensity of the effect of atmospheric disturbance. In other words, the predetermined or greater intensity of the effect of atmospheric disturbance is, in this embodiment, the degree of concentration of the histogram distribution when the haze is dense, as shown in Figure 10(A). Also, setting the haze correction intensity to "weak" is an example of setting it to a value corresponding to a predetermined or less intensity of the effect of atmospheric disturbance.

[0050] In step S604, the calculation unit sets a high confidence level for the subject detection result obtained from the visible light image. For example, the confidence level for the subject detection result from the visible light image is set to 0.9, and the confidence level for the subject detection result from the radar is set to 0.1. Figure 11(A) shows an image where the histogram distribution of the visible light image is dispersed in a clear or lightly hazy (low haze density) situation. In a clear or lightly hazy situation, the histogram distribution is smoothly dispersed even without haze correction processing, and there are no adverse effects. Therefore, in this case, the calculation unit performs subject detection using high-resolution color information from the visible light image. By setting a high confidence level for the subject detection result from the visible light image, the most accurate subject detection result can be obtained.

[0051] In step S603, the calculation unit proceeds with branching processing according to the subject distance obtained from the subject detection result of the radar information. If the subject distance is far, the process proceeds to step S605; if the subject distance is close, the process proceeds to step S606. The processing in step S603 is an example of a second determination regarding the distance to the subject among the information about the subject.

[0052] The determination process in step 603 may be performed by user selection, similar to the determination process in step 602. Alternatively, the calculation unit may perform at least one of the determination processes in steps S602 and S603. If the calculation unit performs the determination process in S603, a threshold for the subject distance should be set, and the determination process should be performed based on that threshold.

[0053] In step S605, the calculation unit sets a high reliability level for the subject detection result obtained from the visible light image. In step S606, the calculation unit sets a high reliability level for the subject detection result obtained from the radar information. Figure 11(B) shows an image where haze is present, strong haze correction processing has been applied, and the histogram distribution is dispersed. As shown for seabirds 710 and reefs 712, for distant subjects, the haze removal effect by the haze correction processing is performed appropriately, and the evaluation values ​​are high for seabirds and low for reefs. On the other hand, for nearby subjects, it is desirable that the evaluation value of seabird 711 be high, but due to overcorrection, the evaluation value is low. Therefore, the calculation unit sets a high reliability level for the subject detection result from the visible light image for distant subjects where the haze correction processing is performed appropriately, and sets a high reliability level for the subject detection result from the radar information for nearby subjects where overcorrection is occurring. For example, for distant subjects, the confidence level of the visible light image results is set to 0.9, and the confidence level of the radar information results is set to 0.1. For nearby subjects, the confidence level of the radar information results is set to 0.9, and the confidence level of the visible light image results is set to 0.1.

[0054] The calculation unit performs the steps described above for all subject detection results, thereby enabling it to calculate the confidence level proposed in this embodiment.

[0055] Referring to Figure 2, in step S207, the subject discrimination unit 107 calculates an integrated evaluation value based on the subject detection results obtained from the first detection unit 104 and the second detection unit 105, and the confidence level of one or more subjects, and then discriminates the subject. Here, the integrated evaluation value is a value obtained based on the confidence level calculated in step S206 for the evaluation values ​​of the visible light image and radar information of each subject. Specifically, as will be explained below, weighted addition is performed on the evaluation value.

[0056] For example, in Figure 10(A), the evaluation value of the subject detection result from radar information is 0.9, and the evaluation value from the visible light image is 0.1. For seabird 701 (704), the evaluation value from the visible light image is 40, and the evaluation value from the radar information is 60. Therefore, the combined evaluation value for seabird 701 (704) is 40 x 0.1 + 60 x 0.9 = 58. Using a similar calculation method, the combined evaluation values ​​for seabird 702 (705) and reef 703 (706) are 58 and 40, respectively. Thus, even under conditions where it is difficult to distinguish the subject using only the evaluation value from the visible light image, considering the evaluation value from radar information creates a significant difference in the evaluation value, making discrimination possible.

[0057] The subject discrimination unit 107 can, for example, use the detection result of a subject directly if the evaluation value of the subject obtained from the first detection unit 104 or the second detection unit 105 is above a threshold. If the evaluation value of the subject is below the threshold, the subject discrimination unit 107 may provide the user with information indicating that the subject has that evaluation value (a low evaluation value). Alternatively, two or more threshold values ​​for the evaluation value may be set, and the subject discrimination unit 107 can distinguish subjects using various known methods according to the evaluation values ​​obtained from the first detection unit 104 or the second detection unit 105.

[0058] Similarly, when the integrated evaluation values ​​are calculated in the same way in Figure 11(A), seabird 707 is 78, seabird 708 is 87, and reef 709 is 13. In clear weather conditions like Figure 11(A), by placing more emphasis on the reliability of subject detection in visible light images, accurate subject identification becomes possible.

[0059] Furthermore, in Figure 11(B), the evaluation value of the subject detection result from the distant radar information is 0.1, and the evaluation value of the distant visible light image is 0.9. On the other hand, the evaluation value of the subject detection result from the nearby radar information is 0.9, and the evaluation value of the distant visible light image is 0.1. Here, for the distant seabird 710, the evaluation value of the visible light image is 80 and the evaluation value of the radar information is 60 (Figure 10(B)), so the combined evaluation value is 80 x 0.9 + 60 x 0.1 = 78. On the other hand, for the nearby seabird 711, the evaluation value of the visible light image is 20 and the evaluation value of the radar information is 60 (Figure 10(B)), so the combined evaluation value is 20 x 0.1 + 60 x 0.9 = 56. Also, using the same calculation method, the combined evaluation value for the distant reef 712 is 13.

[0060] As described above, using only the evaluation value of the visible light image after haze correction processing resulted in low evaluation values ​​for nearby seabirds, making accurate identification impossible. However, by using the haze correction intensity and the radar subject detection results, it becomes possible to correctly identify the subject.

[0061] The technology described in Patent Document 1 above does not take into account the problems that may arise from correction processing when the parameters for image processing (luminance correction curve) are dynamically changed (generated), and there is a risk of missed detection or false detection. In contrast, in this embodiment, the reliability of subject detection is calculated based on the content of the correction by haze correction processing and subject information obtained from sources other than visible light, thereby realizing subject detection that is robust to the external environment. In other words, according to this embodiment, subject detection processing can be performed appropriately.

[0062] [Second Embodiment] In the first embodiment, an example of a method for calculating reliability in the reliability calculation unit 106 based on haze correction intensity information and subject distance information obtained from the subject detection result of radar information was shown. In the second embodiment, an example of a method for calculating reliability based on haze correction intensity information and noise information superimposed on the subject obtained from the subject detection result of the visible light image will be described. Components that are the same as in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0063] In this embodiment, the flow of steps S201 to S205 and step 207 in Figure 2 is the same as in the first embodiment, so its explanation is omitted. Figure 12 is a flowchart of the confidence calculation process in this embodiment. In Figure 12, the processes other than step S803 are the same as in the first embodiment (steps S601, S602, and S604 to S607), so their explanation is omitted.

[0064] Haze correction processing expands the signal, thus expanding both the signal component and the noise component of the subject image. Therefore, under conditions where the noise component is significantly larger than the signal component, such as in nighttime photography, haze correction processing may further amplify the noise, potentially reducing the subject detection performance. As shown in step S803, if strong haze correction processing is performed and a large amount of noise is present, the calculation unit sets a high reliability level for the subject detection result of the radar information. On the other hand, if strong haze correction processing is performed but the noise level is low, the calculation unit sets a high reliability level for the subject detection result of the visible light image after haze correction. The noise in the subject image can be calculated using various methods, such as methods that determine the noise from the variance, standard deviation, and time-direction average of the noise calculated from the subject detection result of the visible light image. The processing in step S803 is an example of a second determination regarding the noise contained in the subject image in the visible light image.

[0065] As explained above, in this embodiment, by calculating the reliability based on noise information superimposed on the subject obtained from the subject detection result of the visible light image, it becomes possible to maintain subject detection performance even when haze correction processing is performed during nighttime shooting.

[0066] In this embodiment, the evaluation value indicating noise (whether it is high or low) is calculated from the noise superimposed on the image, but the configuration does not necessarily have to be this way. For example, a configuration could be used in which the confidence ratio is increased when a specific ISO setting is reached, based on known information such as the noise characteristics of the visible light image sensor and the sensor size.

[0067] [Other embodiments] In the first and second embodiments described above, subject information (information about the subject) was given as an example of information about the size, distance, and direction of the subject measured by the radar 12. However, information about the subject may also include information such as the temperature, position, and speed of the subject. This information can be measured by, for example, a thermal camera, sonar, GNSS, AIS, etc.

[0068] The subject detection device 100 may acquire external environmental information such as weather information. In this case, the reliability calculation unit 106 may calculate the reliability based on the acquired external environmental information in addition to the correction content of the haze correction processing unit 103. Here, weather information includes information such as rainfall and snowfall. For example, if the rainfall or snowfall is above a threshold, the reliability calculation unit 106 may weight the evaluation value of subject detection using a reliability set low from the visible light image.

[0069] In the first and second embodiments described above, the degree of concentration or dispersion of the histogram distribution was used as an example of information indicating the strength of the effect of atmospheric pollution. However, the information indicating the strength of the effect of atmospheric pollution may also be the contrast value of the visible light image. In this case as well, the contrast value is the contrast value (contrast ratio) of the entire visible light image or of a partial region of the visible light image. Specifically, the haze correction intensity is set to be strong when the contrast value of the visible light image is low, and weak when the contrast value is high. If the contrast value is the contrast value of a partial region of the visible light image, haze correction processing can be performed for each region. The haze correction processing unit may apply processing such as a high-pass filter to the visible light image and set the haze correction intensity to be stronger as the intensity of the edge components increases.

[0070] Alternatively, the haze correction processing unit may use the Dark Channel Prior method, a known haze removal method. In this case, the haze correction processing unit may calculate the dark channel value for each region of the visible light image to estimate the atmospheric transmittance, and set the haze correction intensity to be stronger in regions with higher dark channel values. The estimated atmospheric transmittance is an example of information indicating the strength of the effect of atmospheric obstruction.

[0071] In the first and second embodiments described above, the haze correction processing unit 103 performed correction processing on the entire visible light image, or on a region-by-region basis. However, the haze correction processing unit 103 may calculate the strength of the atmospheric obstruction effect for each region of the visible light image and perform correction only on a portion of the region where the strength of the atmospheric obstruction effect is above a threshold.

[0072] In the first and second embodiments described above, the haze correction intensity in the haze correction process was set to two levels, "strong" and "weak," but it may also be three or more levels. For example, in S602, four levels of correction parameters may be set: "inactive," "weak," "medium," and "strong," and the reliability may also be set to two or three or more levels depending on these correction parameters. Similarly, in S603 (Figure 9) and S803 (Figure 12), it may also be three or more levels instead of just two levels, "far" and "close."

[0073] The subject detection device 100 can also provide the user with an image that highlights the detected subject. Highlighting may include, for example, coloring the subject, surrounding the subject with a rectangle or other shaped line, or displaying a predetermined image (e.g., an arrow) that points to the subject. The predetermined image or the subject image itself may also be displayed as flashing. The subject detection device 100 may generate images with different highlighting depending on the confidence level calculated by the confidence level calculation unit 106.

[0074] At least one of the visible light camera 11 and the radar 12 may be integrally incorporated into the subject detection device 100.

[0075] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0076] This embodiment includes the following configurations, methods, and programs. (Composition 1) A first acquisition means for acquiring a visible light image including the subject, A second acquisition means for acquiring information about the subject that is different from the visible light image, A correction means for correcting the region of the visible light image affected by atmospheric interference, A first detection means for performing a first detection of the subject from the visible light image, A second detection means for performing a second detection of the subject based on the information relating to the subject, A calculation means for calculating the reliability of the detection results of the first detection and the second detection performed on the visible light image corrected by the correction means, The system includes a discrimination means for determining the subject based on the detection results of the first detection and the second detection, and the calculated confidence level. A subject detection device characterized by the following features. (Configuration 2) The correction means performs the correction based on the histogram distribution of the brightness values ​​of the visible light image. A subject detection device according to configuration 1, characterized by the above. (Composition 3) The correction means performs the correction based on the contrast value of the visible light image. A subject detection device according to configuration 1, characterized by the above. (Composition 4) The correction means performs the correction based on the estimated atmospheric transmittance in the Dark Channel Prior method. A subject detection device according to configuration 1, characterized by the above. (Composition 5) The calculation means performs a first determination regarding the setting of correction parameters in the correction for suppressing the effect of atmospheric interference on the visible light image. A subject detection device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The calculation means performs a second determination regarding the distance to the subject from the information about the subject. A subject detection device according to configuration 5, characterized by the features described herein. (Composition 7) The calculation means performs a second determination regarding the noise contained in the image of the subject in the visible light image. A subject detection device according to configuration 5, characterized by the features described herein. (Composition 8) If the calculation means determines in the first determination that the correction parameter has been set to a value corresponding to a predetermined or greater intensity of the atmospheric disturbance, it performs the second determination. A subject detection device according to configuration 6 or 7, characterized by the above. (Composition 9) If the calculation means determines in the first determination that the correction parameter is set to a value corresponding to a strength of less than a predetermined level of the effect of the atmospheric disturbance, it sets the reliability of the first detection result to be greater than the reliability of the second detection result. A subject detection device according to any one of the configurations 6 to 8, characterized in that it is a subject detection device. (Composition 10) In the calculation means, if the distance to the subject is below a threshold in the second determination, the confidence level of the second detection result is set to be greater than the confidence level of the first detection result. A subject detection device according to any one of the configurations 6 to 9, characterized in that it is a subject detection device. (Composition 11) The correction means sets the correction parameters based on user instructions. A subject detection device according to any one of the configurations 6 to 10, characterized in that it is a subject detection device. (Composition 12) The information relating to the subject is at least one of the size, orientation, position, and distance to the subject. A subject detection device according to any one of configurations 1 to 11, characterized by the above. (Composition 13) It further has a third means for acquiring external environmental information, The calculation means further calculates the reliability based on the acquired external environmental information. A subject detection device according to any one of configurations 1 to 12, characterized by the above. (Composition 14) The correction means performs correction on the region of the visible light image that is affected by haze. A subject detection device according to any one of configurations 1 to 13, characterized by the above. (method) A method performed by a subject detection device, A first acquisition step involves acquiring a visible light image including the subject, A second acquisition step for acquiring information about the subject that is different from the visible light image, A correction step is performed on the region of the visible light image that is affected by atmospheric interference. A first detection step is performed to perform a first detection of the subject from the visible light image, A second detection step is performed to perform a second detection of the subject based on the information relating to the subject, A calculation means for calculating the reliability of the detection results of the first detection and the second detection performed on the visible light image corrected by the correction step, The process includes a determination step of determining the subject based on the detection results of the first detection and the second detection, and the calculated confidence level. A method characterized by the following: (program) A program that causes a computer to perform the above method. [Explanation of symbols]

[0077] 100: Subject detection device 101: Visible light image acquisition unit 102: Radar Information Acquisition Unit 103: Haze Correction Processing Unit 104: First detection unit 105: Second detection unit 106: Confidence Calculation Unit 107: Subject discrimination section

Claims

1. A first acquisition means for acquiring a visible light image including the subject, A second acquisition means for acquiring information about the subject that is different from the visible light image, A correction means for correcting the region of the visible light image affected by atmospheric interference, A first detection means for performing a first detection of the subject from the visible light image, A second detection means for detecting a subject from information relating to the subject, A calculation means for calculating the reliability of the detection results of the first detection and the second detection performed on the visible light image corrected by the correction means, The system includes a discrimination means for determining the subject based on the detection results of the first detection and the second detection, and the calculated confidence level. A subject detection device characterized by the following features.

2. The correction means performs the correction based on the histogram distribution of the brightness values ​​of the visible light image. The subject detection device according to feature 1.

3. The correction means performs the correction based on the contrast value of the visible light image. The subject detection device according to feature 1.

4. The correction means performs the correction based on the estimation result of atmospheric transmittance using the Dark Channel Prior method. The subject detection device according to feature 1.

5. The calculation means performs a first determination regarding the setting of correction parameters in the correction for suppressing the effect of atmospheric interference on the visible light image. The subject detection device according to feature 1.

6. The calculation means performs a second determination regarding the distance to the subject from the information about the subject. The subject detection device according to claim 5.

7. The calculation means performs a second determination regarding the noise contained in the image of the subject in the visible light image. The subject detection device according to claim 5.

8. If the calculation means determines in the first determination that the correction parameter has been set to a value corresponding to a predetermined or greater intensity of the atmospheric disturbance, it performs the second determination. The subject detection device according to feature 6.

9. If the calculation means determines in the first determination that the correction parameter is set to a value corresponding to a strength of less than a predetermined level of the effect of the atmospheric disturbance, it sets the reliability of the first detection result to be greater than the reliability of the second detection result. The subject detection device according to feature 6.

10. In the second determination, the calculation means sets the confidence level of the second detection result to be greater than the confidence level of the first detection result if the distance to the subject is below a threshold. The subject detection device according to feature 6.

11. The correction means sets the correction parameters based on user instructions. The subject detection device according to feature 6.

12. The information relating to the subject is at least one of the size, orientation, position, and distance to the subject. The subject detection device according to feature 1.

13. It further has a third means for acquiring external environmental information, The calculation means further calculates the reliability based on the acquired external environmental information. The subject detection device according to feature 1.

14. The correction means is a correction means that performs correction on the region of the visible light image that is affected by haze. The subject detection device according to feature 1.

15. A method performed by a subject detection device, A first acquisition step involves acquiring a visible light image including the subject, A second acquisition step for acquiring information about the subject that is different from the visible light image, A correction step is performed on the region of the visible light image that is affected by atmospheric interference. A first detection step is performed to perform a first detection of the subject from the visible light image, A second detection step is performed to perform a second detection of the subject based on the information relating to the subject, A calculation means for calculating the reliability of the detection results of the first detection and the second detection performed on the visible light image corrected by the correction step, The process includes a determination step of determining the subject based on the detection results of the first detection and the second detection, and the calculated confidence level. A method characterized by the following:

16. A program that causes a computer to perform the method described in claim 15.

Images