Image processing device and image processing method

Abstract

To provide a technique for enabling exposure control suitable for subject detection.SOLUTION: An image processing device identifies an area of a subject detected from a first image obtained by imaging using non-visible light that does not correspond to an area of the subject detected from a second image obtained by imaging using visible light, and controls exposure for imaging using visible light on the basis of the identified area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to image processing technology. [Background technology]

[0002] An imaging system having a visible light camera and a non-visible light (infrared) camera is known (for example, Patent Document 1). In Patent Document 1, the visible light camera and the non-visible light (infrared) camera capture images of a target area for object detection from approximately the same angle and with approximately the same imaging magnification. In Patent Document 1, object detection is performed in both a visible light image captured by the visible light camera and an infrared image (temperature distribution image) captured by the non-visible light camera. In Patent Document 1, the scores of the detection results for the infrared image and the visible light image are weighted and added based on the surrounding infrared energy, the area where the detection target exists is determined based on the result, and exposure correction of the visible light image is performed based on that area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6356925 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not provide detailed descriptions of exposure correction methods when multiple subjects are detected. Patent Document 1 states that when exposure correction is performed on multiple subjects simultaneously, suitable exposure correction may not be performed on all of the subjects depending on the brightness of the subjects. Furthermore, even if exposure correction is performed on selected subjects, suitable exposure may not be achieved for other subjects. The present invention provides a technology that enables exposure control suitable for subject detection. [Means for solving the problem]

[0005] One aspect of the present invention is characterized by comprising an identification means for identifying an area of ​​a subject detected from a first image obtained by imaging using non-visible light that does not correspond to an area of ​​a subject detected from a second image obtained by imaging using visible light, and a control means for controlling exposure for imaging using visible light based on the area identified by the identification means. [Effects of the Invention]

[0006] According to the present invention, it is possible to realize exposure control suitable for subject detection. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 2] 4 is a flowchart of exposure control of the visible light camera 101 by the image processing device 103. [Figure 3] FIG. 1A is a diagram showing an example of a visible light image, and FIG. 1B is a diagram showing an example of a non-visible light image. [Figure 4] FIG. 1A is a diagram showing an example of a visible light image, and FIG. 1B is a diagram showing an example of a non-visible light image. [Figure 5] FIG. 1A is a diagram showing an example of a visible light image, and FIG. 1B is a diagram showing an example of a non-visible light image. [Figure 6] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 7] 10 is a flowchart of a subject tracking process performed by the image processing device 103. [Figure 8] FIG. 1A is a diagram showing an example of a visible light image, and FIG. 1B is a diagram showing an example of a non-visible light image. [Figure 9] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device applicable to the image processing device 103. [Figure 10] FIG. 1A is a diagram showing an example of a visible light image, and FIG. 1B is a diagram showing an example of a non-visible light image. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] [First embodiment] An example of the functional configuration of a system according to this embodiment will be described with reference to the block diagram of FIG. 1. As shown in FIG. 1, the system according to this embodiment includes a visible light camera 101, which is an imaging device that captures images using visible light, a non-visible light camera 102, which is an imaging device that captures images using non-visible light that cannot be captured by the visible light camera 101, and an image processing device 103 that controls the exposure of the visible light camera 101. The visible light camera 101 and the image processing device 103 are connected directly or indirectly via a network such as a LAN or the Internet. Similarly, the non-visible light camera 102 and the image processing device 103 are connected directly or indirectly via a network such as a LAN or the Internet. However, the connection between the visible light camera 101 and the image processing device 103 and between the non-visible light camera 102 and the image processing device 103 are not limited to a specific connection form.

[0010] The visible light camera 101 includes an imaging optical system including one or more lenses, a visible light imaging element (visible light sensor) that captures an optical image formed by the imaging optical system and converts it into an electrical signal, and an image processing circuit that generates a captured image based on the electrical signal. The visible light sensor detects visible light with wavelengths ranging from approximately 380 nm to approximately 750 nm, for example. The visible light sensor may be sensitive to at least a portion of the near-infrared wavelength range.

[0011] As described above, the non-visible light camera 102 is an imaging device that captures images using non-visible light that cannot be captured by the visible light camera 101. Non-visible light includes, for example, infrared light, millimeter waves, and terahertz waves. In this embodiment, the non-visible light camera 102 captures infrared light. The non-visible light camera 102 includes an imaging optical system including one or more lenses, an infrared imaging element (infrared sensor) that captures an optical image formed by the imaging optical system and converts it into an electrical signal, and an image processing circuit that generates a captured image based on the electrical signal. The infrared sensor detects infrared light with a wavelength ranging from 0.83 μm to 1000 μm, for example. In this embodiment, the infrared sensor detects far-infrared light with a wavelength ranging from 6 μm to 1000 μm. The infrared sensor may be a thermal infrared sensor such as a microbolometer or an SOI (Silicon on Insulator) diode. The visible light camera 101 and the non-visible light camera 102 capture images of approximately the same imaging area.

[0012] Next, we will explain the image processing device 103. The image processing device 103 controls the exposure of the visible light camera 101 based on an area of ​​the subject detected from the captured image obtained by the non-visible light camera 102 that does not correspond to the area of ​​the subject detected from the captured image obtained by the visible light camera 101.

[0013] The acquisition unit 201 acquires, as a visible light image, an image captured by the visible light camera 101. The acquisition unit 202 acquires, as a non-visible light image, an image captured by the non-visible light camera 102.

[0014] The detection unit 203 detects a subject from the visible light image. Various methods can be used to detect a subject from the visible light image, such as a pattern matching method, a method using a brightness gradient within a local region, or a method based on machine learning such as deep learning. In this embodiment, the detection unit 203 detects a subject from the visible light image using, as an example, a trained model that has been trained in advance by deep learning.

[0015] The detection unit 204 detects a subject from the non-visible light image. The method for detecting a subject from the non-visible light image may be the same as the method for detecting a subject from a visible light image, or may be a method different from the method for detecting a subject from a visible light image. In the present embodiment, as an example, the detection unit 204 detects a subject from the non-visible light image using a method similar to that used by the detection unit 203.

[0016] Note that even if the detection unit 203 and the detection unit 204 use the same subject detection method, the trained machine learning model, such as deep learning, may be different or the same between the detection unit 203 and the detection unit 204. In this embodiment, as an example, the detection unit 203 performs subject detection processing on a visible light image using a "trained model (first model) that has been trained to detect people from images using facial and color features." Also, in this embodiment, as an example, the detection unit 204 performs subject detection processing on an invisible light image using a "trained model (second model) that has been trained to detect people from images using the silhouette of a human body." Furthermore, in this embodiment, detection reliability is also calculated for the region of the detected subject. Here, the detection reliability is calculated as a value normalized between 0 and 1, and the larger the value, the more likely it is that the subject is the detection target.

[0017] The comparison unit 205 identifies an area of ​​the subject detected by the detection unit 204 that does not correspond to the subject area detected by the detection unit 203, and generates area information that defines the identified area.

[0018] The exposure control unit 206 determines the amount of exposure correction for the visible light camera 101 based on the area in the visible light image defined by the area information, and controls the exposure of the visible light camera 101 based on the amount of correction.

[0019] Next, exposure control of the visible light camera 101 by the image processing device 103 will be described with reference to the flowchart in Fig. 2. Note that in the flowchart in Fig. 2 as well as in the flowcharts used in the following description, the order of some of the processes may be changed as appropriate, or some of the processes may be executed in parallel.

[0020] In step S301, the acquisition unit 201 acquires, as a visible light image, an image captured by the visible light camera 101. In step S302, the acquisition unit 202 acquires, as a non-visible light image, an image captured by the non-visible light camera 102.

[0021] In step S303, the detection unit 203 inputs the visible light image acquired in step S301 into a first model and performs calculations on the first model, thereby detecting one or more subjects from the visible light image.

[0022] In step S304, the detection unit 204 inputs the non-visible light image acquired in step S302 into a second model and performs calculations on the second model, thereby detecting one or more subjects from the non-visible light image.

[0023] Fig. 3(a) shows an example of a visible light image captured of a scene in a dark environment at night, with a spotlight 401 lit, a person 402 in the dark area, and a person 403 under the light. Fig. 3(b) shows an example of an invisible light image captured of such a scene.

[0024] In the visible light image of Fig. 3(a), person 402 cannot be detected due to crushed blacks, and person 403 cannot be detected due to blown-out highlights. On the other hand, in the invisible light image of Fig. 3(b), far-infrared light is captured, so it is less affected by lighting and it is possible to detect both person 402 and person 403. In the example of Fig. 3(b), the area within frame 404 is detected as the area of ​​person 402, and the area within frame 405 is detected as the area of ​​person 403.

[0025] In step S305, comparison unit 205 identifies, as non-corresponding regions, regions other than a detected subject region (described later) in the region of the subject detected from the non-visible light image in step S304 that do not correspond to the region of the subject detected from the visible light image in step S303. Various methods can be used to identify non-corresponding regions.

[0026] For example, the comparison unit 205 calculates the IoU (Intersection over Union) between the region of the subject of interest detected from the non-visible light image (first region) and the regions of all subjects detected from the visible light image (second region), and if any of the calculated IoUs is less than a threshold, the comparison unit 205 identifies the first region as a non-corresponding region. The comparison unit 205 performs this process for all subject regions detected from the non-visible light image. Note that a measure other than IoU may be used as the ratio of overlap between the first region and the second region.

[0027] In the example of Fig. 3, since no subject is detected in the visible light image of Fig. 3(a), there is no area for calculating the IoU with the area within frame 404 or the area within frame 405 in the non-visible light image of Fig. 3(b). Therefore, in this case, the area within frame 404 or the area within frame 405 is identified as a non-corresponding area.

[0028] The comparison unit 205 then generates information defining the non-corresponding region in the non-visible light image as region information. The region information may be, for example, information indicating the coordinate positions of the upper left and lower right corners of the non-corresponding region, or information indicating the coordinate position of the upper left corner of the non-corresponding region and the vertical and horizontal sizes of the non-corresponding region.

[0029] In step S306, the exposure control unit 206 identifies the region in the visible light image defined by the region information, that is, the region in the visible light image that corresponds to the non-corresponding region, as a region subject to exposure control.

[0030] In step S307, the exposure control unit 206 calculates the average value (average brightness value) of the brightness values ​​of the exposure control target area. For example, the exposure control unit 206 calculates the average brightness value of the exposure control target area according to the following equation (1): Calculate TIFF2025174827000002.tif1014.

[0031]

number

[0032] … (1) Here, I(x, y) represents the luminance value of the pixel at coordinate position (x, y) in the visible light image (the horizontal direction is the x-axis direction, and the vertical direction is the y-axis direction). f represents the number of exposure control target areas, s represents the index of the exposure control target area, ks represents the horizontal size of the exposure control target area with index s, and ls represents the vertical size of the exposure control target area with index s. vs represents the x-coordinate position of the center pixel in the exposure control target area with index s, and hs represents the y-coordinate position of the center pixel in the exposure control target area with index s.

[0033] Next, the exposure control unit 206 determines the exposure correction amount EVcorrection based on the average brightness value. First, the exposure control unit 206 calculates the average brightness value EVcorrection according to the following equation (2): A difference value ΔDiff between TIFF2025174827000004.tif1014 and the target luminance value Iobjecttarget is calculated.

[0034]

number

[0035] … (2) The target brightness value Iobjecttarget may be set arbitrarily by the user, or may be set to a value that increases accuracy in consideration of subject detection and detection accuracy.

[0036] Next, the exposure control unit 206 determines the correction amount EVcorrection according to the following equation (3): EVcurrent is an APEX-converted EV value based on the subject luminance value (BV value), and is set based on a program diagram related to exposure control that is stored in advance in the image processing device 103.

[0037]

number

[0038] … (3) Here, β is a preset coefficient that affects the degree (speed) of exposure correction when correcting exposure to underexposure or overexposure, centered around the current exposure value EVcurrent, and Th is a preset threshold value.

[0039] By setting the value of β large, the processing speed (or time) required for the exposure to reach the target will be faster, but if an erroneous detection occurs in the detection result or if the subject detection is unstable, the brightness of the entire screen will fluctuate sharply. On the other hand, by setting the value of β small, the processing speed (or time) required for the exposure to reach the target will be slower, but the system will be more robust to erroneous detection and shooting conditions. β is set as an exposure correction value for the current exposure value EVcurrent when the difference value ΔDiff is equal to or greater than the threshold value Th.

[0040] The exposure control unit 206 then sets an exposure setting value that satisfies the correction amount EVcorrection for the visible light camera 101, thereby controlling the exposure of the visible light camera 101. The processing of step S307 ends when EVcorrection=EVcurrent in equation (3).

[0041] The processes from step S308 onwards are performed after the exposure control of the visible light camera 101 in step S307. In step S308, the acquisition unit 201 acquires, as a visible light image, the image captured by the visible light camera 101. In step S309, the acquisition unit 202 acquires, as a non-visible light image, the image captured by the non-visible light camera 102.

[0042] In step S310, the detection unit 203 inputs the visible light image acquired in step S308 into a first model and performs calculations on the first model, thereby detecting one or more subjects from the visible light image.

[0043] In step S311, the detection unit 204 inputs the non-visible light image acquired in step S309 into a second model and performs calculations on the second model, thereby detecting one or more subjects from the non-visible light image.

[0044] In step S312, comparison unit 205 identifies, as a corresponding region, a region of the subject detected in the non-visible light image in step S311 that corresponds to the subject region detected in the visible light image in step S310. Various methods can be used to identify the corresponding region.

[0045] For example, the comparison unit 205 calculates the IoU (Intersection over Union) between the region of the subject of interest detected from the non-visible light image (first region) and the regions of all subjects detected from the visible light image (second region), and if any of the calculated IoUs is equal to or greater than a threshold, the comparison unit 205 identifies the first region as a corresponding region. The comparison unit 205 performs this process for all subject regions detected from the non-visible light image. Note that a measure other than IoU may be used as the ratio of overlap between the first region and the second region.

[0046] The comparison unit 205 then determines the identified corresponding region as the region of the detected subject (detected subject region) and stores region information defining the detected subject region in memory. The region information defining the detected subject region may be, for example, information indicating the coordinate positions of the upper left and lower right corners of the detected subject region, or information indicating the coordinate position of the upper left corner and the vertical and horizontal sizes of the detected subject region. The memory that stores the region information defining the detected subject region may be a memory within the image processing device 103, or may be a memory device connected to the image processing device 103 or capable of communicating with the image processing device 103.

[0047] The processing in step S312 will be described using the specific example shown in Fig. 4. Fig. 4(a) shows an example of a visible light image of the same scene as in Fig. 3 captured by the visible light camera 101 after the exposure control in step S307. Fig. 4(b) shows an example of a non-visible light image of the same scene captured by the non-visible light camera 102.

[0048] 4(a), a person 402 is detected in the visible light image captured after exposure control, and the area within frame 406 is detected as the area of ​​person 402. If the IoU between the area within frame 404 and the area within frame 406 is equal to or greater than a threshold, the area within frame 404, i.e., the area of ​​person 402, is determined to be a detected subject area, and area information about the detected subject area is stored in memory.

[0049] In step S313, comparison unit 205 determines whether region information for all subject regions in the non-visible light image has been stored in memory. If the result of this determination is that region information for all subject regions in the non-visible light image has been stored in memory, the processing according to the flowchart in Fig. 2 ends. On the other hand, if there are still subject regions in the non-visible light image for which region information has not yet been stored in memory, the processing proceeds to step S305.

[0050] In the example of FIG. 4, the region information for the region within frame 405 in the non-visible light image has not yet been stored in memory, so the process proceeds to step S305. In the case of FIG. 4, the region within frame 405 is a non-corresponding region, so in step S305 the region within frame 405 is identified as a non-corresponding region, and region information for the non-corresponding region is generated. In step S306, the exposure control unit 206 identifies a region in the visible light image that corresponds to the "region within frame 405" as a region subject to exposure control. Then, in step S307, the exposure control unit 206 controls the exposure of the visible light camera 101 based on the exposure control region. After this exposure control, the processes of steps S308 to S312 are performed.

[0051] The processing in step S312 for the second time will be described using the specific example shown in Fig. 5. Fig. 5(a) shows an example of a visible light image of the same scene as in Fig. 3 captured by the visible light camera 101 after the exposure control in step S307 for the second time. Fig. 5(b) shows an example of a non-visible light image of the same scene captured by the non-visible light camera 102.

[0052] 5(a), a person 403 is detected in the visible light image captured after exposure control, and the area within frame 407 is detected as the area of ​​person 403. If the IoU between the area within frame 405 and the area within frame 407 is equal to or greater than a threshold, the area within frame 405, i.e., the area of ​​person 403, is determined to be a detected subject area, and information defining this detected subject area is stored in memory.

[0053] 5, the region information for all subject regions in the non-visible light image has been stored in memory, so in step S313 the comparison unit 205 determines that the region information for all subject regions in the non-visible light image has been stored in memory, and therefore the processing according to the flowchart in FIG. 2 ends.

[0054] In this manner, in this embodiment, exposure control is performed based on the area of ​​the subject detected in the non-visible light image that is not detected in the visible light image. Furthermore, by recording a subject that has been detected once as a detected subject, it is possible to prevent exposure control from being repeatedly performed on the same subject. This makes it possible, for example, to sequentially provide visible light images with exposure appropriately controlled for multiple detected subjects to a subsequent system (e.g., a system that authenticates people based on facial features, clothing color, etc.).

[0055] In the present embodiment, a case where two cameras, a visible light camera and a non-visible light camera, are used has been described as an example. However, the present invention is not limited to this. For example, a single imaging device having two sensors, a visible light sensor and a non-visible light sensor, may be used, or an imaging device having a single sensor with pixels for visible light and pixels for non-visible light may be used. In this way, regardless of the method for acquiring a visible light image or a non-visible light image, the image processing device 103 identifies an area of ​​the subject detected from a first image obtained by imaging using non-visible light that does not correspond to an area of ​​the subject detected from a second image obtained by imaging using visible light, and controls the exposure for imaging using visible light based on the identified area.

[0056] In addition, in this embodiment, the case has been described where the visible light camera 101, the invisible light camera 102, and the image processing device 103 are separate devices. However, the visible light camera 101, the invisible light camera 102, and the image processing device 103 may be integrated into a single device.

[0057] Furthermore, in this embodiment, the case where the subject is a person has been described as an example, but the subject is not limited to a specific subject, and various objects such as animals, ships, and cars may also be used as the subject.

[0058] The exposure control unit 206 may output the visible light image captured by the visible light camera 101 after exposure control. For example, the exposure control unit 206 may transmit the visible light image captured by the visible light camera 101 after exposure control to an external device via a network such as a LAN or the Internet. The external device is, for example, a device / system that authenticates a person using the person's facial features, clothing color, etc.

[0059] Furthermore, the exposure control unit 206 may cause the visible light image captured by the visible light camera 101 after exposure control to be displayed on a display unit of the image processing device 103, or may store the visible light image in a memory of the image processing device 103.

[0060] The exposure control unit 206 may output a non-visible light image in addition to or instead of the visible light image. As with the visible light image, the output destination and output form of the non-visible light image are not limited to a specific output destination or a specific output form.

[0061] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. The system according to this embodiment is a system that suppresses repeated exposure control for the same subject by tracking a detected subject using the technology according to the first embodiment.

[0062] An example of the functional configuration of the system according to this embodiment will be described using the block diagram of Fig. 6. In Fig. 6, functional units similar to those shown in Fig. 1 are given the same reference numerals as those functional units, and descriptions of those functional units will be omitted.

[0063] The subject tracking unit 207 performs subject tracking processing, which is processing for tracking the area (detected subject area) indicated by the "area information of the detected subject area" stored in memory by the comparison unit 205, based on the characteristics of the subject, positional continuity, etc.

[0064] Next, the subject tracking process performed by the image processing device 103 will be described with reference to the flowchart in Fig. 7. In Fig. 7, the same processing steps as those in Fig. 2 are assigned the same step numbers as those in Fig. 2, and descriptions of those processing steps will be omitted.

[0065] In step S501, the subject tracking unit 207 performs subject tracking processing, which is processing for tracking the area (detected subject area) indicated by the "area information of the detected subject area" stored in the memory in step S312, based on the features of the subject, positional continuity, etc. In the example of Fig. 4, the subject tracking unit 207 performs subject tracking processing for tracking the area within the frame 404.

[0066] In step S502, comparison unit 205 determines whether region information defining all regions of the subject in the non-visible light image has been stored in memory. If this determination determines that region information defining all regions of the subject in the non-visible light image has been stored in memory, processing according to the flowchart in Fig. 7 ends. On the other hand, if it determines that there are still regions of the subject in the non-visible light image for which region information has not yet been stored in memory, processing proceeds to step S305.

[0067] In the example of FIG. 4, region information for the region within frame 405 in the non-visible light image has not yet been stored in memory, so the process proceeds to step S305. In the case of FIG. 4, the region within frame 405 is a non-corresponding region, so in step S305 the region within frame 405 is identified as a non-corresponding region, and region information for the non-corresponding region is generated. In step S306, the exposure control unit 206 identifies a region in the visible light image that corresponds to the "region within frame 405" as a region subject to exposure control. Then, in step S307, the exposure control unit 206 controls the exposure of the visible light camera 101 based on the exposure control region. After this exposure control, the processes from step S308 onwards are performed.

[0068] The processing from step S308 onwards for the second time will be described with reference to a specific example shown in Fig. 8. Fig. 8(a) shows an example of a visible light image captured by the visible light camera 101 of the same scene as in Fig. 3 (except that the person 402 is moving) after the exposure control in step S307 for the second time. Fig. 8(b) shows an example of a non-visible light image captured of the same scene by the non-visible light camera 102.

[0069] 8(a), a person 403 is detected in the visible light image captured after exposure control, and the area within frame 407 is detected as the area of ​​person 403. If the IoU between the area within frame 405 and the area within frame 407 is equal to or greater than a threshold, the area within frame 405, i.e., the area of ​​person 403, is determined to be a detected subject area, and information defining this detected subject area is stored in memory.

[0070] 8(b), since person 402 is moving, area information for the area of ​​person 402 after the movement (area within frame 404) is not stored in memory, but since the area within frame 404 is the tracking target, in this case it is determined that "area information for the area within frame 404 is stored in memory." Therefore, at this point, area information for each of the areas within frame 404 and area 405 has been stored in memory, and the processing according to the flowchart in FIG. 7 ends.

[0071] In step S501, the subject tracking unit 207 may track the area of ​​the invisible light image corresponding to the area for which exposure control by the exposure control unit 206 has been completed, based on the characteristics of the subject, the continuity of the position, etc. By doing so, it is possible to prevent repeated exposure adjustment to the same area, for example, even if there is an erroneous detection determination of the invisible light image (the subject cannot be detected in the visible light image even after exposure control).

[0072] [Third embodiment] The following describes the differences from the first and second embodiments, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first and second embodiments. The system according to this embodiment is a system in which, when multiple regions are identified as subject regions, the comparison unit 205 generates region information of non-corresponding regions that satisfy conditions as regions to be processed preferentially in the processing from step S306 onwards. The processing from step S301 to step S304 in Figure 2 is the same as in the first embodiment, so description thereof will be omitted.

[0073] In step S305, the comparison unit 205 identifies, as a non-corresponding area, an area other than a detected object area (described later) in the area of ​​the object detected in the non-visible light image in step S304 that does not correspond to the area of ​​the object detected in the visible light image in step S303. As in the first embodiment, various methods can be applied to identify a non-corresponding area.

[0074] Here, if multiple areas are identified as non-corresponding areas, the comparison unit 205 may generate area information for the non-corresponding areas that satisfy the conditions as areas to be processed preferentially or limitedly in the processing from step S306 onwards.

[0075] For example, the comparison unit 205 may generate non-corresponding regions by prioritizing regions with higher detection reliability based on the detection reliability obtained from the detection unit 204. Generating non-corresponding regions in this manner can reduce the impact of erroneous detection of the non-visible light image, which is likely to occur in no small part in step S304. In the example of FIG. 10 , no subject is detected in the visible light image of FIG. 10( a), so there are no regions for calculating the IoU with the regions within frame 405, frame 406, and frame 407 in the non-visible light image of FIG. 10( b). Therefore, in this case, the regions within frame 405, 406, and 407 are identified as non-corresponding regions. Here, as shown in FIG. 10( b), when the subject detection target in each region is a person, the detection reliability is set to 1.0 for person 403, 0.8 for person 402, and 0.2 for animal 404. By generating non-corresponding regions only for regions with detection reliability exceeding a certain threshold of 0.5, subsequent processing is not performed on animal 404, which is not a person, thereby reducing erroneous detection. Alternatively, a threshold value may not be set, and processing may be prioritized for those with high detection reliability, while processing for those with low detection reliability may be postponed.

[0076] In addition, for example, if the brightness of non-corresponding regions differs significantly, it may not be possible to simultaneously control the exposure to an optimal level for multiple non-corresponding regions. In such cases, the comparison unit 205 may generate region information by prioritizing non-corresponding regions that are determined to have similar brightness using a specified index. By generating region information in this manner, it is possible to reduce the amount of exposure change in step S306 (described below) and subsequent steps, enabling exposure control to be performed quickly.

[0077] Alternatively, for example, the comparison unit 205 may generate region information by prioritizing subject regions that move quickly between frames, or by prioritizing subjects that are close to the edge of the angle of view. This is based on the idea of ​​generating region information for subjects that tend to move outside the angle of view when they move. For example, if a fast-moving subject crosses the frame, it becomes possible to recognize that subject with priority.

[0078] Alternatively, for example, the comparison unit 205 may generate region information by prioritizing multiple subject regions based on the learning model used in the detection unit 204. In this embodiment, as described above, detection is performed using a first trained model using face and color information and a second trained model using a human body silhouette. Here, compared with the detection results of the second trained model, the first trained model is capable of detecting details of the subject and is considered to have a higher probability of being the relevant subject. Therefore, for example, the comparison unit 205 may generate region information by prioritizing regions detected by the first trained model over the second trained model. The processing from step S306 onwards is the same as in the first embodiment, and therefore description thereof will be omitted.

[0079] As described above, in this embodiment, when multiple regions are identified as non-corresponding regions, region information that satisfies the conditions is prioritized or limited and generated as the region for subsequent processing. By performing such processing, even when there are multiple recognition targets on the screen, it is possible to limit the subjects to be recognized and to perform subsequent recognition processing in order of subjects considered to have the highest priority.

[0080] [Fourth embodiment] Each functional unit of the image processing device 103 shown in Figures 1 and 6 may be implemented by hardware or software (computer program). In the former case, each functional unit may be implemented by hardware such as an ASIC or a programmable logic array (PLA). ASIC stands for Application Specific Integrated Circuit. Note that some of the functional units may be implemented by hardware.

[0081] In the latter case, a computer device capable of executing such a computer program is applicable to the image processing device 103. An example of the hardware configuration of a computer device applicable to the image processing device 103 will be described with reference to the block diagram of Fig. 9. Applicable computer devices include PCs, tablet terminal devices, and smartphones.

[0082] The CPU 901 executes various processes using computer programs and data stored in the RAM 902 and ROM 903. As a result, the CPU 901 controls the operation of the entire computer device, and executes or controls the various processes described as processes performed by the image processing device 103. Note that a programmable processor such as an MPU may be used instead of the CPU 901. CPU stands for Central Processing Unit. MPU stands for Micro-Processing Unit.

[0083] The RAM 902 has an area for storing computer programs and data loaded from the ROM 903 or the storage device 906, and an area for storing computer programs and data received from the outside via the I / F 907. The RAM 902 also has a work area used by the CPU 901 when executing various processes. In this way, the RAM 902 can provide various areas as needed.

[0084] The ROM 903 stores setting data for the computer device, computer programs and data relating to the startup of the computer device, computer programs and data relating to the basic operation of the computer device, and the like.

[0085] The operation unit 904 is a user interface such as a keyboard, a mouse, a touch panel screen, etc., and allows the user to input various instructions and information to the computer device by operating it.

[0086] The display unit 905 has a liquid crystal screen or a touch panel screen, and can display the processing results by the CPU 901 as images, text, etc. The display unit 905 may also be a projection device such as a projector that projects images and text.

[0087] The storage device 906 is a large-capacity information storage device such as a hard disk drive. The storage device 906 stores an OS (operating system), computer programs and data for causing the CPU 901 to execute or control the various processes described as processes performed by the image processing device 103. The computer programs stored in the storage device 906 may include computer programs for causing the CPU 901 to execute or control the functions of the functional units of the image processing device 103 shown in FIGS. 1 and 6. The data stored in the storage device 906 may also include known parameters such as the threshold values ​​described above. Images captured by the visible light camera 101 and the invisible light camera 102 may be stored in the storage device 906, and the CPU 901 may read and process the images as needed. The above-mentioned memories mentioned in the descriptions of the first and second embodiments may be, for example, the RAM 902 or the storage device 906.

[0088] The I / F 907 is a communication interface for performing data communication with an external device via a network such as a LAN or the Internet. For example, the computer device can acquire images captured by the visible light camera 101 or the invisible light camera 102 via the I / F 907. The computer device can also output various pieces of information described above as information output by the exposure control unit 206 to the external device via the I / F 907.

[0089] The CPU 901, RAM 902, ROM 903, operation unit 904, display unit 905, storage device 906, and I / F 907 are all connected to a system bus 908. Note that the hardware configuration of a computer applicable to the image processing device 103 is not limited to the configuration shown in Fig. 9, and can be modified / changed as appropriate.

[0090] The system configuration described in each of the above embodiments can be modified or changed as appropriate depending on the specifications of the devices applied to the system and various conditions (usage conditions, usage environment, etc.), and the configurations shown in each of the above embodiments are merely examples.

[0091] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0092] Furthermore, some or all of the above-described embodiments may be used in appropriate combination. Furthermore, some or all of the above-described embodiments may be used selectively. Furthermore, not all of the configurations of the above-described embodiments are necessarily required.

[0093] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0094] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) a specifying means for specifying an area of ​​the subject detected from the first image obtained by imaging using non-visible light that does not correspond to an area of ​​the subject detected from the second image obtained by imaging using visible light; a control means for controlling exposure for imaging using visible light based on the area specified by the specifying means; An image processing device comprising: (Item 2) Item 1. The image processing device according to item 1, wherein the identification means identifies an area of ​​the subject detected from the first image where the overlap rate with the subject detected from the second image is less than a threshold value. (Item 3) 3. The image processing device according to item 1 or 2, wherein the control means controls exposure for imaging using visible light based on an area determined to have similar brightness according to a specified index among the areas identified by the identification means. (Item 4) 3. The image processing device according to item 1 or 2, wherein the control means controls exposure for imaging using visible light based on an area of ​​the subject identified by the identification means, the area having a moving speed equal to or greater than a threshold between frames. (Item 5) 3. The image processing device according to item 1 or 2, wherein the control means controls exposure for imaging using visible light based on the area of ​​the subject identified by the identification means that is within a specified distance from an end of the angle of view. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the control means calculates an average luminance value of the area identified by the identification means, and controls exposure for imaging using visible light based on the difference between the average luminance value and a target luminance value. (Item 7) moreover, 7. The image processing device according to any one of items 1 to 6, further comprising a tracking unit that tracks the area identified by the identifying unit. (Item 8) 8. The image processing device according to item 7, wherein the tracking means tracks an area after exposure control by the control means. (Item 9) The first image and the second image are captured by a single imaging device having a visible light sensor and a non-visible light sensor. 9. The image processing device according to any one of items 1 to 8, characterized in that: (Item 10) The first image and the second image are captured by an imaging device having a single sensor that includes pixels for visible light and pixels for non-visible light. 9. The image processing device according to any one of items 1 to 8, characterized in that: (Item 11) An image processing method performed by an image processing device, a specifying step in which a specifying means of the image processing device specifies an area of ​​the subject detected from the first image obtained by imaging using invisible light that does not correspond to an area of ​​the subject detected from the second image obtained by imaging using visible light; a control step in which a control means of the image processing device controls exposure for imaging using visible light based on the area specified in the specifying step; An image processing method comprising: (Item 12) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 10.

[0095] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0096] 101: Visible light camera 102: Invisible light camera 103: Image processing device 201: Acquisition unit 202: Acquisition unit 203: Detection unit 204: Detection unit 205: Comparison unit 206: Exposure control unit

Claims

1. a specifying means for specifying an area of ​​the subject detected from the first image obtained by imaging using non-visible light that does not correspond to an area of ​​the subject detected from the second image obtained by imaging using visible light; a control means for controlling exposure for imaging using visible light based on the area specified by the specifying means; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the identifying means identifies an area of ​​the subject detected from the first image where the ratio of overlap with the subject area detected from the second image is less than a threshold value.

3. 2. The image processing device according to claim 1, wherein the control means controls exposure for imaging using visible light based on an area determined to have similar brightness according to a specified index among the areas identified by the identification means.

4. 2. The image processing device according to claim 1, wherein the control means controls exposure for imaging using visible light based on an area of ​​the subject identified by the identification means, the area having a moving speed equal to or greater than a threshold between frames.

5. 2. The image processing device according to claim 1, wherein the control means controls exposure for imaging using visible light based on the area of ​​the subject within a specified distance from an end of the angle of view, among the areas identified by the identification means.

6. 2. The image processing device according to claim 1, wherein the control means calculates an average luminance value of the area identified by the identification means, and controls exposure for imaging using visible light based on the difference between the average luminance value and a target luminance value.

7. moreover, 2. The image processing apparatus according to claim 1, further comprising: a tracking unit that tracks the area identified by the identifying unit.

8. 8. The image processing apparatus according to claim 7, wherein said tracking means tracks an area after exposure control by said control means.

9. The first image and the second image are captured by a single imaging device having a visible light sensor and a non-visible light sensor.

2. The image processing device according to claim 1, wherein:

10. The first image and the second image are captured by an imaging device having a single sensor that includes pixels for visible light and pixels for non-visible light.

2. The image processing device according to claim 1, wherein:

11. An image processing method performed by an image processing device, a specifying step in which a specifying means of the image processing device specifies an area of ​​the subject detected from the first image obtained by imaging using non-visible light that does not correspond to an area of ​​the subject detected from the second image obtained by imaging using visible light; a control step in which a control means of the image processing device controls exposure for imaging using visible light based on the area specified in the specifying step; An image processing method comprising:

12. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 10.

Images