Image processing device and image processing method

Abstract

An image processing device includes an exposure control unit, a Δ posture estimation unit, and an integration unit. The exposure control unit controls exposure by sequentially performing positive correction of increasing exposure from an appropriate exposure or negative correction of decreasing exposure from an appropriate exposure in a predetermined execution order as exposure when each of a plurality of image frames is acquired in a time series. The Δ posture estimation unit estimates a first position posture of the image processing device based on matching between image frames subjected to positive correction, and a second position posture of the image processing device based on matching between image frames subjected to negative correction. The integration unit integrates the first position posture and the second position posture.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing method. Background Art

[0002] There is a known technology called simultaneous localization and mapping (SLAM) for performing self-localization and map creation at the same time. SLAM can obtain its own position from a state without prior information such as map information, and thus greatly contributes to autonomous driving in unknown environments of cars, robots, unmanned aerial vehicles (UAVs), etc.

[0003] Furthermore, in SLAM, a technology for performing self-localization and map construction by extracting feature points from a captured image such as a camera image and tracking the extracted feature points in time series is referred to as "visual SLAM".

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP 2016-045874 A,

[0007] Patent document 2: JP2018-112936A,

[0008] Patent document 3: JP2017-118551A. Summary of the invention

[0009] Technical issues

[0010] The self-position estimated by "visual SLAM" or the like is geometrically derived based on the captured image (such as the camera image). Therefore, the success and accuracy of the self-localization depends on the content of the captured image.

[0011] Therefore, the present disclosure proposes an image processing device and an image processing method, which can improve the accuracy of self-positioning without being affected by the content of the captured camera image.

[0012] Solution to the problem

[0013] In order to solve the above-mentioned problems, an image processing device in one form of the present disclosure has an exposure control unit, an estimation unit, and an integration unit. The exposure control unit controls the exposure by sequentially performing positive correction of increasing exposure from an appropriate exposure or negative correction of decreasing exposure from an appropriate exposure in a predetermined execution order as the exposure when each of a plurality of image frames is acquired in a time series. The estimation unit estimates a first position posture of the image processing device based on matching between image frames subjected to positive correction, and a second position posture of the image processing device based on matching between image frames subjected to negative correction. The integration unit integrates the first position posture and the second position posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 are diagrams showing comparative examples of camera images with different exposures.

[0015] Figure 2 is a diagram showing a configuration example of an image processing apparatus according to an embodiment.

[0016] Figure 3 : is a diagram illustrating an operation example of the image processing apparatus according to the embodiment.

[0017] Figure 4 : is a diagram showing an operation example of the image processing apparatus according to the present embodiment.

[0018] Figure 5 : is a flowchart showing an example of a processing procedure of the image processing apparatus according to the present embodiment.

[0019] Figure 6 : is a flowchart showing an example of a processing procedure of the image processing apparatus according to the present embodiment.

[0020] Figure 7 is a diagram showing a configuration example of an image processing device according to a modification.

[0021] Figure 8 is a flowchart showing an example of a processing procedure of an image processing apparatus according to a modification. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. Note that in the following embodiments, there are cases where redundant descriptions are omitted by denoting the same components with the same symbols.

[0023] Note that in the present specification and the drawings, a plurality of components having substantially the same functional configuration can be distinguished by attaching different numbers after the same symbol. For example, when necessary, a plurality of components having substantially the same functional configuration as in the camera 101a and the camera 101b are distinguished. However, in the case where it is not particularly necessary to distinguish each of the plurality of components having substantially the same functional configuration, only the same symbol is attached. For example, in the case where it is not particularly necessary to distinguish the camera 101a and the camera 101b, they are simply referred to as the imaging unit 101.

[0024] Furthermore, the present disclosure will be described in the order of the following items.

[0025] 1. Introduction

[0026] 2. Function configuration example

[0027] 2-1. Operation example of image processing device (1)

[0028] 2-2. Operation example of image processing device (2)

[0029] 3. Processing example

[0030] 4. Modifications

[0031] 4-1. Modifications of device configuration

[0032] 4-2. Modification of the processing procedure

[0033] 4-3. Others

[0034] 5. Conclusion

[0035] <<1. Introduction>>

[0036] Conventionally, in "visual SLAM" and the like, the result of self-localization is geometrically derived based on the information of the scene captured in the camera image. Therefore, whether the self-localization is successful and its accuracy is affected by how much useful information is captured in the image and how much resolution is obtained (whether the geometric error is small).

[0037] For example, in "visual SLAM" and the like, characteristic points (feature points) that can be used for self-positioning are detected from camera images. Feature points typically correspond to areas in the image that have high contrast and no structures similar to the surrounding environment.

[0038] Camera images have a limited dynamic range, and in the case of imaging in a high-contrast environment, overexposed highlights appear as bright spots and occluded shadows appear as dark spots. High-contrast environments correspond, for example, to the outdoors in good weather, indoors such as operating rooms or stages using strong lighting, and looking from indoors to outdoors such as tunnels. It is difficult to extract feature points from camera images where overexposed highlights or occluded shadows appear, and feature points are an obstacle to successful self-localization and its accuracy. If feature points cannot be extracted from the camera image, self-localization fails.

[0039] In the operation of "visual SLAM" and the like, the brightness of the imaging environment may greatly change depending on various conditions such as location, time, and weather. In order to stably achieve self-positioning in an environment with any brightness, it is necessary to maintain the camera image at an appropriate brightness level (avoiding overexposing bright spots and blocking shadows). Therefore, when acquiring a camera image, the exposure of the camera is controlled based on the brightness level of the camera image so as to maintain the brightness of the camera image at an appropriate level.

[0040] However, even if the camera's exposure control is performed, there are cases where overexposed highlights or blocked shadows cannot be avoided in environments with high contrast. In the case where each pixel of a camera image is represented by an 8-bit digital signal, the expressible contrast ratio is 255 times, but in environments with high contrast, areas with overexposed highlights or blocked shadows may occur. Figure 1 are diagrams showing comparative examples of camera images with different exposures.

[0041] like Figure 1 As shown, in Figure 1 The diagram shown on the left side of is an example of a camera image captured in such a manner that the brightness level of the camera image is appropriate, for example, by adjusting the exposure amount (hereinafter referred to as “appropriate exposure”). Figure 1 The diagram shown in the center is an example of a camera image captured by adjusting the exposure ("underexposing") so that the brightness level of the camera image is lower than the appropriate brightness. Figure 1 The diagram shown on the right side of is an example of a camera image captured with the exposure adjusted ("overexposed") so that the brightness of the camera image is higher than the appropriate brightness level.

[0042] As in Figure 1 In the camera image shown in the left diagram of , even when the exposure is adjusted to be appropriate, overexposed highlights appear in bright areas such as the sky, and blocked shadows appear in dark areas such as the shadows of trees. That is, the image is a typical camera image with insufficient dynamic range.

[0043] On the other hand, Figure 1In the camera image shown in the center of , the overexposed bright areas can be prevented by adjusting the exposure to underexpose, but the blocked shadow areas cannot be prevented. Figure 1 In the camera image shown in the right figure of , the appearance of the blocked shadow area can be prevented by adjusting the exposure to overexposure, but the appearance of the overexposed bright area cannot be prevented. As described above, due to the limitation of the dynamic range of the camera, in the camera image captured in an environment with high contrast, there is a case where the appearance of both the overexposed bright area and the blocked shadow area cannot be prevented even if the exposure adjustment is performed.

[0044] As a method for solving the problem of insufficient dynamic range of the camera as described above, there is a method of capturing multiple images with different exposures and combining the images. That is, this technology is a method of obtaining an image without overexposed bright spots or blocked shadows by capturing multiple images with different exposures (such as "underexposed" or "overexposed") (that is, capturing multiple images with different dynamic ranges and combining the images). This technology has no problem in the case where the camera or the subject does not change its position between the multiple images, however, in other cases, there is a problem that items whose positions have changed appear to be shifted and overlap. Although measures such as estimating movement and performing alignment are considered, it is difficult to perform perfect alignment, and errors in alignment appear in the composite image as artifacts such as false edges.

[0045] In addition, as another method to solve the problem of insufficient dynamic range of the camera, there is a method of obtaining a composite image with a high range by combining pixels with different sensitivity characteristics. In this method, by using pixels with different sensitivities, an image that captures bright spots to dark spots can be obtained, that is, an image without overexposed bright spots or blocked shadows can be obtained. In addition, compared with the above-mentioned method of combining multiple images, this method has the following advantages: there is no alignment problem even if the subject, etc. moves. On the other hand, since the imaging surface includes multiple pixels, the pixel interval is wider than that of a single sensitivity pixel. That is, the spatial resolution is reduced. In the case where such an image is used as an input for self-positioning, a quantization error or matching deviation caused by low resolution causes an error in self-positioning, which leads to a problem.

[0046] In view of the above problems, the purpose of the image processing device according to the embodiment of the present disclosure is to improve the accuracy of self-positioning without being affected by the content of the captured camera image. Specifically, the purpose of the image processing device of the present disclosure is to avoid the influence of overexposed bright spots or blocked shadows appearing in the camera image.

[0047] <<2. Functional Configuration Example>>

[0048] Will refer to Figure 2A configuration example of the image processing apparatus 1 according to the embodiment is described. Figure 2 is a diagram showing a configuration example of an image processing apparatus according to the present embodiment.

[0049] like Figure 2 As shown, the image processing device 1 includes an imaging unit 101, an exposure control unit 102, a feature point detection unit 103, a parallax matching unit 104, a distance estimation unit 105, a three-dimensional information storage unit 106, and a two-dimensional information storage unit 107. Figure 2 As shown, the image processing apparatus 1 includes a motion matching unit 108 , a Δ pose estimating unit 109 , a brightness level detecting unit 110 , and an integrating unit 111 .

[0050] Each block (imaging unit 101 to integration unit 111) included in the image processing device 1 is implemented by a controller that controls each unit of the image processing device 1. The controller is implemented by, for example, a processor such as a central processing unit (CPU) or a microprocessing unit (MPU). For example, the controller is implemented by a processor that executes various programs stored in a storage device inside the image processing device 1 using a random access memory (RAM) or the like as a work area. Note that the controller can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Any of the CPU, MPU, ASIC, and FPGA can be regarded as a controller.

[0051] Each block (imaging unit 101 to integration unit 111) included in the image processing device 1 is a functional block indicating the function of the image processing device 1. These functional blocks may be software blocks or hardware blocks. For example, each of the functional blocks described above may be a software module implemented by software (including microprograms), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The functional blocks may be configured in any manner. It should be noted that each block included in the image processing device 1 may be implemented by a processor or an integrated circuit. Figure 2 The examples shown in FIG. 1 are different functional unit configurations.

[0052] The imaging unit 101 is implemented by a stereo camera including a camera 101a and a camera 101b. The camera image captured by the imaging unit 101 is appropriately processed by various corrections such as optical distortion and gain adjustment (such as peripheral dimming correction), and then subjected to parallelization that cancels out the posture deviation between the stereo cameras. The camera image captured by the camera 101a is sent to the parallax matching unit 104 described later. The camera image captured by the camera 101b is sent to the feature point detection unit 103 described later.

[0053] The exposure control unit 102 controls the exposure of the imaging unit 101. Note that controlling the exposure of the imaging unit 101 means controlling the amount of light entering the imaging unit 101 (cameras 101a and 101b). Specifically, the brightness of the camera image is controlled as a result of adjusting the shutter speed (=exposure time), sensitivity, aperture, etc.

[0054] The exposure control unit 102 selectively performs, for example, exposure control based on the brightness level of the camera image and exposure control by exposure correction depending on the size of the area of ​​overexposed bright spots and / or blocked shadow areas included in the camera image.

[0055] For example, in a case where the area of ​​the overexposed bright spot and / or the blocked shadow region included in the camera image is less than or equal to the threshold value, the exposure control unit 102 controls the exposure so that the brightness level of the camera image detected by the brightness level detection unit 110 is constant (at the target level). Based on such a premise, in an environment where the area of ​​the overexposed bright spot and / or the blocked shadow region is less than or equal to the threshold value, that is, in a case where the contrast is not high, in a case where a feature point is detected from an image frame, there is a case where it is advantageous to continue imaging with appropriate exposure.

[0056] When the brightness level is higher than the target level, the exposure control unit 102 controls the exposure of the imaging unit 101 by, for example, increasing the shutter speed so that the camera image becomes darker. On the other hand, when the brightness level is lower than or equal to the target level, the exposure control unit 102 controls the exposure of the imaging unit 101 by, for example, decreasing the shutter speed so that the camera image becomes brighter.

[0057] When feature points cannot be obtained from the camera image, the processing of the feature point detection unit 103, the parallax matching unit 104, and the motion matching unit 108 described later cannot be performed. In addition, in the case where feature points cannot be obtained from the camera image, the estimation of the own position and posture fails. Therefore, as described above, the exposure control unit 102 maintains the brightness of the camera image by controlling the exposure of the imaging unit 101 to an appropriate exposure according to the brightness level of the camera image in order to cope with changes in the brightness of the environment in which the image processing device 1 operates. In this way, the appearance of (a) overexposed bright spots and / or occluded shadow areas in the camera image is suppressed as much as possible, and the feature points of the camera image are prevented from being buried in the overexposed bright spots and / or occluded shadow areas, thereby enabling stable detection of feature points from the camera image.

[0058] Meanwhile, as described above, even when exposure control is performed according to the brightness level of the camera image, in the case where the image processing apparatus 1 is operated in an environment with a high contrast, there are cases where (a) the occurrence of overexposed bright spots and / or blocked shadow areas cannot be avoided due to a lack of dynamic range (see Figure 1 ). In addition, in the above-mentioned method of capturing a plurality of camera images with different exposure levels and combining the camera images, there is a problem in which alignment errors appear as noise on the image. In addition, in the method of obtaining a wide range of synthetic images by combining pixels with different sensitivity characteristics, there is a problem in which quantization errors increase due to reduced resolution. In addition, the processing of the distance estimation unit 105 and the Δ posture estimation unit 109 described later is based on the premise that each point on the camera image is captured at a geometrically correct position. Therefore, in the case where the resolution of the camera image is reduced and there is an error (quantization error) between the points on the camera image, this leads to an error in the estimation result of the own position and posture.

[0059] Therefore, in the case where the area of ​​the overexposed bright spot and / or the blocked shadow area included in the camera image is greater than the threshold value, the exposure control unit 102 sequentially performs a positive correction of increasing the exposure from the proper exposure or a negative correction of reducing the exposure from the proper exposure according to a predetermined execution order. As a result, the exposure when the imaging unit 101 is imaging, that is, the exposure when each of the multiple image frames is acquired in a time series, is controlled. Based on the premise that in an environment where the area of ​​the overexposed bright spot and / or the blocked shadow area is greater than the threshold value, that is, in an environment with high contrast, it is difficult to stably extract feature points from the camera image in imaging under proper exposure. As a result, exposure control of the camera image is performed so as to compensate for the lack of dynamic range and stably detect feature points from the camera image even in an environment with high contrast.

[0060] For example, a mode of alternately performing positive correction (overexposure) or negative correction (underexposure) may be adopted as the predetermined execution order. In this case, when the area of ​​the overexposed bright spot and / or the blocked shadow area included in the camera image is larger than the threshold value, the exposure control unit 102 alternately performs, for example, positive correction (overexposure) and negative correction (underexposure) (see the method described later). Figure 3 ). As a result, exposure correction of positive correction or negative correction is alternately performed in synchronization with the imaging process of the imaging unit 101, and image frames on which positive correction is performed and image frames on which negative correction is performed are alternately acquired.

[0061] For example, in the case where the frame rate is 60 frames / second, the exposure control unit 102 performs exposure control so that underexposure and overexposure are periodically repeated alternately in synchronization with the acquisition of an image frame every sixtieth of a second. That is, as exposure correction of the image frame, underexposure and overexposure are alternately applied to every other frame. The exposure control unit 102 can control the exposure, for example, by calculating an appropriate exposure (exposure value) from the brightness level detected by the brightness level detection unit 110 and applying a positive correction or a negative correction to the appropriate exposure using a predetermined correction value.

[0062] By alternately performing positive correction (overexposure) or negative correction (underexposure) for every image frame, feature points contained in the overexposed highlight area and the occluded shadow area can be effectively detected. Note that regarding the execution order when performing positive correction and negative correction alternately, the execution order of exposure correction in which positive correction and negative correction are performed first can be set according to any desired rule. For example, in the case where only the overexposed highlight area exceeds the threshold, it is conceivable to set the execution order so that the negative correction is performed first so as to detect many feature points including feature points lurking in the overexposed highlight area from the camera image. At the same time, in the case where only the occluded shadow area exceeds the threshold, it is conceivable to set the execution order so that the positive correction is performed first so as to detect many feature points including feature points lurking in the occluded shadow area from the camera image. Note that in the case where both the overexposed highlight area and the occluded shadow area exceed the threshold, which of the positive correction and the negative correction is performed first can be randomly set.

[0063] In addition, the exposure control unit 102 can perform positive correction or negative correction on every other frame based on the number of feature points included in the image frame. As described above, by alternately performing positive correction and negative correction on every other frame, it is possible to know which of the camera image subjected to positive correction and the camera image subjected to negative correction includes more feature points.

[0064] Therefore, the exposure control unit 102 can modify the execution order of the positive correction and the negative correction based on the ratio between the number of feature points included in the image frame subjected to the positive correction and the number of feature points included in the image frame subjected to the negative correction. Generally, the number of feature points detected from the overexposed camera image is compared with the number of feature points detected from the underexposed camera image, and if there is a difference in the number of feature points included in the camera images, it is considered that the estimation of the own position and posture using the image including more feature points is more advantageous.

[0065] Specifically, the exposure control unit 102 determines that the number of feature points included in the image frame subjected to positive correction is greater than the number of feature points included in the image frame subjected to negative correction. In this case, the exposure control unit 102 modifies the execution order of the positive correction and the negative correction so that more image frames subjected to positive correction are acquired than image frames subjected to negative correction. That is, the execution ratio of the positive correction is increased compared to the execution ratio of the negative correction. On the other hand, the exposure control unit 102 determines that the number of feature points included in the image frame subjected to positive correction is less than the number of feature points included in the image frame subjected to negative correction. In this case, the exposure control unit 102 modifies the execution order of the positive correction and the negative correction so that more image frames subjected to negative correction are acquired than image frames subjected to positive correction. That is, the execution ratio of the negative correction is increased compared to the execution ratio of the positive correction.

[0066] For example, the exposure control unit 102 determines that the ratio of the number of feature points included in the image frame subjected to positive correction to the number of feature points included in the image frame subjected to negative correction is 1 to 2. In this case, the exposure control unit 102 modifies the execution order of the positive correction and the negative correction so that the positive correction and the negative correction are sequentially performed at a ratio of 1 to 2. That is, the execution order is modified so that negative correction (underexposure) -> negative correction (underexposure) -> positive correction (overexposure) is periodically repeated (for example, see the following description of the execution order). Figure 4 ). Note that if positive correction and negative correction are performed in the order of a ratio of 1 to 2, the execution order can be modified to any order. For example, the execution order can be modified so that negative correction (underexposure) -> positive correction (overexposure) -> negative correction (underexposure) is repeated periodically.

[0067] Note that the control of the execution ratio of the positive correction and the negative correction by the exposure control unit 102 can also be represented by a loop of inputting these image frames based on the (coordinated exposure) image frames to be input to the Δ posture estimation unit 109. For example, in a case where the frame rate of the image frames acquired by the imaging unit 101 is 60 (fps) and the positive correction (overexposure) and the negative correction (underexposure) are repeated alternately, that is, when the execution ratio of the positive correction (overexposure) / negative correction (underexposure) is 1:1, only the image frames on the positive correction (overexposure) side are focused, and the frame period of the image frames on the positive correction (overexposure) side can be represented as 1 / 30 (seconds). On the other hand, when only the image frames on the negative correction (underexposure) side are focused, the frame period of the image frames on the negative correction (underexposure) side can be represented as 1 / 30 (seconds). In addition, when the frame rate is 60 (fps) and the execution ratio of positive correction (overexposure) to negative correction (underexposure) is 2:1, only the image frame on the positive correction (overexposure) side is focused, and the frame period of the image frame on the positive correction (overexposure) side can be expressed as 1 / 60 (seconds) and 1 / 30 (seconds). On the other hand, when only the image frame on the negative correction (underexposure) side is focused, the frame period of the image frame on the negative correction (underexposure) side can be expressed as 1 / 20 (seconds).

[0068] The feature point detection unit 103 detects feature points that can be used to estimate the own position and posture (the position and posture of the image processing device 1). Feature points generally correspond to areas on the camera image that have high contrast and do not have similar structures around them. Feature points can be matched due to their uniqueness and are used for processing based on the function of the motion matching unit 108 described later. The feature point detection unit 103 stores the positions of each feature point on the camera image captured by the camera 101b as two-dimensional position information in the two-dimensional information storage unit 107.

[0069] For each feature point (first feature point group) on one of the camera images detected by the feature point detection unit 103, the disparity matching unit 104 searches for corresponding points (second feature point group) from the other camera image. The disparity matching unit 104 performs a search for a second feature point group corresponding to the first feature point group from the other camera image by template matching. For example, the first feature point group on the camera image captured by the camera 101a and the second feature point group on the camera image captured by the camera 101b can be rephrased as the same object viewed from two viewpoints. Based on the first feature point group in one camera image and the second feature point group in the other camera image, the disparity matching unit 104 obtains a disparity which is a difference in appearance when the same object is viewed from the camera 101a and the camera 101b.

[0070] The distance estimation unit 105 obtains the distance to each feature point (camera 101a and camera 101b) based on the parallax of each of the feature points obtained by the parallax matching unit 104. If the installation interval between the camera 101a and the camera 101b is known, the distance estimation unit 105 can calculate the distance to each feature point (camera 101a and camera 101b) based on the principle of triangulation. The distance estimation unit 105 obtains the distance to each feature point (camera 101a and camera 101b), and then obtains the position of each feature point in the three-dimensional space and stores the position as three-dimensional position information in the three-dimensional information storage unit 106.

[0071] The three-dimensional position information stored in the three-dimensional information storage unit 106 and the two-dimensional position information stored in the two-dimensional information storage unit 107 are used as a previous camera image used as a reference for a change in the own position posture (posture difference (Δposture)). The three-dimensional information storage unit 106 and the two-dimensional information storage unit 107 are implemented by a storage device capable of reading and writing data, such as DRAM, SRAM, flash memory, or a hard disk.

[0072] When the imaging unit 101 captures the next image in terms of time, the motion matching unit 108 searches for corresponding points from the current camera image for each of the feature points on the previous camera image based on the two-dimensional position information stored in the two-dimensional information storage unit 107. For example, the motion matching unit 108 performs a search for points corresponding to the respective feature points on the previous camera image from the current camera image by template matching. The points corresponding to the respective feature points on the previous camera image, which have been searched from the current camera image, correspond to the points when the same object is viewed from the camera 101a and the camera 101b. The difference in appearance is caused by the temporal change in the position posture of the image processing device 1 (camera 101a and camera 101b). The two-dimensional position information of the points corresponding to the respective feature points on the previous camera image, which have been searched from the current camera image by the motion matching unit 108, is sent to the Δ posture estimation unit 109.

[0073] The Δ posture estimation unit 109 estimates a first position posture of the image processing device 1 based on matching between image frames subjected to positive correction, and a second position posture of the image processing device 1 based on matching between image frames subjected to negative correction. The first position posture and the second position posture correspond to a change (posture difference, also referred to as "Δ posture") of the position posture of the image processing device 1 (camera 101a and camera 101b) from a previous image capturing time to a current image capturing time. The first position posture and the second position posture include, for example, information of three degrees of freedom indicating a position change of the image processing device 1 and information of three degrees of freedom indicating a rotation change of the image processing device 1. The Δ posture estimation unit 109 can estimate the change (posture difference) of the position posture of the image processing device 1 by, for example, the following method based on the three-dimensional position information of each feature point on the previous camera image and the two-dimensional position information of the corresponding point on the current camera image.

[0074] That is, when a point group (three-dimensional position) in a three-dimensional space and a point group (two-dimensional position) on a two-dimensional plane obtained by projecting the point group in the three-dimensional space are given, the position and posture of the projection plane can be obtained by solving a minimization problem in which the error caused when the three-dimensional position is projected to the two-dimensional position is a cost. That is, the position and posture based on the projection plane is the position and posture of the image processing device 1 (camera 101a and camera 101b), and the three-dimensional position of each feature point on the previous camera image and the two-dimensional position of the corresponding point on the current camera image are known. Then, a minimization problem is solved in which the error on the camera image when the three-dimensional position of each feature point on the previous camera image is projected to the two-dimensional position of the corresponding point on the current camera image is used as a cost. As a result, the change in the position and posture of the image processing device 1 (camera 101a and camera 101b) from the previous image capture time to the current image capture time can be estimated. Note that the three-dimensional position information of each feature point on the previous camera image is acquired from the three-dimensional information storage unit 106, and the two-dimensional position information of the corresponding point on the current camera image is acquired from the motion matching unit 108.

[0075] The first position and posture (posture difference) and the second position and posture (posture difference) of the image processing device 1 as the estimation result by the Δposture estimation unit 109 are respectively sent to the integration unit 111 .

[0076] The brightness level detection unit 110 detects the brightness level of the camera image captured by the camera 101 b , for example, by pixel integration, etc. The brightness level of the camera image detected by the brightness level detection unit 110 is sent to the exposure control unit 102 .

[0077] The integration unit 111 integrates the posture differences acquired from the Δ posture estimation unit 109 through filtering processing such as a Kalman filter.

[0078] <2-1. Operation Example of Image Processing Device (1)>

[0079] Will refer to Figure 3 and Figure 4 An operation example of the image processing apparatus 1 is described. Figure 3 and Figure 4 : is a diagram illustrating an operation example of the image processing apparatus according to the embodiment.

[0080] Figure 3 is a diagram showing an operation example in the case where exposure is controlled so that underexposure (negative correction) and overexposure (positive correction) are repeated periodically and alternately every image frame.

[0081] like Figure 3As shown, in a case where the area of ​​an overexposed bright spot or an obscured shadow area included in the camera image is greater than a threshold value, the exposure control unit 102 performs exposure control so that underexposure and overexposure are periodically repeated alternately for each image frame. The exposure control unit 102 performs underexposure (negative correction) and overexposure (positive correction) alternately by following a predetermined execution order. That is, in a case where the frame rate is 60 frames per second, the exposure control unit 102 performs exposure control so that underexposure and overexposure are periodically repeated alternately in synchronization with the acquisition of an image frame every sixtieth of a second. The exposure control unit 102 can control the exposure, for example, by calculating an appropriate exposure (exposure value) from the brightness level detected by the brightness level detection unit 110 and applying a positive correction or a negative correction to the appropriate exposure using a predetermined correction value.

[0082] The parallax matching unit 104 , the distance estimating unit 105 , and the motion matching unit 108 perform matching between image frames having the same exposure for camera inputs that alternately repeat underexposure (negative correction) and overexposure (positive correction) every image frame.

[0083] The Δ pose estimation unit 109 estimates the pose difference based on the underexposed image frame (an example of the first position pose) and the pose difference based on the overexposed image frame (an example of the second position pose) alternately. The Δ pose estimation unit 109 solves a minimization problem in which the error on the camera image when the three-dimensional position of each feature point on the previous camera image is projected to the two-dimensional position of the corresponding point on the current camera image is used as a cost. As a result, the change in the position pose of the image processing device 1 (camera 101a and camera 101b) from the previous image capture time to the current image capture time, that is, the pose difference, can be estimated.

[0084] The integration unit 111 integrates the estimated posture differences between image frames with the same exposure through filtering processing such as a Kalman filter. The Kalman filter is a filter that integrates multiple observations (posture differences) with a modeling error probability distribution (error variance) and estimates the current state with the highest probability. The Kalman filter integrates two posture differences input thereto (i.e., a posture difference obtained from a pair of underexposed image frames and a posture difference obtained from a pair of overexposed image frames) and estimates the posture difference with the highest probability as the current state. The Kalman filter updates the state by the following equations (1) and (2).

[0085] The state quantity after observation = the state quantity before observation + the error variance of the state quantity before observation ÷ (the error variance of the state quantity before observation + the error variance of the observed value) × (observed value - predicted value)...(1).

[0086] The error variance of the state quantity after observation = the error variance of the observed value ÷ (the error variance of the state quantity before observation + the error variance of the observed value) × the error variance of the state quantity before observation...(2).

[0087] In the above equations (1) and (2), the observed value corresponds to the posture difference estimated by the Δ posture estimation unit 109. The predicted value in the above equation (1) can be obtained by linear interpolation from the previous state. Alternatively, in the case where the image processing device 1 includes an inertial measurement unit (IMU) and the IMU is used in combination with the camera 101a and the camera 101b, the predicted value in the above equation (1) can be obtained from the integrated value of the detection value of the IMU.

[0088] In the above equations (1) and (2), the error variance of the observed value can be obtained as the inverse (inverse matrix) of the partial derivative (matrix, because it is multivariable), and the partial derivative indicates the inclination to the optimal solution when estimating the posture difference (solving the minimization problem). That is, the error variance of the observed value is obtained by the inverse matrix of the matrix indicating the inclination of the optimal solution in solving the minimization problem for estimating the posture difference. For example, a smaller inclination relative to the optimal solution means a lower sensitivity to the error, and the error variance of its inverse is larger. On the other hand, when the inclination is large, the sensitivity to the error is high and the error variance is small.

[0089] Although the case has been described where the integration unit 111 integrates a plurality of observation values ​​(posture differences) by filtering processing by a Kalman filter, the plurality of observation values ​​may be integrated by a particle filter or simpler weighting.

[0090] <2-2. Operation Example of Image Processing Device (2)>

[0091] Figure 4 : is a diagram showing an example of operation in a case where the execution order of positive correction and negative correction is modified according to the number of feature points included in an image frame and exposure control is performed in accordance with the modified execution order. For example, the number of feature points detected from a camera image with overexposure (positive correction) is compared with the number of feature points detected from a camera image with underexposure (negative correction). As a result, if the number of feature points included in one of the camera images is larger, the use of the camera image with more feature points is considered to be more advantageous for estimating the own position and posture. Therefore, the execution order of positive correction and negative correction can be modified using the number of detected feature points as an index.

[0092] exist Figure 4, an example of a case where, when it is determined that more feature points exist in the underexposed camera image as a result of comparison between an underexposed camera image and an overexposed camera image as input, exposure control is performed by modifying the execution ratio of positive correction and negative correction.

[0093] The number of feature points detected from a camera image is excellent as an indicator of the brightness of a camera image that is optimal for estimating the own position and posture. However, from a camera image where an overexposed bright spot or an obscured shadow occurs, it is not possible to know what type of feature points are potential in the area where the overexposed bright spot or the obscured shadow occurs. Therefore, it is not possible to determine whether to increase the exposure to brighten the camera image or to decrease the exposure to darken the camera image, and it is difficult to use the number of feature points for exposure control in conventional techniques.

[0094] On the other hand, because the image processing device 1 according to the embodiment of the present disclosure captures both underexposed camera images and overexposed camera images, it is possible to grasp the number of potential feature points in the area where overexposed bright spots and blocked shadows appear in the camera image captured with appropriate exposure. In this way, the image processing device 1 according to the embodiment of the present disclosure can use the number of feature points detectable from the camera image for exposure control.

[0095] As a method of determining the execution order of underexposure (negative correction) or overexposure (positive correction), a method of determining based on the ratio of the number of feature points detected using each of them may be adopted. For example, let us assume that the ratio of the number of feature points obtained from the most recent underexposed camera image to the number of feature points obtained from the most recent overexposed camera image is 2:1. In this case, the exposure control unit 102 modifies the execution order of the positive correction (overexposure) and the negative correction (underexposure) so that the positive correction and the negative correction are sequentially executed at a ratio of 1 to 2. That is, as Figure 4 As shown, the execution order is modified so that negative correction (underexposure) -> negative correction (underexposure) -> positive correction (overexposure) is repeated periodically. Note that if positive correction and negative correction are performed in the order of a ratio of 1 to 2, the execution order can be modified to any order. For example, the execution order can be modified so that negative correction (underexposure) -> positive correction (overexposure) -> negative correction (underexposure) is repeated periodically.

[0096] Note that in this example, the method of determining the execution ratio of the negative correction and the positive correction assigned to the image frame does not have to be particularly limited. In addition, the frames to be compared for the number of feature points may not be limited to the most recent frame of each, for example, the number of feature points included in a plurality of underexposed image frames and the number of feature points included in a plurality of overexposed image frames may be averaged in the time direction and thus used. In addition, as the number of feature points, the number of feature points that have been matched among the feature points detected from the camera image may be adopted.

[0097] Note that, depending on the ratio of the number of feature points, the execution ratio of negative correction and positive correction implemented as exposure correction in synchronization with the imaging of the imaging unit 101 may be unbalanced without a limit such as 100:1. However, as described above, in order to know how many feature points are latent in the area where overexposed bright spots or blocked shadows occur, it is desirable to perform imaging at least once by at least one of positive correction and negative correction within a certain period of time. That is, for the execution ratio of positive correction and negative correction, a lower limit may be set for the side where the ratio is smaller.

[0098] For example, based on the premise that the frame rate is 60 (fps) and the execution ratio of the positive correction (overexposure) to the negative correction (underexposure) is 119:1, the negative correction (underexposure) is input once at 120 frames (input once at 2 seconds). Therefore, when focusing only on the negative correction (underexposure), the frame rate of acquiring the image frame of the negative correction (underexposure) can be expressed as 0.5 (fps). At this time, the image frame of the negative correction (underexposure) corresponds to the side of the smaller ratio being executed, and it is desired to perform control so that imaging is performed at least once using one of the positive correction and the negative correction within a specific time period by setting the lower limit of the ratio. The lower limit of the side of the smaller ratio being executed can be rewritten as the lower limit of the input frame rate of each of the positive correction (overexposure) and the negative correction (underexposure) of the Δ posture estimation unit 109. Then, when the lower limit of the input frame rate for each of the positive correction (overexposure) and negative correction (underexposure) of the Δ posture estimation unit 109 is set to 10 (fps), control can be performed so that imaging using the positive correction (overexposure) or negative correction (underexposure) exposure is performed once in six frames.

[0099] The lower limit of the ratio to be executed may be determined as a constant value or may be dynamically modified according to the amount of change (speed) of the own position posture to be estimated. When the overlapping area of ​​the field of view between frames decreases, the matching process of the motion matching unit 108 becomes difficult. Therefore, if the speed is high, the lower limit of the ratio may be increased so as to ensure that sufficient overlap of the field of view is obtained even on the side where the ratio to be executed is smaller, and if the speed is low, the lower limit of the ratio may be decreased.

[0100] The number of feature points is also useful in determining how much the target level will be negatively or positively corrected (underexposed or overexposed) relative to the proper exposure. For example, one could imagine trying a slight negative correction (or positive correction) and if the increase or decrease in the number of feature points is slight compared to the proper exposure, then correcting the correction value more.

[0101] In addition, the above description is a description of each of underexposure and overexposure performed step by step as an example, however, each of underexposure and overexposure can be performed with multi-level brightness. That is, in addition to underexposure and overexposure, darker underexposure or brighter overexposure can be sequentially used for imaging, and these images can be used.

[0102] <<3. Handler Example>>

[0103] Will refer to Figure 5 and Figure 6 An example of a processing procedure of the image processing apparatus 1 according to the present embodiment will be described. Figure 5 and Figure 6 : is a flowchart showing an example of a processing procedure executed by the image processing apparatus according to the present embodiment.

[0104] Figure 5 1 is a flowchart showing an example of a process for controlling exposure by alternately performing positive correction and negative correction by the exposure control unit 102. Figure 5 As shown, the exposure control unit 102 acquires an image frame from the imaging unit 101 (step S101) and calculates the areas of overexposed bright spots and blocked shadow regions in the acquired image frame (step S102).

[0105] The exposure control unit 102 determines whether the area of ​​the overexposed bright spot and / or the blocked shadow region calculated in step S102 is greater than a threshold value (step S103 ).

[0106] If the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow region is larger than the threshold value (step S103; Yes), the process proceeds to exposure bracketing (step S104).

[0107] The exposure control unit 102 controls the exposure so that positive correction and negative correction are performed alternately (step S105). That is, the exposure control unit 102 alternately performs underexposure (negative correction) and overexposure (positive correction) by following a predetermined execution order. That is, in the case where the frame rate is 60 frames / second, the exposure control unit 102 performs exposure control so that underexposure and overexposure are periodically repeated alternately in synchronization with the acquisition of an image frame every sixtieth of a second. Then, the exposure control unit 102 returns to the processing of the above-mentioned step S101.

[0108] In the above step S103, if the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow area is less than or equal to the threshold (step S103; No), if exposure bracketing is being performed, the exposure bracketing is released (step S106).

[0109] Then, the exposure control unit 102 performs control to achieve appropriate exposure based on the brightness level of the camera image (step S107 ), and returns to the processing procedure of the above-described step S101 .

[0110] Figure 6 FIG. 2 shows an example of a process in which the exposure control unit 102 modifies the execution order of the positive correction or the negative correction according to the number of feature points contained in the camera image and controls the exposure in the modified execution order. Figure 6 As shown, the exposure control unit 102 acquires an image frame from the imaging unit 101 (step S201) and calculates the areas of overexposed bright spots and blocked shadow regions in the acquired image frame (step S202).

[0111] The exposure control unit 102 determines whether the area of ​​the overexposed bright spot and / or blocked shadow region calculated in step S202 is greater than a threshold value (step S203 ).

[0112] If the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow region is larger than the threshold value (step S203; Yes), the process proceeds to exposure bracketing (step S204).

[0113] The exposure control unit 102 acquires the number of feature points of the current frame and the number of feature points of the previous frame (step S205). The previous frame corresponds to, for example, an image frame immediately before the current frame.

[0114] The exposure control unit 102 determines whether the number of feature points of the current frame is greater than the number of feature points of the previous frame (step S206 ).

[0115] If it is determined that the number of feature points of the current frame is greater than the number of feature points of the previous frame (step S206; yes), the exposure control unit 102 modifies the execution order of the exposure correction so that the proportion of the exposure correction corresponding to the current frame increases (step S207). For example, let us assume that the exposure correction of the current frame is underexposure (negative correction), the exposure correction of the previous frame is overexposure (positive correction), and the ratio of the number of feature points of the current frame to the number of feature points of the previous frame is 2 to 1. In this case, the exposure control unit 102 modifies the execution order of underexposure (negative correction) and overexposure (positive correction) so that negative correction and positive correction are performed sequentially at a ratio of 2 to 1. That is, the execution order is modified so as to periodically repeat underexposure (negative correction) -> underexposure (negative correction) -> overexposure (positive correction) (see Figure 4 ). Then, the exposure control unit 102 controls the exposure in the modified execution order in synchronization with the imaging of the imaging unit 101, and returns to the processing procedure of step S201.

[0116] If it is determined that the number of feature points of the current frame is less than or equal to the number of feature points of the previous frame (step S206; No), the exposure control unit 102 maintains the current execution order of exposure correction (step S208), and the process returns to the processing procedure of step S201.

[0117] In the above step S203, if the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow area is not less than or equal to the threshold (step S203; No), if exposure bracketing is being performed, the exposure bracketing is released (step S209).

[0118] Then, the exposure control unit 102 performs control to achieve appropriate exposure based on the brightness level of the camera image (step S210 ), and returns to the processing procedure of the above-described step S201 .

[0119] <<4. Modifications>>

[0120] Note that the above-described embodiments are examples, and various modifications and applications can be made.

[0121] <4-1. Variation of device configuration>

[0122] The image processing device 1 of the present embodiment is not limited to the device described in the above embodiment. Figure 7 is a diagram showing a configuration example of an image processing device according to a modification. Figure 7 The image processing device 1 shown in Figure 2 The differences from the configuration example shown in are described below.

[0123] like Figure 7As shown, the image processing device 1 according to the modified example may include an imaging unit 121 including a monocular camera, instead of an imaging unit 101 including a stereo camera (see Figure 2 ). In addition, in the case where the imaging unit 121 includes a monocular camera, the image processing device 1 includes a motion parallax-based distance estimation unit 122 instead of the distance estimation unit 105. The motion parallax-based distance estimation unit 122 estimates the distance by motion parallax (parallax caused by the movement of the camera) based on a combination of the two-dimensional position information of the corresponding point on the current image obtained by the motion matching unit 108 and the posture difference from the previous image capture time to the current image capture time obtained by the Δ posture estimation unit 109.

[0124] The image processing device 1 according to the modification is similar to the image processing device 1 according to the above embodiment regarding other functional configurations except for the imaging unit 121 and the method of distance estimation by the motion parallax-based distance estimation unit 122 , and can perform the exposure control described in the above embodiment.

[0125] <4-2. Modification of the Process>

[0126] In the above embodiment, the image processing device 1 can perform positive correction and negative correction alternately when transitioning to the start of exposure bracketing, and perform exposure correction based on the number of feature points after extracting feature points from the image frame by exposure bracketing. Hereinafter, an example of the processing procedure in this case will be described. Figure 8 is a flowchart showing an example of a processing procedure of the image processing device 1 according to the modification.

[0127] like Figure 8 As shown, the exposure control unit 102 acquires an image frame from the imaging unit 101 (step S301), and calculates the areas of overexposed bright spots and blocked shadow regions in the acquired image frame (step S302).

[0128] The exposure control unit 102 determines whether the area of ​​the overexposed bright spot and / or the blocked shadow region calculated in step S302 is greater than a threshold value (step S303 ).

[0129] If the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow region is larger than the threshold value (step S303; Yes), it determines whether exposure bracketing is being performed (step S304).

[0130] When the exposure control unit 102 determines that exposure bracketing is not being executed (step S304 ; No), the process proceeds to exposure bracketing (step S305 ).

[0131] Then, the exposure control unit 102 controls the exposure so that the positive correction and the negative correction are performed alternately (step S306). That is, the exposure control unit 102 performs underexposure (negative correction) and overexposure (positive correction) alternately by following a predetermined execution order. That is, in the case where the frame rate is 60 frames / second, the exposure control unit 102 performs exposure control so that underexposure and overexposure are periodically repeated alternately in synchronization with the acquisition of an image frame every sixtieth of a second. Then, the exposure control unit 102 returns to the processing of the above-mentioned step S301.

[0132] In step S304, if the exposure control unit 102 determines that exposure bracketing is being performed (step S304; Yes), the number of feature points of the current frame and the number of feature points of the previous frame are acquired (step S307). The previous frame corresponds to, for example, an image frame immediately before the current frame.

[0133] The exposure control unit 102 determines whether the number of feature points of the current frame is greater than the number of feature points of the previous frame (step S308 ).

[0134] If it is determined that the number of feature points of the current frame is greater than the number of feature points of the previous frame (step S308; yes), the exposure control unit 102 modifies the execution order of the exposure correction so that the ratio of the exposure correction corresponding to the current frame increases (step S309). For example, let us assume that the exposure correction of the current frame is underexposure (negative correction), the exposure correction of the previous frame is overexposure (positive correction), and the ratio of the number of feature points of the current frame to the number of feature points of the previous frame is 2 to 1. In this case, the exposure control unit 102 modifies the execution order of underexposure (negative correction) and overexposure (positive correction) so that the negative correction (underexposure) and the positive correction (overexposure) are sequentially executed at a ratio of 2 to 1. That is, the execution order is modified so as to periodically repeat underexposure (negative correction) -> underexposure (negative correction) -> overexposure (positive correction) (see Figure 4 ). Then, the exposure control unit 102 controls the exposure in the modified execution order in synchronization with the imaging of the imaging unit 101, and returns to the processing procedure of step S301.

[0135] On the other hand, if it is determined that the number of feature points of the current frame is less than or equal to the number of feature points of the previous frame (step S308; No), the exposure control unit 102 maintains the current execution order of exposure correction (step S310), and the processing returns to the processing process of step S301.

[0136] In the above step S303, if the exposure control unit 102 determines that the area of ​​the overexposed bright spot and / or blocked shadow area is less than or equal to the threshold (step S303; No), if exposure bracketing is being performed, the exposure bracketing is released (step S311).

[0137] Then, the exposure control unit 102 performs control to achieve appropriate exposure based on the brightness level of the camera image (step S312 ) and returns to the processing procedure of the above-described step S301 .

[0138] <4-3. Others>

[0139] In the above-mentioned embodiment, instead of determining whether the area of ​​the overexposed bright spot and / or the occluded shadow region in the image frame is greater than the threshold value, the image processing device 1 may determine whether the contrast ratio of the camera image is greater than the threshold value. As a result, it is not necessary to calculate the area of ​​the overexposed bright spot and / or the occluded shadow region, so as to be more advantageous in dealing with the case where the image frame is acquired with appropriate exposure to extract feature points.

[0140] Meanwhile, the image processing apparatus 1 according to the present embodiment may be implemented by a dedicated computer system or by a general-purpose computer system.

[0141] For example, a program for executing the operation of the image processing device 1 of the present embodiment may be stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a floppy disk. In addition, for example, the control device is configured by a program installed in a computer and performs the above processing. In this case, the control device may be the image processing device 1 according to the embodiment.

[0142] In addition, the program may be stored in a disk device of a server device on a network such as the Internet so that the program can be downloaded to a computer. In addition, the above functions may be implemented by the cooperation of an operating system (OS) and application software. In this case, the parts other than the OS may be stored and distributed in a medium, or the parts other than the OS may be stored in a server device to allow downloading to a computer, etc.

[0143] In the processes described in the above embodiments, all or part of the processes described as being automatically performed may be performed manually, or all or part of the processes described as being manually performed may be performed automatically by a known method. In addition, unless otherwise specified, the process, specific name, and information including various data or parameters described above or shown in the drawings may be modified as needed. For example, the various types of information shown in the drawings are not limited to the information that has been shown.

[0144] In addition, each component of each device shown in the drawings is conceptual in function and does not necessarily need to be physically configured as shown in the drawings. That is, the specific forms of distribution and integration of the devices are not limited to those shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit according to various loads, usage states, etc.

[0145] In addition, the above-mentioned embodiments can be combined as appropriate as long as the processing contents are not contradictory. In addition, the order of the steps shown in the sequence diagram or flowchart of this embodiment can be modified as appropriate.

[0146] It should be noted that the self-position and posture estimation technology realized by the exposure control of the image processing device 1 according to the embodiment can be applied to any industrial field such as autonomous driving of automobiles, surgery support, XR experience stage, etc.

[0147] <<5. Conclusion>>

[0148] As described above, according to an embodiment of the present disclosure, the image processing device 1 includes the exposure control unit 102, the Δ posture estimation unit 109 (an example of an estimation unit), and the integration unit 111. The exposure control unit 102 controls the exposure by sequentially performing positive correction of increasing exposure from an appropriate exposure or negative correction of decreasing exposure from an appropriate exposure in a predetermined execution order as exposure when each of a plurality of image frames is acquired in a time series. The Δ posture estimation unit 109 estimates a first position posture of the image processing device 1 based on matching between image frames subjected to positive correction, and a second position posture of the image processing device 1 based on matching between image frames subjected to negative correction. The integration unit 111 integrates the first position posture with the second position posture.

[0149] As a result, the accuracy of self-positioning can be enhanced without being affected by the content of the captured camera image. That is, it is possible to avoid the influence of overexposed bright spots or blocked shadows appearing in the camera image, detect feature points from the camera image, and improve the accuracy of self-positioning.

[0150] For example, the image processing device 1 may perform positive correction and negative correction alternately. As a result, even in an environment with high contrast, it is possible to compensate for the lack of dynamic range and perform exposure control capable of stably detecting feature points from a camera image.

[0151] In addition, for example, the image processing device 1 can modify the execution order based on the ratio between the number of feature points contained in the image frame subjected to positive correction and the number of feature points contained in the image frame subjected to negative correction. As a result, even in an environment with high contrast, it is possible to compensate for the lack of dynamic range and perform exposure control that can more stably detect feature points from camera images.

[0152] In addition, for example, in a case where the number of feature points included in an image frame subjected to positive correction is greater than the number of feature points included in an image frame subjected to negative correction, the image processing device 1 may modify the execution order so that more image frames subjected to positive correction are acquired compared to image frames subjected to negative correction. On the other hand, in a case where the number of feature points included in an image frame subjected to positive correction is less than the number of feature points included in an image frame subjected to negative correction, the image processing device 1 may modify the execution order so that more image frames subjected to negative correction are acquired compared to image frames subjected to positive correction. As a result, even in an environment with high contrast, it is possible to compensate for the lack of dynamic range and perform exposure control capable of more stably detecting many feature points from a camera image.

[0153] In addition, for example, the image processing device 1 can determine whether to control the exposure in the execution order based on the area of ​​the overexposed bright spot and / or the blocked shadow area included in the image frame captured with the appropriate exposure. As a result, it is possible to cope with the case where the image frame is acquired with the appropriate exposure in an environment with low contrast to extract feature points.

[0154] In addition, for example, the image processing device 1 can determine whether to control the exposure in the execution order based on the contrast ratio of the image frame captured with the appropriate exposure. As a result, it is not necessary to calculate the area of ​​the overexposed bright spot and / or the blocked shadow area, so as to be more advantageous in dealing with the case of acquiring the image frame with the appropriate exposure to extract the feature point.

[0155] In addition, for example, the image processing device 1 can integrate the first position and the second position by a Kalman filter. As a result, the estimation result of the own position and the posture of the image processing device 1 can be obtained from the feature points detected from a plurality of image frames with different exposures.

[0156] Furthermore, the first position and the second position include information indicating three degrees of freedom of position change of image processing device 1 and information indicating three degrees of freedom of rotation change of image processing device 1. As a result, the versatility of processing based on the estimation result of the own position and orientation of image processing device 1 can be improved.

[0157] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.

[0158] Furthermore, the effects of the embodiments described herein are merely examples and are not restrictive, and other effects may be achieved.

[0159] It should be noted that the present technology may also have the following configurations.

[0160] (1) An image processing device comprising:

[0161] an exposure control unit that controls exposure by sequentially performing, in a predetermined execution order, a positive correction for increasing exposure from an appropriate exposure or a negative correction for decreasing exposure from an appropriate exposure as exposure when each of a plurality of image frames is acquired in time series;

[0162] an estimating unit that estimates a first position and posture of the image processing device based on matching between image frames subjected to the positive correction, and a second position and posture of the image processing device based on matching between image frames subjected to the negative correction; and

[0163] An integration unit integrates the first position and posture with the second position and posture.

[0164] (2) The image processing device according to 1 above,

[0165] Wherein, the exposure control unit,

[0166] The positive correction and the negative correction are performed alternately.

[0167] (3) The image processing device according to item 1 above,

[0168] Wherein, the exposure control unit,

[0169] The execution order is modified based on a ratio between the number of feature points contained in the image frame subjected to the positive correction and the number of feature points contained in the image frame subjected to the negative correction.

[0170] (4) The image processing device according to 3 above,

[0171] wherein the exposure control unit, when the number of feature points included in the image frame subjected to the positive correction is greater than the number of feature points included in the image frame subjected to the negative correction,

[0172] modifying the execution order so as to acquire more image frames subjected to the positive correction than image frames subjected to the negative correction; and

[0173] In a case where the number of feature points included in the image frame subjected to the positive correction is smaller than the number of feature points included in the image frame subjected to the negative correction,

[0174] The execution order is modified so that more image frames subjected to the negative correction are acquired than image frames subjected to the positive correction.

[0175] (5) The image processing device according to 1 above,

[0176] Wherein, the exposure control unit,

[0177] Whether to control the exposure in the execution order is determined based on the area of ​​the overexposed bright spot region and / or the blocked shadow region included in the image frame captured with the appropriate exposure.

[0178] (6) The image processing device according to item 1 above,

[0179] Wherein, the exposure control unit,

[0180] Based on the contrast ratio of the image frame captured with the appropriate exposure, it is determined whether to control the exposure in the execution order.

[0181] (7) The image processing device according to item 1 above,

[0182] Wherein, the integration unit,

[0183] The first position and posture and the second position and posture are integrated through a Kalman filter.

[0184] (8) The image processing device according to item 1 above,

[0185] The first position and the second position and the posture include information of three degrees of freedom indicating a change in a position of the image processing device and information of three degrees of freedom indicating a change in a rotation of the image processing device.

[0186] (9) An image processing method,

[0187] By the image processing device:

[0188] controlling exposure by sequentially performing, in a predetermined execution order, a positive correction for increasing exposure from an appropriate exposure or a negative correction for decreasing exposure from an appropriate exposure as exposure when each of a plurality of image frames is acquired in time series;

[0189] estimating a first position and orientation of the image processing device based on matching between image frames subjected to positive correction, and a second position and orientation of the image processing device based on matching between image frames subjected to negative correction; and

[0190] The first position and posture and the second position and posture are integrated.

[0191] Reference Numbers List

[0192] 1 Image processing device

[0193] 101 Imaging Unit

[0194] 102 exposure control unit

[0195] 103 Feature point detection unit

[0196] 104 Parallax Matching Unit

[0197] 105 Distance Estimation Unit

[0198] 106 Three-dimensional information storage unit

[0199] 107 Two-dimensional information storage unit

[0200] 108 Motion Matching Unit

[0201] 109 Δ pose estimation unit

[0202] 110 Brightness level detection unit

[0203] 111 Integration Unit

[0204] 121 Imaging Unit

[0205] 122 Motion parallax based distance estimation unit.

Claims

1. An image processing device, comprising: an exposure control unit that controls exposure by sequentially performing, in a predetermined execution order, a positive correction for increasing exposure from an appropriate exposure or a negative correction for decreasing exposure from the appropriate exposure as exposure when each of a plurality of image frames is acquired in time series; a feature point detection unit that detects feature points that can be used to estimate the position and posture of the image processing device using the exposure controlled by the exposure control unit; an estimating unit that estimates, based on the detected feature points, a first position and posture of the image processing device based on matching between the image frames subjected to the positive correction, and a second position and posture of the image processing device based on matching between the image frames subjected to the negative correction; as well as An integration unit integrates the first position and posture with the second position and posture.

2. The image processing device according to claim 1, in, the exposure control unit, The positive correction and the negative correction are performed alternately.

3. The image processing device according to claim 1, in, the exposure control unit, The execution order is modified based on a ratio between the number of feature points contained in the image frame subjected to the positive correction and the number of feature points contained in the image frame subjected to the negative correction.

4. The image processing device according to claim 3, in, The exposure control unit, in a case where the number of the feature points included in the image frame subjected to the positive correction is greater than the number of the feature points included in the image frame subjected to the negative correction, modifying the execution order so as to acquire more image frames subjected to the positive correction than image frames subjected to the negative correction; as well as In a case where the number of the feature points included in the image frame subjected to the positive correction is smaller than the number of the feature points included in the image frame subjected to the negative correction, The execution order is modified so that more image frames subjected to the negative correction are acquired than image frames subjected to the positive correction.

5. The image processing device according to claim 1, in, the exposure control unit, Whether to control the exposure in the execution order is determined based on the area of ​​the overexposed bright spot region and / or the blocked shadow region included in the image frame captured with the appropriate exposure.

6. The image processing device according to claim 1, in, the exposure control unit, Based on a contrast ratio of an image frame captured with the appropriate exposure, it is determined whether to control the exposure in accordance with the execution order.

7. The image processing device according to claim 1, in, The integration unit, The first position and posture and the second position and posture are integrated through a Kalman filter.

8. The image processing device according to claim 1, in, The first position and the second position and the posture include information of three degrees of freedom indicating a change in a position of the image processing device and information of three degrees of freedom indicating a change in a rotation of the image processing device.

9. An image processing method, controlling exposure by sequentially performing, in a predetermined execution order, a positive correction for increasing exposure from an appropriate exposure or a negative correction for decreasing exposure from the appropriate exposure as exposure when each of a plurality of image frames is acquired in time series; Using the controlled exposure, detecting feature points that can be used to estimate the position and posture of the image processing device; estimating, based on the detected feature points, a first position and posture of the image processing device based on matching between image frames subjected to the positive correction, and a second position and posture of the image processing device based on matching between image frames subjected to the negative correction; as well as The first position and posture and the second position and posture are integrated.

Images