Image processing device, image processing method, and program

Abstract

To improve processing when there is a flicker.SOLUTION: An image processing device includes: image signal acquisition means for acquiring an image signal photographed by an imaging device; feature point calculation means for calculating a feature point from the acquired image signal; flicker detection signal generation means for generating a flicker detection signal indicating a pixel in which a flicker is generated in the acquired image signal; saturation detection signal generation means for generating a saturation detection signal indicating the pixel in which a pixel value exceeds the upper limit value and is saturated in the acquired image signal; feature amount signal calculation means for calculating a feature amount signal indicating a feature amount of the feature point on the basis of the acquired image signal; and feature amount mask signal generation means for generating a feature amount mask signal indicating a mask of the feature amount signal on the basis of the flicker detection signal and the saturation detection signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Digital cameras mounted on vehicles and industrial robots are required to have a function for generating wide dynamic range images (hereinafter referred to as HDR function) to accurately recognize environments even under conditions with large differences in brightness. HDR is an abbreviation for High Dynamic Range. Digital cameras capture images at multiple signal levels to obtain wide dynamic range images. In recent years, image sensors with HDR function have sometimes been unable to capture images of traffic lights and other objects due to LED flicker. LED is an abbreviation for Light Emitting Diode. Patent Document 1 proposes an image sensor with a function for detecting and eliminating flicker in such cases. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-180459 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the method described in Patent Document 1, although the flicker removal function can suppress the phenomenon where a lit LED appears to be off, there is a possibility that pixel values ​​may fluctuate between frames. This fluctuation in pixel values ​​can reduce the accuracy of feature point matching in, for example, traffic lights that use LED light sources. In other words, there was previously room for improvement in processing when flicker is present.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to improve processing when flicker is present. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present invention comprises: image signal acquisition means for acquiring an image signal captured by an imaging device; feature point calculation means for calculating feature points from the acquired image signal; flicker detection signal generation means for generating a flicker detection signal indicating pixels in the acquired image signal where flicker is occurring; saturation detection signal generation means for generating a saturation detection signal indicating pixels in the acquired image signal where the pixel value exceeds an upper limit value and is saturated; feature signal calculation means for calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; and feature mask signal generation means for generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal. [Effects of the Invention]

[0007] The present invention allows for improved handling in the presence of flicker. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an image processing system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of an image processing system according to a first embodiment of the present invention, and is a block diagram showing the internal configuration of an image processing unit 120 shown in FIG. [Figure 3] 3 is a flowchart showing the operation of the image processing device 10 according to the first embodiment of the present invention. [Figure 4] 3A and 3B are diagrams illustrating a method for generating a feature signal and a feature mask signal according to the first embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating a feature point matching method according to the first embodiment of the present invention. [Figure 6]FIG. 10 is a block diagram showing the configuration of an image processing system according to a second embodiment of the present invention. [Figure 7] FIG. 7 is a block diagram of an image processing system according to a second embodiment of the present invention, and is a block diagram showing the internal configuration of an image processing unit 504 shown in FIG. 6. [Figure 8] 10 is a flowchart showing the operation of an image processing device 502 according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] [Embodiment 1] In a first embodiment of the present invention, in order to estimate the orientation of an imaging device having an imaging unit, a pixel structure for detecting flicker is used, and flicker is detected for each pixel to calculate a flicker detection signal and a saturation state of pixel values, and multiple feature points are extracted using the image signal. In this embodiment, a feature amount is calculated for each feature point using the image signal and the flicker detection signal around the feature point. In this embodiment, the calculated feature amount is used to perform feature point matching, thereby estimating the orientation of the imaging device.

[0011] Fig. 1 is a block diagram showing the configuration of an image processing system according to a first embodiment of the present invention. Fig. 2 is a block diagram showing the internal configuration of an image processing unit 120 shown in Fig. 1, which is a block diagram of the image processing system according to the first embodiment of the present invention. The image processing system according to this embodiment has an imaging unit 100 and an image processing device 10. The image processing device 10 may have a configuration including the imaging unit 100.

[0012] <Image capture unit> The imaging unit 100 has a lens 101 and an imaging element 102. The lens 101 is a photographing lens, and is a device that forms an image of a subject on the imaging element 102, which is the imaging surface of the imaging unit 100. The imaging element 102 captures images of the input subject in a plurality of different dynamic ranges, and transmits the images as image signals to the signal processing unit 110 of the image processing device 10. The imaging element 102 is an image sensor configured from a CMOS (complementary metal-oxide semiconductor) or a CCD (charge-coupled device). CMOS is an abbreviation for Complementary Metal-Oxide-Semiconductor. CCD is an abbreviation for Charge Coupled Device.

[0013] An image of a subject formed on an image sensor 102 via a lens 101 is converted by the image sensor 102 into image signals for each dynamic range for each pixel, and transmitted to a signal processing unit 110. In the first embodiment, image signals captured in two different dynamic ranges by exposing one pixel multiple times in the time direction are transmitted to the signal processing unit 110.

[0014] The image signals captured with two different dynamic ranges are set as an image signal exposed for a long time and an image signal exposed for a short time. However, the image signal exposed for a long time is an image signal with low-sensitivity light-receiving characteristics (hereinafter referred to as a low-sensitivity image signal). On the other hand, the image signal exposed for a short time is an image signal with high-sensitivity light-receiving characteristics and a brighter dynamic range than the image signal exposed for a long time (hereinafter referred to as a high-sensitivity image signal). The image signal exposed for a long time, i.e., a low-sensitivity image signal, is exposed for a time longer than the expected flicker period. The low-sensitivity image signal is an example of a long-exposure image signal. The image signal exposed for a short time, i.e., a high-sensitivity image signal, is exposed for a time shorter than the expected flicker period. The high-sensitivity image signal is an example of a short-exposure image signal.

[0015] The present invention is not limited to a configuration for acquiring two different dynamic ranges, and may be a configuration for acquiring image signals of two or more dynamic ranges. As a configuration for acquiring image signals of two or more dynamic ranges, pixels having at least two or more different sensitivities may be acquired, or the signals may be acquired by exposing multiple times in the time direction.

[0016] <Image processing device> The image processing device 10 includes a signal processing unit 110 and an image processing unit 120 . <Signal processing section> The following describes the signal processing unit 110. The signal processing unit 110 includes a state detection unit 111 and a signal synthesis unit 112.

[0017] The state detection unit 111 receives a high-sensitivity image signal and a low-sensitivity image signal from the imaging unit 100. The state detection unit 111 is an example of an image signal acquisition unit that acquires an image signal. The state detection unit 111 detects whether a pixel requires flicker removal using two image signals, the high-sensitivity image signal and the low-sensitivity image signal. When flicker does not occur, the pixel values ​​of the low-sensitivity image signal are larger than the pixel values ​​of the high-sensitivity image signal because the exposure time is long. In this case, the ratio of pixel values ​​is the exposure time ratio. On the other hand, when flicker occurs, the high-sensitivity image signal with a short exposure time may or may not receive light within the exposure time due to the influence of the flicker period, resulting in fluctuations in pixel values ​​between frames. For the low-sensitivity image signal with a long exposure time, setting the exposure time longer than the flicker period reduces fluctuations in pixel values ​​between frames. As a result, when flicker occurs, the ratio of pixel values ​​between the low-sensitivity image signal and the high-sensitivity image signal does not match the exposure time ratio. The state detection unit 111 can detect the occurrence of flicker by detecting a change in the pixel value ratio of the image signals due to the occurrence of flicker. The state detection unit 111 outputs a flicker detection signal to the signal synthesis unit 112, with a signal value of 1 if a flicker is detected and a signal value of 0 if a flicker is not detected. The state detection unit 111 is an example of a flicker detection signal generation means that generates a flicker detection signal that indicates pixels in the acquired image signal where flicker is occurring. The above detection method is an example of a flicker detection method, and other detection methods may be used in the present invention.

[0018] The state detection unit 111 further detects whether the low-sensitivity image signal received from the imaging unit 100 is saturated. Whether the low-sensitivity image signal is saturated can be determined by detecting a pixel whose pixel value exceeds a predetermined upper limit as a saturated pixel. The state detection unit 111 outputs a saturation detection signal to the signal synthesis unit 112, the signal value of which is 1 if the low-sensitivity image signal is saturated and 0 if the low-sensitivity image signal is not saturated. The state detection unit 111 is an example of a saturation detection signal generation unit that generates a saturation detection signal indicating pixels in the acquired image signal whose pixel value exceeds an upper limit. As an output signal format, the state detection unit 111 outputs, for example, a high-sensitivity image signal, a low-sensitivity image signal, a flicker detection signal, and a saturation detection signal in separate signals. However, the output format is not limited to this format.

[0019] The signal synthesis unit 112 synthesizes the high-sensitivity image signal and the low-sensitivity image signal received from the state detection unit 111 to generate a wide dynamic range image. The signal synthesis unit 112 determines a synthesis ratio, which is the ratio used to synthesize each of the two image signals, the high-sensitivity image signal and the low-sensitivity image signal, based on their signal values. The synthesis ratio is set between 0 and 1, with α1 representing the synthesis ratio for the high-sensitivity image signal and α2 representing the synthesis ratio for the low-sensitivity image signal, so that the sum of α1 and α2 is 1. The signal synthesis unit 112 synthesizes signal values ​​for all pixels using the synthesis ratio to generate a wide dynamic range image. The signal synthesis unit 112 outputs a synthesized image signal, which is the signal for the generated wide dynamic range image, to the image processing unit 120. This synthesized image signal, which is the signal for the wide dynamic range image, is also referred to as an HDR synthesized signal. The signal synthesis unit 112 also outputs the flicker detection signal and saturation detection signal received from the state detection unit 111 to the image processing unit 120.

[0020] When the signal synthesizing unit 112 synthesizes two image signals, it is necessary to increase the synthesis ratio α1 for the high-sensitivity image signal at pixels with high brightness values ​​where the low-sensitivity image signal saturates. As described above, the high-sensitivity image signal is significantly affected by flicker, so when the low-sensitivity image signal saturates, the synthesized signal is likely to be affected by flicker. Therefore, if saturation occurs in a pixel where flicker is occurring, pixel values ​​are likely to fluctuate between frames. The image processing unit 120 can detect pixels where such pixel value fluctuations may occur based on the flicker detection signal and saturation detection signal.

[0021] <Image processing unit> Next, a description will be given of the image processing unit 120. The image processing unit 120 includes an image signal storage unit 121, a feature point calculation unit 122, a feature amount generation unit 123, a feature point signal storage unit 124, a matching unit 125, and a posture estimation unit 126.

[0022] The image signal storage unit 121 stores the synthesized image signal, which is a signal of an image with a wide dynamic range, the flicker detection signal, and the saturation detection signal received from the signal processing unit 110.

[0023] The feature point calculation unit 122 reads out the composite image signal from the image signal storage unit 121 and calculates feature points. The feature point calculation unit 122 is an example of a feature point calculation means that calculates feature points from the acquired image signal. A feature point is coordinate information of a characteristic pixel in an image. The feature point calculation unit 122 outputs the calculated feature points as feature point signals to the feature amount generation unit 123 and the feature point signal storage unit 124.

[0024] The feature amount generation unit 123 acquires feature points from the received feature point signal, reads out a composite image signal for the feature point peripheral region for each feature point, and calculates a feature amount. The feature amount generation unit 123 is an example of a feature amount signal calculation means that calculates a feature amount signal indicating the feature amount of the feature point based on the acquired image signal. The feature amount generation unit 123 further reads out a flicker detection signal and a saturation detection signal for the feature point peripheral region, and generates a feature amount mask signal. The feature amount generation unit 123 is an example of a feature amount mask signal generation means that generates a feature amount mask signal indicating a mask of the feature amount signal based on the flicker detection signal and the saturation detection signal. The feature amount generation unit 123 outputs the feature amount signal, which is a signal for the calculated feature amount, and the generated feature amount mask signal to the feature point signal storage unit 124. Details of the feature amount mask signal will be described later.

[0025] The feature point signal storage unit 124 stores the feature point signals from the feature point calculation unit 122, and the feature signal and feature mask signal from the feature generation unit 123. For storage, a memory such as an SRAM may be used, or a configuration may be made to use a DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The feature point signal storage unit 124 outputs the feature point signals, feature signal, and feature mask signal to the matching unit 125 in response to a request from the matching unit 125.

[0026] The matching unit 125 performs feature point matching using the feature point signal, feature amount signal, and feature amount mask signal, calculates motion vector information indicating the movement of the imaging device, and outputs the calculated motion vector information to the posture estimation unit 126.

[0027] The orientation estimation unit 126 estimates a homography matrix using the motion vector information calculated by the matching unit 125, and outputs it as orientation information indicating the orientation of the image capture device.

[0028] <Flowchart> Fig. 3 is a flowchart showing the operation of the image processing device 10 according to the first embodiment of the present invention. The control operation of the image processing device 10 will be described step by step using Fig. 3. In this embodiment, an example will be described in which the posture is estimated from the amount of movement of the imaging device between frames.

[0029] In step S201, the state detection unit 111 generates and outputs a flicker detection signal and a saturation detection signal based on the high-sensitivity image signal and the low-sensitivity image signal received from the imaging unit 100.

[0030] In step S202, the feature point calculation unit 122 calculates feature points from the HDR synthesis signal and outputs them as a feature point signal. A known method can be used to calculate the feature points. For example, the feature point calculation unit 122 can use a method called FAST, which detects only the corners of an object as feature points. FAST is an abbreviation for Features from Accelerated Segment Test.

[0031] In step S203, the feature generation unit 123 calculates and generates a feature signal and a feature mask signal for each feature point calculated in step S202. A known method can be used to calculate the feature. For example, the feature generation unit 123 can use a method called BRIEF, which randomly arranges multiple line segments around a feature point and generates bits by comparing the brightness values ​​at both ends of the line segments. BRIEF is an abbreviation for Binary Robust Independent Element Features.

[0032] Here, a method for generating a feature signal and a feature mask signal will be described with reference to FIG. 4. FIG. 4 is a diagram illustrating a method for generating a feature signal and a feature mask signal according to the first embodiment of the present invention. FIG. 4(A) is a 7×7 pixel wide dynamic range image centered on pixel 301, which is one of the feature points calculated in step S202. FIG. 4(B) is a diagram illustrating a flicker detection signal for each pixel in FIG. 4(A). FIG. 4(C) is a diagram illustrating a saturation detection signal for each pixel in FIG. 4(A). FIG. 4(D) is a diagram illustrating a feature signal for pixel 301, which is the feature point in FIG. 4(A). FIG. 4(E) is a diagram illustrating a feature mask signal for pixel 301, which is the feature point in FIG. 4(A).

[0033] In FIG. 4A, 7×7 pixels centered on pixel 301 are pixels (peripheral pixels) in a peripheral region of pixel 301, which is a feature point. In this embodiment, the peripheral region of the feature point is 7×7 pixels centered on pixel 301, but can be set to a predetermined range. Furthermore, pixels 302, 303, 304, and 305 included in the 7×7 pixels centered on pixel 301 are reference pixels for calculating feature amounts for each feature point. In this embodiment, the number of reference pixels is four, but may be any other number. In this embodiment, the positions of the reference pixels are the four corners of the peripheral pixels, but may be any other positions.

[0034] In this embodiment, a comparison is made between the luminance values ​​of pixel 301, which is a feature point, and those of four surrounding pixels 302, 303, 304, and 305, and bits 306, 307, 308, and 309, which are four-bit strings, are calculated as the feature amount of the feature point. This feature amount calculation is performed for all feature points. In this embodiment, a feature amount signal is generated by setting a corresponding bit to 0 when the luminance value of a surrounding pixel is greater than the luminance value of the feature point, and 1 when the luminance value is less than the luminance value of the feature point. Bit 306 is a bit that stores the comparison result of the luminance value comparison between pixel 301 and pixel 302. Bit 307 is a bit that stores the comparison result of the luminance value comparison between pixel 301 and pixel 303. Bit 308 is a bit that stores the comparison result of the luminance value comparison between pixel 301 and pixel 304. Bit 309 is a bit that stores the comparison result of the luminance value comparison between pixel 301 and pixel 305. FIG. 4D is an example of a feature signal when only pixel 304 has a luminance value greater than that of pixel 301, which is a feature point.

[0035] FIG. 4(E) shows the feature mask signal, which is generated as a 4-bit bit string consisting of bits 310, 311, 312, and 313. The feature mask signal is a bit string with the same bit length as the feature signal in FIG. 4(D), with bits 310, 311, 312, and 313 corresponding to bits 306, 307, 308, and 309, respectively. The feature mask signal is a signal that indicates whether or not to use the feature corresponding to each bit in feature point matching. By setting the bit positions of the feature mask signal corresponding to unused bits of the 4-bit feature signal to 0, the feature signal bits at the corresponding positions are not used in feature point matching.

[0036] The feature mask signal is generated based on the flicker detection signal and saturation detection signal of two pixels that are compared when calculating the feature. In this embodiment, if flicker is detected in both pixels and saturation is detected in either pixel, the feature mask signal is set to 0; otherwise, the feature mask signal is set to 1.

[0037] In the example of Fig. 4(A), pixel 301 and pixel 304 are both pixels for which flicker is detected, as shown in Fig. 4(B), and pixel 304 is a pixel for which saturation is detected, as shown in Fig. 4(C). Therefore, bit 312, which is the bit of the feature mask signal corresponding to pixel 301 and pixel 304, is set to 0, and the other bits of the feature mask signal are set to 1.

[0038] The method for generating the feature mask signal is not limited to the above method. For example, the feature mask signal may be generated by a method in which the bits of the feature mask signal for pixels in which flicker is detected are set to 0, regardless of whether saturation is detected or not. Furthermore, the positions and number of pixels that are compared in brightness with the pixel of the feature point, and the bit length of the feature are not limited to those in this embodiment.

[0039] In step S204, the feature generation unit 123 generates feature signals and feature mask signals for all feature points and determines whether generation is complete. If the feature generation unit 123 determines that generation is complete, the process proceeds to step S205. If the feature generation unit 123 determines that generation is not complete, the process proceeds to step S203.

[0040] In step S205, the matching unit 125 acquires the feature signal and feature mask signal of the previous frame required for feature point matching from the feature signal storage unit 124. In step S206, the feature generation unit 123 stores the feature signal and feature mask signal of the current frame to be used in the next frame in the feature signal storage unit 124.

[0041] In step S207, the matching unit 125 performs matching using the feature signal and feature mask signal of the current frame generated in step S203 and the feature signal and feature mask signal of the previous frame acquired in step S205. The matching unit 125 calculates the degree of match in the feature point matching through this matching, and calculates motion vector information that indicates the movement of the imaging device. The matching unit 125 outputs the calculated motion vector information to the posture estimation unit 126. The matching in the matching unit 125 is performed using only bits of the 4-bit feature signal where the feature mask signal is 1.

[0042] Here, a method in which the matching unit 125 performs matching for each feature point using the feature signal and the feature mask signal in step S207 will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the feature point matching method according to the first embodiment of the present invention.

[0043] Feature point 401 is an example of a feature point calculated from the current frame. Feature points 402, 403, 404, and 405 are examples of feature points in the previous frame that are compared with feature point 401. The matching unit 125 calculates the similarity between feature point 401 and each of feature points 402, 403, 404, and 405. The similarity is determined by comparing the feature amount of feature point 401 with the feature amount of each feature point bit by bit and determining the number of different bits. When counting the number of different bits (referred to as the number of different bits), bit positions where the feature amount mask signal is 0 are not counted as the number of different bits. The positions where the feature amount mask signal is 0 correspond to the positions of feature amounts calculated from pixels where flicker and saturation are detected. As described above, pixels where flicker and saturation are detected are prone to fluctuations in pixel value between frames, and therefore the magnitude of pixel values ​​is easily reversed, making the 0 / 1 determination prone to change from frame to frame. If the feature changes for each frame, the number of bits that differ from the feature point that you want to match increases, which can lead to incorrect matching, where the wrong feature point is selected. Therefore, by calculating the similarity without using the bit signal of the feature specified by the feature mask signal, it is possible to suppress incorrect matching due to fluctuations in the feature.

[0044] When calculating similarity, feature points calculated as feature point 405 using only three of the four bits of the feature mask signal cannot be accurately calculated, and are therefore excluded from matching. In this way, the matching unit 125 excludes feature points that do not use more than a predetermined number of bits from matching candidates. Furthermore, when there are multiple feature points with the fewest number of difference bits using the feature mask signal, such as feature points 402 and 403, feature point 402, whose feature mask signal has the fewest number of 0 bits, is calculated as the feature point with the highest similarity. Well-known methods can be used to calculate the feature point with the highest similarity. The BF method is a method for calculating similarities with all feature points and finding the feature point with the highest similarity. BF stands for Brute Force. Alternatively, the FLANN method, which finds similar feature points using an approximate nearest neighbor search, may be used. FLANN stands for Fast Library for Approximate Nearest Neighbors.

[0045] In step S208, the orientation estimation unit 126 estimates the orientation of the image capture device from the motion vector information calculated by the matching unit 125. A known method can be used to estimate the orientation. For example, there is a method called an 8-point algorithm that uses eight or more motion vectors to estimate a fundamental matrix that represents orientation information.

[0046] As described above, according to this embodiment, by using the detection signals of pixels where flickering occurs and pixels where saturation occurs, it is possible to detect feature amount bits of feature points whose feature amounts are likely to fluctuate. This allows feature points to be matched taking feature amount fluctuations into consideration, thereby improving the accuracy of feature point matching.

[0047] [Embodiment 2] An image processing system according to a second embodiment of the present invention will be described. In the second embodiment, an example will be described in which the amount of parallax is calculated from the result of feature point matching for a plurality of images that include the effects of flicker, in order to calculate the amount of parallax for images captured by a stereo camera.

[0048] Fig. 6 is a block diagram showing the configuration of an image processing system according to embodiment 2 of the present invention. Fig. 7 is a block diagram showing the internal configuration of the image processing unit 504 shown in Fig. 6, which is a block diagram of the image processing system according to embodiment 2 of the present invention. The second embodiment has two sets of the image capturing unit 100 and the signal processing unit 110 of embodiment 1, and since the constituent blocks have overlapping functions, the same reference numerals are used and descriptions of the blocks with the same reference numerals will be omitted.

[0049] <Image capture unit> The imaging unit 501 has two pairs of lenses 101 and imaging elements 102. The imaging unit 501 is, for example, a stereo camera, and the two pairs of lenses 101 and imaging elements 102 are arranged so that they can capture images of the same subject with parallax. The imaging unit 501 transmits high-sensitivity image signals and low-sensitivity image signals captured by the two pairs of lenses 101 and imaging elements 102 to the image processing device 502. Note that the combination of the lenses 101 and imaging elements 102 may be called a camera.

[0050] <Image processing device> The image processing device 502 includes a signal processing unit 503 and an image processing unit 504 . <Signal processing section> The signal processing unit 503 will now be described. The signal processing unit 503 has two sets of state detection units 111 and signal synthesis units 112. In the second embodiment, two sets of image synthesis signals, a flicker detection signal, and a saturation detection signal generated by the two sets of state detection units 111 and signal synthesis units 112 are output to the image processing unit 504.

[0051] <Image processing unit> Next, we will explain the image processing unit 504. The image processing unit 504 has two sets of image signal storage units 121, a feature point calculation unit 122, a feature amount generation unit 123, a feature point signal storage unit 124, a matching unit 505, and a disparity calculation unit 506.

[0052] The matching unit 505 performs feature point matching using the two sets of feature point signals, feature amount signals, and feature amount mask signals output from the two sets of feature point signal storage units 124, and calculates a positional relationship vector between the two cameras, i.e., the two image capture devices. The matching unit 505 outputs the calculated positional relationship vector to the disparity calculation unit 506.

[0053] The parallax calculation unit 506 calculates the amount of parallax using the positional relationship vector calculated by the matching unit 505, and outputs it as parallax information.

[0054] <Flowchart> Fig. 8 is a flowchart showing the operation of an image processing device 502 according to the second embodiment of the present invention. The control operation of the image processing device 502 will be described step by step using Fig. 8. In this embodiment, an example will be described in which feature point matching is performed using two sets of feature point signals, feature amount signals, and feature amount mask signals calculated from images from a stereo camera, and the amount of parallax is calculated. However, functions that overlap with those in the previous embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0055] The processing from step S201 to step S205 is executed in parallel for two sets of high-sensitivity image signals and low-sensitivity image signals input to the image processing device 502 from two sets of lenses 101 and image sensors 102.

[0056] In step S201, the state detection unit 111 generates and outputs a flicker detection signal and a saturation detection signal based on the high-sensitivity image signal and low-sensitivity image signal received from the imaging unit 501. In step S202, the feature point calculation unit 122 calculates feature points from the HDR synthesis signal and outputs them as feature point signals. In step S203, the feature generation unit 123 calculates and generates a feature signal and a feature mask signal for each feature point calculated in step S202.

[0057] In step S204, the feature generation unit 123 generates feature signals and feature mask signals for all feature points and determines whether generation is complete. If the feature generation unit 123 determines that generation is complete, the process of step S205 is executed. If the feature generation unit 123 determines that generation is not complete, the process of step S203 is executed. In step S205, the feature generation unit 123 stores the feature signals and feature mask signals of the current frame to be used in step S601 in the feature point signal storage unit 124.

[0058] In step S601, the matching unit 505 performs feature point matching between the cameras using the two sets of feature signals and feature mask signals saved in step S205. In the first embodiment, feature points of the current frame are matched with feature points of the previous frame, whereas in this embodiment, feature points of the same frame calculated from images from the two cameras are matched between the cameras. The matching method is the same as in the first embodiment, so a description thereof will be omitted. As a result of feature point matching, a positional relationship vector between the two cameras is calculated. In step S602, the parallax calculation unit 506 calculates the amount of parallax using the positional relationship vector calculated in step S601.

[0059] As described above, according to this embodiment, in comparing feature points and feature amounts calculated from images captured by two cameras, a feature amount mask signal is generated, thereby making it possible to suppress erroneous matching due to fluctuations in feature amounts. As a result, the parallax amount of a flicker area can be calculated with high accuracy.

[0060] (Other embodiments) The technology disclosed in this specification can also be realized by executing the following process. That is, the disclosed technology can also be realized by supplying a software program that realizes the functions of the above-described embodiments to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the program. The computer includes a CPU, an MPU, etc. CPU is an abbreviation for Central Processing Unit. MPU is an abbreviation for Micro Processing Unit.

[0061] A computer may have one or more processors or circuits, and may include separate computers or a network of separate processors or circuits, for reading and executing computer-executable instructions. The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an ASIC, or an FPGA. ASIC stands for application-specific integrated circuit. FPGA stands for field-programmable gateway. The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0062] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0063] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention.

[0064] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image signal acquisition means for acquiring an image signal captured by an imaging device; a feature point calculation means for calculating feature points from the acquired image signal; a flicker detection signal generating means for generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating means for generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit value and are saturated; a feature signal calculation means for calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; a feature mask signal generating means for generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; 1. An image processing device comprising: (Configuration 2) The acquired image signals include a long-exposure image signal obtained by capturing an image of a subject with a first exposure time longer than the flicker period, and a short-exposure image signal obtained by capturing an image of the subject with a second exposure time shorter than the flicker period. 2. The image processing device according to configuration 1, (Configuration 3) The feature signal is a signal generated according to a comparison result between the luminance value of the pixel at the feature point and the luminance values ​​of the pixels surrounding the feature point. 4. The image processing device according to any one of the configurations 1, 2, and 3, characterized in that: (Configuration 4) a matching means for performing feature point matching based on the feature signal and the feature mask signal and calculating a motion vector of the imaging device; 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) The feature mask signal is a signal indicating whether or not the feature signal is to be used when calculating the degree of match in the feature point matching. 5. The image processing device according to configuration 4. (Configuration 6) The feature mask signal generating means does not use the feature signal when calculating the degree of match in the feature point matching, if the pixel of the feature point is a pixel where flickering occurs, a pixel surrounding the feature point is a pixel where flickering occurs, and either the pixel of the feature point or the pixel surrounding the feature point is a saturated pixel. 6. The image processing device according to configuration 4 or 5, (Configuration 7) The matching means calculates a similarity between the feature point and a matching target based on the feature signal and the feature mask signal. 7. The image processing device according to any one of configurations 4 to 6, (Configuration 8) the image signal acquisition means includes a first image signal acquisition means for acquiring a first image signal captured by a first imaging device, and a second image signal acquisition means for acquiring a second image signal captured by a second imaging device; the feature point calculation means includes first feature point calculation means for calculating feature points from the first image signal and second feature point calculation means for calculating feature points from the second image signal; the flicker detection signal generating means includes a first flicker detection signal generating means for generating a first flicker detection signal that indicates a pixel in the first image signal where flicker is occurring, and a second flicker detection signal generating means for generating a second flicker detection signal that indicates a pixel in the second image signal where flicker is occurring, the saturation detection signal generating means includes a first saturation detection signal generating means for generating a first saturation detection signal that indicates a pixel in the first image signal whose pixel value exceeds an upper limit value and is saturated, and a second saturation detection signal generating means for generating a second saturation detection signal that indicates a pixel in the second image signal whose pixel value exceeds an upper limit value and is saturated, the feature signal calculation means includes first feature signal calculation means for calculating a first feature signal indicating a feature amount of the feature point based on the first image signal, and second feature signal calculation means for calculating a second feature signal indicating the feature amount of the feature point based on the second image signal, a matching means for performing feature point matching based on the first feature amount signal, the second feature amount signal, and the first feature amount mask signal, and the second feature amount mask signal, and calculating a positional relationship vector between the first image pickup device and the second image pickup device; 4. The image processing device according to any one of configurations 1 to 3. (Method 1) an image signal acquisition step of acquiring an image signal captured by the imaging device; a feature point calculation step of calculating feature points from the acquired image signal; a flicker detection signal generating step of generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating step of generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit value and are saturated; a feature signal calculation step of calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; a feature mask signal generating step of generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; An image processing method comprising: (Program 1) Computer, an image signal acquisition means for acquiring an image signal captured by the imaging device; a feature point calculation means for calculating feature points from the acquired image signal; a flicker detection signal generating means for generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating means for generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit and are saturated; a feature signal calculation means for calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; and a feature mask signal generating means for generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; A program characterized by functioning as [Explanation of symbols]

[0065] 10 Image processing device 100 Imaging unit 101 Lens 102 Image sensor 110 Signal processing section 111 Status detection unit 112 Signal synthesis unit 120 Image processing unit 121 Image signal storage unit 122 Feature point calculation unit 123 Feature Generation Unit 124 Feature point signal storage section 125 Matching Section 126 Posture estimation section

Claims

1. an image signal acquisition means for acquiring an image signal captured by an imaging device; a feature point calculation means for calculating feature points from the acquired image signal; a flicker detection signal generating means for generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating means for generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit value and are saturated; a feature signal calculation means for calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; a feature mask signal generating means for generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; 1. An image processing device comprising:

2. The acquired image signals include a long-exposure image signal obtained by capturing an image of a subject with a first exposure time longer than the flicker period, and a short-exposure image signal obtained by capturing an image of the subject with a second exposure time shorter than the flicker period.

2. The image processing device according to claim 1, wherein:

3. The feature signal is a signal generated according to a comparison result between the luminance value of the pixel at the feature point and the luminance values ​​of the pixels surrounding the feature point.

2. The image processing device according to claim 1, wherein:

4. a matching means for performing feature point matching based on the feature signal and the feature mask signal and calculating a motion vector of the imaging device; 2. The image processing device according to claim 1, wherein:

5. The feature mask signal is a signal indicating whether or not the feature signal is to be used when calculating the degree of match in the feature point matching.

5. The image processing apparatus according to claim 4.

6. The feature mask signal generating means does not use the feature signal when calculating the degree of match in the feature point matching, if the pixel of the feature point is a pixel where flickering occurs, a pixel surrounding the feature point is a pixel where flickering occurs, and either the pixel of the feature point or the pixel surrounding the feature point is a saturated pixel.

5. The image processing apparatus according to claim 4.

7. The matching means calculates a similarity between the feature point and a matching target based on the feature signal and the feature mask signal.

5. The image processing apparatus according to claim 4.

8. the image signal acquisition means includes a first image signal acquisition means for acquiring a first image signal captured by a first imaging device, and a second image signal acquisition means for acquiring a second image signal captured by a second imaging device; the feature point calculation means includes first feature point calculation means for calculating feature points from the first image signal and second feature point calculation means for calculating feature points from the second image signal, the flicker detection signal generating means includes a first flicker detection signal generating means for generating a first flicker detection signal indicating a pixel in the first image signal where flicker is occurring, and a second flicker detection signal generating means for generating a second flicker detection signal indicating a pixel in the second image signal where flicker is occurring, the saturation detection signal generating means includes a first saturation detection signal generating means for generating a first saturation detection signal that indicates a pixel in the first image signal whose pixel value exceeds an upper limit value and is saturated, and a second saturation detection signal generating means for generating a second saturation detection signal that indicates a pixel in the second image signal whose pixel value exceeds an upper limit value and is saturated, the feature signal calculation means includes first feature signal calculation means for calculating a first feature signal indicating a feature amount of the feature point based on the first image signal, and second feature signal calculation means for calculating a second feature signal indicating the feature amount of the feature point based on the second image signal, a matching unit that performs feature point matching based on the first feature amount signal, the second feature amount signal, and the first feature amount mask signal, and the second feature amount mask signal, and calculates a positional relationship vector between the first image pickup device and the second image pickup device; 2. The image processing device according to claim 1, wherein:

9. an image signal acquisition step of acquiring an image signal captured by the imaging device; a feature point calculation step of calculating feature points from the acquired image signal; a flicker detection signal generating step of generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating step of generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit value and are saturated; a feature signal calculation step of calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; a feature mask signal generating step of generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; An image processing method comprising:

10. Computer, an image signal acquisition means for acquiring an image signal captured by the imaging device; a feature point calculation means for calculating feature points from the acquired image signal; a flicker detection signal generating means for generating a flicker detection signal indicating a pixel in which flicker occurs from the acquired image signal; a saturation detection signal generating means for generating a saturation detection signal indicating pixels in the acquired image signal whose pixel values ​​exceed an upper limit and are saturated; a feature signal calculation means for calculating a feature signal indicating a feature amount of the feature point based on the acquired image signal; and a feature mask signal generating means for generating a feature mask signal indicating a mask of the feature signal based on the flicker detection signal and the saturation detection signal; A program characterized by functioning as

Images