Image processing apparatus

Abstract

To calculate the shape of a flicker region with high accuracy.SOLUTION: An image processing apparatus has: a first state detection part 111 which detects a flicker detection signal of each pixel based upon a first image signal captured for a first exposure time and a second image signal captured for a second exposure time different from the first exposure time; and a first clicker shape calculation part 121 which calculates the shape of a flicker region based upon the flicker detection signal detected by the first state detection part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing system, a processing method for an image processing device, and a program. [Background technology]

[0002] Digital cameras mounted on vehicles and industrial robots are required to have the ability to generate wide dynamic range images (hereinafter referred to as HDR (High Dynamic Range) function) in order to accurately recognize the environment even under conditions with large differences in brightness. To obtain wide dynamic range images, images are acquired at multiple signal levels. In recent years, image sensors with HDR function have sometimes been unable to capture images due to flickering caused by LEDs, such as those in traffic lights.

[0003] Patent Document 1 discloses an imaging element that has a function of detecting and eliminating flicker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-180459 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Patent Document 1 is unable to acquire the shape of the pixel region where flickering occurs. By acquiring the shape of the pixel region where flickering occurs, it can be used for pose estimation by feature point matching using shape information. The Canny method is generally used to extract shapes, but there is a problem in that accuracy is low because the threshold for the gradient of pixel values ​​that is regarded as shape cannot be uniquely determined.

[0006] An object of the present disclosure is to enable the shape of a flicker region to be calculated with high accuracy. [Means for solving the problem]

[0007] The image processing device has a first state detection unit that detects a flicker detection signal for each pixel based on a first image signal captured with a first exposure time and a second image signal captured with a second exposure time different from the first exposure time, and a first flicker shape calculation unit that calculates the shape of a flicker area based on the flicker detection signal detected by the first state detection unit. [Effects of the Invention]

[0008] According to the present disclosure, the shape of a flicker region can be calculated with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing system. [Figure 2] 10 is a flowchart illustrating an example of the operation of an image processing unit. [Figure 3] 10 is a flowchart illustrating an example of the operation of a flicker shape calculation unit. [Figure 4] FIG. 10 is a diagram illustrating a method for calculating a shape image of a flicker region. [Figure 5] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing system. [Figure 6] 10 is a flowchart illustrating an example of the operation of an image processing unit. [Figure 7] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing system. [Figure 8] 10 is a flowchart illustrating an example of the operation of an image processing unit. [Figure 9] FIG. 10 is a diagram illustrating a motion vector removal method performed by the image processing unit. [Figure 10] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing system. [Figure 11] 10 is a flowchart illustrating an example of the operation of an image processing unit. [Figure 12] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing system. [Figure 13] 10 is a flowchart illustrating an example of the operation of an image processing unit. DETAILED DESCRIPTION OF THE INVENTION

[0010] (First embodiment) FIG. 1 is a block diagram showing an example of the configuration of an image processing system 130 according to the first embodiment. The image processing system 130 according to the first embodiment will be described in detail below with reference to the drawings. To estimate the posture of the image capturing unit 100, the image processing system 130 calculates a flicker shape by detecting flicker for each pixel, and calculates feature amounts by superimposing the feature points calculated using the image signal on a flicker shape image. An example will be described in which the image processing system 130 estimates the posture of the image capturing unit 100 by performing feature point matching using the calculated feature amounts.

[0011] The image processing system 130 includes an imaging unit 100 and an image processing device 10. The image processing device 10 includes a signal processing unit 110 and an image processing unit 120.

[0012] <Image capture unit> The imaging unit 100 includes a lens 101 and an imaging element 102 .

[0013] The lens 101 is a photographing lens, and is a device that forms an image of a subject on the image sensor 102 that is the image pickup surface of the image pickup unit 100 .

[0014] The image sensor 102 captures an input subject in a plurality of different dynamic ranges and transmits the captured images as image signals to the signal processing unit 110. The image sensor 102 is an image sensor configured from a CMOS (complementary metal oxide semiconductor) or a CCD (charge coupled device).

[0015] An object image formed on an image sensor 102 via a lens 101 is converted by the image sensor 102 into an image signal for each dynamic range for each pixel, and is sent to a signal processing unit 110 .

[0016] In the first embodiment, the image sensor 102 transmits image signals captured in two different dynamic ranges for one pixel to the signal processing unit 110. The image signals captured in the two different dynamic ranges are set as an image signal captured with a long exposure time and an image signal captured with a short exposure time.

[0017] However, the image signal obtained by long-time exposure is an image signal captured with low-sensitivity light-receiving characteristics (hereinafter referred to as a low-sensitivity image signal), while the image signal obtained by short-time exposure is an image signal captured with high-sensitivity light-receiving characteristics and has a brighter dynamic range than the image signal obtained by long-time exposure (hereinafter referred to as a high-sensitivity image signal).

[0018] The long-exposure image signal is an image signal captured with an exposure time longer than the frequency of the detected flicker. The short-exposure image signal is an image signal captured with an exposure time shorter than the frequency of the detected flicker. The image sensor 102 generates a high-sensitivity image signal and a low-sensitivity image signal.

[0019] Furthermore, the present embodiment is not limited to a configuration for acquiring two different dynamic ranges, and may be a configuration for acquiring image signals of two or more dynamic ranges. As a configuration for acquiring image signals of two or more dynamic ranges, pixels having at least two or more different sensitivities may be acquired, or the signals may be acquired by exposing multiple times in the time direction.

[0020] <Signal processing section> Next, a description will be given of the signal processing unit 110. The signal processing unit 110 includes a state detection unit 111 and a signal synthesis unit 112.

[0021] The state detection unit 111 receives a high-sensitivity image signal and a low-sensitivity image signal from the imaging unit 100. The state detection unit 111 uses the high-sensitivity image signal and the low-sensitivity image signal to detect whether or not a pixel is experiencing flicker.

[0022] Pixels where flickering occurs cannot receive light from the high-sensitivity image signal with a short exposure time due to the influence of the flicker frequency, so the signal value becomes small.On the other hand, pixels where flickering occurs can receive light from the low-sensitivity image signal with a long exposure time, so the signal value becomes large.

[0023] Therefore, for a pixel where flickering occurs, the pixel value of the low-sensitivity image signal, which is different from normal, is larger than the pixel value of the high-sensitivity image signal. In this case, the state detection unit 111 detects the pixel where flickering occurs as a flicker detection signal.

[0024] The state detection unit 111 outputs a flicker detection signal of 1 for each pixel when it detects a pixel in which flicker is occurring, or 0 when it does not detect a pixel in which flicker is occurring, to the signal synthesis unit 112. The output format may be, for example, such that the high-sensitivity image signal, low-sensitivity image signal, and flicker detection signal are output as independent signals, although the output format is not limited to this.

[0025] The signal synthesis unit 112 generates a wide dynamic range image (wide dynamic range image signal) for each frame by synthesizing the high-sensitivity image signal and the low-sensitivity image signal received from the state detection unit 111. At this time, the signal synthesis unit 112 determines the ratio (synthesis ratio) used to synthesize the high-sensitivity image signal and the low-sensitivity image signal from the signal values ​​of the high-sensitivity image signal and the low-sensitivity image signal.

[0026] The combination ratios are given as a combination ratio α1 for the high-sensitivity image signal and a combination ratio α2 for the low-sensitivity image signal. The signal combination unit 112 sets α1 and α2 between 0 and 1 so that the sum of α1 and α2 is 1.

[0027] The signal synthesis unit 112 synthesizes the high-sensitivity image signal and the low-sensitivity image signal for all pixels using synthesis ratios α1 and α2, thereby generating an image with a wide dynamic range.

[0028] Then, the signal synthesis unit 112 generates a flicker detection image based on the flicker detection signal. The flicker detection image is an image in which pixels where flicker occurs are 1 and pixels where flicker does not occur are 0, as shown in Fig. 4(b), for example.

[0029] Then, the signal synthesis unit 112 outputs the generated wide dynamic range image and flicker detection image to the image processing unit 120.

[0030] <Image processing unit> Next, a description will be given of the image processing unit 120. The image processing unit 120 has a flicker shape calculation unit 121, a feature point calculation unit 122, a feature point superimposition unit 123, a feature amount calculation unit 124, a matching unit 125, a feature amount storage unit 126, and a posture estimation unit 127.

[0031] The flicker shape calculation unit 121 calculates a shape image of the flicker region based on the flicker detection image. The flicker shape calculation unit 121 sets each pixel as a target pixel, starting from the coordinates of the top left of the flicker detection image. The flicker shape calculation unit 121 compares the values ​​of the target pixel, the pixel immediately to the left of the target pixel, and the pixel immediately above the target pixel. If any of the three pixel values ​​is different from the other pixels, the flicker shape calculation unit 121 outputs a shape image of the flicker region with a pixel value of 1 to the feature point superimposition unit 123. If the three pixel values ​​are the same, the flicker shape calculation unit 121 outputs a shape image of the flicker region with a pixel value of 0 to the feature point superimposition unit 123.

[0032] The feature point calculation unit 122 calculates feature points based on the wide dynamic range image, and outputs the calculated feature points to the feature point superimposition unit 123 as first feature point information.

[0033] The feature point superimposing unit 123 superimposes the shape image of the flicker region on the first feature point information, and outputs the result to the feature amount calculating unit 124 as second feature point information.

[0034] The feature amount calculation unit 124 calculates the feature amount based on the second feature point information, and outputs the calculated feature amount to the matching unit 125 and the feature amount storage unit 126 as feature amount information.

[0035] The feature amount storage unit 126 stores the feature amount information calculated by the feature amount calculation unit 124. A memory such as an SRAM is used for storage. When the matching unit 125 requires the feature amount information and requests output, the feature amount storage unit 126 outputs the feature amount information to the matching unit 125.

[0036] Matching unit 125 performs feature point matching using the feature amount information, calculates a motion vector, and outputs it to posture estimation unit 127.

[0037] The posture estimation unit 127 uses the calculated motion vector to estimate a homography matrix and outputs it as posture information of the image capture unit 100.

[0038] <Flowchart> Fig. 2 is a flowchart showing an example of a processing method of the image processing unit 120 according to the first embodiment. Each step of the processing method will be described using Fig. 2. In this embodiment, an example will be described in which the posture is estimated from the amount of movement of the imaging unit 100 between frames.

[0039] In step S201, flicker shape calculation unit 121 calculates a shape image of the flicker region from the flicker detection image. The control operation for the process of calculating the shape image will be described with reference to FIG.

[0040] Fig. 3 is a flowchart showing the details of the process of step S201 in Fig. 2. The flicker shape calculation unit 121 processes the target pixel for each pixel of the flicker detection image.

[0041] In step S301, the flicker shape calculation unit 121 compares the values ​​of the target pixel, the pixel immediately to the left of the target pixel, and the pixel immediately above the target pixel in the flicker detection image.

[0042] In step S302, if any of the values ​​of the target pixel, the pixel immediately to the left of the target pixel, and the pixel immediately above the target pixel is different from the other pixels, the flicker shape calculation unit 121 proceeds to step S303. If the values ​​of the target pixel, the pixel immediately to the left of the target pixel, and the pixel immediately above the target pixel are the same, the flicker shape calculation unit 121 proceeds to step S304.

[0043] In step S303, the flicker shape calculation unit 121 sets the pixel value of the target pixel in the shape image of the flicker region to 1, and the process proceeds to step S305.

[0044] In step S304, the flicker shape calculation unit 121 sets the pixel value of the target pixel in the shape image of the flicker region to 0, and the process proceeds to step S305.

[0045] In step S305, flicker shape calculation unit 121 determines whether or not processing of all pixels in the flicker detection image has been completed. If processing of all pixels has not been completed, flicker shape calculation unit 121 returns to step S301 with the next pixel as the target pixel. If processing of all pixels has been completed, flicker shape calculation unit 121 ends the processing of the flowchart in FIG. 3 and proceeds to step S202 in FIG. 2.

[0046] Note that any method capable of acquiring the shape of the flicker region may be used other than the process in Fig. 3. Here, an example of a method for calculating a shape image of a flicker region for one image will be described with reference to Fig. 4.

[0047] Fig. 4(a) shows a wide dynamic range image output from the signal processing unit 110. Fig. 4(b) shows a flicker detection image output from the signal processing unit 110. Fig. 4(c) shows an image extracted from the flicker detection image of Fig. 4(b) to show an example of calculating a shape image of the flicker region. Fig. 4(d) shows a shape image of the flicker region.

[0048] In the flicker detection image of FIG. 4(c), each pixel is filled with white if the value is 1 and black if the value is 0, and a calculation method using this flicker detection image will be described.

[0049] In step S301, the flicker shape calculation unit 121 compares the values ​​of the target pixel, the pixel immediately to the left of the target pixel, and the pixel immediately above the target pixel, in the flicker detection image of FIG. 4(c), starting from the coordinate (1,1).

[0050] In step S302, the flicker shape calculation unit 121 determines that the values ​​of the three pixels at coordinates (1,1) to (6,5) are the same, so the process proceeds to step S304. In step S304, the flicker shape calculation unit 121 sets the pixel values ​​of the shape image of the flicker area to 0 for coordinates (1,1) to (6,5), as shown in FIG. 4(d).

[0051] In step S302, the flicker shape calculation unit 121 proceeds to step S303 because the value of the target pixel for the coordinates (7,5) is different from the values ​​of the pixel to the left and the pixel above. In step S303, the flicker shape calculation unit 121 sets the pixel value of the shape image of the flicker area for the coordinates (7,5) to 1, as shown in FIG. 4(d).

[0052] Similarly, in step S302, for the coordinates (8,5) and (7,6), the value of the target pixel differs from the value of the pixel above or to the left, so the flicker shape calculation unit 121 proceeds to step S303. In step S303, the flicker shape calculation unit 121 sets the pixel value of the shape image of the flicker area to 1 for the coordinates (8,5) and (7,6), as shown in FIG.

[0053] In step S302, the flicker shape calculation unit 121 determines that the values ​​of the three pixels at coordinates (1,6) to (6,6) are the same, and therefore proceeds to step S304. In step S304, the flicker shape calculation unit 121 sets the pixel values ​​of the shape image of the flicker area to 0 for coordinates (1,6) to (6,6), as shown in FIG. 4(d).

[0054] In step S302, the flicker shape calculation unit 121 determines that the values ​​of the three pixels at the coordinate (8,6) are the same, so the process proceeds to step S304. In step S304, the flicker shape calculation unit 121 sets the pixel value of the shape image of the flicker area at the coordinate (8,6) to 0, as shown in FIG. 4(d).

[0055] As a result of the above, a shape image of the flicker region is generated as shown in Fig. 4(d). The shape image of the flicker region shows the outline of the flicker region.

[0056] 2, the feature point calculation unit 122 calculates feature points from the wide dynamic range image for each frame and outputs the calculated feature points as first feature point information to the feature point superimposition unit 123. A known method can be used to calculate the feature points. For example, there is a method called FAST (Features from Accelerated Segment Test) that detects only the corners of an object as feature points.

[0057] In step S203, the feature point superimposing unit 123 superimposes, for each frame, the feature points of the shape image of the flicker region in step S201 on the first feature point information in step S202 to obtain second feature point information. Since the coordinates where the pixel value of the shape image of the flicker region is 1 become feature points, the feature point superimposing unit 123 converts the feature points into a feature point format and superimposes the converted feature points on the first feature point information to obtain second feature point information.

[0058] In step S204, if there are any coordinates where the coordinates of the shape image of the flicker region overlap with the coordinates of the first feature point information, the feature point superimposition unit 123 removes the overlapping feature points from the second feature point information, updates the second feature point information, and outputs it to the feature amount calculation unit 124.

[0059] In step S205, the feature amount calculation unit 124 calculates the feature amount of each feature point in the second feature point information for each frame, and outputs the calculated feature amount as feature amount information to the matching unit 125 and the feature amount storage unit 126. A known method can be used to calculate the feature amount. For example, a method called BRIEF (Binary Robust Independent Element Features) can be used, in which multiple line segments are randomly arranged around a feature point, and bits are generated by comparing the magnitude of the luminance values ​​at both ends of the line segments.

[0060] In step S206, the feature amount calculation unit 124 determines whether feature amount calculation for all feature points of the second feature point information has been completed. If feature amount calculation for all feature points has not been completed, the feature amount calculation unit 124 returns to step S205, and if feature amount calculation for all feature points has been completed, the feature amount calculation unit 124 proceeds to step S207.

[0061] In step S207, the matching unit 125 acquires, from the feature storage unit 126, feature information of the previous frame required for performing feature point matching.

[0062] In step S208, the feature amount storage unit 126 stores the feature amount information of the current frame to be used in the next frame in step S205.

[0063] In step S209, the matching unit 125 performs feature point matching using the feature amount information of the current frame calculated in step S205 and the feature amount information of the previous frame acquired in step S207, calculates a motion vector, and outputs the motion vector to the posture estimation unit 127. A known method can be used for matching. For example, the BF (Brute Force) method can be used, which calculates the distance to all feature points using Euclidean distance or the like to find the feature point with the shortest distance. Alternatively, the FLANN (Fast Library for Approximate Nearest Neighbors) method can be used, which finds similar feature points using an approximate nearest neighbor search.

[0064] In step S210, the posture estimation unit 127 estimates the posture of the image capture unit 100 based on the calculated motion vector. A known method can be used to estimate the posture. For example, a method called an 8-point algorithm can be used, which uses eight or more motion vectors to estimate a fundamental matrix representing posture information.

[0065] As described above, according to this embodiment, the image processing device 10 can improve the accuracy of shape extraction by using a shape image of a flicker region where flickering occurs. This enables matching using better feature points using the shape image of the flicker region, thereby improving the accuracy of pose estimation.

[0066] (Second embodiment) 5 is a block diagram showing an example of the configuration of an image processing system 130 according to the second embodiment. Functions of the second embodiment that overlap with those of the first embodiment are given the same reference numerals, and descriptions thereof will be omitted.

[0067] In the second embodiment, an example will be described in which posture estimation is performed using a motion vector that superimposes matching results of feature points calculated using an acquired shape image of a flicker region and a wide dynamic range image. By performing feature point matching using feature points calculated from the shape image of the flicker region and feature points calculated from the wide dynamic range image, it is possible to reduce the number of combinations to be compared.

[0068] 5 is obtained by deleting the feature point superimposing unit 123 and adding a motion vector superimposing unit 521 compared to FIG. 1. The image processing system 130 includes an imaging unit 100 and an image processing device 10. The image processing device 10 includes a signal processing unit 110 and an image processing unit 520. The image processing unit 520 is obtained by deleting the feature point superimposing unit 123 and adding the motion vector superimposing unit 521 compared to the image processing unit 120 in FIG. 1. Furthermore, the image processing unit 520 includes first and second feature amount calculation units 124 and first and second matching units 125.

[0069] The imaging unit 100 and the signal processing unit 110 have functions that overlap with those of the first embodiment, and output a wide dynamic range image and a flicker detection image to the image processing unit 520.

[0070] <Image processing unit> The image processing unit 520 will now be described. The first feature amount calculation unit 124 calculates the feature amount of each feature point of the shape image of the flicker region calculated by the flicker shape calculation unit 121, and outputs the calculated feature amount information to the first matching unit 125 and the feature amount storage unit 126. The second feature amount calculation unit 124 calculates the feature amount of each feature point of the first feature point information calculated by the feature point calculation unit 122, and outputs the calculated feature amount information to the second matching unit 125 and the feature amount storage unit 126.

[0071] As in the first embodiment, the first and second matching units 125 perform matching using the feature amount information calculated by the first and second feature amount calculation units 124 and the feature amount information stored in the feature amount storage unit 126, calculate a motion vector, and output it to the motion vector superimposition unit 521. At this time, the first matching unit 125 calculates a motion vector based on the feature amount of each feature point in the shape image of the flicker area and outputs it to the motion vector superimposition unit 521. The second matching unit 125 calculates a motion vector based on the feature amount of each feature point in the first feature point information and outputs it to the motion vector superimposition unit 521.

[0072] The motion vector superimposition unit 521 superimposes a motion vector based on the feature amount of each feature point of the shape image of the flicker area with a motion vector based on the feature amount of each feature point of the first feature point information, and outputs the superimposed motion vector to the posture estimation unit 127.

[0073] The posture estimation unit 127 uses the superimposed motion vector to estimate a homography matrix and outputs it as posture information of the image capture unit 100.

[0074] <Flowchart> Fig. 6 is a flowchart showing an example of a processing method of the image processing unit 520 according to the second embodiment. Each step of the processing method will be described using Fig. 6. In this embodiment, an example will be described in which the posture is estimated from the amount of movement of the imaging unit 100 between frames.

[0075] In step S601, flicker shape calculation unit 121 calculates a shape image of the flicker region from the flicker detection image. Flicker shape calculation unit 121 calculates a shape image of the flicker region from the flicker detection image. The process for calculating the shape image is the same as that described above with reference to FIG. 3.

[0076] In step S602, the first feature calculation unit 124 calculates a feature for each frame by using pixels in the shape image of the flicker region whose pixel value is 1 as feature points, and outputs the calculated feature as feature information to the first matching unit 125 and the feature storage unit 126. A known method can be used to calculate the feature. For example, a method called BRIEF (Binary Robust Independent Element Features) can be used, in which multiple line segments are randomly arranged around the feature points and bits are generated by comparing the magnitude of the luminance values ​​at both ends of the line segments.

[0077] In step S603, the first feature amount calculation unit 124 determines whether feature amount calculation has been completed for all feature points in the shape image of the flicker region. If feature amount calculation for all feature points has not been completed, the first feature amount calculation unit 124 returns to step S602, and if feature amount calculation for all feature points has been completed, the first feature amount calculation unit 124 proceeds to step S604.

[0078] In step S604, the first matching unit 125 acquires, from the feature storage unit 126, feature amount information of the previous frame required for performing feature point matching on the shape image of the flicker region.

[0079] In step S605, the feature amount storage unit 126 stores the feature amount information of the current frame, which is to be used in the next frame in step S602, for the shape image of the flicker region.

[0080] In step S606, the first matching unit 125 determines parameters required for matching of the shape image of the flicker region according to the number of feature points. Because the number of feature points obtained from flicker varies greatly for each scene, the recursive parameter setting parameters of the FLANN method may be set according to the number of feature points. Alternatively, if the number of feature points is smaller than a preset threshold, the BF method may be used, and if the number is larger, the FLANN method may be used.

[0081] In step S607, the first matching unit 125 performs feature point matching using the matching method determined in step S606 and parameters required for matching. A known matching method can be used. For example, the BF method or the FLANN method can be used. The first matching unit 125 performs feature point matching using the feature amount information of the current frame calculated in step S602 and the feature amount information of the previous frame acquired in step S604, calculates a motion vector, and outputs it to the posture estimation unit 127. Then, the process proceeds to step S614.

[0082] In step S608, the feature point calculation unit 122 calculates feature points from the wide dynamic range image for each frame and outputs the calculated feature points as first feature point information to the feature amount calculation unit 124. A known method can be used to calculate the feature points. For example, there is a method called FAST (Features from Accelerated Segment Test) that detects only the corners of an object as feature points.

[0083] In step S609, the second feature amount calculation unit 124 calculates the feature amount of each feature point in the first feature amount information for each frame, and outputs the calculated feature amount as feature amount information to the second matching unit 125 and the feature amount storage unit 126. A known method can be used to calculate the feature amount. For example, a method called BRIEF (Binary Robust Independent Element Features) can be used, in which multiple line segments are randomly arranged around a feature point, and bits are generated by comparing the magnitude of the luminance values ​​at both ends of the line segments.

[0084] In step S610, the second feature amount calculation unit 124 determines whether feature amount calculation for all feature points of the first feature point information has been completed. If feature amount calculation for all feature points has not been completed, the second feature amount calculation unit 124 returns to step S609, and if feature amount calculation for all feature points has been completed, the second feature amount calculation unit 124 proceeds to step S611.

[0085] In step S611, the second matching unit 125 acquires, from the feature storage unit 126, feature amount information of the previous frame that is required for performing feature point matching for the first feature point information.

[0086] In step S612, the feature amount storage unit 126 stores the feature amount information of the current frame to be used in the next frame in step S609, for the first feature point information.

[0087] In step S613, the second matching unit 125 performs feature point matching using the feature amount information of the current frame calculated in step S609 and the feature amount information of the previous frame acquired in step S611, calculates a motion vector, and outputs it to the motion vector superimposition unit 521. A known method can be used for matching. For example, a BF method can be used, which calculates the distance to all feature points using Euclidean distance or the like to find the feature point with the shortest distance. Alternatively, a FLANN method can be used, which finds similar feature points using an approximate nearest neighbor search.

[0088] In step S614, motion vector convolution section 521 convolves the motion vector calculated in step S607 and the motion vector calculated in step S613, and outputs the convolved motion vector to posture estimation section 127.

[0089] In step S615, the posture estimation unit 127 estimates the posture of the image capture unit 100 based on the motion vectors convolved in step S614. A known method can be used to estimate the posture. For example, a method called an 8-point algorithm can be used, which uses eight or more motion vectors to estimate a fundamental matrix representing posture information.

[0090] As described above, according to this embodiment, the image processing device 10 can reduce the number of combinations of feature point matching while maintaining better posture estimation accuracy by using a shape image of a flicker region, thereby enabling processing with low delay.

[0091] (Third embodiment) 7 is a block diagram showing an example of the configuration of an image processing system 130 according to the third embodiment. Functions of the third embodiment that overlap with those of the second embodiment are given the same reference numerals, and descriptions thereof will be omitted.

[0092] In the third embodiment, to perform posture estimation, a motion vector obtained from a matching result at feature points calculated using an acquired shape image of a flicker region where flicker is occurring is used as a representative vector. An example of a case where posture estimation is performed by removing motion vectors with large differences from the representative vector from the matching result at feature points calculated using a wide dynamic range image will be described.

[0093] 7 is obtained by deleting the motion vector superimposing unit 521 and adding a motion vector removal unit 721 compared to FIG. 5. The image processing system 130 includes an imaging unit 100 and an image processing device 10. The image processing device 10 includes a signal processing unit 110 and an image processing unit 720. The image processing unit 720 is obtained by deleting the motion vector superimposing unit 521 and adding a motion vector removal unit 721 compared to the image processing unit 520 in FIG. 5.

[0094] <Image processing unit> The motion vector removal unit 721 determines outlier motion vectors using the motion vectors calculated using the shape image of the flicker area, removes the outlier motion vectors from the motion vectors calculated using the wide dynamic range image, and outputs the result to the posture estimation unit 127.

[0095] <Flowchart> Fig. 8 is a flowchart showing an example of a processing method of the image processing unit 720 according to the third embodiment. Each step of the processing method will be described using Fig. 8. In this embodiment, an example will be described in which the posture is estimated from the amount of movement of the imaging unit 100 between frames.

[0096] In Fig. 8, step S801 is added instead of step S614 in Fig. 6. Functions of the third embodiment that overlap with those of the second embodiment are given the same reference numerals, and descriptions thereof will be omitted.

[0097] In step S801, the motion vector removal unit 721 uses the motion vector calculated in step S607 to determine outliers of the motion vector calculated in step S613 and removes them from the motion vectors used for posture estimation. The motion vectors used for determining outliers are those calculated from the shape image of the flicker region.

[0098] For example, Fig. 9(a) shows motion vectors v1 to v4 calculated from a shape image of a flicker region in a certain scene, while Fig. 9(b) shows motion vectors v1' to v7' calculated from a wide dynamic range image of the same scene as Fig. 9(a).

[0099] The motion vector v used to determine the outlier is calculated from the motion vectors v1 to v4, and for example, the average value of the motion vectors v1 to v4 can be used. Of the motion vectors v1' to v7' calculated from the wide dynamic range image, the motion vector removal unit 721 determines the motion vectors v2' and v5' that have a large difference from the motion vector v as outliers. Then, the motion vector removal unit 721 removes the outlier motion vectors v2' and v5' from the motion vectors v1' to v7', and outputs the remaining motion vectors v1', v3', v4', v6', and v7' to the posture estimation unit 127.

[0100] As described above, the motion vector removal unit 721 removes, for each frame, any motion vectors calculated in step S613 that differ by a threshold or more from the determination value based on the motion vector calculated in step S607. The determination value is, for example, the average value of the motion vectors calculated in step S607.

[0101] In step S615, the orientation estimation unit 127 estimates the orientation of the image capture unit 100 based on the motion vector output in step S801.

[0102] As described above, according to this embodiment, the image processing device 10 can improve the accuracy of the motion vectors used for pose estimation by using the motion vectors calculated from the shape image of the flicker region to remove outliers of the motion vectors calculated from the wide dynamic range image.

[0103] (Fourth embodiment) 10 is a block diagram showing an example of the configuration of an image processing system 130 according to the fourth embodiment. Functions of the fourth embodiment that overlap with those of the first embodiment are given the same reference numerals, and descriptions thereof will be omitted.

[0104] In the fourth embodiment, an example will be described in which posture estimation is performed from a matching result using shape images of a plurality of flicker regions in order to use the images from a stereo camera for rectification.

[0105] The image processing system 130 includes an imaging unit 100 and an image processing device 10. The image processing device 10 includes a signal processing unit 110 and an image processing unit 1020.

[0106] The imaging unit 100 is a stereo camera and has two pairs of lenses 101 and imaging elements 102. The two lenses 101 are a first lens 101 and a second lens 101. The two imaging elements 102 are a first imaging element 102 and a second imaging element 102, which output a high-sensitivity image signal and a low-sensitivity image signal, respectively.

[0107] The signal processing unit 110 has two sets of a state detection unit 111 and a signal synthesis unit 112. The two state detection units 111 are a first and a second state detection unit 111. The two signal synthesis units 112 are a first and a second signal synthesis unit 112, which output a flicker detection image and a wide dynamic range image, respectively.

[0108] The first state detection unit 111 detects a flicker detection signal for each pixel based on the high-sensitivity image signal and low-sensitivity image signal captured by the first imaging element 102. The second state detection unit 111 detects a flicker detection signal for each pixel based on the high-sensitivity image signal and low-sensitivity image signal captured by the second imaging element 102.

[0109] The first signal synthesis unit 112 outputs a flicker detection image and a wide dynamic range image for each frame based on the output signal of the first state detection unit 111. The second signal synthesis unit 112 outputs a flicker detection image and a wide dynamic range image for each frame based on the output signal of the second state detection unit 111.

[0110] The image processing unit 1020 has two flicker shape calculation units 121, two feature amount calculation units 1021, a matching unit 1022, a posture estimation unit 127, and a parallelization unit 1023. The two flicker shape calculation units 121 are the first and second flicker shape calculation units 121. The two feature amount calculation units 1021 are the first and second feature amount calculation units 1021.

[0111] The first flicker shape calculation unit 121 calculates the shape of the flicker region based on the output signal of the first signal synthesis unit 112. The second flicker shape calculation unit 121 calculates the shape of the flicker region based on the output signal of the second signal synthesis unit 112.

[0112] <Image processing unit> The two feature amount calculation units 1021 calculate feature amounts from the shape images of the flicker regions calculated by the two flicker shape calculation units 121, respectively, and output the calculated feature amounts to the matching unit 1022 as feature amount information.

[0113] The first feature amount calculation unit 1021 calculates feature amounts for each frame based on feature points of the shape of the flicker area calculated by the first flicker shape calculation unit 121. The second feature amount calculation unit 1021 calculates feature amounts for each frame based on feature points of the shape of the flicker area calculated by the second flicker shape calculation unit 121.

[0114] The matching unit 1022 performs feature point matching using feature amount information calculated from the shape images of the two flicker regions captured by the stereo camera, and calculates a vector indicating the positional relationship (hereinafter referred to as a positional relationship vector). Then, the matching unit 1022 outputs the calculated positional relationship vector to the posture estimation unit 127.

[0115] The posture estimation unit 127 estimates the posture from the positional relationship vector, and outputs a fundamental matrix representing the posture information to the parallelization unit 1023 .

[0116] The rectification unit 1023 calculates the tilt of the imaging unit 100 using the first wide dynamic range image (image signal) and the second wide dynamic range image (image signal) output from the first and second signal synthesis units 112 and a fundamental matrix representing the posture information output from the posture estimation unit 127. Then, the rectification unit 1023 rectifies the first wide dynamic range image and the second wide dynamic range image output from the first and second signal synthesis units 112, and outputs a first rectified image and a second rectified image.

[0117] <Flowchart> 11 is a flowchart showing an example of a processing method of the image processing unit 1020 according to the fourth embodiment. Each step of the processing method will be described with reference to FIG.

[0118] In this embodiment, an example will be described in which the posture is estimated from the tilt between the stereo cameras of the imaging unit 100 and the image is parallelized. However, functions of the fourth embodiment that overlap with those of the above-described embodiments are denoted by the same reference numerals, and description thereof will be omitted.

[0119] In step S201a, first flicker shape calculation unit 121 calculates a shape image of the first flicker region from the first flicker detection image generated by first signal synthesis unit 112. The process for calculating the shape image is the same as that described above in FIG.

[0120] In step S205a, the first feature amount calculation unit 1021 calculates the feature amount of each feature point in the shape image of the first flicker region and outputs the calculated feature amount as first feature amount information to the matching unit 1022. A known method can be used to calculate the feature amount. For example, a method called BRIEF can be used, in which multiple line segments are randomly arranged around the feature point and bits are generated by comparing the brightness values ​​at both ends of the line segments.

[0121] In step S206a, the first feature amount calculation unit 1021 determines whether feature amount calculation for all feature points of the shape image of the first flicker region is complete. If feature amount calculation for all feature points is not complete, the first feature amount calculation unit 1021 returns to step S205a, and if feature amount calculation for all feature points is complete, the first feature amount calculation unit 1021 proceeds to step S1101.

[0122] In step S201b, second flicker shape calculation unit 121 calculates a shape image of the second flicker region from the second flicker detection image generated by second signal synthesis unit 112. The process for calculating the shape image is the same as that described above in FIG.

[0123] In step S205b, the second feature amount calculation unit 1021 calculates the feature amount of each feature point in the shape image of the second flicker region and outputs the calculated feature amount as second feature amount information to the matching unit 1022. A known method can be used to calculate the feature amount. For example, a method called BRIEF can be used, in which multiple line segments are randomly arranged around the feature point and bits are generated by comparing the brightness values ​​at both ends of the line segments.

[0124] In step S206b, the second feature amount calculation unit 1021 determines whether feature amount calculation for all feature points of the shape image of the second flicker region is complete. If feature amount calculation for all feature points is not complete, the second feature amount calculation unit 1021 returns the process to step S205b, and if feature amount calculation for all feature points is complete, the second feature amount calculation unit 1021 proceeds to step S1101.

[0125] In step S1101, matching unit 1022 performs matching using the first feature amount information and the second feature amount information, calculates a positional relationship vector, and outputs it to posture estimation unit 127. A known method can be used for matching. For example, it is possible to use the BF method, which calculates the distances to all feature points using the Euclidean distance or the like to find the feature point with the shortest distance, or the FLANN method, which finds similar feature points using an approximate nearest neighbor search.

[0126] In step S210, posture estimation unit 127 estimates the posture from the positional relationship vector, and outputs a fundamental matrix representing the posture information to rectification unit 1023.

[0127] In step S1102, the rectification unit 1023 calculates the tilt of the imaging unit 100 using the fundamental matrix representing the first and second wide dynamic range images output from the first and second signal synthesis units 112 and the posture information output from the posture estimation unit 127. Then, the rectification unit 1023 rectifies the first and second wide dynamic range images and outputs the first and second rectified images.

[0128] This embodiment may be configured to perform template matching using a first rectified image and a second rectified image, which are the results of rectifying the first wide dynamic range image and the second wide dynamic range image, and calculate the amount of parallax. A known method can be used for template matching. For example, a method called SSD (Sum of Squared Difference), which uses the sum of squares of the differences between pixels as an evaluation value, can be used.

[0129] As described above, according to this embodiment, the image processing device 10 can process the parallelization of images from the stereo cameras of the imaging unit 100 with low delay by calculating a positional relationship vector from a shape image of the flicker area between the multiple cameras of the imaging unit 100.

[0130] (Fifth embodiment) 12 is a block diagram showing an example of the configuration of an image processing system 130 according to the fifth embodiment. Functions of the fifth embodiment that overlap with those of the fourth embodiment are given the same reference numerals, and descriptions thereof will be omitted.

[0131] In the fifth embodiment, an example will be described in which the amount of parallax is calculated from the result of feature point matching using shape images of a plurality of flicker regions in order to calculate the amount of parallax of images captured by the stereo camera of the imaging unit 100.

[0132] The image processing system 130 includes an imaging unit 100 and an image processing device 10. The image processing device 10 includes a signal processing unit 110 and an image processing unit 1200. The image processing unit 1200 is obtained by deleting the posture estimation unit 127 and the parallelization unit 1023 from the image processing unit 1020 in FIG. 10 and adding a disparity calculation unit 1201. The other parts in FIG. 12 are the same as those in FIG. 10.

[0133] <Image processing unit> The disparity calculation unit 1201 calculates the amount of disparity using the positional relationship vector output from the matching unit 1022, and outputs the calculated amount as disparity information.

[0134] <Flowchart> 13 is a flowchart showing an example of a processing method of the image processing unit 1200 according to the fifth embodiment. Each step of the processing method will be described with reference to FIG.

[0135] In this embodiment, an example will be described in which feature point matching is performed using a shape image of a flicker region calculated from images taken by the stereo camera of the imaging unit 100, and the amount of parallax is calculated. However, functions of the fifth embodiment that overlap with those of the fourth embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0136] Figure 13 is obtained by deleting steps S210 and S1102 and adding step S1301 to Figure 11. Step S1301 is performed after step S1101.

[0137] In step S1301, the disparity calculation unit 1201 calculates the amount of disparity using the positional relationship vector output from the matching unit 1022, and outputs the calculated amount as disparity information.

[0138] As described above, according to this embodiment, the image processing device 10 can calculate the amount of parallax in the flicker area with low delay by calculating the amount of parallax using the positional relationship vector calculated from the shape image of the flicker area between multiple cameras of the imaging unit 100.

[0139] According to the first to fifth embodiments, the image processing system 130 can improve the accuracy of extracting the flicker shape by acquiring a shape image of the flicker region where flicker is occurring.

[0140] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0141] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0142] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first state detection unit that detects a flicker detection signal for each pixel based on a first image signal captured with a first exposure time and a second image signal captured with a second exposure time different from the first exposure time; a first flicker shape calculation unit that calculates the shape of a flicker area based on the flicker detection signal detected by the first state detection unit; 1. An image processing device comprising: (Configuration 2) the first exposure time is an exposure time longer than a frequency of flicker; 2. The image processing device according to configuration 1, wherein the second exposure time is shorter than the frequency of flicker. (Configuration 3) a first feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker area calculated by the first flicker shape calculation unit; 3. The image processing device according to configuration 1 or 2, further comprising: a first matching unit that performs feature point matching based on the feature of the current frame calculated by the first feature calculation unit and the feature of the previous frame, and calculates a motion vector. (Configuration 4) a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a feature point calculation unit that calculates feature points of the third image signal for each frame; a feature point superimposing unit that superimposes the feature point of the third image signal and the feature point of the shape of the flicker region for each frame, the first feature amount calculation unit calculates, for each frame, a feature amount of the feature point superimposed by the feature point superimposition unit; The image processing device according to configuration 3, wherein the first matching unit performs feature point matching based on the feature of the current frame and the feature of the previous frame calculated by the first feature calculation unit, and calculates a motion vector. (Configuration 5) 5. The image processing device according to configuration 3 or 4, further comprising an attitude estimation unit that estimates an attitude of an imaging unit that generated the first image signal and the second image signal, based on the motion vector calculated by the first matching unit. (Configuration 6) a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a feature point calculation unit that calculates feature points of the third image signal for each frame; a second feature amount calculation unit that calculates, for each frame, a feature amount of the feature point of the third image signal calculated by the feature point calculation unit; The image processing device according to configuration 3, further comprising a second matching unit that performs feature point matching based on the feature of the current frame calculated by the second feature calculation unit and the feature of the previous frame, and calculates a motion vector. (Configuration 7) a motion vector superimposing unit that superimposes, for each frame, the motion vector calculated by the first matching unit and the motion vector calculated by the second matching unit; 7. The image processing device according to configuration 6, further comprising: an attitude estimation unit that estimates an attitude of an imaging unit that generated the first image signal and the second image signal, based on the motion vector superimposed by the motion vector superimposition unit. (Configuration 8) a motion vector removal unit that removes, for each frame, motion vectors calculated by the second matching unit that differ by a threshold or more from a determination value based on the motion vector calculated by the first matching unit, and outputs the motion vectors; 7. The image processing device according to configuration 6, further comprising an orientation estimation unit that estimates an orientation of an imaging unit that generated the first image signal and the second image signal, based on the motion vector output by the motion vector removal unit. (Configuration 9) 9. The image processing device according to configuration 8, wherein the determination value is an average value of the motion vectors calculated by the first matching unit. (Configuration 10) the first image signal and the second image signal are image signals captured by a first image sensor, a second state detection unit that detects a flicker detection signal for each pixel based on a fourth image signal captured by a second image sensor with the first exposure time and a fifth image signal captured by the second image sensor with the second exposure time; a second flicker shape calculation unit that calculates the shape of a flicker area based on the flicker detection signal detected by the second state detection unit; a first feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker area calculated by the first flicker shape calculation unit; a second feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker area calculated by the second flicker shape calculation unit; 3. The image processing device according to configuration 1 or 2, further comprising: a matching unit that performs matching based on the feature amount calculated by the first feature amount calculation unit and the feature amount calculated by the second feature amount calculation unit, and calculates a vector indicating a positional relationship. (Configuration 11) a posture estimation unit that estimates a posture based on the vector indicating the positional relationship calculated by the matching unit; a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a second signal synthesis unit that synthesizes the fourth image signal and the fifth image signal for each frame to generate a sixth image signal; 11. The image processing device according to configuration 10, further comprising a parallelization unit that parallelizes the third image signal and the sixth image signal based on the orientation estimated by the orientation estimation unit. (Configuration 12) 11. The image processing device according to configuration 10, further comprising a parallax calculation unit that calculates a parallax amount based on the vector indicating the positional relationship calculated by the matching unit. (Configuration 13) The image processing device according to any one of configurations 1 to 12, an imaging unit that generates the first image signal and the second image signal; An image processing system comprising: (Method 1) a first state detection step of detecting a flicker detection signal for each pixel based on a first image signal captured with a first exposure time and a second image signal captured with a second exposure time different from the first exposure time; a first flicker shape calculation step of calculating a shape of a flicker area based on the flicker detection signal detected in the first state detection step; 10. A processing method for an image processing apparatus, comprising: (Program 1) 13. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12. [Explanation of symbols]

[0143] 111 state detection unit, 112 signal synthesis unit, 121 flicker shape calculation unit, 122 feature amount calculation unit, 123 feature point superposition unit, 124 feature amount calculation unit, 125 matching unit, 126 feature amount storage unit, 126 posture estimation unit

Claims

1. a first state detection unit that detects a flicker detection signal for each pixel based on a first image signal captured with a first exposure time and a second image signal captured with a second exposure time different from the first exposure time; a first flicker shape calculation unit that calculates the shape of a flicker area based on the flicker detection signal detected by the first state detection unit; 1. An image processing device comprising:

2. the first exposure time is an exposure time longer than a frequency of flicker; 2. The image processing apparatus according to claim 1, wherein the second exposure time is shorter than the frequency of flicker.

3. a first feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker area calculated by the first flicker shape calculation unit; 2. The image processing device according to claim 1, further comprising a first matching unit that performs feature point matching based on the feature of the current frame calculated by the first feature calculation unit and the feature of the previous frame, and calculates a motion vector.

4. a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a feature point calculation unit that calculates feature points of the third image signal for each frame; a feature point superimposing unit that superimposes the feature point of the third image signal and the feature point of the shape of the flicker region for each frame, the first feature amount calculation unit calculates, for each frame, a feature amount of the feature point superimposed by the feature point superimposition unit; 4. The image processing device according to claim 3, wherein the first matching unit performs feature point matching based on the feature of the current frame and the feature of the previous frame calculated by the first feature calculation unit, and calculates a motion vector.

5. 4. The image processing device according to claim 3, further comprising an attitude estimation unit that estimates an attitude of an imaging unit that generated the first image signal and the second image signal based on the motion vector calculated by the first matching unit.

6. a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a feature point calculation unit that calculates feature points of the third image signal for each frame; a second feature amount calculation unit that calculates, for each frame, a feature amount of the feature point of the third image signal calculated by the feature point calculation unit; 4. The image processing device according to claim 3, further comprising a second matching unit that performs feature point matching based on the feature of the current frame calculated by the second feature calculation unit and the feature of the previous frame, and calculates a motion vector.

7. a motion vector superimposing unit that superimposes, for each frame, the motion vector calculated by the first matching unit and the motion vector calculated by the second matching unit; 7. The image processing device according to claim 6, further comprising: an orientation estimation unit that estimates an orientation of an imaging unit that generated the first image signal and the second image signal based on the motion vector superimposed by the motion vector superimposition unit.

8. a motion vector removal unit that removes, for each frame, motion vectors calculated by the second matching unit that differ by a threshold or more from a determination value based on the motion vector calculated by the first matching unit, and outputs the motion vectors; 7. The image processing device according to claim 6, further comprising: an orientation estimation unit that estimates an orientation of an imaging unit that generated the first image signal and the second image signal based on the motion vector output by the motion vector removal unit.

9. 9. The image processing device according to claim 8, wherein the determination value is an average value of the motion vectors calculated by the first matching unit.

10. the first image signal and the second image signal are image signals captured by a first image sensor, a second state detection unit that detects a flicker detection signal for each pixel based on a fourth image signal captured by a second image sensor with the first exposure time and a fifth image signal captured by the second image sensor with the second exposure time; a second flicker shape calculation unit that calculates a shape of a flicker area based on the flicker detection signal detected by the second state detection unit; a first feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker area calculated by the first flicker shape calculation unit; a second feature amount calculation unit that calculates a feature amount for each frame based on feature points of the shape of the flicker region calculated by the second flicker shape calculation unit; 2. The image processing device according to claim 1, further comprising a matching unit that performs matching based on the feature amount calculated by the first feature amount calculation unit and the feature amount calculated by the second feature amount calculation unit, and calculates a vector indicating a positional relationship.

11. a posture estimation unit that estimates a posture based on the vector indicating the positional relationship calculated by the matching unit; a first signal synthesis unit that synthesizes the first image signal and the second image signal for each frame to generate a third image signal; a second signal synthesis unit that synthesizes the fourth image signal and the fifth image signal for each frame to generate a sixth image signal; 11. The image processing device according to claim 10, further comprising a parallelization unit that parallelizes the third image signal and the sixth image signal based on the orientation estimated by the orientation estimation unit.

12. 11. The image processing device according to claim 10, further comprising a parallax calculation unit that calculates a parallax amount based on the vector indicating the positional relationship calculated by the matching unit.

13. An image processing device according to any one of claims 1 to 12; an imaging unit that generates the first image signal and the second image signal; An image processing system comprising:

14. a first state detection step of detecting a flicker detection signal for each pixel based on a first image signal captured with a first exposure time and a second image signal captured with a second exposure time different from the first exposure time; a first flicker shape calculation step of calculating a shape of a flicker area based on the flicker detection signal detected in the first state detection step; 10. A processing method for an image processing apparatus, comprising:

15. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12.

Images