Signal Processing Device, Signal Processing Method, and Program

Abstract

To provide a signal processing apparatus that suppresses reduction in accuracy of phase difference detection.SOLUTION: An imaging apparatus 100 comprises: a pixel 303 that has two photoelectric conversion elements 301, 302 receiving light beams with different incident directions from an object, and outputs image signals of a plurality of frames photoelectrically converted from the light beams; saturation detection means 102 that generates saturation information indicating whether the image signals output from the pixel 303 are saturated; filter operation means 104 that performs filter processing on the image signals of the plurality of frames to calculate image signals after the filter processing; saturation evaluation value calculation means 105 that calculates, from the saturation information on the image signals of the plurality of frames, a multivalued saturation evaluation value indicating reliability of the image signals after the filter processing; and image displacement amount calculation means 107 that, on the basis of, the calculated saturation evaluation value, determines whether to use the image signals after the filter processing for phase difference detection.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a signal processing apparatus, a signal processing method, and a program for detecting a phase difference of image signals output from a plurality of photoelectric conversion elements of pixels of an imaging device.

Background Art

[0002] Conventionally, at least two image signals obtained by photoelectrically converting light beams having different incident directions are acquired from pixels of an imaging device, and a phase difference (shift amount) between these image signals is detected to calculate a distance to an object. There is known an imaging device that performs phase difference type focus detection. Further, there is also known an imaging device that simultaneously calculates the distance to an object and creates a recorded image using the image signal acquired from a pixel (for example, Patent Document 1). In the imaging device of Patent Document 1, two photoelectric conversion elements divided to receive light beams having different incident directions that pass through one microlens are provided in each pixel of the imaging device. Then, image signals are individually acquired from each photoelectric conversion element. Thereby, a phase difference detection signal and an image creation signal are simultaneously acquired from one pixel.

[0003] In addition, the amount of light received by a pixel may become excessive, and the acquired image signal may be saturated. In this case, by not using the image signal for phase difference detection, the accuracy of calculating the distance to an object, and thus the accuracy of the focus adjustment function, is improved. For example, when charges are saturated as a result of photoelectric conversion in one of the two divided photoelectric conversion elements and charge leakage occurs into the other photoelectric conversion element, image breakdown occurs and the accuracy of phase difference detection decreases. Therefore, the image signal acquired from a pixel in which charge leakage has occurred is excluded from phase difference detection (see, for example, Patent Document 2).

[0004] Incidentally, it is known that using the image signal after filtering the time axis of multiple image signals output from the same pixel can reduce noise, mainly in low light conditions, and improve the accuracy of the phase-difference focusing function. In this case, if even one of the image signals among the multiple frames that are filtered is saturated, the accuracy of phase-difference detection using the filtered image signal may decrease. Therefore, it is also conceivable to exclude such filtered image signals from phase-difference detection. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2012-27390 [Patent Document 2] Patent No. 6317548 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, if even a single frame of image signal is saturated, excluding the filtered image signal from phase difference detection can reduce the amount of filtered image signal used for phase difference detection, potentially leading to a decrease in the accuracy of phase difference detection.

[0007] The object of the present invention is to provide a signal processing device, a signal processing method, and a program that suppress the decrease in the accuracy of phase difference detection. [Means for solving the problem]

[0008] To achieve the above objective, the signal processing apparatus of the present invention is characterized by comprising: a plurality of photoelectric conversion means for receiving light beams from an object with different incident directions; a pixel that outputs image signals of a plurality of frames converted photoelectrically from the light beams; an information generation means for generating saturation information indicating whether or not the image signals output by the pixels are saturated; a filter calculation means for applying a filter to the image signals of the plurality of frames and calculating the image signal after filtering; an evaluation value calculation means for calculating a multi-level saturation evaluation value indicating the reliability of the image signal after filtering from the saturation information of the image signals of the plurality of frames; and a phase difference detection means for determining whether or not to use the image signal after filtering for phase difference detection based on the calculated saturation evaluation value. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress a decrease in the accuracy of phase difference detection. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic block diagram showing the configuration of an imaging device as a signal processing device according to the first embodiment of the present invention. [Figure 2] This figure illustrates the configuration of each pixel in the image sensor of the imaging device shown in Figure 1. [Figure 3] This is a flowchart showing the image shift amount calculation process as a signal processing method according to the first embodiment of the present invention. [Figure 4] This diagram illustrates the format of the image signal with saturation information output by the saturation detection means. [Figure 5] This diagram illustrates the moving average processing performed by the filter calculation means. [Figure 6] This is a block diagram schematically showing the configuration of the filter calculation means in the first embodiment of the present invention. [Figure 7] Figure 3 is a flowchart showing the saturation evaluation value calculation process in step S205. [Figure 8]This is a graph of phase characteristics showing the effect that the image signal of the frame of interest has on the filtered image signal when an IIR filter configuration is applied to the filtering calculation means. [Figure 9] This is a schematic block diagram showing the configuration of an imaging device as a signal processing device according to a second embodiment of the present invention. [Figure 10] This diagram illustrates the moving average processing performed by the saturation evaluation value calculation means. [Figure 11] This is a schematic block diagram showing the configuration of an imaging device as a signal processing device according to a third embodiment of the present invention. [Figure 12] This diagram illustrates the format of a single pixel held in frame memory. [Figure 13] This diagram illustrates the configuration of a modified image sensor pixel. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First, a first embodiment of the present invention will be described. Figure 1 is a schematic block diagram showing the configuration of an imaging device as a signal processing device according to this embodiment. In this embodiment, a saturation evaluation value of the filtered image signal is calculated based on the saturation information of multiple frames of image signals output by the same pixel of the image sensor of the imaging device, and it is determined whether or not to use the filtered image signal for phase difference detection according to the saturation evaluation value. Saturation information and saturation evaluation value will be described later.

[0012] In Figure 1, the imaging device 100 includes an image sensor 101, a saturation detection means 102 (information generation means), a separation means 103, a filter calculation means 104, and a saturation evaluation value calculation means 105. Furthermore, the imaging device 100 includes a saturation evaluation value determination means 106, an image shift amount calculation means 107 (phase difference detection means), a control means 108, and a frame memory 109 (storage means).

[0013] The imaging device 101 has a plurality of pixels, and each pixel includes one microlens and a plurality of, for example, two photoelectric conversion elements typified by a photodiode that converts incident light passing through the microlens into electric charges. Each photoelectric conversion element receives light fluxes from objects (subjects) with different incident directions, photoelectrically converts the received light fluxes, and outputs an image signal to the saturation detection means 102.

[0014] FIG. 2 is a diagram for explaining the configuration of each pixel of the imaging device 101 included in the imaging apparatus 100 of FIG. 1. In FIG. 2, two pixels 303 and 307 are shown. The imaging device 101 includes a large number of pixels having the same configuration as the pixels 303 and 307, and each pixel is arranged vertically and horizontally. For example, the pixel 303 has one microlens 304 and two photoelectric conversion elements 301 and 302 (photoelectric conversion means). In the pixel 303, the microlens 304 is disposed near the image plane of the optical system (not shown) of the imaging apparatus 100, and the microlens 304 condenses light fluxes with different incident directions from the subject onto each of the two photoelectric conversion elements 301 and 302. The two photoelectric conversion elements 301 and 302 output image signals of a plurality of frames along the time axis. At this time, the image signals individually acquired from the two photoelectric conversion elements 301 and 302 are used for phase difference detection, and the image signals acquired simultaneously are used for creating a recorded image. The pixel 307 has one microlens 308 and two photoelectric conversion elements 305 and 306. In the pixel 307, the microlens 308 is disposed near the image plane of the optical system (not shown) of the imaging apparatus 100, and the microlens 308 condenses light fluxes with different incident directions from the subject onto each of the two photoelectric conversion elements 305 and 306. Similar to the two photoelectric conversion elements 301 and 302, the two photoelectric conversion elements 305 and 306 output image signals of a plurality of frames along the time axis. At this time, the image signals individually acquired from the two photoelectric conversion elements 305 and 306 are used for phase difference detection, and the image signals acquired simultaneously are used for creating a recorded image.

[0015] Returning to FIG. 1, the saturation detection means 102 detects whether the image signal of each frame acquired from each pixel of the imaging device 101 is saturated, and generates saturation information indicating whether the image signal is saturated. Further, the saturation detection means 102 sequentially outputs the saturation information corresponding to the image signal of each frame to the separation means 103. The separation means 103 separates the saturation information corresponding to the image signal of each frame output by the saturation detection means 102, outputs the image signal to the filter operation means 104, and outputs the saturation information to the saturation evaluation value calculation means 105.

[0016] The filter operation means 104 performs filter processing using the image signal output from the separation means 103 and the image signal output from the frame memory 109, and outputs the image signal after the filter processing to the image displacement amount calculation means 107 and the frame memory 109. The frame memory 109 holds the image signals of a plurality of frames before the image signal of the current output frame of a certain pixel of the imaging device 101. The frame memory 109 outputs the image signals of these plurality of frames to the filter operation means 104 during the filter processing. The details of the filter processing in the present embodiment will be described later.

[0017] The saturation evaluation value calculation means 105 calculates a saturation evaluation value using the saturation information output from the separation means 103 and the count-up setting value set by the control means 108, and outputs the calculated saturation evaluation value to the saturation evaluation value determination means 106. Further, the saturation evaluation value calculation means 105 holds the calculated saturation evaluation value. The calculated saturation evaluation value may be held by the frame memory 109 instead of the saturation evaluation value calculation means 105. The details of the calculation of the saturation evaluation value in the present embodiment will be described later. The saturation evaluation value determination means 106 determines the contribution rate of the image signal of the saturated frame in the image signal after the filter processing based on the saturation evaluation value output from the saturation evaluation value calculation means 105 and the threshold value described later set by the control means 108, and generates new saturation information. Further, the saturation evaluation value determination means 106 outputs the new saturation information to the image displacement amount calculation means 107.

[0018] The image shift amount calculation means 107 calculates the image shift amount of the image signal based on the filtered image signal and new saturation information, and performs phase difference detection. The control means 108 controls the entire imaging device 100. In this embodiment, the control means 108 sets the filter coefficients used in filtering and the count-up setting values ​​used in calculating the saturation evaluation value, and sets the threshold used in generating new saturation information.

[0019] The frame memory 109 holds the filtered image signal output from the filter calculation means 104. Furthermore, as described above, the frame memory 109 holds the image signals of multiple frames prior to the image signal output by a particular pixel of the image sensor 101. When filtering is performed, the frame memory 109 outputs these previous image signals to the filter calculation means 104. In this embodiment, the image signal of the frame output by a particular pixel of the image sensor 101 will be referred to as the "image signal of the frame of interest," and the image signals of frames prior to the image signal output by the aforementioned pixel will be referred to as the "image signals of previous frames."

[0020] Figure 3 is a flowchart illustrating the image shift amount calculation process as a signal processing method according to the first embodiment of the present invention. This process is mainly performed when the imaging device 100 performs autofocus operation. For example, if the imaging device 100 is a digital camera, this process is started in response to the pressing of the shutter release button. This process is realized by the control means 108 executing a control program to control each part of the signal processing device. In this embodiment, each time an image signal of the frame of interest is output from a pixel of the image sensor 101, filtering is performed using the image signal of the frame of interest and the image signals of multiple previous frames, and then phase difference detection is performed using the filtered image signal. This process corresponds to this phase difference detection and is repeatedly performed each time an image signal of the frame of interest is output.

[0021] In Figure 3, first, the image sensor 101 outputs the image signal of the frame of interest to the saturation detection means 102 (step S201). The saturation detection means 102 detects whether or not the image signal of the frame of interest is saturated and generates saturation information (step S202). Specifically, if the signal level of the image signal of the frame of interest is above a predetermined value, the saturation detection means 102 considers the image signal of the frame of interest to be saturated and generates saturation information as information that can identify saturation. If the saturation detection means 102 considers the image signal of the frame of interest to be saturated, it sets the saturation information to "1", and if the image signal of the frame of interest is not considered to be saturated, it sets the saturation information to "0". After that, the saturation detection means 102 adds the saturation information to the image signal of the frame of interest and outputs it to the separation means 103 as an image signal with saturation information.

[0022] Figure 4 is a diagram illustrating the format of the image signal with saturation information output by the saturation detection means 102. Figure 4 schematically shows the format of the image signal for one pixel. In this embodiment, the image signal output by each pixel is output as digital data in which the image signal per pixel is represented by a predetermined number of bits for ease of handling. For example, as shown in Figure 4, if the digital data 400, which is the image signal for one pixel, consists of 16 bits, the digitized signal value of the image signal for one pixel is stored in the 12-bit bit sequence 401. Saturation information is stored in one bit 402 adjacent to bit sequence 401. Furthermore, the saturation evaluation value generated by the saturation evaluation value calculation means 105 is stored in the 3-bit bit sequence 403 adjacent to bit 402. Note that the saturation evaluation value may not be stored in the digital data 400, but may be stored in a counter frame memory (not shown) of the saturation evaluation value calculation means 105.

[0023] In this embodiment, the bit precision of the saturated evaluation value is set to 3 bits, but the bit precision of the saturated evaluation value may be changed according to the filter coefficients used in the filter calculation means 104. For example, if the filter response is slow, more frames need to be counted compared to when the filter response is fast. Therefore, if the filter response is slow, the threshold of the saturated evaluation value is increased by increasing the bit precision of the saturated evaluation value. For example, if the bit precision is 3 bits, the threshold of the saturated evaluation value is 7, and if the bit precision is 4 bits, the threshold of the saturated evaluation value is 15. Also, if the threshold of the saturated evaluation value remains constant at 1, the saturated evaluation value changes in increments of 0.125 if the bit precision is 3 bits. If the bit precision is 4 bits, the saturated evaluation value changes in increments of 0.0625. As a result, the rate at which the saturated evaluation value increases with respect to the threshold becomes slower, and by increasing the bit precision, the operation approximates that of a slow filter response.

[0024] Returning to Figure 3, the separation means 103 separates the saturation information from the image signal with saturation information output by the saturation detection means 102 and outputs it to the saturation evaluation value calculation means 105 (step S203). It also separates only the signal value from the image signal with saturation information as an image signal and outputs it to the filter calculation means 104. Specifically, it separates the bit sequence 401 and bit 402 from the digital data 400, outputting the former as the image signal of the frame of interest, i.e., an image signal for each pixel. It also outputs the saturation information of the image signal of the frame of interest, i.e., saturation information for each pixel. In other words, since the information can be handled on a pixel-by-pixel basis, it is possible to separate the image signal and saturation information while maintaining the positional information of the pixel in the image sensor 101 that outputs the image signal of the frame of interest. Hereafter, the saturation information of the image signal of the frame of interest will be referred to as "saturation information of the frame of interest".

[0025] Next, the filter calculation means 104 performs filtering using the separated image signals (step S204). This filtering is a digital filtering process of multiple frames of image signals along the time axis, specifically a moving average processing of multiple frames of image signals along the time axis. The filter calculation means 104 then outputs the filtered image signals to the frame memory 109 and the image shift amount calculation means 107.

[0026] Figure 5 is a diagram illustrating the moving average processing performed by the filter calculation means 104. In Figure 5, image signals F0 to F6 are image signals from seven consecutive frames along the time axis, output from a pixel of the image sensor 101. The filtered image signal F0' is a moving average filter output with 5 taps (number of frames used for filtering) obtained by applying moving average processing to five consecutive image signals F0 to F4 along the time axis. The filtered image signal F1' is a moving average filter output with 5 taps obtained by applying moving average processing to five consecutive image signals F1 to F5 along the time axis. Similarly, the filtered image signal F2' is a moving average filter output with 5 taps obtained by applying moving average processing to five consecutive image signals F2 to F6 along the time axis. Each moving average filter output corresponds to a filtered image signal. Each time this process is executed, one filtered image signal is calculated. These filtered image signals F0' to F2' are calculated using filter coefficients set by the control means 108. The filtering performed by the filter calculation means 104 is not limited to moving average processing; for example, it may also be an additive decimation process that applies additive averaging to five consecutive frames of image signals along the time axis.

[0027] Figure 6 is a block diagram schematically showing the configuration of the filter calculation means 104 in this embodiment. Figure 6(A) shows the case when an FIR filter (Finite Impulse Filter) configuration is applied to the filter calculation means 104. Figure 6(B) shows the case when an IIR filter (Infinite Impulse Filter) configuration is applied to the filter calculation means 104.

[0028] In the FIR filter shown in Figure 6(A), the image signal of the frame of interest output from the separation means 103 is input through the input terminal. The FIR filter also has frame memories 601a to 601d, each holding the image signals of multiple previous frames output by the frame memory 109. In the FIR filter, the image signals of the five frames, which are the image signal of the frame of interest and the image signals of the previous frames held in each of the frame memories 601a to 601d, are multiplied by the filter coefficients coef0 to coef4. The adder 602 combines each image signal multiplied by the filter coefficients coef0 to coef4, and the normalization processing unit 603 normalizes each combined image signal to calculate the filtered image signal. For example, when calculating the filtered image signal F0', the image signal F4 is input from the separation means 103 through the input terminal. Then, the five image signals, which are the input image signal F4 and the image signals F3 to F0 held in the frame memories 601a to 601d respectively, are multiplied by the filter coefficients coef0 to coef4. After that, the adder 602 combines the image signals F4 to F0 that have been multiplied by the filter coefficients coef0 to coef4. The normalization processing unit 603 normalizes the combined image signals F4 to F0 to calculate the filtered image signal F0'. Note that in an FIR filter, the number of taps (the number of image signal frames used) is fixed, and in Figure 6(A), the number of taps is 5.

[0029] In the IIR filter shown in Figure 6(B), the image signal of the frame of interest output from the separation means 103 is input through the input terminal. The IIR filter also has a frame memory 601e that holds filtered image signals calculated from the image signals of several previous frames (hereinafter referred to as "previous filtered image signals"). In the IIR filter, the filtered image signal is calculated using the image signal of the frame of interest and the previous filtered image signals held in the frame memory 601e by the following equation 1. Filtered image signal = A*coef + B *(1-coef)…(1)

[0030] A is the image signal of the frame of interest, and B is the image signal after the previous filtering process. After calculating the filtered image signal, the previously filtered image signal held in the frame memory 601e is replaced with the calculated filtered image signal.

[0031] The IIR filter in Figure 6(B) requires fewer frame memories compared to the FIR filter in Figure 6(A), thus reducing costs. Furthermore, the IIR filter is preferable because its response characteristics continue indefinitely, and the filter coefficient coef remains constant, eliminating unwanted influences and resulting in a stable filtered image signal. In this embodiment, the explanation will proceed based on the IIR filter shown in Figure 6(B).

[0032] Returning to Figure 3, the saturation evaluation value calculation means 105 calculates the saturation evaluation value using the saturation evaluation value it already possesses, the saturation information output from the separation means 103, and the count-up setting value (count value) set by the control means 108 (step S205). Hereinafter, the saturation evaluation value it already possesses will be referred to as the "previous saturation evaluation value".

[0033] Figure 7 is a flowchart showing the saturation evaluation value calculation process in step S205 of Figure 3. In Figure 7, the saturation evaluation value calculation means 105 determines whether or not the image signal of the frame of interest is saturated based on the saturation information of the frame of interest output from the separation means 103 (step S701). Specifically, if the saturation information is "1", it is determined that the image signal of the frame of interest is saturated, and the process proceeds to step S702. If the saturation information is "0", it is determined that the image signal of the frame of interest is not saturated, and the process proceeds to step S703.

[0034] In step S702, a new saturation evaluation value is calculated by adding the count-up setting value to the previous saturation evaluation value and counting up. In step S703, a new saturation evaluation value is calculated by subtracting (counting down) the count-up setting value from the previous saturation evaluation value. Here, the previous saturation evaluation value is the saturation evaluation value of the filtered image signal calculated using the image signal of the previous frame, obtained when step S205 of the previous image shift amount calculation process (processing in Figure 3) was executed. For example, if the filtered image signal F1' is calculated by the processing in Figure 3, the previous saturation evaluation value is the saturation evaluation value of the filtered image signal F0' calculated using the image signals F0 to F4 of the frame. The new saturation evaluation value is the saturation evaluation value of the filtered image signal F1' calculated using the image signals F1 to F5 of the frame.

[0035] Therefore, a large saturation evaluation value indicates that there are many saturated image signals in each frame used to calculate the filtered image signal. Conversely, a small saturation evaluation value indicates that there are few saturated image signals in each frame used to calculate the filtered image signal. In other words, a large saturation evaluation value of the filtered image signal indicates low reliability of the filtered image signal, while a small saturation evaluation value indicates high reliability of the filtered image signal.

[0036] Furthermore, if the number of image signals from previous frames used to calculate the filtered image signal is small, the count-up setting may be increased. For example, normally, five image signals are used to calculate the filtered image signal, but if the shutter release button has just been pressed and only one image signal from the previous frame exists, the count-up setting may be increased. In this case, if the image signal of the frame of interest is saturated, the saturation evaluation value will suddenly increase even though only two image signals (the image signal from one previous frame and the image signal from the frame of interest) are used to calculate the filtered image signal. This indicates that the reliability of the filtered image signal is low, and it is possible to exclude filtered image signals that have been greatly affected by abrupt saturation, such as flash light that occurred only in the very first frame of shooting, from phase difference detection.

[0037] Subsequently, the saturation evaluation value calculation means 105 outputs the newly calculated saturation evaluation value to the saturation evaluation value determination means 106, and also retains the calculated saturation evaluation value as the saturation evaluation value before it is used in the next image shift amount calculation process.

[0038] Returning to Figure 3, the saturation evaluation value determination means 106 determines whether the new saturation evaluation value output from the saturation evaluation value calculation means 105 is greater than the threshold (predetermined threshold) output from the control means 108 (step S206). Figure 8 is a graph of the phase characteristics 700 showing the influence of the image signal of the frame of interest on the filtered image signal when an IIR filter configuration is applied to the filter calculation means 104. The influence of the image signal of the frame of interest on the filtered image signal is, in other words, the contribution rate of the image signal of the frame of interest to the filtered image signal. In this graph, the horizontal axis shows the number of image signal frames used for filtering, and the vertical axis shows the saturation evaluation value as the contribution rate of the image signal of the frame of interest to the filtered image signal. As the phase characteristics 700 shows, the more image signal frames used for filtering there are, the smaller the influence of the image signal of the frame of interest on the filtered image signal. In this embodiment, when the number of frames of the image signal used for filtering is greater than the number of frames at the centroid of the phase characteristic 700, the influence of the image signal of the frame of interest on the filtered image signal becomes smaller. Therefore, the saturation evaluation value corresponding to the number of frames at the centroid of the phase characteristic 700 is set as the threshold as described above.

[0039] In step S206, if it is determined that the new saturation evaluation value is greater than the threshold, "1" is set as the new saturation information for the filtered image signal (step S207). If it is determined that the new saturation evaluation value is less than or equal to the threshold, "0" is set as the new saturation information for the filtered image signal (step S208). Setting "1" as the new saturation information indicates that the saturated image signal of the frame of interest has a large influence on the filtered image signal, and the reliability of the filtered image signal is low. Setting "0" as the new saturation information indicates that the saturated image signal of the frame of interest has a small influence on the filtered image signal, and the reliability of the filtered image signal is high. Subsequently, the saturation evaluation value determination means 106 outputs the new saturation information to the image shift amount calculation means 107. Alternatively, the image shift amount calculation means 107 may directly obtain the new saturation evaluation value from the saturation evaluation value calculation means 105 and determine whether the new saturation evaluation value is greater than the threshold. In this case, the saturation evaluation value determination means 106 becomes unnecessary, and there is no need to store new saturation information in the frame memory 109, thus reducing the capacity of the frame memory 109.

[0040] Next, the image shift amount calculation means 107 calculates the image shift amount of the image signal based on the filtered image signal output from the filter calculation means 104 and the new saturation information output from the saturation evaluation value determination means 106, and performs phase difference detection (step S209).

[0041] Specifically, if the new saturation information is "0", the filtered image signal output from the filter calculation means 104 is used to calculate the image shift amount of the image signal. On the other hand, if the new saturation information is "1", the filtered image signal output from the filter calculation means 104 is not used to calculate the image shift amount of the image signal.

[0042] As mentioned above, the image signal output by each pixel includes two image signals output by two photoelectric conversion elements that receive light beams with different incident directions from the subject. Therefore, the image signal after filtering also includes two image signals corresponding to light beams with different incident directions. In step S209, a correlation calculation is performed between these two image signals, and the amount of image shift is calculated from the correlation calculation result. If the new saturation information is "0", the correlation calculation between the two image signals is performed; if the new saturation information is "1", the correlation calculation between the two image signals is not performed. After that, this process is terminated.

[0043] According to this embodiment, a new saturation evaluation value indicating the reliability of the filtered image signal is used to determine whether or not to use the filtered image signal for phase difference detection. Furthermore, the new saturation evaluation value is calculated using not only the saturation information of the frame of interest, but also the saturation evaluation value of the filtered image signal calculated using the image signals of frames prior to the frame of interest. Moreover, this saturation evaluation value is not a binary value represented by 0 or 2, but is obtained as three or more values ​​through counting up or counting down a count-up setting value for each image signal of multiple frames. In other words, the saturation evaluation value does not simply indicate whether or not the image signal of one of the multiple frames subjected to filtering is saturated, but rather indicates the proportion of pixels that are saturated in multiple frames. Therefore, even if the image signal of the frame of interest is saturated, it is possible to avoid a situation where the filtered image signal calculated using the image signal of the frame of interest is immediately not used for phase difference detection. This prevents the amount of filtered image signal used for phase difference detection from decreasing, and suppresses a decrease in the accuracy of phase difference detection.

[0044] Furthermore, in this embodiment, the saturation evaluation threshold for determining whether or not to use the filtered image signal for phase difference detection is set based on the phase characteristic 700, which indicates the contribution rate of the image signal of the frame of interest to the filtered image signal. As a result, even if the image signal of the frame of interest is saturated, the filtered image signal can be used for phase difference detection if the contribution rate of the image signal of the frame of interest is small. Also, if the image signal of the frame of interest is saturated and the contribution rate of the image signal of the frame of interest is large, the filtered image signal will not be used for phase difference detection. In other words, it is possible to appropriately determine whether or not to use the filtered image signal for phase difference detection while considering the influence of the image signal of the frame of interest on the filtered image signal.

[0045] Next, a second embodiment of the present invention will be described. The second embodiment is basically the same as the first embodiment described above in terms of its configuration and operation, and differs from the first embodiment in that the saturated evaluation value is calculated by filtering. Therefore, the explanation of overlapping configurations and operations will be omitted, and the different configurations and operations will be described below.

[0046] Figure 9 is a schematic block diagram showing the configuration of an imaging device 800 as a signal processing device according to a second embodiment of the present invention. In Figure 9, the frame memory 802 holds the filtered image signal output from the filter calculation means 104 and the new saturation information output from the saturation evaluation value determination means 106. The frame memory 802 also holds the image signal of the previous frame and multiple pieces of saturation information of the image signal of the previous frame (hereinafter referred to as the saturation information of the previous frame). For example, it holds the image signal of the previous frame and the saturation information of the previous frame for the number of taps minus 1 in the moving average processing described later. When filtering is performed, the frame memory 802 outputs the multiple image signals of the previous frames it holds to the filter calculation means 104 and outputs the multiple saturation information of the previous frames it holds to the saturation evaluation value calculation means 801.

[0047] The saturation evaluation value calculation means 801 calculates a saturation evaluation value based on the saturation information of the frame of interest output from the separation means 103, the saturation information of multiple previous frames output from the frame memory 802, and the filter coefficient set by the control means 108. The saturation evaluation value calculation means 801 also outputs the calculated saturation evaluation value to the saturation evaluation value determination means 106. The saturation evaluation value calculation means 801 has an FIR filter configuration and applies a moving average processing as a filtering process to the saturation information of the frame of interest output from the separation means 103 and the saturation information of multiple previous frames output from the frame memory 802. In the second embodiment, the filter calculation means 104 also has an FIR filter configuration.

[0048] Figure 10 is a diagram illustrating the moving average processing performed by the saturation evaluation value calculation means 801. In Figure 10, the saturation information S0 to S6 is saturation information of seven consecutive frames of image signals output from one pixel of the image sensor 101 along the time axis. The saturation evaluation value S0' is a moving average filter output with 5 taps (number of frames used for filtering) obtained by applying moving average processing to the saturation information S0 to S4 of five consecutive frames along the time axis. The saturation evaluation value S1' is a moving average filter output with 5 taps obtained by applying moving average processing to the saturation information S1 to S5 of five consecutive frames along the time axis. Similarly, the saturation evaluation value S2' is a moving average filter output with 5 taps obtained by applying moving average processing to the saturation information S2 to S6 of five consecutive frames along the time axis. Each moving average filter output corresponds to a saturation evaluation value. These saturation evaluation values ​​S0' to S2' are calculated using filter coefficients set by the control means 108.

[0049] Therefore, if there are many saturated images in each frame's image signal (images with saturation information of "1" instead of "0"), the saturation evaluation value calculated by the moving average process will be large. In other words, a large saturation evaluation value indicates that there are many saturated images in each frame's image signal used to calculate the filtered image signal. To put it another way, a large saturation evaluation value indicates low reliability of the filtered image signal, similar to the first embodiment. Note that the filtering process performed by the saturation evaluation value calculation means 801 is not limited to moving average processing; for example, it may be an additive decimation process that applies additive averaging to the saturation information of five consecutive frames along the time axis.

[0050] The processing in the saturation evaluation value determination means 106 and the image displacement amount calculation means 107 after the saturation evaluation value has been calculated by the moving average processing is the same as the processing in the first embodiment.

[0051] Similar to the first embodiment, this embodiment also determines whether or not to use the filtered image signal for phase difference detection based on a saturation evaluation value indicating the reliability of the filtered image signal. This prevents the amount of filtered image signal used for phase difference detection from decreasing, thereby suppressing a decrease in the accuracy of phase difference detection.

[0052] Furthermore, in this embodiment, since the saturation evaluation value is calculated by moving average processing (filtering) of the saturation information of each frame, even if the image signal of the frame of interest is saturated and the saturation information is "1", the saturation evaluation value is not necessarily greater than the threshold. In other words, it is possible to determine whether or not to use the filtered image signal for phase difference detection after taking into account the contribution rate of the saturation information of the frame of interest to the saturation evaluation value.

[0053] In this embodiment, since the saturation evaluation value is calculated by a process similar to the moving average processing performed by the filter calculation means 104, the phase characteristics of the filtered image signal and the saturation evaluation value can be matched.

[0054] Next, a third embodiment of the present invention will be described. The third embodiment calculates the saturation evaluation value by filtering, similar to the second embodiment, but differs from the second embodiment in that the saturation evaluation value calculation means has an IIR filter configuration. Therefore, the description of overlapping configurations and operations will be omitted, and the different configurations and operations will be described below.

[0055] Figure 11 is a schematic block diagram showing the configuration of an imaging device 1000 as a signal processing device according to a third embodiment of the present invention. In Figure 11, the frame memory 1002 holds the filtered image signal output from the filter calculation means 104 and the saturation evaluation value output from the saturation evaluation value calculation means 1001. The frame memory 1002 also holds the image signals of multiple previous frames. When filtering is performed, the frame memory 1002 outputs the image signals of the multiple previous frames it holds to the filter calculation means 104 and outputs the saturation evaluation value it holds to the saturation evaluation value calculation means 1001.

[0056] The saturation evaluation value calculation means 1001 calculates a saturation evaluation value based on the saturation information of the frame of interest output from the separation means 103, the previous saturation evaluation value, and the filter coefficient set by the control means 108. The saturation evaluation value calculation means 1001 also outputs the calculated saturation evaluation value to the saturation evaluation value determination means 106. Here, the previous saturation evaluation value is the saturation evaluation value calculated by filtering using the saturation information of frames prior to the frame of interest. The saturation evaluation value calculation means 1001 has a configuration similar to the IIR filter shown in Figure 6(B) and has one frame memory. This one frame memory holds the previous saturation evaluation value. In the third embodiment, the filter calculation means 104 also has an IIR filter configuration.

[0057] In the saturation evaluation value calculation means 1001, a new multi-value saturation evaluation value is calculated using the above formula 1. In this case, the calculation result of formula 1 is the new saturation evaluation value, where A in formula 1 is the saturation information of the frame of interest, and B is the previous saturation evaluation value. After the calculation of the new saturation evaluation value, the previous saturation evaluation value held in the frame memory is replaced with the new saturation evaluation value.

[0058] The processing in the saturation evaluation value determination means 106 and the image displacement amount calculation means 107 after a new saturation evaluation value has been calculated is the same as the processing in the first embodiment.

[0059] Figure 12 is a diagram illustrating the format for one pixel held by the frame memory. Figure 12(A) shows the format 1200 for one pixel in the second embodiment, and Figure 12(B) shows the format 1201 for one pixel in the third embodiment.

[0060] In the second embodiment, the filter calculation means 104 and the saturation evaluation value calculation means 801 have an FIR filter configuration. Since only saturation information is used in calculating the saturation evaluation value, the format 1200 for one pixel only needs to have a bit sequence 401 that stores the digitized signal value of the image signal and a single bit 402 that stores saturation information, which is either "0" or "1". In other words, the amount of data per pixel is small. On the other hand, in the third embodiment, the filter calculation means 104 and the saturation evaluation value calculation means 1001 have an IIR filter configuration. Since the saturation evaluation value is calculated by filtering, it may be multi-bit data with a fractional component. To address this, in the third embodiment, the format 1201 for one pixel has multiple bits, for example a 3-bit bit sequence 403, for storing the saturation evaluation value instead of a single bit 402 that stores saturation information. Therefore, the amount of data per pixel is smaller in the second embodiment than in the third embodiment.

[0061] However, while an FIR filter requires a frame memory of the number of taps minus 1, an IIR filter only requires one frame memory. As a result, the third embodiment can be implemented at a lower cost than the second embodiment.

[0062] In the third embodiment as well, if there are many saturated frames in the image signal of each frame (frames where the saturation information is "1" instead of "0"), the new saturation evaluation value will be large. Therefore, a large saturation evaluation value indicates that the reliability of the filtered image signal is low, similar to the first embodiment.

[0063] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist.

[0064] For example, in each of the embodiments described above, phase difference detection was performed by filtering the two image signals output by two photoelectric conversion elements that receive light beams with different incident directions in a single pixel. However, as shown in Figure 13, the present invention can also be applied when two pixels 303 and 307 each have only one photoelectric conversion element 301 and 302, and the photoelectric conversion elements 301 and 302 receive light beams with different incident directions from the same subject. In this case, phase difference detection is performed by filtering the two image signals output by the photoelectric conversion elements 301 and 302. In this case, the saturation information of the two image signals output by the photoelectric conversion elements 301 and 302 is used to calculate the saturation evaluation value in the manner described in each embodiment.

[0065] Furthermore, while the embodiments described above have described examples of applying the present invention to an imaging device, the devices to which the present invention can be applied are not limited to imaging devices; for example, it may also be applied to a distance measuring device.

[0066] The present invention can also be realized by supplying a program that implements one or more of the functions of each of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0067] 100,800,1000 imaging device 101 Image sensor 102 Saturation detection means 104 Filter calculation means 105,801,1001 Saturation evaluation value calculation method 107 Means for calculating image displacement 109,802,1002 frame memory 303,307 pixels 301, 302, 305, 306 Photoelectric conversion elements 700 Phase characteristics

Claims

1. a pixel having a plurality of photoelectric conversion means for receiving light beams incident from an object in different directions, and outputting a plurality of frames of image signals photoelectrically converted from the light beams; an information generating means for generating saturation information indicating whether the image signal output from the pixel is saturated; a filter calculation means for performing a filter process on the image signals of the plurality of frames to calculate a filtered image signal; evaluation value calculation means for calculating a multi-value saturation evaluation value indicating the reliability of the image signal after the filtering process using the saturation information of the image signal of the currently output frame; and a phase difference detection unit that determines whether or not the filtered image signal is to be used for phase difference detection based on the calculated saturation evaluation value.

2. 2. The signal processing device according to claim 1, wherein the phase difference detection means determines that the filtered image signal is not to be used for phase difference detection when the calculated saturation evaluation value is greater than a predetermined threshold value.

3. 3. The signal processing device according to claim 1, wherein the evaluation value calculation means counts up or down the saturation evaluation value based on the saturation information of the image signal of the currently output frame to calculate a new saturation evaluation value.

4. 4. The signal processing device according to claim 3, wherein the evaluation value calculation means counts up the saturation evaluation value when the saturation information indicates that the image signal of the currently output frame is saturated, and counts down the saturation evaluation value when the saturation information indicates that the image signal of the currently output frame is not saturated.

5. 5. The signal processing device according to claim 3, wherein a count value used when counting up or counting down the saturation evaluation value is set in accordance with the number of the image signals used in the filtering process.

6. 6. The signal processing device according to claim 5, wherein the count value is increased when the number of image signals of a frame preceding the image signal of the currently output frame used in the filtering process is equal to or less than a predetermined value.

7. 3. The signal processing device according to claim 2, wherein the predetermined threshold value is set based on the center of gravity of the phase characteristic of the filtering process.

8. 3. The signal processing device according to claim 1, wherein the evaluation value calculation means calculates a saturation evaluation value by performing a filtering process similar to the filtering process performed by the filter calculation means on saturation information of the image signals of the plurality of frames.

9. 9. The signal processing device according to claim 1, further comprising a storage unit for storing the calculated saturation evaluation value and the image signal after the filtering process.

10. 10. The signal processing device according to claim 1, wherein the filtering process is a moving average process.

11. 11. The signal processing device according to claim 1, wherein the filter calculation means has an IIR filter configuration.

12. 11. The signal processing device according to claim 1, wherein the filter calculation means has an FIR filter configuration.

13. a signal output step in which pixels each having a plurality of photoelectric conversion means for receiving light beams incident from an object in different directions output image signals of a plurality of frames photoelectrically converted from the light beams; an information generating step of generating saturation information indicating whether the image signal output by the pixel is saturated; a filter calculation step of filtering the image signals of the plurality of frames to calculate filtered image signals; an evaluation value calculation step of calculating a multi-value saturation evaluation value indicating reliability of the filtered image signal using the saturation information of the image signal of the currently output frame; and a phase difference detection step of determining whether or not the filtered image signal is to be used for phase difference detection based on the calculated saturation evaluation value.

14. A program for causing a computer to execute a signal processing method, The signal processing method includes: a signal output step in which pixels each having a plurality of photoelectric conversion means for receiving light beams incident from an object in different directions output image signals of a plurality of frames photoelectrically converted from the light beams; an information generating step of generating saturation information indicating whether the image signal output by the pixel is saturated; a filter calculation step of filtering the image signals of the plurality of frames to calculate filtered image signals; an evaluation value calculation step of calculating a multi-value saturation evaluation value indicating reliability of the filtered image signal using the saturation information of the image signal of the currently output frame; and a phase difference detection step of determining whether or not the filtered image signal is to be used for phase difference detection based on the calculated saturation evaluation value.

Images