Information processing system

Abstract

To provide an information processing system capable of obtaining information regarding the accuracy of optical flow calculations. [Solution] The system includes a weighting unit that obtains a first dot product of time-series pixel data, which includes a plurality of time-series pixel values ​​as elements, and a first coefficient sequence, which includes a plurality of weight coefficients corresponding to each of the plurality of pixel values ​​as elements, and obtains a second dot product of the time-series pixel data and a second coefficient sequence, which includes a plurality of weight coefficients corresponding to each of the plurality of pixel values ​​as elements and is different from the first coefficient sequence, and a reliability calculation unit that calculates the reliability of the accuracy of calculating the optical flow from the plurality of pixel values ​​based on the first dot product and the second dot product.

Description

[Technical Field]

[0001] This invention relates to an information processing system. [Background technology]

[0002] Non-Patent Document 1 discloses a time-correlated image sensor. Furthermore, Non-Patent Document 1 discloses the structure of a pixel circuit capable of detecting time correlation and a method for calculating optical flow. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Shigeru Ando and Akira Kurumi, "Time-Correlation Imaging and its Applications," Transactions of the Institute of Electrical Engineers of Japan, Vol. 129, No. 5, May 1, 2009, pp. 129-137. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Time-correlated imaging provides information about the optical motion of the subject in the image. The magnitude of the subject's optical motion can affect the accuracy of optical flow calculation.

[0005] Therefore, the present invention aims to provide an information processing system capable of obtaining information regarding the calculation accuracy of optical flow. [Means for solving the problem]

[0006] According to one disclosure of this specification, a first inner product of time-series pixel data including a plurality of pixel values in time series as elements and a first coefficient sequence including a plurality of weighting coefficients corresponding to the plurality of pixel values as elements is obtained, and a weighting unit that obtains a second inner product of the time-series pixel data and a second coefficient sequence including a plurality of weighting coefficients corresponding to the plurality of pixel values as elements and different from the first coefficient sequence, and based on the first inner product and the second inner product, a reliability calculation unit that calculates a reliability regarding the calculation accuracy of optical flow from the plurality of pixel values is provided, and an information processing system is provided.

Effects of the Invention

[0007] According to this disclosure, an information processing system capable of obtaining information regarding the calculation accuracy of optical flow is provided.

Brief Description of the Drawings

[0008] [Figure 1] It is a block diagram showing the overall configuration of the information processing system according to the first embodiment. [Figure 2] It is a functional block diagram of the information processing apparatus according to the first embodiment. [Figure 3] It is a schematic diagram showing the operation of the weighting unit according to the first embodiment. [Figure 4] It is a graph showing the inner product calculation according to the first embodiment. [Figure 5] It is a diagram showing an example of a reliability map and an optical flow map according to the first embodiment. [Figure 6] It is a diagram showing an example of a reliability map and an optical flow map according to the second embodiment. [Figure 7] It is a schematic diagram showing the overall configuration of the photoelectric conversion device according to the fourth embodiment. [Figure 8] It is a schematic block diagram showing a configuration example of the sensor substrate according to the fourth embodiment. [Figure 9] It is a schematic block diagram showing a configuration example of the circuit board according to the fourth embodiment. [Figure 10]This is a schematic block diagram showing an example of the configuration of one pixel in the photoelectric conversion unit and pixel signal processing unit according to the fourth embodiment. [Figure 11] This is a diagram illustrating the operation of an avalanche photodiode according to the fourth embodiment. [Figure 12] This figure shows the pixel configuration according to the fourth embodiment. [Figure 13] This is a schematic diagram illustrating the configuration of the photoelectric converter according to the fifth embodiment. [Figure 14] This figure shows the pixel configuration according to the fifth embodiment. [Figure 15] This figure shows the pixel arrangement according to the fifth embodiment. [Figure 16] This is a functional block diagram of the information processing device according to the sixth embodiment. [Figure 17] This is a block diagram showing the schematic configuration of the device according to the seventh embodiment. [Figure 18] This is a block diagram showing the schematic configuration of the device according to the eighth embodiment. [Modes for carrying out the invention]

[0009] Embodiments will be described below with reference to the drawings. Elements identical or corresponding to each other across multiple drawings are denoted by the same reference numerals, and their descriptions may be omitted or simplified.

[0010] [First Embodiment] Figure 1 is a block diagram showing the overall configuration of the information processing system according to this embodiment. The information processing system includes a photoelectric converter 100 and an information processing device 300. The photoelectric converter 100 includes a plurality of pixels arranged in a plurality of rows and a plurality of columns. Each of the plurality of pixels generates a signal corresponding to the incident light by photoelectric conversion. The information processing device 300 acquires image data from the photoelectric converter 100 and processes it.

[0011] The information processing system may have a configuration other than that shown in Figure 1. For example, the information processing system may be an integrated device that includes the functions of a photoelectric converter 100 and an information processing device 300. Furthermore, the information processing system does not necessarily have to include the photoelectric converter 100; in that case, the information processing system acquires and processes image data previously generated by an external photoelectric converter from memory or the like.

[0012] In the following description, the photoelectric conversion device is assumed to be an imaging device that captures images. However, the photoelectric conversion device to which the technology of each embodiment can be applied is not limited to imaging devices, but may be other devices. For example, the technology of each embodiment can also be applied to a distance measuring device or a photometric device. A distance measuring device is, for example, a device that measures distance using a focus detection device or ToF (Time of Flight). A photometric device is a device that measures the amount of incident light.

[0013] The information processing device 300 is, for example, a computer. The information processing device 300 performs calculations such as image processing on image data acquired from the photoelectric converter 100. The information processing device 300 may also have a function to control the photoelectric converter 100.

[0014] The information processing device 300 includes a processor 301, RAM (Random Access Memory) 302, ROM (Read Only Memory) 303, recording medium 304, and interface 305. These components are interconnected via a bus or the like.

[0015] The processor 301 may be, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Alternatively, the processor 301 may be an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Multiple processors 301 may be provided within the information processing device 300. Furthermore, multiple processors 301 may perform processing in a distributed manner. The processor 301 reads programs stored in the ROM 303 and recording medium 304 into the RAM 302 and executes them, performing arithmetic processing, controlling various parts of the information processing device 300, and so on.

[0016] RAM 302 is a volatile storage medium and functions as work memory when the processor 301 executes a program. ROM 303 is a non-volatile storage medium and stores firmware and other data necessary for the operation of the information processing device 300. The recording medium 304 is a non-volatile storage medium such as an HDD (Hard Disk Drive), SSD (Solid State Drive), optical disc, or magneto-optical disc. The recording medium 304 stores programs, image data, and other data used in the processing of this embodiment. The function of storing image data and other information in the recording medium 304 may be provided in another device rather than within the information processing device 300.

[0017] Interface 305 may include a communication device that communicates with other devices via wired or wireless means. Interface 305 acquires image data from the photoelectric converter 100. Interface 305 may also include user interfaces such as input devices and output devices.

[0018] Figure 2 is a functional block diagram of the information processing device 300 according to this embodiment. The information processing device 300 includes a data input unit 311, a weighting unit 312, an optical flow calculation unit 313, and a reliability calculation unit 314.

[0019] The processor 301 performs predetermined arithmetic processing by executing a program. This enables the processor 301 to implement the functions of the weighting unit 312, the optical flow calculation unit 313, and the reliability calculation unit 314. Furthermore, the processor 301 controls the interface 305 by executing a program to acquire image data from the photoelectric converter 100. This causes the interface 305 to function as a data input unit 311.

[0020] Figure 3 is a schematic diagram showing the operation of the weighting unit 312 according to this embodiment. In this embodiment, the photoelectric converter 100 captures a normal image without weighting and supplies the captured image to the data input unit 311 of the information processing device 300.

[0021] The data input unit 311 receives a time-series input of images containing N subframes SF-1, SF-2, ..., SF-N, which are repeatedly captured by the photoelectric converter 100 (N is an integer greater than or equal to 2). Each of the subframes SF-1, SF-2, ..., SF-N is array data containing multiple pixel values ​​generated at multiple pixels. The N subframes SF-1, SF-2, ..., SF-N are used to generate one image. The image generated by integrating the pixel values ​​of the N subframes SF-1, SF-2, ..., SF-N is called a mainframe. The exposure period for generating one mainframe is called the mainframe period. One mainframe period is divided into N subframe periods. Each of the N subframe periods is the exposure period for generating a corresponding subframe. For example, suppose the frame rate of the mainframe is 30 fps and the number of subframes constituting one mainframe is 100. In this case, the length of one mainframe period can be calculated as approximately 33.3 ms (1 s ÷ 30 fps), and the length of one subframe period can be calculated as approximately 333 μs (33.3 ms ÷ 100).

[0022] The weighting unit 312 generates a main frame by multiplying the pixel values ​​of the N subframes SF-1, SF-2, ..., SF-N by N weight coefficients for each pixel, and then accumulating the N values ​​obtained from the multiplication. Here, the N weight coefficients are arranged in time series from the first weight coefficient to the nth weight coefficient, and the i-th subframe SF-i is multiplied by the i-th weight coefficient (i is an integer between 1 and N). The pixel values ​​of identical pixels included in the N subframes SF-1, SF-2, ..., SF-N can be represented by time-series pixel data of N elements, and the N weight coefficients can be represented by a coefficient sequence of N elements. Therefore, the processing of the weighting unit 312 can be rephrased as the process of calculating the dot product of time-series pixel data containing the time-series pixel values ​​of subframes SF-1, SF-2, ..., SF-N as elements and a coefficient sequence containing the weight coefficients as elements.

[0023] Multiple sets of weight coefficients are used in the processing of the weighting unit 312. Figure 3 shows an example of weighting using three sets of weight coefficients WN-1, WN-2, ..., WN-N, weight coefficients WS-1, WS-2, ..., WS-N, and weight coefficients WC-1, WC-2, ..., WC-N. The weight coefficients WN-1, WN-2, ..., WN-N have a constant value during the mainframe period. The weight coefficients WS-1, WS-2, ..., WS-N change in a waveform of a sinusoidal function for one period during the mainframe period. The weight coefficients WC-1, WC-2, ..., WC-N change in a waveform of a cosine function for one period during the mainframe period. That is, the coefficient sequence (second coefficient sequence) including the weight coefficients WN-1, WN-2, ..., WN-N has a constant value with respect to the acquisition time of the corresponding pixel value. The coefficient sequence containing weight coefficients WS-1, WS-2, ..., WS-N (first coefficient sequence) and the coefficient sequence containing weight coefficients WC-1, WC-2, ..., WC-N (third coefficient sequence) change periodically with respect to the acquisition time of the corresponding pixel value. By applying weights using the three sets of weight coefficients as described above, three types of mainframes are generated that have the same time resolution but different weights for each time point within the mainframe period.

[0024] The normal image IMN is generated by integrating the dot product values ​​(second dot product) calculated by performing a dot product operation between the weight coefficients WN-1, WN-2, ..., WN-N and the pixel values ​​of subframes SF-1, SF-2, ..., SF-N. Since the weight coefficients WN-1, WN-2, ..., WN-N are constant values, the normal image IMN is similar to the image obtained by normal exposure without time weighting.

[0025] The correlation image IMS is generated by integrating the dot product values ​​(first dot product) calculated by performing a dot product operation between the weight coefficients WS-1, WS-2, ..., WS-N and the pixel values ​​of subframes SF-1, SF-2, ..., SF-N. Since the weight coefficients WS-1, WS-2, ..., WS-N change over time due to the waveform of a sine function, the correlation image IMS has time-dependent information added to it.

[0026] The correlation image IMC is generated by integrating the dot product (third dot product) calculated by performing a dot product operation between the weight coefficients WC-1, WC-2, ..., WC-N and the pixel values ​​of subframes SF-1, SF-2, ..., SF-N. Since the weight coefficients WC-1, WC-2, ..., WC-N change over time due to the waveform of the cosine function, the correlation image IMC has time-dependent information added to it, with a different phase than the correlation image IMS.

[0027] The weighting described above is illustrative, and for example, waveforms of periodic functions other than sine and cosine functions may be used for weighting. For example, the waveform used for weighting may be a triangular wave, a ramp wave, or a random pattern. Also, the number of sets of weight coefficients may be two, four or more.

[0028] The weighting unit 312 generates the normal image IMN and the correlated image IMS and IMC by calculating the dot product of the weight coefficients and the pixel values ​​of the subframes, as described above. The weighting unit 312 outputs the normal image IMN and the correlated image IMS and IMC to the optical flow calculation unit 313 and the confidence calculation unit 314.

[0029] The optical flow calculation unit 313 calculates the horizontal motion vector v in the normal image IMN according to the optical flow constraint equations shown in equations (1) to (5) below. x , and vertical motion vector v y Calculate.

number

number

number

number

number

[0030] Here, the time integral terms of f(x,y,t) in equations (2) and (3) correspond to the normal image IMN. The time integral terms of f(x,y,t)sinωt in equations (1), (4), and (5) correspond to the correlated image IMS. The time integral terms of f(x,y,t)cosωt in equations (1), (2), and (3) correspond to the correlated image IMC. In equations (2) to (5), the horizontal difference component ∂x and the vertical difference component ∂y of the normal image IMN can be calculated using a Sobel filter or the like.

[0031] The confidence calculation unit 204 calculates the confidence level of each pixel using the normal image IMN and the correlated image IMS and IMC. The confidence calculation unit 204 outputs the calculated confidence level for each pixel, for example, in the form of a confidence map.

[0032] Figures 4(a), 4(b), and 4(c) are graphs showing the dot product calculation according to this embodiment. Figures 4(a), 4(b), and 4(c) show the time variation of the product of light intensity and weight coefficient over the mainframe period when the output signal of a single pixel is weighted in the three ways described above. In the examples of Figures 4(a), 4(b), and 4(c), it is assumed that there is no movement of the subject during the mainframe period and that a constant amount of light is incident on the photoelectric converter 100.

[0033] Figure 4(a) shows an example of weighting using weight coefficients WN-1, WN-2, ..., WN-N, each having a constant value. Here, the weight coefficients WN-1, WN-2, ..., WN-N are all 1, and the product of the light intensity and the weight coefficient is H for all subframe periods. In this case, the dot product value within the mainframe period is HN. This corresponds to the sum of the areas of the hatched regions in Figure 4(a).

[0034] Figure 4(b) shows an example of weighting using weight coefficients WS-1, WS-2, ..., WS-N, which vary with a sinusoidal waveform. Since the amount of incident light is the same as in Figure 4(a), the amplitude of the product of the light amount and the weight coefficient is H. In this case, the area of ​​the hatched region in Figure 4(b) is half that of the example in Figure 4(a), so it can be calculated as 0.5HN. Therefore, the dot product value within the mainframe period when weighting is performed using a sinusoidal waveform is 0.5HN.

[0035] Figure 4(c) shows an example of weighting using weighting coefficients WC-1, WC-2, ..., WC-N, which vary with the waveform of a cosine function. Since the amount of incident light is the same as in Figure 4(a), the amplitude of the product of the light amount and the weighting coefficient is H. In this case, the area of ​​the hatched region in Figure 4(c) is half that of the example in Figure 4(a), so it can be calculated as 0.5HN. Therefore, the dot product value within the mainframe period when weighting is performed using the waveform of a cosine function is 0.5HN.

[0036] When the amount of incident light during the main frame period is constant, the inner product value P obtained by weighting with a constant value nor , the inner product value P obtained by weighting with a sine function sin and the inner product value P obtained by weighting with a cosine function cos satisfy the following relationship of Equation (6).

Equation

[0037] Equation (6) shows that the ratio value of the inner product value P nor to the inner product value P sin is 0.5, and the ratio value of the inner product value P nor to the inner product value P cos is 0.5. In pixels that satisfy Equation (6), since the amount of incident light during the main frame period is constant, the pixel value of the sub-frame does not change with time. This situation typically occurs when the subject at the coordinates of the pixel is stationary. Also, even if the subject at the coordinates of the pixel is moving, an optical flow that cannot be distinguished from the state where the subject is stationary is calculated. Therefore, when Equation (6) is satisfied, or when inner product values P nor , P sin , P cos close to Equation (6) are obtained, the calculation accuracy of the optical flow decreases. On the other hand, when inner product values P nor , P sin , P cos far from the conditions of Equation (6) are obtained, it is highly likely that the subject is moving at the coordinates of the pixel, and the optical flow can be calculated with high accuracy.

[0038] Therefore, by calculating the reliability defined by an equation that indicates whether the inner product values P nor , P sin , P cos have a relationship close to the relational expression of Equation (6), an index indicating the calculation accuracy of the optical flow can be obtained. Specifically, as an example of an index of the calculation accuracy of the optical flow, the reliability R1 of the following Equation (7) can be cited.

number

[0039] The expression inside the absolute value sign in equation (7) is the dot product P. sin and the dot product P cos The sum of the dot products P nor This involves subtracting the inner product value P. nor , P sin , P cos The value is close to 0 when the relationship between the values ​​is close to equation (6). Therefore, the closer the confidence level R1 is to 0, the higher the probability that the coordinates of the pixel are in the stationary region, and the lower the accuracy of the optical flow calculation. Also, the larger the confidence level R1, the higher the probability that the coordinates of the pixel are in the dynamic region. By calculating the confidence level R1 for each of multiple pixels, a confidence map can be generated in which the confidence levels of each pixel are mapped in a matrix.

[0040] The formulas that can be used to calculate the confidence level are not limited to formula (7), but also include the dot product P. nor , P sin , P cos Any formula that can evaluate the closeness to formula (6) can be used as appropriate. For example, the confidence level R² of formula (8) below may be used.

number

[0041] The confidence level R² is the inner product value P nor The inner product value P sin The ratio value, or dot product value P nor The inner product value P cos The closer the ratio of P is to 0.5, the closer it approaches 0. Also, when both of these ratios are close to 0.5, the confidence level R² approaches 0 even closer. In other words, equation (8) is obtained by adding the dot product value P to the conditions of equation (7). sin and the dot product P cos It is also an indicator that evaluates proximity.

[0042] The optical flow calculated by the optical flow calculation unit 313 and the confidence map calculated by the confidence calculation unit 314 can be used for image processing. This image processing may, for example, be a process that evaluates the movement of a subject based on the optical flow and the confidence map. The image processing using the optical flow and the confidence map may be performed inside the information processing device 300 or in another information processing device outside the information processing device 300.

[0043] Figure 5(a) shows an example of a confidence map according to this embodiment, and Figure 5(b) shows an example of an optical flow map according to this embodiment. Figures 5(a) and 5(b) show the two-dimensional distribution of confidence and optical flow generated from the video, respectively. The imaging range in Figures 5(a) and 5(b) includes trees, a vehicle OB2, and the background BG. The trees are divided into leaf portion OB1a and trunk portion OB1b. Vehicle OB2 is traveling at a predetermined speed. The leaf portion OB1a is moving due to wind, while the trunk portion OB1b is not moving.

[0044] Figure 5(a) shows the distribution of confidence levels. Since vehicle OB2 is moving at a relatively high speed, a high confidence level is obtained in the region of vehicle OB2. Since the leaf portion OB1a is moving at a relatively low speed, a moderate confidence level is obtained. Since the trunk portion OB1b and background BG are not moving, a low confidence level is obtained. In the confidence map, the position within the imaging range is associated with the confidence level at each position.

[0045] The confidence map obtained in this way can be used to detect moving areas using region extraction methods, such as extracting high-confidence regions where a confidence level of a predetermined value or higher is obtained. In the case of an image of a moving vehicle OB2, as shown in Figure 5(a), the approximate range containing the moving vehicle OB2 can be detected by detecting a rectangular region that includes the moving area.

[0046] Figure 5(b) shows the high-confidence region RH1, extracted from the confidence map, overlaid on the optical flow map. The hatched areas in Figure 5(b) represent regions with a moderate or higher confidence level, indicating a high probability of subject movement. In these hatched regions, optical flow can be calculated with relatively high accuracy. Furthermore, within the high-confidence region RH1, optical flow can be calculated with even higher accuracy.

[0047] Furthermore, the optical flow calculation unit 313 may omit the optical flow calculation in regions other than the high-reliability region RH1. This reduces the computational load and processing time. In this way, the optical flow calculation unit 313 may control whether or not to perform the optical flow calculation based on the reliability calculation result in the reliability calculation unit 314. For example, the optical flow calculation unit 313 may employ an operation control that performs the optical flow calculation when the reliability is above a predetermined threshold, and does not perform the optical flow calculation when the reliability is below the predetermined threshold. In addition, the reliability calculation result in the reliability calculation unit 314 may be used to control whether or not to perform image processing other than optical flow or recognition processing of captured images.

[0048] In this embodiment, in image processing where weighting is performed by the dot product of time-series pixel data including subframe pixel values ​​as elements and a sequence of coefficients including weight coefficients as elements, confidence is obtained based on two or more dot product values ​​obtained using multiple different weight coefficients. This confidence indicates the likelihood that the pixel values ​​are changing over time and can be used in other processing as information indicating the accuracy of optical flow calculation. Therefore, according to this embodiment, an information processing device 300 is provided that can more favorably obtain information regarding the accuracy of optical flow calculation.

[0049] [Second Embodiment] This embodiment describes modified examples of the process for calculating and determining reliability. In this embodiment, elements common to the first embodiment may be omitted or simplified in their explanation.

[0050] In the first embodiment, an example is shown in which the confidence level is calculated as a continuous value using a mathematical formula. In contrast, this embodiment describes an example in which the confidence level determination result is output as binary data by comparing the calculated confidence level with a threshold.

[0051] In this embodiment, the reliability calculation unit 314 calculates the reliability values ​​R3 and R4 using the following equations (9) and (10).

number

number

[0052] The formula for calculating the confidence level R3 is the dot product P obtained by applying a certain weighting. nor And the inner product P obtained by weighting with a sine function. sin This includes the dot product P. The confidence level R3 is the inner product value P. nor The inner product value P sin The closer the ratio is to 0.5, the closer it approaches 0. The formula for calculating the confidence level R4 is the dot product P obtained by a certain weighting. nor And the inner product P obtained by weighting the cosine function. cos This includes the dot product P. The confidence level R4 is the dot product value P. nor The inner product value P cos The closer the ratio is to 0.5, the closer it approaches 0.

[0053] The confidence calculation unit 314 then determines whether both confidence levels R3 and R4 are less than the threshold TH, and outputs the determination result as binary data. For example, the confidence calculation unit 314 outputs a logical value of 0 if both confidence levels R3 and R4 are less than the threshold TH, and outputs a logical value of 1 if at least one of confidence levels R3 and R4 is greater than or equal to the threshold TH.

[0054] A logical value of 0 indicates that the coordinates of the pixel are likely to be in a stationary region, resulting in low accuracy in calculating the optical flow. A logical value of 1 indicates that the coordinates of the pixel are likely to be in a dynamic region, resulting in high accuracy in calculating the optical flow. In other words, the confidence calculation unit 314 of this embodiment outputs the confidence information in a binarized form. By calculating the above logical value for each of a plurality of pixels, a confidence map can be generated in which the confidence judgment results for each pixel are mapped in a matrix. The threshold TH may be a pre-set value, or it may be dynamically set based on information from the input image or information from the external imaging environment. The magnitude of confidence can also change depending on circumstances other than the movement of the subject (for example, the brightness of the imaging environment), so the judgment accuracy can be improved by dynamically setting the threshold TH by acquiring information from the input image, imaging environment, etc.

[0055] The optical flow calculation unit 313 may, for example, calculate a representative optical flow vector by excluding the output signals of pixels whose logical value in the confidence map is 0. Alternatively, for pixels whose logical value in the confidence map is 0, interpolation processing may be performed using the output signals of surrounding pixels whose logical value is 1. Furthermore, the optical flow calculation unit 313 may exclude the output signals of pixels whose logical value in the confidence map is 0 from the calculation process.

[0056] Figure 6(a) shows an example of a confidence map according to this embodiment, and Figure 6(b) shows an example of an optical flow map according to this embodiment. The types of subjects present in the image are generally the same as in the first embodiment, but in this embodiment, it is assumed that there is little color change due to texture, gradient, etc., within the body panel surface of the vehicle OB2. When there is little color change within a certain area of ​​the subject, even if the subject is moving, the amount of light incident on the pixel of interest of the photoelectric converter 100 does not change over time, which may reduce the accuracy of the optical flow calculation. Figure 6(a) shows an example of a confidence map obtained in this situation.

[0057] Because vehicle OB2 is moving at a relatively high speed, a logical value of 1 is obtained in the region of the vehicle OB2's contour. However, a logical value of 0 is obtained in other regions. In other words, in this embodiment, the region in which high-precision optical flow is calculated is only the contour portion of vehicle OB2. Therefore, in order to obtain an appropriate optical flow, it is necessary to perform a process that interpolates the central portion of vehicle OB2. Specifically, since high-precision optical flow is calculated in the contour portion of vehicle OB2, a process can be performed to propagate the optical flow vector from the contour toward the inner region of vehicle OB2. For example, morphological processing used in the field of image analysis can be applied to this process.

[0058] In this embodiment, the reliability is obtained in the same way as in the first embodiment. Furthermore, in this embodiment, data indicating the reliability determination result can also be obtained by comparing the reliability with a threshold. Therefore, according to this embodiment, an information processing device 300 is provided that can more preferably obtain information regarding the calculation accuracy of optical flow.

[0059] [Third Embodiment] This embodiment describes a modified example of the process for calculating the confidence level. In this embodiment, elements common to the first embodiment may be omitted or simplified in their explanation.

[0060] In the first and second embodiments, the dot product value P nor , P sin , P cos The confidence levels R1, R2, R3, and R4 based on these levels are illustrated. In calculating these confidence levels, the dot product P is used. nor Normalization using [a specific method / function] may also be performed.

[0061] First, an example of normalization being performed when calculating the confidence level in the processing of the second embodiment will be explained. The confidence level calculation unit 314 calculates the confidence levels R5 and R6 using the following equations (11) and (12).

number

number

[0062] The formulas for calculating confidence levels R5 and R6 are the dot product of equations (9) and (10), respectively, P. nor It is obtained by normalizing by dividing by . The confidence calculation unit 314 then determines whether both confidence levels R5 and R6 are smaller than the threshold TH and outputs the determination result as binary data. For example, the confidence calculation unit 314 outputs a logical value of 0 if both confidence levels R5 and R6 are smaller than the threshold TH, and outputs a logical value of 1 if at least one of confidence levels R5 and R6 is greater than or equal to the threshold TH.

[0063] In the case of using the reliability values ​​R3 and R4 of the second embodiment, the reliability value increases as the imaging environment becomes brighter. Therefore, in order to ensure judgment accuracy, it may be necessary to dynamically change the threshold TH according to the brightness of the imaging environment. In contrast, the reliability values ​​R5 and R6 of this embodiment use the dot product value P nor Because it is normalized, there is no need to change the threshold TH according to the brightness of the imaging environment, and the determination can be made using a constant threshold. Therefore, according to this embodiment, an information processing device 300 is provided that can more favorably acquire information regarding the calculation accuracy of optical flow.

[0064] In the processing of the first embodiment, normalization may be performed when calculating the confidence level. In this modified example, the confidence level calculation unit 314 calculates the confidence level R7 or confidence level R8 using the following formula (13) or formula (14).

number

number

[0065] The formulas for calculating confidence levels R7 and R8 are the dot product of equations (7) and (8), respectively, P. norThis is obtained by normalizing by dividing by [the specified factor]. In these modified examples, as with equations (11) and (12), a reliability independent of the brightness of the imaging environment is obtained.

[0066] [Fourth Embodiment] This embodiment describes a modified version of the photoelectric converter 100. In this embodiment, elements common to the first to third embodiments may be omitted or simplified in their description.

[0067] In the first to third embodiments, an example was described in which the dot product calculation is performed within the information processing device 300 to generate a normal image and a correlated image. However, the process of generating the normal image and the correlated image may be performed outside the information processing device 300. In this embodiment, an example is described in which a signal generation operation equivalent to the dot product calculation is performed within the photoelectric converter 100.

[0068] Figure 7 is a schematic diagram showing the overall configuration of the photoelectric converter 100 according to this embodiment. The photoelectric converter 100 has a sensor substrate 11 (first substrate) and a circuit board 21 (second substrate) stacked on top of each other. The sensor substrate 11 and the circuit board 21 are electrically interconnected. The sensor substrate 11 has a pixel region 12 on which a plurality of pixel circuits 101 are arranged in a plurality of rows and a plurality of columns. The circuit board 21 may include a first circuit region 22 on which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns, and a circuit for controlling the first circuit region elementary signal processing unit 103. The sensor substrate 11 has a light incident surface that receives incident light and a connection surface that faces the light incident surface. The sensor substrate 11 is connected to the circuit board 21 on the connection surface side. In other words, the photoelectric converter 100 is a so-called back-illuminated type.

[0069] In this specification, "plan view" refers to viewing from a direction perpendicular to the surface opposite to the light incidence surface. Similarly, "cross-section" refers to the surface of the sensor substrate 11 perpendicular to the surface opposite to the light incidence surface. While the light incidence surface may appear rough at a microscopic level, in such cases, the plan view is defined based on the light incidence surface as viewed macroscopically.

[0070] In the following description, the sensor substrate 11 and the circuit board 21 are assumed to be diced chips, but the sensor substrate 11 and the circuit board 21 are not limited to chips. For example, the sensor substrate 11 and the circuit board 21 may be wafers. Furthermore, if the sensor substrate 11 and the circuit board 21 are diced chips, the photoelectric converter 100 may be manufactured by stacking wafers and then dicing them, or by stacking wafers after dicing them.

[0071] Figure 8 is a schematic block diagram showing an example of the arrangement of the sensor substrate 11. Multiple pixel circuits 101 are arranged in multiple rows and multiple columns in the pixel region 12. Each of the multiple pixel circuits 101 has a photoelectric conversion unit 102 on the substrate, which includes an avalanche photodiode (hereinafter referred to as APD) as a photoelectric conversion element.

[0072] In an APD (Automated Precipitator), the conductivity type corresponding to the charge used as the signal charge is called the first conductivity type. The first conductivity type refers to a conductivity type in which the majority carriers are charges of the same polarity as the signal charge. Conversely, the conductivity type opposite to the first conductivity type, i.e., a conductivity type in which the majority carriers are charges of a different polarity than the signal charge, is called the second conductivity type. In the APD described below, the anode of the APD is at a fixed potential, and the signal is extracted from the cathode of the APD. Therefore, the semiconductor region of the first conductivity type is the N-type semiconductor region, and the semiconductor region of the second conductivity type is the P-type semiconductor region. Alternatively, the cathode of the APD may be at a fixed potential, and the signal may be extracted from the anode of the APD. In this case, the semiconductor region of the first conductivity type is the P-type semiconductor region, and the semiconductor region of the second conductivity type is the N-type semiconductor region. Furthermore, the following description focuses on the case where one node of the APD is at a fixed potential, but a configuration in which the potentials of both nodes fluctuate is also possible.

[0073] Figure 9 is a schematic block diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a first circuit region 22 in which a plurality of pixel signal processing units 103 are arranged in a plurality of rows and a plurality of columns.

[0074] Furthermore, the circuit board 21 includes a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, a pixel output signal line 113, an output circuit 114, a control signal generation unit 115, and a weight control unit 116. The multiple photoelectric conversion units 102 shown in Figure 8 and the multiple pixel signal processing units 103 shown in Figure 9 are electrically connected via connecting wiring provided for each pixel circuit 101.

[0075] The control signal generation unit 115 is a control circuit that generates and supplies control signals to drive the vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, and the weight control unit 116. In this way, the control signal generation unit 115 controls the drive timing and other aspects of each component.

[0076] The vertical scanning circuit 110 supplies control signals to each of the multiple pixel signal processing units 103 based on the control signals supplied from the control signal generation unit 115. The vertical scanning circuit 110 supplies control signals to each pixel signal processing unit 103 row by row via drive lines provided for each row of the first circuit region 22. As will be described later, there may be multiple drive lines for each row. Logic circuits such as shift registers and address decoders may be used in the vertical scanning circuit 110. This allows the vertical scanning circuit 110 to select the row from which the pixel signal processing unit 103 will output a signal.

[0077] The signal output from the photoelectric conversion unit 102 of the pixel circuit 101 is processed by the pixel signal processing unit 103. The pixel signal processing unit 103 acquires and stores a digital signal based on the pulse output from the APD included in the photoelectric conversion unit 102.

[0078] The weight control unit 116 controls the weighting coefficient (weighting amount) applied to the output signal from the APD in the pixel signal processing unit 103. A reference signal containing information about the weighting amount is supplied from the weight control unit 116 to each of the multiple pixel signal processing units 103.

[0079] The horizontal scanning circuit 111 supplies control signals to the readout circuit 112 based on control signals supplied from the control signal generation unit 115. The pixel signal processing unit 103 is connected to the readout circuit 112 via pixel output signal lines 113, which are provided for each column of the first circuit region 22. The pixel output signal line 113 of one column is shared by multiple pixel signal processing units 103 of the corresponding column. The pixel output signal line 113 includes multiple wires and has at least the function of outputting a digital signal from each pixel signal processing unit 103 to the readout circuit 112, and the function of supplying a control signal to the pixel signal processing unit 103 for selecting the column from which to output a signal. The readout circuit 112 outputs a signal to the information processing device 300 via the output circuit 114 based on control signals supplied from the control signal generation unit 115.

[0080] The information processing device 300 performs signal processing on the signal output from the photoelectric converter 100. The information processing device 300 can perform processing related to time correlation imaging using the signal output from the photoelectric converter 100. The processing related to time correlation imaging includes the processing performed by the optical flow calculation unit 313 described in the first to third embodiments and the processing performed by the reliability calculation unit 314.

[0081] The photoelectric conversion units 102 in the pixel region 12 may be arranged in a one-dimensional manner. Furthermore, the pixel signal processing unit 103 does not necessarily have to be provided for every pixel circuit 101. For example, one pixel signal processing unit 103 may be shared by multiple pixel circuits 101. In this case, the pixel signal processing unit 103 provides signal processing functionality to each pixel circuit 101 by sequentially processing the signals output from each photoelectric conversion unit 102.

[0082] As shown in Figures 8 and 9, a first circuit region 22, in which multiple pixel signal processing units 103 are arranged, is located in a region that overlaps with the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, a control signal generation unit 115, and a weight control unit 116 are arranged so as to overlap between the edge of the sensor substrate 11 and the edge of the pixel region 12 in a plan view. In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A second circuit region 23 is located in the circuit substrate 21, in a region that overlaps with the non-pixel region in a plan view, and in which the vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, control signal generation unit 115, and weight control unit 116 are arranged.

[0083] Note that the arrangement of the pixel output signal lines 113, the readout circuit 112, and the output circuit 114 is not limited to those shown in Figure 9. For example, the pixel output signal lines 113 may be arranged to extend in the row direction and be shared by multiple pixel signal processing units 103 in the corresponding row. The readout circuit 112 may be arranged so that the pixel output signal lines 113 of each row are connected.

[0084] Figure 10 is a schematic block diagram showing an example of the configuration of one pixel of the photoelectric conversion unit 102 and the pixel signal processing unit 103 according to this embodiment. Figure 10 schematically shows a more specific configuration example, including the connection relationship between the photoelectric conversion unit 102 arranged on the sensor substrate 11 and the pixel signal processing unit 103 arranged on the circuit board 21. In Figure 10, the drive lines between the vertical scanning circuit 110 and the pixel signal processing unit 103 in Figure 9 are shown as drive lines 213 and 214.

[0085] The photoelectric conversion unit 102 has an APD 201. The pixel signal processing unit 103 has a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. Note that the pixel signal processing unit 103 only needs to have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0086] The APD201 generates charge pairs corresponding to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD201. The cathode of the APD201 is connected to the first terminal of the quench element 202 and the input terminal of the waveform shaping unit 210. A voltage VH (second voltage), which is higher than the voltage VL supplied to the anode, is supplied to the cathode of the APD201. As a result, a reverse bias voltage is supplied to the anode and cathode of the APD201, causing the APD201 to perform avalanche multiplication. When a charge is generated by incident light in the APD201 with the reverse bias voltage supplied, this charge undergoes avalanche multiplication, generating an avalanche current.

[0087] When a reverse bias voltage is supplied to the APD201, there are two operating modes: Geiger mode and linear mode. Geiger mode is a mode in which the anode and cathode potential difference is greater than the breakdown voltage, while linear mode is a mode in which the anode and cathode potential difference is near or below the breakdown voltage.

[0088] An APD operating in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In this case, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V. The APD201 may operate in linear mode or Geiger mode. In the case of a SPAD, the potential difference is larger compared to a linear-mode APD, and the avalanche multiplication effect is more pronounced, so it is preferable to use a SPAD.

[0089] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. The quench element 202 suppresses the voltage supplied to the APD201, thereby suppressing avalanche multiplication (quench operation). The quench element 202 also restores the voltage supplied to the APD201 to voltage VH by supplying a current corresponding to the voltage drop caused by the quench operation (recharge operation). The quench element 202 may be, for example, a transistor.

[0090] The waveform shaping unit 210 shapes the cathode potential change of the APD201 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 210. Figure 10 shows an example in which one inverter is used as the waveform shaping unit 210, but the waveform shaping unit 210 may also be a circuit in which multiple inverters are connected in series, or it may be any other circuit that has a waveform shaping effect.

[0091] The counter circuit 211 counts the pulse signals output from the waveform shaping unit 210 and holds a digital signal indicating the count value. When a control signal is supplied from the vertical scanning circuit 110 via the drive line 213, the counter circuit 211 resets the signal it is holding.

[0092] The selection circuit 212 receives a control signal from the vertical scanning circuit 110 shown in Figure 9 via the drive line 214 shown in Figure 10. In response to this control signal, the selection circuit 212 switches between electrically connecting and disconnecting the counter circuit 211 and the pixel output signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal corresponding to the value held in the counter circuit 211.

[0093] In the example shown in Figure 10, the selection circuit 212 switches between the electrical connection and disconnection of the counter circuit 211 and the pixel output signal line 113. However, the method for controlling the signal output to the pixel output signal line 113 is not limited to this. For example, switches such as transistors may be placed at nodes such as between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the pixel signal processing unit 103, to switch between electrical connection and disconnection and control the signal output to the pixel output signal line 113. Alternatively, the signal output to the pixel output signal line 113 may be controlled by changing the value of the voltage VH or voltage VL supplied to the photoelectric conversion unit 102 using a switch such as a transistor.

[0094] Figures 11(a), 11(b), and 11(c) illustrate the operation of the APD201 according to this embodiment. Figure 11(a) is a diagram showing the APD201, quench element 202, and waveform shaping unit 210 extracted from Figure 10. As shown in Figure 11(a), the connection node of the input terminals of the APD201, quench element 202, and waveform shaping unit 210 is designated as nodeA. Also, as shown in Figure 11(a), the output side of the waveform shaping unit 210 is designated as nodeB.

[0095] Figure 11(b) is a graph showing the time evolution of the potential of node A in Figure 11(a). Figure 11(c) is a graph showing the time evolution of the potential of node B in Figure 11(a). During the period from time t0 to time t1, a voltage of VH-VL is applied to APD201 in Figure 11(a). When a photon is incident on APD201 at time t1, avalanche multiplication occurs in APD201. This causes an avalanche current to flow in the quench element 202, and the potential of node A drops. Subsequently, the amount of potential drop increases further, and the voltage applied to APD201 gradually decreases. Then, at time t2, avalanche multiplication in APD201 stops. As a result, the voltage level of node A no longer drops below a certain value. Subsequently, during the period from time t2 to time t3, a current flows through nodeA to compensate for the voltage drop from the node with voltage VH, and at time t3, nodeA settles back to its original potential.

[0096] In the process described above, the potential of nodeB becomes high during the period when the potential of nodeA is below a certain threshold. In this way, the waveform of the potential drop at nodeA caused by the photon incidence is shaped by the waveform shaping unit 210 and output as a pulse to nodeB.

[0097] Figure 12 shows the configuration of a pixel according to this embodiment. Figure 12 shows the photoelectric conversion unit 102 and the pixel signal processing unit 103 in Figure 10 in more detail. Hereinafter, the combined element of the photoelectric conversion unit 102 and the pixel signal processing unit 103 may be referred to as pixel 200. In Figure 12, elements having the same function as those shown in Figure 10 are denoted by the same reference numerals as in Figure 10, and the explanation of these elements may be omitted or simplified.

[0098] Pixel 200 includes an APD 201, a quench element 202, a waveform shaping unit 210, and a counter circuit 211. The quench element 202 has a P-type MOS transistor 202a. The counter circuit 211 includes an AND circuit 230, multiplexers 241 and 242, and integrators 251, 252, and 253. Pixel 200 receives signals PCLKB and P_RES from the vertical scanning circuit 110.

[0099] The signal PCLKB is input to the gate of MOS transistor 202a. Voltage VH is supplied to the source of MOS transistor 202a. The drain of MOS transistor 202a is connected to the cathode of APD201 and the input terminal of the waveform shaping unit 210.

[0100] The signal PCLKB is, for example, a pulse signal with a frequency of 1 MHz to 200 MHz. The number of pulses within one mainframe period can be set appropriately according to the number of bits in the counter circuit 211. When the counter circuit 211 performs counting with 11 bits, the number of pulses within one mainframe period is less than 2048. The signal PCLKB controls the timing of the recharge operation in the APD201. When the signal PCLKB goes low, the MOS transistor 202a turns on, and the recharge operation is performed in the APD201.

[0101] The signal PCLKB is input to the first input terminal of the AND circuit 230, and the output signal of the waveform shaping unit 210 is input to the second input terminal of the AND circuit 230. The AND circuit 230 outputs the logical AND of the signal PCLKB and the output signal of the waveform shaping unit 210 to the integration circuit 251, the control terminal of the multiplexer 241, and the control terminal of the multiplexer 242.

[0102] The integration circuit 251 counts the number of pulses by integrating the pulses of the signal input from the AND circuit 230. The integration circuit 251 stores the count value obtained in this way. The count value stored in the integration circuit 251 corresponds to the number of photons incident on the APD201. That is, the count value equivalent to the dot product when the weighting coefficient is constant is stored in the integration circuit 251.

[0103] The weight control unit 116 includes a sine wave generation unit 116a and a cosine wave generation unit 116b. The sine wave generation unit 116a generates a pulse signal that represents a time-varying value based on a sine function waveform, with the main frame period as one period. The cosine wave generation unit 116b generates a pulse signal that represents a time-varying value based on a cosine function waveform, with the main frame period as one period. The sine wave generation unit 116a outputs the pulse signal as a bus signal to the first input terminal of the multiplexer 241. The cosine wave generation unit 116b outputs the pulse signal as a bus signal to the first input terminal of the multiplexer 242. The second input terminal of the multiplexer 241 and the second input terminal of the multiplexer 242 are connected to a ground potential line having a ground potential corresponding to a logical value of 0.

[0104] Multiplexers 241 and 242 select the signal at the first input terminal (labeled "1" in Figure 12) when a high-level signal is input to the control terminal, and select the signal at the second input terminal (labeled "0" in Figure 12) when a low-level signal is input to the control terminal. The output signal of multiplexer 241 is input to the integration circuit 252, and the output signal of multiplexer 242 is input to the integration circuit 253.

[0105] Therefore, when the output signal of the AND circuit 230 is at a high level, the multiplexer 241 outputs the pulse signal output from the sinusoidal waveform generation unit 116a to the integrator circuit 252. The integrator circuit 252 counts the number of pulses by counting up according to the value of the signal input from the multiplexer 241. The integrator circuit 252 stores the count value obtained in this way. Also, when the output signal of the AND circuit 230 is at a low level, the multiplexer 241 outputs a signal with a logical value of 0 to the integrator circuit 252. In this case, no count-up is performed. Through these operations, the integrator circuit 252 stores a count value that is weighted according to the number of photons incident on the APD201, corresponding to the weighting coefficients WS-1, WS-2, ..., WS-N in Figure 3.

[0106] Furthermore, if the output signal of the AND circuit 230 is at a high level, the multiplexer 242 outputs the pulse signal output from the cosine waveform generation unit 116b to the integrator circuit 253. The integrator circuit 253 counts the number of pulses by counting up according to the value of the signal input from the multiplexer 242. The integrator circuit 253 stores the count value obtained in this way. Also, if the output signal of the AND circuit 230 is at a low level, the multiplexer 242 outputs a signal with a logical value of 0 to the integrator circuit 253. In this case, no count-up is performed. Through these operations, the integrator circuit 253 stores a count value that is weighted according to the number of photons incident on the APD201, corresponding to the weighting coefficients WC-1, WC-2, ..., WC-N in Figure 3.

[0107] At the same time, the weighting coefficients for the sinusoidal waveform of the signal input to integration circuit 252 and the weighting coefficients for the cosine waveform of the signal input to integration circuit 253 are different. It is not necessary for the two weighting coefficients to be different at all times within a mainframe period; they may be the same at some times.

[0108] The vertical scanning circuit 110 outputs the signal P_RES to the integration circuits 251, 252, and 253. When the signal P_RES reaches a high level, the count values ​​held in each of the integration circuits 251, 252, and 253 are reset to their initial values. The signal P_RES reaches a high level at the start of the mainframe period shown in Figure 3, and then goes to a low level. Due to this operation, the integration circuits 251, 252, and 253 are reset at the start of each mainframe period, and the counting operation for that mainframe period begins. Therefore, the integration circuits 251, 252, and 253 perform weighted integration, accumulating the weighted count values ​​from the start to the end of one mainframe period.

[0109] As described above, integration circuits 251, 252, and 253 each hold different count values. The count value held in integration circuit 251 corresponds to the number of photons incident on APD201. This is equivalent to weighting using a constant weighting coefficient. In other words, integration circuit 251 holds count values ​​that are weighted with a constant weighting coefficient WN-1, WN-2, ..., WN-N for the number of photons incident on APD201.

[0110] The integration circuit 252 stores count values ​​weighted to correspond to the weight coefficients WS-1, WS-2, ..., WS-N, which change with a sinusoidal waveform, for the number of photons incident on the APD201. The integration circuit 253 stores count values ​​weighted to correspond to the weight coefficients WC-1, WC-2, ..., WC-N, which change with a cosine waveform, for the number of photons incident on the APD201. Thus, integration circuits 251, 252, and 253 store count values ​​weighted differently to the output of the APD201. These count values ​​correspond to the pixel values ​​of the normal image IMN, the correlated image IMS, and the correlated image IMC, respectively, in Figure 3.

[0111] As described above, in this embodiment, by storing count values ​​with different weights in the integration circuits 251, 252, and 253, a signal generation operation equivalent to the dot product calculation process in the first to third embodiments is performed within the photoelectric converter 100. Therefore, in this embodiment, a photoelectric converter 100 capable of realizing a signal output for time correlation imaging is provided. Furthermore, similar to the first to third embodiments, the reliability calculation unit 314 calculates the reliability of the output signal of the photoelectric converter 100, thereby allowing for the appropriate acquisition of information regarding the calculation accuracy of the optical flow. Also, similar to the first to third embodiments, the optical flow calculation unit 313 can calculate the optical flow of the output signal of the photoelectric converter 100.

[0112] Furthermore, in this embodiment, the photoelectric converter 100 includes an APD201 with high detection accuracy for incident light, enabling time-correlated imaging of the output signal of the APD201. Therefore, time-correlated imaging can be suitably performed in low-luminance shooting scenes.

[0113] The sine wave generation unit 116a and the cosine wave generation unit 116b may be provided in correspondence to each of the multiple pixels 200. Alternatively, the multiple pixels 605 may be divided into multiple pixel blocks, in which case the sine wave generation unit 116a and the cosine wave generation unit 116b may be provided in correspondence to each of the multiple pixel blocks.

[0114] Furthermore, the configuration of pixel 200 may use a photodiode commonly used in CMOS image sensors instead of an avalanche photodiode. In such a pixel, the amount of light is converted into an analog electrical signal. In this case, a correlated image can be obtained by multiplying the analog reference signal by the analog electrical signal to weight it.

[0115] [Fifth Embodiment] In this embodiment, a modified example of the pixel configuration of the photoelectric converter 100 of the fourth embodiment will be described. In this embodiment, elements common to the first to fourth embodiments may be omitted or simplified in their description.

[0116] In the first to fourth embodiments, three values ​​are generated by weighting the output signal of a single photoelectric conversion unit by a constant value, a sine function, and a cosine function. In contrast, this embodiment describes a configuration in which one of the following weightings is applied to the output signal of a single pixel: weighting by a constant value, weighting by a sine function, and weighting by a cosine function.

[0117] Figures 13(a) and 13(b) are schematic diagrams illustrating the configuration of the photoelectric converter 100 according to this embodiment. Figure 13(a) shows the configuration of the sensor board 11 and circuit board 21 in the fourth embodiment as a comparative example to this embodiment. The sensor board 11 has a plurality of pixel circuits 101, and the circuit board 21 has a plurality of pixel signal processing units 103, each corresponding to one of the pixel circuits 101. As described in the fourth embodiment, one pixel signal processing unit 103 has three integration circuits 251, 252, and 253. As a result, three signals are obtained from one pixel, weighted by a constant value, weighted by a sine function, and weighted by a cosine function, and three types of signals used to calculate reliability can be obtained from one pixel. However, in this circuit configuration, it is necessary to provide three counter circuits for one pixel, which can increase the circuit size per pixel.

[0118] Figure 13(b) shows the configuration of the sensor board 11 and circuit board 21 in this embodiment. In this embodiment, one of the three integration circuits 251, 252, and 253 is provided in one pixel signal processing unit 103. That is, in this embodiment, three types of pixels—pixels weighted by a constant value, pixels weighted by a sine function, and pixels weighted by a cosine function—are arranged in a two-dimensional array. In this circuit configuration, one counter circuit is provided for each pixel, so the circuit size per pixel can be reduced compared to the comparative example in Figure 13(a).

[0119] Figures 14(a), 14(b), and 14(c) show the pixel configuration according to this embodiment. Each of Figures 14(a), 14(b), and 14(c) shows the photoelectric conversion unit 102 and the pixel signal processing unit 103 in Figure 13(b) in more detail. Hereinafter, the combination of the photoelectric conversion unit 102 and the pixel signal processing unit 103 including the integration circuit 251 may be referred to as pixel 200N. Similarly, the combination of the photoelectric conversion unit 102 and the pixel signal processing unit 103 including the integration circuit 252 may be referred to as pixel 200S, and the combination of the photoelectric conversion unit 102 and the pixel signal processing unit 103 including the integration circuit 253 may be referred to as pixel 200C.

[0120] Pixel 200N has a configuration similar to that of pixel 200 in Figure 12, but with the elements related to multiplexers 241 and 242 and integration circuits 252 and 253 omitted. Therefore, pixel 200N generates a count value that is weighted by a constant value.

[0121] Pixel 200S has a configuration similar to that of pixel 200 in Figure 12, but with the elements related to the multiplexer 242 and the integration circuits 251 and 253 omitted. Therefore, pixel 200S generates a count value that is weighted by a sine function.

[0122] Pixel 200C has a configuration similar to that of pixel 200 in Figure 12, but with the elements related to the multiplexer 241 and integration circuits 252 and 253 omitted. Therefore, pixel 200C generates a count value that is weighted by a cosine function.

[0123] Figures 15(a), 15(b), and 15(c) show the pixel arrangements according to this embodiment. Figures 15(a), 15(b), and 15(c) show a 4x4 array extracted from an array in which multiple pixels are repeatedly arranged. The pixel arrangement in Figure 15(a) has a repeating arrangement in which one unit consists of one pixel 200S, two pixels 200N, and one pixel 200C. Pixel 200S is located in the upper left of one unit, pixels 200N are located in the upper right and lower left of one unit, and pixel 200C is located in the lower right of one unit. The pixel arrangement in Figure 15(b) also has a repeating arrangement in which one unit consists of one pixel 200S, two pixels 200N, and one pixel 200C. Figure 15(b) is the same arrangement as Figure 15(a) but reversed horizontally. Thus, the order of the arrangement of pixels 200S, 200N, and 200C can be changed as appropriate.

[0124] In the configuration of this embodiment, the signal obtained from a single pixel is only one type of signal obtained by one type of weighting. For example, the pixel 200S does not output a signal weighted by a constant value or a signal weighted by a cosine function. Similarly, the pixel 200N does not output a signal weighted by a sine function or a signal weighted by a cosine function. Therefore, information from some pixels is lost when generating the normal image IMN and the correlated images IMS and IMC. Furthermore, since the dot product of pixels at the same position cannot be used when calculating the confidence score, the accuracy of the confidence score calculation may decrease. Therefore, a process may be performed to calculate the pixel value of the missing coordinates by interpolation using the values ​​of surrounding pixels. In this interpolation process, algorithms used for demosaicing of output signals from imaging devices in which red, green, and blue color filters are arranged in a Bayer array or the like can be applied. Examples of such algorithms include the nearest neighbor method and the bilinear method.

[0125] Furthermore, instead of, or in addition to, the interpolation process described above, control may be performed to dynamically change the value of the threshold TH described in the second and third embodiments according to the signal generated by the photoelectric converter 100 or the imaging environment. For example, consider a case where there are defective pixels (white spot defective pixels) that always output high pixel values ​​or defective pixels (black spot defective pixels) that always output low pixel values ​​due to pixel defects, etc. If pixel 200N is a defective pixel, the pixel value of pixel 200N will be significantly different from the pixel values ​​of the surrounding pixels 200S and 200C. In such a situation, a high degree of reliability can be calculated even if the subject is not moving. Therefore, if it is known from the output signal or imaging environment information that a defective pixel exists, the possibility of misjudgment can be reduced by setting the value of the threshold TH around the defective pixel higher than in other areas.

[0126] In the pixel arrays of Figures 15(a) and 15(b), the occupancy rate of pixel 200N is greater than that of both pixel 200S and pixel 200C. That is, in the entire array of multiple pixels, the number of first photoelectric conversion units used to generate the normal image IMN is greater than the number of second photoelectric conversion units used to generate the correlated image IMS (or correlated image IMC). In such a pixel array, the amount of information in the normal image IMN is greater than that of the correlated image IMS and the correlated image IMC. Therefore, the pixel arrays of Figures 15(a) and 15(b) are more effective in use cases where the image quality of the normal image IMN is prioritized over the image quality of the correlated image IMS and IMC. However, there are also cases where the image quality of the correlated image IMS and IMC is prioritized over the image quality of the normal image IMN, such as when the main application is the calculation of optical flow. In such cases, a pixel array may be adopted in which the occupancy ratio of pixel 200S and the occupancy ratio of pixel 200C are greater than the occupancy ratio of pixel 200N.

[0127] The pixel array in Figure 15(c) has a repeating arrangement in which one pixel 200S and three pixels 200N form one unit. Thus, a pixel array in which one of the pixels 200S and one of the pixels 200C are arranged may also be adopted. The pixel array in Figure 15(c) is more effective than the pixel arrays in Figures 15(a) and 15(b) in use cases where the image quality of the normal image IMN is even more important. In this case, if the amount of incident light during the mainframe period is constant, the dot product value P obtained by weighting by a constant value is obtained. nor And the inner product P obtained by weighting the sine function sin The following relationship (15) holds between them.

number

[0128] In this case, the confidence level R9 can be calculated using, for example, the following equation (16).

number

[0129] As described above, in this embodiment, the output signal of a single pixel is weighted by one of the following: a constant value, a sine function, or a cosine function. As a result, one counter circuit is provided for each pixel, which can reduce the circuit size per pixel compared to the pixel configuration of the fourth embodiment.

[0130] In the first to fifth embodiments, examples are given where there are three or two types of weights used to calculate the confidence level, but the number of weight types can be any number of two or more. Equations (6) and (15) can be generalized to the following equation (17).

number

[0131] [Sixth Embodiment] This embodiment describes modified configurations of the information processing device 300 according to the first to fifth embodiments. In this embodiment, elements common to the first to fifth embodiments may be omitted or simplified in their description.

[0132] In the first to third embodiments, a method was described in which a dot product value is generated by sequentially multiplying subframes generated in the photoelectric converter 100 by weight coefficients. In the fourth and fifth embodiments, a configuration was described in which sequentially weighted count values ​​are generated within the photoelectric converter 100. In contrast, this embodiment describes a configuration in which multiple subframes generated in the photoelectric converter 100 are acquired in advance, and then weighting calculations are performed.

[0133] Figure 16 is a functional block diagram of the information processing device 300 according to this embodiment. In addition to the configuration shown in Figure 2, the information processing device 300 has a data storage unit 315. The function of the data storage unit 315 is realized by the memory within the information processing device 300. The data storage unit 315 may be, for example, the RAM 302 or recording medium 304 shown in Figure 1. The data storage unit 315 has a storage capacity that can hold N subframes that constitute at least one mainframe.

[0134] When multiple subframes are generated by the photoelectric converter 100 through continuous imaging, the multiple subframes are stored in the data storage unit 315 via the data input unit 311. The data storage unit stores a number of subframes equal to or greater than the amount of data that constitutes one mainframe. Subsequently, the weighting unit 312 performs a process to calculate the dot product using the method described in the first to third embodiments to generate a normal image and a correlated image. After that, the optical flow calculation unit 313 and the confidence calculation unit 314 calculate the optical flow and confidence, respectively, using the methods described in the first to third embodiments.

[0135] In the information processing device 300 of this embodiment, it is not necessary to perform time-correlated imaging signal processing in real time during imaging, and signal processing can be performed at any timing after imaging. For example, even when the photoelectric converter 100 is a high-frame-rate imaging device such as a high-speed camera, sufficient time can be secured for signal processing.

[0136] The memory constituting the data storage unit 315 does not necessarily have to be built into the information processing device 300. For example, the data storage unit 315 may be provided within the photoelectric converter 100.

[0137] [Seventh Embodiment] The device according to the seventh embodiment will be described with reference to Figure 17. Figure 17 is a block diagram showing the schematic configuration of the device according to this embodiment.

[0138] Figure 17 is a schematic diagram showing an instrument EQP including a photoelectric converter APR. The photoelectric converter APR has the functions of the photoelectric converter 100 of the first to sixth embodiments. All or part of the photoelectric converter APR is a semiconductor device IC. The photoelectric converter APR in this example can be used as, for example. The semiconductor device IC has a pixel area PX in which pixel circuits PXC including a photoelectric conversion unit are arranged in a matrix. The semiconductor device IC may have a peripheral area PR around the pixel area PX. Circuits other than pixel circuits can be arranged in the peripheral area PR.

[0139] The photoelectric converter APR may have a stacked structure (chip stacking structure) comprising a first semiconductor chip provided with multiple photoelectric conversion units and a second semiconductor chip provided with peripheral circuits. The peripheral circuits on the second semiconductor chip can each be a column circuit corresponding to a pixel row of the first semiconductor chip. Alternatively, the peripheral circuits on the second semiconductor chip can each be a matrix circuit corresponding to a pixel or pixel block of the first semiconductor chip. For connecting the first and second semiconductor chips, through-swivel electrodes (TSVs), direct bonding of conductors such as copper for inter-chip wiring, connection by microbumps between chips, or connection by wire bonding can be employed.

[0140] The photoelectric converter APR may include a semiconductor device IC as well as a package PKG that houses the semiconductor device IC. The package PKG may include a substrate on which the semiconductor device IC is fixed, a lid made of glass or the like that faces the semiconductor device IC, and connecting members such as bonding wires and bumps that connect terminals provided on the substrate to terminals provided on the semiconductor device IC.

[0141] The EQP device may further comprise at least one of the following: an optical device OPT, a control unit CTRL, a processing unit PRCS, a display unit DSPL, a memory device MMRY, and a mechanical device MCHN. The optical device OPT corresponds to the photoelectric converter APR as a photoelectric converter, and is, for example, a lens, shutter, or mirror. The control unit CTRL controls the photoelectric converter APR and is, for example, a semiconductor device such as an ASIC.

[0142] The processing unit PRCS processes the signals output from the photoelectric converter APR and constitutes either the AFE (analog front end) or DFE (digital front end). The processing unit PRCS is a semiconductor device such as a CPU (central processing unit) or ASIC (application-specific integrated circuit). The display device DSPL is an EL display device, liquid crystal display device, etc., that displays the information (image) obtained from the photoelectric converter APR. The memory device MMRY is a magnetic device, semiconductor device, etc., that stores the information (image) obtained from the photoelectric converter APR. The memory device MMRY is a volatile memory such as SRAM or DRAM, or a non-volatile memory such as flash memory or hard disk drive.

[0143] Furthermore, the processing unit PRCS may have the functions of the information processing unit 300 of the first to sixth embodiments. That is, the processing unit PRCS may acquire optical flow using the signals output by the photoelectric converter 100 of the first to sixth embodiments. For example, the processing unit PRCS may generate a correlation image weighted based on a sine function, a correlation image weighted based on a cosine function, and a normal image, and acquire optical flow from these three images.

[0144] The mechanical device MCHN has moving parts or propulsion parts such as motors and engines. The equipment EQP displays the signals output from the photoelectric converter APR on the display device DSPL, or transmits them to the outside using a communication device (not shown) provided by the equipment EQP. For this purpose, it is preferable that the equipment EQP further includes a memory device MMRY and a processing device PRCS, separate from the memory circuit and arithmetic circuit of the photoelectric converter APR. The mechanical device MCHN may be controlled based on the signals output from the photoelectric converter APR.

[0145] The EQP (Equipment Equipped Device) shown in Figure 17 can be electronic devices such as information terminals with imaging capabilities (e.g., smartphones and wearable devices), cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). In a camera, the mechanical device MCHN can drive components of the optical device OPT for zooming, focusing, and shutter operation. The EQP can also be transportation equipment (mobile devices) such as vehicles, ships, drones, and aircraft. Furthermore, the EQP can be medical equipment such as endoscopes and CT scanners. Additionally, the EQP can be measuring instruments such as distance sensors, analytical instruments such as electron microscopes, office equipment such as photocopiers, and industrial equipment such as robots.

[0146] The mechanical device MCHN in transport equipment can be used as a mobile device. The device EQP as transport equipment is suitable for transporting the photoelectric converter APR, assisting and / or automating driving (operation) through its imaging function, etc. The processing device PRCS for assisting and / or automating driving (operation) can perform processing to operate the mechanical device MCHN as a mobile device based on information obtained from the photoelectric converter APR.

[0147] According to the first to sixth embodiments, good signal acquisition is possible. Therefore, the photoelectric converter APR or processing unit PRCS according to the first to sixth embodiments can provide high value to its designers, manufacturers, distributors, buyers, and / or users. Therefore, by installing the photoelectric converter APR or processing unit PRCS in the EQP, the value of the EQP can also be increased. Thus, when manufacturing and selling the EQP, deciding to install the photoelectric converter APR or processing unit PRCS of this embodiment in the EQP is advantageous in increasing the value of the EQP. Increasing value here includes at least one of the following: addition of functions, improvement of performance, improvement of characteristics, improvement of reliability, improvement of manufacturing yield, reduction of environmental impact, cost reduction, miniaturization, and weight reduction.

[0148] For example, by installing the APR (Photoelectric Converter) in transportation equipment, superior performance can be obtained when photographing the outside of the transportation equipment or measuring the external environment. Therefore, when manufacturing and selling transportation equipment, deciding to install the APR (Photoelectric Converter) according to this embodiment in the transportation equipment is advantageous in improving the performance of the transportation equipment itself. In particular, the APR (Photoelectric Converter) is suitable for transportation equipment that uses information obtained from the APR to provide driving assistance and / or automatic driving.

[0149] [Eighth Embodiment] Figures 18(a) and 18(b) are block diagrams of the equipment related to the in-vehicle camera in this embodiment. Figures 18(a) and 18(b) show an example of applying the photoelectric converter to a moving object such as a vehicle. Equipment 80 includes an imaging device 800 (an example of a photoelectric converter) and a signal processing device (processing device) that processes signals from the imaging device 800. Equipment 80 includes an image processing unit 801 that performs image processing on a plurality of image data acquired by the imaging device 800, and a parallax calculation unit 802 that calculates parallax (phase difference of parallax images) from a plurality of image data acquired by equipment 80.

[0150] Here, the device 80 may include an optical system (not shown) that guides light to the imaging device 800. The optical system may include, for example, lenses, shutters, and mirrors. In addition, multiple photoelectric conversion units that are substantially conjugate to the pupil of the optical system may be arranged in pixels of the imaging device 800. For example, the multiple photoelectric conversion units may be arranged corresponding to one microlens. The multiple photoelectric conversion units receive light beams that have passed through different positions in the pupil of the optical system. As a result, the imaging device 800 outputs multiple image data corresponding to the light beams that have passed through different positions in the pupil of the optical system. The disparity calculation unit 802 may then calculate the disparity using the output multiple image data.

[0151] Furthermore, the device 80 includes a distance measurement unit 803 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax calculation unit 802 and the distance measurement unit 803 are examples of distance information acquisition means that acquire distance information to an object. That is, distance information is information related to parallax, defocus amount, distance to an object, etc. The collision determination unit 804 may use any of this distance information to determine the possibility of collision. Note that the distance information may be acquired using ToF (Time of Flight) technology. The distance information acquisition means may be implemented by specially designed hardware or by a software module. It may also be implemented by FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0152] Device 80 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. Device 80 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate braking force on the vehicle based on the collision determination result of the collision determination unit 804. Furthermore, device 80 is connected to a warning device 830 that issues a warning to the driver based on the collision determination result of the collision determination unit 804. For example, if the collision determination result of the collision determination unit 804 indicates a high probability of collision, the control ECU 820 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, or suppressing engine output. The warning device 830 warns the user by sounding an alarm, displaying warning information on a screen such as a car navigation system, or vibrating the seatbelt or steering wheel. As described above, device 80 functions as a control means that controls the actions that control the vehicle.

[0153] In this embodiment, the equipment 80 images the area around the vehicle, for example, in front of or behind it. Figure 18(b) shows the equipment when imaging the area in front of the vehicle (imaging range 850). The vehicle information acquisition device 810, acting as an imaging control means, sends instructions to the equipment 80 or imaging device 800 to perform the imaging operation. This configuration allows for further improvement of the accuracy of distance measurement.

[0154] The above example described controlling a vehicle to avoid collisions with other vehicles, but it can also be applied to control systems that automatically follow other vehicles, or control systems that automatically stay within their lane. Furthermore, the equipment is not limited to vehicles such as automobiles, but can be applied to mobile objects (mobile devices) such as ships, aircraft, satellites, industrial robots, and consumer robots. In addition, it can be applied not only to mobile objects, but also to a wide range of devices that utilize object recognition or biometric recognition, such as intelligent transportation systems (ITS) and surveillance systems.

[0155] [Modified Embodiment] The present invention is not limited to the embodiments described above and can be modified in various ways. For example, an example in which a part of the configuration of one embodiment is added to another embodiment, or an example in which a part of the configuration of one embodiment is replaced with a part of the configuration of another embodiment, is also an embodiment of the present invention.

[0156] The embodiments described above can be modified as appropriate without departing from the technical concept. Furthermore, the disclosures in this specification include not only what is described herein, but also all matters that can be understood from this specification and the drawings attached thereto. In addition, the disclosures in this specification include the complement of the concepts described herein. That is, if this specification states, for example, "A is greater than B," then even if the statement "A is not greater than B" is omitted, this specification can be said to disclose that "A is not greater than B." This is because the statement "A is greater than B" presupposes that the case where "A is not greater than B" is being considered.

[0157] The disclosures in this specification include the following components: (Composition 1) A weighting unit that obtains a first dot product of time-series pixel data containing multiple time-series pixel values ​​as elements and a first coefficient sequence containing multiple weight coefficients corresponding to each of the multiple pixel values ​​as elements, and obtains a second dot product of the time-series pixel data and a second coefficient sequence different from the first coefficient sequence, which contains multiple weight coefficients corresponding to each of the multiple pixel values ​​as elements, A reliability calculation unit calculates the reliability of the accuracy of calculating the optical flow from the plurality of pixel values ​​based on the first dot product and the second dot product, An information processing system characterized by having the following features. (Configuration 2) The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The information processing system according to configuration 1, characterized by the features described above. (Composition 3) The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. An information processing system according to configuration 1 or 2, characterized by the above. (Composition 4) The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The reliability calculation unit calculates the reliability such that the reliability decreases as the ratio of the first dot product to the second dot product approaches a predetermined value. The information processing system according to configuration 1, characterized by the features described above. (Composition 5) The weighting unit obtains a third inner product of the time-series pixel data and a third coefficient sequence that includes a plurality of weight coefficients corresponding to each of the plurality of pixel values ​​as elements, and is different from both the first coefficient sequence and the second coefficient sequence. The reliability calculation unit further calculates the reliability based on the third dot product. The information processing system according to configuration 1, characterized by the features described above. (Composition 6) The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The information processing system according to configuration 5, characterized by the features described herein. (Composition 7) The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value, with a different phase from the weight coefficients of the first coefficient sequence. The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes terms including the first dot product, the second dot product, and the third dot product. The information processing system according to configuration 5, characterized by the features described herein. (Composition 8) The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes a term obtained by subtracting the second dot product from the sum of the first dot product and the third dot product. The information processing system according to configuration 7, characterized by the features described above. (Composition 9) The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value, with a different phase from the weight coefficients of the first coefficient sequence. The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes the product of a term including the first dot product and the second dot product and a term including the third dot product and the second dot product. The information processing system according to configuration 5, characterized by the features described herein. (Composition 10) The confidence calculation unit determines whether the confidence level is less than a threshold and outputs the determination result. An information processing system according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The reliability calculation unit dynamically determines the threshold based on information contained in the time-series pixel data or information about the imaging environment in which the time-series pixel data is generated. The information processing system according to configuration 10, characterized by the above. (Composition 12) The confidence calculation unit calculates the confidence level based on a value obtained by a normalized formula obtained by dividing the terms including the first dot product and the second dot product by the term including the second dot product. An information processing system according to any one of configurations 1 to 11, characterized by the features described herein. (Composition 13) The confidence calculation unit calculates a smaller confidence value the smaller the time change of the time-series pixel data. An information processing system according to any one of configurations 1 to 12, characterized by the above. (Composition 14) It further includes a photoelectric conversion unit that generates a signal corresponding to the incident light, The weighting unit obtains the first dot product by performing weighted integration based on a plurality of signals input in time series and a reference signal indicating the plurality of weight coefficients. An information processing system according to any one of configurations 1 to 13, characterized by the features described herein. (Composition 15) The photoelectric conversion unit includes an avalanche photodiode. The weighting unit obtains the first inner product by counting the number of pulses of the reference signal when a photon is incident on the avalanche photodiode. The information processing system according to configuration 14, characterized by the features described above. (Composition 16) Having multiple photoelectric conversion units, The plurality of photoelectric conversion units include a first photoelectric conversion unit that generates a signal used to generate the first dot product and does not generate a signal used to generate the second dot product, and a second photoelectric conversion unit that generates a signal used to generate the second dot product and does not generate a signal used to generate the first dot product. The information processing system according to configuration 14 or 15, characterized by the above. (Composition 17) The plurality of photoelectric conversion units form an array in which the first photoelectric conversion unit and the second photoelectric conversion unit are repeatedly arranged. The information processing system according to configuration 16, characterized by the features described above. (Composition 18) In the array, the number of second photoelectric conversion units is greater than the number of first photoelectric conversion units. The information processing system according to configuration 17, characterized by the features described above. (Composition 19) The system further includes a memory for storing the aforementioned time-series pixel data, The weighting unit performs the process of obtaining the first and second dot products after the amount of time-series pixel data exceeding the amount of data constituting a single image has been stored in the memory and the time-series pixel data has been read from the memory. An information processing system according to any one of configurations 1 to 18, characterized by the above. (Composition 20) The system further includes an optical flow calculation unit that calculates the optical flow based on the first dot product and the second dot product. If the confidence level is above the threshold, the optical flow calculation unit calculates the optical flow. If the confidence level is lower than the threshold, the optical flow calculation unit does not calculate the optical flow. An information processing system according to any one of configurations 1 to 19, characterized by the features described herein. (Composition 21) The information processing system described in any one of items 1 to 20, Optical device corresponding to the aforementioned information processing system, A control device for controlling the aforementioned information processing system, A processing unit that processes signals output from the aforementioned information processing system, A display device that displays the information obtained by the aforementioned information processing system. A storage device for storing information obtained by the information processing system, and The apparatus is characterized by comprising at least one of the following: a mechanical device that operates based on information obtained from the aforementioned information processing system. (Composition 22) The processing device acquires distance information from the photoelectric conversion element used to acquire the pixel values ​​to the object. The apparatus according to configuration 21, characterized by the features described above.

[0158] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0159] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of Symbols]

[0160] 300 Information Processing Devices 311 Data Input Section 312 Weighting section 313 Optical Flow Calculation Unit 314 Confidence Calculation Unit

Claims

1. A weighting unit that obtains a first dot product of time-series pixel data, which includes multiple pixel values ​​in a time series as elements, and a first coefficient sequence, which includes multiple weight coefficients corresponding to each of the multiple pixel values ​​as elements, and obtains a second dot product of the time-series pixel data and a second coefficient sequence, which includes multiple weight coefficients corresponding to each of the multiple pixel values ​​as elements, and is different from the first coefficient sequence, A reliability calculation unit calculates the reliability of the accuracy of calculating the optical flow from the plurality of pixel values ​​based on the first dot product and the second dot product, An information processing system characterized by having the following features.

2. The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The information processing system according to feature 1.

3. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The information processing system according to feature 1.

4. The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The reliability calculation unit calculates the reliability such that the reliability decreases as the ratio of the first dot product to the second dot product approaches a predetermined value. The information processing system according to feature 1.

5. The weighting unit obtains a third inner product of the time-series pixel data and a third coefficient sequence that includes a plurality of weight coefficients corresponding to each of the plurality of pixel values ​​as elements, and is different from both the first coefficient sequence and the second coefficient sequence. The reliability calculation unit further calculates the reliability based on the third dot product. The information processing system according to feature 1.

6. The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The information processing system according to feature 5.

7. The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value, with a different phase from the weight coefficients of the first coefficient sequence. The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes terms including the first dot product, the second dot product, and the third dot product. The information processing system according to feature 5.

8. The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes a term that subtracts the second dot product from the sum of the first dot product and the third dot product. The information processing system according to feature 7.

9. The weight coefficients of the first coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the second coefficient sequence are constant with respect to the acquisition time of the corresponding pixel value. The weight coefficients of the third coefficient sequence change periodically with respect to the acquisition time of the corresponding pixel value, with a different phase from the weight coefficients of the first coefficient sequence. The reliability calculation unit calculates the reliability based on a value obtained by an expression that includes the product of a term including the first dot product and the second dot product and a term including the third dot product and the second dot product. The information processing system according to feature 5.

10. The confidence calculation unit determines whether the confidence level is less than a threshold and outputs the determination result. The information processing system according to feature 1.

11. The reliability calculation unit dynamically determines the threshold based on information contained in the time-series pixel data or information about the imaging environment in which the time-series pixel data is generated. The information processing system according to feature 10.

12. The confidence calculation unit calculates the confidence level based on a value obtained by a normalized formula obtained by dividing the terms including the first dot product and the second dot product by the term including the second dot product. The information processing system according to feature 1.

13. The confidence calculation unit calculates a smaller confidence value the smaller the time change of the time-series pixel data. The information processing system according to feature 1.

14. It further includes a photoelectric conversion unit that generates a signal corresponding to the incident light, The weighting unit obtains the first dot product by performing weighted integration based on a plurality of signals input in time series and a reference signal indicating the plurality of weight coefficients. The information processing system according to feature 1.

15. The photoelectric conversion unit includes an avalanche photodiode. The weighting unit obtains the first inner product by counting the number of pulses of the reference signal when a photon is incident on the avalanche photodiode. The information processing system according to feature 14.

16. Having multiple photoelectric conversion units, The plurality of photoelectric conversion units include a first photoelectric conversion unit that generates a signal used to generate the first dot product and does not generate a signal used to generate the second dot product, and a second photoelectric conversion unit that generates a signal used to generate the second dot product and does not generate a signal used to generate the first dot product. The information processing system according to feature 14.

17. The plurality of photoelectric conversion units form an array in which the first photoelectric conversion unit and the second photoelectric conversion unit are repeatedly arranged. The information processing system according to feature 16.

18. In the array, the number of second photoelectric conversion units is greater than the number of first photoelectric conversion units. The information processing system according to feature 17.

19. The system further includes a memory for storing the aforementioned time-series pixel data, The weighting unit performs the process of obtaining the first and second dot products after the amount of time-series pixel data exceeding the amount of data constituting a single image has been stored in the memory and the time-series pixel data has been read from the memory. The information processing system according to feature 1.

20. The system further includes an optical flow calculation unit that calculates the optical flow based on the first dot product and the second dot product. If the confidence level is above the threshold, the optical flow calculation unit calculates the optical flow. If the confidence level is lower than the threshold, the optical flow calculation unit does not calculate the optical flow. The information processing system according to feature 1.

21. An information processing system according to any one of claims 1 to 20, Optical device corresponding to the aforementioned information processing system, A control device for controlling the aforementioned information processing system, A processing unit that processes signals output from the aforementioned information processing system, A display device that displays the information obtained by the aforementioned information processing system. A storage device for storing information obtained by the information processing system, and The apparatus is characterized by comprising at least one of the following: a mechanical device that operates based on information obtained from the aforementioned information processing system.

22. The processing device acquires distance information from the photoelectric conversion element used to acquire the pixel values ​​to the object. The apparatus according to feature 21.

Images