Photoelectric conversion device and device

Abstract

To enable an execution of an optimal control when outputting a gradation signal and an event signal.SOLUTION: A photoelectric conversion device comprises: a pixel array part in which a plurality of first pixels that can generate a gradation signal expressing a light amount of an incident light and a plurality of second pixels that can generate an event signal are disposed in a two-dimensional shape; a reading part that reads the gradation signal and the event signal from the pixel array part; and a control part that changes an output data rate of the gradation signal and an output data rate of the event signal. In the photoelectric conversion device, a first pattern where the even signal is output at a second output data rate and the gradation signal is output at a first output data rate and a second pattern where the gradation signal is output at a third output data rate that is different from the first output data rate, and the event signal is output at a fourth output data rate that is different from the second output data rate are switched.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device and an instrument. [Background technology]

[0002] With the recent spread of IoT, AI, and autonomous driving, there is a demand for image sensors that consume less power and are faster than ever before. One of these is an event-based sensor (also called a dynamic vision sensor). Patent Document 1 discloses a solid-state imaging element that can generate a grayscale signal and an event signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-140650 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a photoelectric conversion device capable of generating a grayscale signal and an event signal, the optimization of operation during signal output has not been sufficiently considered.

[0005] An object of the present invention is to provide a photoelectric conversion device capable of optimally controlling the output of a grayscale signal and an event signal. [Means for solving the problem]

[0006] According to one disclosure of the present specification, there is provided a photoelectric conversion device comprising: a pixel array section in which first pixels capable of generating a gradation signal representing the amount of incident light and second pixels capable of generating an event signal are arranged in a two-dimensional array; a readout section that reads out the gradation signals and the event signals from the pixel array section; and a control section that changes an output data rate of the gradation signals and an output data rate of the event signals, wherein the photoelectric conversion device is characterized in that it switches between a first pattern in which the gradation signals are output at a first output data rate and the event signals are output at a second output data rate, and a second pattern in which the gradation signals are output at a third output data rate different from the first output data rate and the event signals are output at a fourth output data rate different from the second output data rate. Effect of the Invention

[0007] According to the present invention, it is possible to perform optimal control when outputting a grayscale signal and an event signal. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a photoelectric conversion device according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a photoelectric conversion device according to a first embodiment. [Diagram 3] FIG. 1 is a block diagram illustrating a photoelectric conversion device according to a first embodiment. [Figure 4] FIG. 1 is a block diagram illustrating a photoelectric conversion device according to a first embodiment. [Diagram 5] FIG. 1 is a diagram for explaining output data rate control of a photoelectric conversion device according to a first embodiment. [Figure 6] FIG. 11 is a block diagram illustrating a photoelectric conversion device according to a second embodiment. [Figure 7] FIG. 13 is a diagram for explaining output data rate control of a photoelectric conversion device according to a second embodiment. [Figure 8] FIG. 13 is a schematic diagram illustrating a device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment will be described below with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these features are necessarily essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicated descriptions are omitted. Furthermore, in each embodiment described below, a sensor for imaging will be mainly described as an example of a photoelectric conversion device. However, each embodiment is not limited to a sensor for imaging, and can be applied to other examples of photoelectric conversion devices. For example, there are an imaging device, a distance measuring device (a device for distance measurement using focus detection or TOF (Time Of Flight), a photometric device (a device for measuring the amount of incident light, etc.), and the like.

[0010] First Embodiment A photoelectric conversion device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 5. FIG.

[0011] FIG. 1 is an example of a block diagram of a photoelectric conversion device according to this embodiment.

[0012] As shown in FIG. 1, the photoelectric conversion device 100 includes a pixel array section 101, a readout section 102, and a control section 103. In the pixel array section 101, a plurality of pixels are arranged two-dimensionally to form a plurality of rows and a plurality of columns. The plurality of pixels have a photoelectric conversion section that generates and accumulates a signal charge based on incident light, and each of the pixels outputs a signal charge according to the incident light. The plurality of pixels also includes gradation pixels 110 (first pixels) capable of generating a gradation signal representing the amount of incident light, and event pixels 120 (second pixels) capable of generating an event signal. For example, the pixel array section 101 includes a mixture of the gradation pixels 110 and the event pixels 120 arranged in a two-dimensional array. The event signal is a signal generated when an event in which the amount of light changes suddenly is detected in the event pixel 120. This event may be, for example, a local change in the amount of light caused by a change in the position of a subject within a shooting range.

[0013] A microlens and a color filter may be disposed on the pixel. The R pixel 110a, the G pixel 110b, and the B pixel 110c disposed in the pixel array unit 101 respectively indicate gradation pixels 110 that receive light through red, green, and blue color filters. The types of color filters used in the gradation pixels 110 are merely an example, and color filters of other colors may be used. For example, complementary color filters may be used, and color filters of cyan, magenta, and yellow may be used. Also, a clear pixel that does not use a color filter may be used. The E pixel 120 indicates the event pixel 120. The event pixel 120 may be configured without a color filter in order to prioritize sensitivity in a dark place. However, the event pixel 120 may also be configured to receive light through a color filter of any color in order to obtain color information. The arrangement of the gradation pixels 110 and the event pixels 120 shown in the pixel array unit 101 is merely an example. For example, the event pixels 120 are arranged at a ratio of 1 in 4 pixels, but may be arranged at any other ratio, such as 1 in 16 pixels. The pixels may also be arranged in any other arbitrary arrangement. Furthermore, each of the multiple pixels arranged in the pixel array unit 101 may be capable of generating a grayscale signal representing the amount of incident light and an event signal representing a change in the amount of incident light.

[0014] The readout unit 102 sequentially reads out each of the gradation pixels 110 and the event pixels 120 arranged in the pixel array unit 101, and outputs the signals to the outside of the photoelectric conversion device 100. The readout unit 102 may be capable of reading out signals output from a plurality of pixels, for example, in units of one row, in parallel. For the event pixels 120, for example, rows in which no event has occurred may be skipped and not read out. In this case, a row address may be added to each row so that the position of the pixel in which the event has occurred can be identified. Similarly, it is also possible to skip areas in the column direction in which no event has occurred, and to further add a column address to output the event signal.

[0015] The control unit 103 can control the read cycle and the binning process method of the gradation pixels 110 and the event pixels 120 by controlling the pixel array unit 101. Also, the control unit 103 can control the read cycle and the binning process method of the gradation pixels 110 and the event pixels 120 by controlling the read unit 102. Here, the control unit 103 can reduce the time resolution of at least one of the corresponding gradation signals and event signals by lowering the read cycle. That is, the control unit 103 can variably control the time resolution of the output gradation signals and event signals by controlling the read cycle. Also, the control unit 103 can reduce the spatial resolution of at least one of the corresponding gradation signals and event signals by increasing the number of pixels on which the control unit 103 performs the binning process. That is, the control unit 103 can variably control the spatial resolution of the output gradation signals and event signals by controlling the binning process method. As described above, the control unit 103 can change the output data rate of the gradation signals and the output data rate of the event signals, for example, by changing at least one of the spatial resolution and the time resolution of the signals to be output. Here, the output data rate of the grayscale signal is the data rate at which the grayscale signal is output from the pixel array unit 101 to the outside of the photoelectric conversion device 100 via the readout unit 102. Also, the output data rate of the event signal is the data rate at which the event signal is output from the pixel array unit 101 to the outside of the photoelectric conversion device 100 via the readout unit 102. Note that, when there are multiple lanes capable of outputting signals at the boundary between the photoelectric conversion device 100 and the outside of the photoelectric conversion device 100, the output data rate represents the maximum data rate at which signals are output using the multiple lanes. Note that the control unit 103 controls the pixel array unit 101 or the readout unit 102 to control the binning processing method, which will be described in detail later.

[0016] Note that the control unit 103 controls the pixel array unit 101 to control the signal readout period, which means changing the readout period of at least one of the grayscale signal and the event signal from the pixel array unit 101 to the readout unit 102.

[0017] In addition, the control unit 103 controls the readout unit 102 to control the readout period of the signal means changing the readout period of at least one of the grayscale signal and the event signal from the readout unit 102 to the outside of the photoelectric conversion device 100. For example, after reading out the signal from the pixel array unit 101 to the readout unit 102 at a certain period, the readout period can be changed by averaging or thinning out the signal amount in the time direction in the readout unit 102. Here, averaging the signal amount in the time direction means performing an averaging process on signals output from the same pixel at different timings to reduce the period for outputting the signal. In addition, the above process is not limited to signals output from the same pixel at different timings, but may be performed on signals output from the same region including multiple pixels at different timings. In addition, when averaging the event signal in the time direction is performed, regarding the readout result of the event signal of multiple periods, a pixel in which an event has occurred once or more is regarded as having one event occurrence, and a pixel in which no event has occurred is regarded as having no event occurrence. After that, the multiple periods are converted into an event signal newly regarded as one period.

[0018] Furthermore, the pixel array unit 101 can be divided into a plurality of pixel regions, each including at least one gradation pixel and at least one event pixel. The control unit 103 can control the binning method and readout cycle of the gradation pixels 110 and the event pixels 120 for each of the divided pixel regions. The binning method and readout cycle for each of the divided pixel regions can be set in advance for the control unit 103 from outside the photoelectric conversion device 100, for example, as an output data rate control parameter.

[0019] The control unit 103 may include a vertical scanning circuit, which controls the gradation pixels 110 and the event pixels 120 for each pixel row, thereby performing vertical scanning to read out signals from the corresponding pixels.

[0020] FIG. 2 is an example of a block diagram of the event pixel 120 included in the photoelectric conversion device 100 according to this embodiment.

[0021] As shown in FIG. 2, one event pixel 120 has a photodiode 200, a logarithmic I / V conversion unit 201, a subtraction unit 202, and a comparison unit 203.

[0022] In the event pixel 120, the photodiode 200 generates a photocurrent Ip according to the amount of incident light. For example, the logarithmic I / V converter 201 converts the photocurrent IP into a potential and obtains a logarithmic potential V1 by conversion using a logarithmic function. The output V1 of the logarithmic I / V converter 201 does not necessarily need to be a logarithmic potential, and may be any other value that changes according to the amount of light, such as a potential that changes linearly according to the amount of incident light.

[0023] The subtraction unit 202 calculates the difference Vd between the logarithmic potential V1 and the reference potential. The comparison unit 203 compares the difference Vd calculated by the subtraction unit with a predetermined threshold. The comparison unit 203 uses two thresholds, a positive threshold T1 and a negative threshold T2, and when the difference Vd exceeds the positive threshold or falls below the negative threshold, an event signal having a value other than 0 is generated and detected as an event. Here, the event signal E of the pixel P(x, y) at time t can be expressed using the following formula (1).

[0024]

number

[0025] Here, the time resolution of the event detection of the event pixel 120 at time t is, for example, 1 μs. When the event signal E has a value of 1, it indicates that the event pixel 120 has changed to become brighter, i.e., an ON event has occurred. When the event signal E has a value of −1, it indicates that the event pixel 120 has changed to become darker, i.e., an OFF event has occurred.

[0026] The above formula (1) is an example of an expression of the event signal E, and other expressions may be used, such as assigning different numerical values ​​to each state of an on event, an off event, and no event, or treating an on event and an off event as one type of event signal without distinguishing between them. The event signal E is a signal that is generated independently (asynchronously) of the frame synchronization signal used in a normal image sensor, and the use of the event signal E enables event detection with high time resolution. The detected event signal E is sent to the readout unit 102. The reference potential is updated using a reset signal as a trigger, and the output potential of the logarithmic I / V conversion unit 201 at the time of input of the reset signal becomes the new reference potential.

[0027] FIG. 3 is an example of a block diagram of the gradation pixel 110 included in the photoelectric conversion device 100 according to this embodiment.

[0028] As shown in FIG. 3, one gradation pixel 110 has a photodiode 300, a charge storage section 301, a voltage conversion section 302, and an AD (Analog Digital) conversion section 303. The photodiode 300 generates a photocurrent according to the amount of incident light, and the charge is stored in the charge storage section 301. The voltage conversion section 302 converts the amount of charge stored in the charge storage section 301 for a predetermined time into a voltage and outputs it. The AD conversion section 303 converts the output potential of the voltage conversion section 302 into a digital value and outputs it as a gradation signal. All the gradation pixels 110 are read out at regular time intervals by a frame synchronization signal, and the time resolution is, for example, 16.7 ms (60 fps). The amount of charge in the charge storage section 301 is cleared by a reset signal. In FIG. 3, the gradation pixel 110 is configured to include the AD conversion section 303, but the AD conversion section 303 may be arranged outside the gradation pixel 110. When the AD conversion unit 303 is disposed outside the gradation pixels 110, for example, the readout unit 102 may include the AD conversion unit 303, and one AD conversion unit 303 may be shared by a plurality of gradation pixels 110.

[0029] The configuration of the gradation pixel and the event pixel shown in this embodiment is an example. Either or both of the gradation pixel and the event pixel may be acquired using any other configuration. For example, a photon counting method may be used to obtain a gradation signal from a photon counter value and an event signal from a change in the photon counter value. For example, the photodiode is an avalanche photodiode that operates in a Geiger mode. A waveform shaping circuit that receives the output of the avalanche photodiode and generates a pulse wave is provided, and a counter (photon counter) that counts the pulse wave is provided. The event signal may be obtained using the photon counter value, which is the value of this counter. Also, the counter may not be used, and in this case, the event signal may be generated using the pulse wave of the waveform shaping circuit. In this case, the event signal may be generated using multiple pulse waves, and for example, an event signal indicating that an event has occurred may be generated when multiple pulse waves are generated at intervals shorter than a predetermined interval.

[0030] FIG. 4 is an example of a block diagram of a binning processing circuit 410 included in the photoelectric conversion device 100 according to this embodiment. The binning processing circuit 410 shown in FIG. 4 is included in, for example, the pixel array unit 101. For the sake of simplicity, FIG. 4 shows the binning processing circuit 410 when binning two gradation pixels 110 in the vertical direction. However, the binning processing circuit 410 shown in FIG. 4 can be extended to a case where an arbitrary number of pixels in an arbitrary direction are binned. In addition, the binning processing circuit 410 shown in FIG. 4 can be extended to a binning process of not only the gradation pixels 110 but also the event pixels 120 by partially modifying the arithmetic circuit included in the binning processing circuit 410.

[0031] Binning is a process for lowering the spatial resolution of an output signal. For example, binning is a process for averaging gradation signals of adjacent gradation pixels 110 of the same color to combine them into one value. This makes it possible to reduce the resolution of the gradation signal and lower the output data rate. Furthermore, adding multiple gradation signals can reduce the effect of noise and improve the signal-to-noise ratio. Similarly, binning of event pixels is a process for combining event signals of adjacent event pixels into one event signal. For example, if at least one on-event is detected in adjacent event pixels, the event signal E is set to 1, and if at least one off-event is detected, the event signal E is set to -1. If one or more on-events and one or more off-events are detected, the one with the greater number of detections is used, and if the number of detections is the same, one may be fixed and prioritized. In this way, the events can be combined into one event value. The binning method shown here is an example, and any other processing method can be used as long as it combines the outputs of multiple pixels into one output.

[0032] As shown in FIG. 4, the binning circuit 410 includes an adder circuit 404, a bit shifter 405, a selector 406, a selector 407, and a selector 408. The signal bin_en is a signal for switching whether or not the binning process is performed, and indicates that vertical two-pixel binning is performed at a high level (1) and that binning is not performed at a low level (0). The signal bin_en is, for example, a signal output from a 1-bit register arranged for each area to be binned in the pixel array unit 101, and the register value can be rewritten by the control unit 103. The signal row_sel is a selection signal for a row to be read out, and is output from the control unit 103 to the binning circuit 410. The gradation pixel 400, the event pixel 401, the gradation pixel 402, and the event pixel 403 indicate four rows of pixels in a certain column, and the gradation pixel 400 and the gradation pixel 402 receive light through a red color filter.

[0033] The selector 406 receives the signal bin_en and the gradation signal of the gradation pixel 402, and selects and outputs the gradation signal of the gradation pixel 402 or a low level (0) according to the input signal bin_en. The adder circuit 404 adds the gradation signal output from the gradation pixel 400 and the output value of the selector 406. Therefore, when the signal bin_en is at a low level (0), the output value of the adder circuit 404 becomes the gradation signal of the gradation pixel 400. Also, when the signal bin_en is at a high level (1), the output value of the adder circuit 404 becomes the sum of the gradation signal of the gradation pixel 400 and the gradation signal of the gradation pixel 402. The bit shifter 405 is a circuit that shifts the multi-bit value input from the adder circuit 404 one bit to the right. As a result, the least significant digit (first digit) is removed, and the bits from the second digit onwards are shifted one bit at a time to the lower side. Therefore, the bit shifter 405 performs a process equivalent to an operation of dividing the input value by 2 (with the remainder discarded), and outputs a signal obtained by dividing the input value of the adder circuit 404 by 2. The output value of the adder circuit 404 and the output value of the bit shifter 405 are input to the selector 407, which selects and outputs the output value of the adder circuit 404 or the output value of the bit shifter 405 according to the input signal bin_en. The selector 407 outputs the output value of the adder circuit 404 when the signal bin_en is low level (0), and selects and outputs the output value of the bit shifter 405 when the signal bin_en is high level (1). Therefore, the selector 407 outputs the gradation signal of the gradation pixel 400 when the signal bin_en is low level (0), and outputs the average value (binning value) of the gradation signal of the gradation pixel 400 and the gradation signal of the gradation pixel 402 when the signal bin_en is high level (1). The selector 408 selects a row in response to the input signal row_sel, and a grayscale signal corresponding to the grayscale pixel arranged in the selected row is output to the readout unit 102.

[0034] 4, when the signal bin_en is at low level (0), the signal row_sel is switched to sequentially read out the first and third rows, thereby making it possible to obtain the grayscale signal of the grayscale pixel 400 and the grayscale signal of the grayscale pixel 402. On the other hand, when the signal bin_en is at high level (1), only the binning values ​​of the grayscale signals corresponding to the grayscale pixels 400 and 402 are obtained by reading out the first row, and reading out the third row is not performed. In this way, the output data rate can be lowered by reducing the number of times the grayscale signals are read out during binning processing.

[0035] Here, in the above, it has been described that the binning process is performed in the pixel array unit 101, and the number of times of reading from the pixel array unit 101 to the readout unit 102 is reduced to reduce the output data rate of the corresponding signal. However, the readout unit 102 may hold a plurality of rows of signals sequentially read from the pixel array unit 101, and perform the binning process in the readout unit 102 to reduce the output data rate of the corresponding signal. That is, the output data rate of the grayscale signal may be reduced by reducing the number of grayscale signals output from the readout unit compared to the number of grayscale signals input to the readout unit 102 in a predetermined length of time. Note that FIG. 4 shows a case where the binning process is performed on the digital signals output from the grayscale pixels 400 and 402, but when AD conversion is performed outside the pixels, the binning process may be performed on the analog signal using a binning process circuit that processes the analog signal. Note that these binning processes are not limited to grayscale signals, and similar processes can be performed on event signals.

[0036] FIG. 5 is an example of a diagram illustrating control of the output data rate of a signal output by the photoelectric conversion device 100 according to this embodiment.

[0037] FIG. 5 shows an example of a binning process method for the gradation pixels 110 and the event pixels 120, which are controlled by the control unit 103, and a setting example of a readout period and divided pixel regions.

[0038] 5(a) shows the pixel array unit 101 divided into two pixel regions. Region A500 is a central pixel region set near the center of the pixel array unit 101, and region B501 is a peripheral pixel region set outside the central pixel region within the pixel array unit 101.

[0039] FIG. 5(b) shows an example in which the control pattern of the output data rate of the gradation pixel 110 and the event pixel 120 is set for each operation mode and pixel region. As a specific setting type corresponding to the output data rate, a setting example of the binning process and the readout cycle is shown. Here, for example, a case in which the photoelectric conversion device 100 is used in an in-vehicle camera and a normal mode and a jump-out detection mode are switched will be described. The jump-out detection mode is a setting example assuming an application in which a person, another vehicle, or the like jumping out from the side is detected with low delay and an automatic brake control of the host vehicle is performed. Note that a mode other than the normal mode and the jump-out detection mode may be available, and three or more modes may be switchable.

[0040] In normal mode, the area A 500 and the area B 501 are set to the same setting. Specifically, the gradation pixels 110 corresponding to each pixel area are both "no binning processing" and "a readout period of 16.7 ms." In addition, the event pixels 120 corresponding to each pixel area are both "no binning processing" and "a readout period of 1.0 ms."

[0041] However, in the protrusion detection mode, the setting of the region A500 remains the same, but the region B501 is set differently from the region B501 in the normal mode. In the protrusion detection mode, the gradation pixels 110 arranged in the region B501 are "binned (2x2)". The 2x2 binning process is a process that performs averaging process on signals corresponding to pixels of the same color for a total of four pixels, for example, two pixels in the horizontal direction and two pixels in the vertical direction, to reduce the number of output signals to one-fourth compared to the normal mode of "no binning process". That is, the protrusion detection mode reduces the spatial resolution compared to the normal mode, thereby reducing the output data rate of the gradation signals output from the gradation pixels 110 arranged in the region B501. In addition, in the protrusion detection mode, the event pixels 120 arranged in the region B501 are "read out at a period of 0.5 ms". By setting the readout period to 0.5 ms, the readout period is reduced by half compared to the normal mode of "readout period of 1.0 ms". That is, in the pop-out detection mode, the time resolution is increased compared to the normal mode, thereby increasing the output data rate of the event signals output from the event pixels 120 arranged in the region B501.

[0042] In this way, the time resolution of the event signal is increased by allocating a part of the output data rate allocated to the gradation signal of the pixel peripheral region B501 to the output data rate of the event signal. This type of control makes it possible to quickly detect, for example, an object to be captured entering from outside the imaging range, such as a person jumping out from the side or another vehicle, and to increase the response speed of the automatic brake of the host vehicle.

[0043] In this embodiment, the region B501 is set differently from the region B501 in the normal mode according to the set mode. That is, a first pattern in which the gradation signal and the event signal are output at the first and second output data rates, respectively, and a second pattern in which the gradation signal and the event signal are output at the third and fourth output data rates, respectively, are switched according to the set mode. In other words, while the region A500 and the region B501 are set to the same in the normal mode, in the protrusion detection mode, the setting of the region B501 is changed from the normal mode, so that in the protrusion detection mode, the region A500 and the region B501 are set to different settings. That is, a first pattern in which the gradation signal and the event signal are output at the first and second output data rates, respectively, and a second pattern in which the gradation signal and the event signal are output at the third and fourth output data rates, respectively, are switched according to the divided pixel region. In this case, it can be considered that one pixel region of the divided pixel regions is set to the first pattern, and the other pixel region is set to the second pattern. Note that the first output data rate is different from the third output data rate, and the second output data rate is different from the fourth output data rate. Note that if the first output data rate is greater than the third output data rate, the second output data rate may be less than the fourth output data rate. Note that if the first output data rate is less than the third output data rate, the second output data rate may be greater than the fourth output data rate.

[0044] As described above, in this embodiment, the output data rates of the grayscale signal and the event signal are changed to different values ​​according to the pattern, so that optimal control can be performed when outputting the grayscale signal and the event signal. In addition, since this embodiment can acquire both the grayscale signal and the event signal, more information can be acquired compared to a photoelectric conversion device that can acquire only the grayscale signal or only the event signal. In addition, by switching the above-mentioned patterns for each set mode and divided pixel region according to the purpose or scene, the output control of the grayscale signal and the event signal can be optimized according to the purpose or scene.

[0045] In addition, when the grayscale signal and the event signal are outputted to the outside of the photoelectric conversion device 100 using the same output unit, if the amount of the outputted signal exceeds the maximum output data amount of the output unit, there is a possibility that a part of the outputted signal will be lost. Here, at least one of the output data rate of the grayscale signal and the output data rate of the event signal is lowered and at least one of the output data rate of the grayscale signal and the output data rate of the event signal is increased according to the switching pattern. By performing such control, it is possible to appropriately allocate the output data rate of the grayscale signal and the output data rate of the event signal within the maximum output data amount of the output unit. For example, when the event signal is more necessary than the grayscale signal, the output data rate of the grayscale signal is lowered and the output data rate of the event signal is increased, thereby making it possible to appropriately control the output data rate. Similarly, when the grayscale signal and the event signal are read out using the same readout unit 102, if the amount of the readout signal exceeds the maximum readout data amount of the readout unit 102, there is a possibility that a part of the readout signal will be lost. Here, depending on the switching pattern, at least one of the output data rate of the gradation signal and the output data rate of the event signal is lowered, and at least one of the output data rate of the gradation signal and the output data rate of the event signal is increased. By performing such control, it is possible to appropriately allocate the output data rate of the gradation signal and the output data rate of the event signal within the maximum read data amount of the reading unit 102. For example, when the event signal is more necessary than the gradation signal, the output data rate can be appropriately controlled by lowering the output data rate of the gradation signal and increasing the output data rate of the event signal.

[0046] The numerical values ​​used in the configuration and processing method shown in this embodiment are merely examples and do not limit the embodiment. The range and read cycle of the binning process of the gradation pixels 110 and the event pixels 120 can take any value. The pixel region setting method shown in FIG. 5(a) is also merely an example, and does not limit the positions, areas, shapes, etc. of the regions A500 and B501. Although the pixel region is divided into two in this embodiment, it is also possible to divide it into a number of pixel regions and configure each region to have a different binning process method and read cycle. In the example shown in FIG. 5, the event pixels 120 are not binned in the pop-out detection mode, but there may be an example in which the binning process is performed. For example, the event pixels 120 in the region B501 are set to "binned (2×2)" to lower the spatial resolution, and the read cycle of the event pixels 120 in the region B501 can be shortened accordingly to increase the time resolution. The sizes of the binning target regions of the gradation pixels 110 and the event pixels 120 may be different from each other. For example, the grayscale pixels 110 may be set to "binned (4x4)" and the event pixels 120 may be set to "binned (2x2)", or any other arbitrary setting may be used.

[0047] Furthermore, although the present embodiment has been described as reading out the event pixels 120 at a constant cycle, only the event signal corresponding to the pixel region where the event occurred asynchronously according to the timing of the event occurrence may be output. In that case, the time resolution for outputting the event signal can be adjusted by adjusting the readout frequency, for example, by setting a minimum readout interval in the time interval for reading out the event signal. In addition, the output data rate of the event signal in this case can be considered as the maximum data rate that can be output. In this case, the control unit 103 may include an arbitration circuit. The arbitration circuit sequentially selects the event pixel 120 in which an event is detected from among the multiple event pixels 120. As a result, the event signals are sequentially read out from the multiple event pixels 120 to the readout unit 102. In other words, the arbitration circuit is a circuit that performs control to arbitrate the readout order from the multiple event pixels 120 in which an event is detected.

[0048] Furthermore, in this embodiment, the grayscale signal and the event signal are generated based on the incident light, but other examples are possible. For example, a combination of grayscale pixels that output grayscale signals to acquire distance information to an imaging target, such as a distance image acquisition sensor, and event pixels that output event signals to acquire changes in the distance information may be used. In this way, if there is a combination of grayscale pixels that output grayscale signals corresponding to an input value and event pixels that output event signals corresponding to changes in the input value, the same effect as this embodiment can be obtained by applying the present invention.

[0049] Second Embodiment A photoelectric conversion device 100 according to a second embodiment of the present invention will be described with reference to Figures 6 and 7. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and the description of these components may be omitted or simplified.

[0050] This embodiment differs from the first embodiment in that the photoelectric conversion device 100 further includes a region setting section that sets a plurality of pixel regions into which the pixel array section 101 is divided.

[0051] FIG. 6 is an example of a block diagram of a photoelectric conversion device 100 according to this embodiment.

[0052] As shown in FIG. 6, the photoelectric conversion device 100 further includes a region setting unit 600. The region setting unit 600 sets a plurality of pixel regions into which the pixel array unit 101 is divided based on, for example, a signal input from the pixel array unit 101, and outputs a setting signal for the plurality of pixel regions to the control unit 103. The control unit 103 controls at least one of the pixel array unit 101 and the readout unit 102 based on the signal output from the region setting unit 600 so that the output data rate is different for each pixel region. The output data rate for each divided pixel region may be calculated by the region setting unit 600, may be calculated by the control unit 103, may be calculated outside the photoelectric conversion device 100, or may be incorporated in the photoelectric conversion device 100 in advance as a design value. The output data rate for each divided pixel region is controlled, for example, by changing the binning processing method and the readout period. It is not necessarily required to output the grayscale signal and the event signal used only in the region setting unit 600 to the outside of the readout unit 102 and the photoelectric conversion device 100.

[0053] The region setting unit 600 may include a spiking neural network (SNN) to which output values ​​of each event pixel 120 in the pixel array unit 101 are input, for example. The SNN is a neural network that processes spike information in a time series as input, and it is possible to input the occurrence of an event of the event pixel 120 as spike information to the SNN. The circuit constituting the SNN is arranged in a lower layer of the pixel array unit 101 by, for example, semiconductor lamination technology. The region setting unit 600 including the SNN finds an area of ​​high importance within the imaging range, and inputs information indicating the position and range of the area as a parameter to the control unit 103. The SNN may be input with a grayscale signal, or may be input with both a grayscale signal and an event signal. However, the SNN uses spike information such as an event signal as input data. Therefore, when a grayscale signal is input, the grayscale signal output by the readout unit 102 is first stored in a memory (not shown) in the photoelectric conversion device 100. After that, the grayscale signal may be converted into spike information, and the converted grayscale signal may be input to the SNN.

[0054] Although the region setting unit 600 includes an SNN, the configuration is not limited. For example, the region setting unit 600 may include other configurations such as a convolutional neural network (CNN). A grayscale signal may be input to the SNN, an event signal may be input, or both a grayscale signal and an event signal may be input. However, the CNN uses input data in an array structure such as frame image data (grayscale signal) output by a conventional image sensor as input data. Therefore, when an event signal is input, if an event signal corresponding to an area where no event occurs is not output, the event signal output by the readout unit 102 is first stored in a memory (not shown) in the photoelectric conversion device 100. After that, the event signal may be converted to an array structure, and the converted event signal may be input to the CNN. Note that, as an example of converting an event sequence into an array structure, there is a 2D histogram format in which a value corresponding to a pixel value of the output image data of a general image sensor is the number of events that occurred at that pixel within a certain period of time. There is also a Time Surface format in which a value corresponding to a pixel value of the output image data of a general image sensor is the latest timestamp of events that occurred at each pixel. These are just examples, and any method may be used to convert an event sequence into array-structured data. When using CNN, a setting value stored in a memory (not shown) in the photoelectric conversion device 100 may be input as an input instead of an event signal.

[0055] FIG. 7 is a diagram illustrating an example of output data rate control of a signal output by the photoelectric conversion device 100 according to this embodiment.

[0056] FIG. 7 shows an example of a binning process method for the gradation pixels 110 and the event pixels 120, which are controlled by the control unit 103, and a setting example of a readout period and divided pixel regions.

[0057] FIG. 7(a) shows pixel regions divided based on the setting result of the region setting unit 600. Region C700a and region C'700b indicate important regions set by the region setting unit 600, and move along the direction of the arrow 702 as time passes. Here, the state before the important region moves is referred to as region C700a, and the state after the important region moves is referred to as region C'700b. Region D701 is a region different from the important region. The region setting unit 600 sequentially executes a process for determining an important region, and updates a signal output to the control unit 103 while region C700a moves to region C'700b. Note that when region C700a is set as an important region, the region corresponding to region C'700b is included in region D701, and when region C'700b is set as an important region, the region corresponding to region C700a is included in region D701.

[0058] The signal used to detect the arrow 702 for moving the important area may be an event signal, a grayscale signal, or both an event signal and a grayscale signal. For example, by using an event signal having a shorter read cycle than a grayscale signal, the important area can be moved at higher speed.

[0059] FIG. 7(b) shows an example in which the control pattern of the output data rate of the gradation pixel 110 and the event pixel 120 is set for each operation mode and pixel region. As a specific setting type corresponding to the output data rate, a setting example of binning processing and readout cycle is shown. Here, a case where the photoelectric conversion device 100 is applied to a network camera that captures an area in a store where products are displayed will be described. The normal mode is used for purposes such as tracking the movement of customers and collecting marketing information. In addition, the moving object tracking mode is a setting example assuming that, for example, a suspicious person is treated as a moving object and an important area is moved according to the movement of the suspicious person, thereby continuously acquiring an image (gradation signal) of the suspicious person and its surroundings with high time resolution. Note that modes other than the normal mode and the moving object tracking mode may be available, and three or more modes may be switchable. In the normal mode, the important area and the area D701 are set to the same setting. Specifically, both the gradation pixels 110 corresponding to each pixel area are "no binning processing" and "the readout cycle is 100.0 ms". Moreover, the event pixels 120 corresponding to the respective pixel regions are both "no binning processing" and "a readout period of 1.0 ms."

[0060] However, in the moving object tracking mode, the important region is set differently from the important region in the normal mode. Also, in the moving object tracking mode, the important region in the normal mode and the important region in the moving object tracking mode are set differently. In other words, while the important region and the region D701 are set to be the same in the normal mode, in the moving object tracking mode, the important region and the region D701 are set to be different.

[0061] In the moving object tracking mode, the event pixels 120 arranged in the important region and the gradation pixels 110 arranged in the region D701 do not output signals. In addition, in the moving object tracking mode, the gradation pixels 110 arranged in the important region have a "readout period of 25.0 ms". By setting the readout period to 25.0 ms, the readout period is 1 / 4 times that of the normal mode in which the "readout period is 100.0 ms". In other words, in the moving object tracking mode, the time resolution is increased compared to the normal mode, thereby increasing the output data rate of the gradation signal output from the gradation pixels 110 arranged in the important region. In addition, in the moving object tracking mode, the event pixels 120 arranged in the region D701 have a "readout period of 2.0 ms". By setting the readout period to 2.0 ms, the readout period is doubled compared to the normal mode in which the "readout period is 1.0 ms". That is, the object tracking mode reduces the time resolution compared to the normal mode, thereby reducing the output data rate of the event signal output from the event pixels 120 arranged in the region D701. Therefore, compared to the normal mode, the object tracking mode reduces the output data rate of the event signal in the important region and the gradation signal and event signal in the region D701, and increases the output data rate of the gradation signal in the important region.

[0062] In this way, the output data rate is controlled by allocating a part of the output data rate allocated to the event signal of the important region, the gradation signal of region D701, and the event signal to the gradation signal of the important region. With this control, in the moving object tracking mode, the accuracy of identifying the face of a suspicious person is improved and detailed analysis of behavior is possible by placing emphasis on pixel data corresponding to the gradation signal of the important region. Meanwhile, a certain amount of information can be obtained from the event signal for regions other than the periphery of the suspicious person (region D701).

[0063] In this embodiment, the important region is set differently from the important region in the normal mode according to the set mode, and the region D701 is set differently from the region D701 in the normal mode. That is, a first pattern in which the gradation signal and the event signal are output at the first and second output data rates, respectively, and a second pattern in which the gradation signal and the event signal are output at the third and fourth output data rates, respectively, are switched according to the set mode. Also, in the moving object tracking mode, the important region and the region D701 are set differently. That is, a first pattern in which the gradation signal and the event signal are output at the first and second output data rates, respectively, and a second pattern in which the gradation signal and the event signal are output at the third and fourth output data rates, respectively, are switched according to the divided pixel region. In this case, it can be considered that one pixel region of the divided multiple pixel regions is set to the first pattern and the other pixel region is set to the second pattern. Note that the first output data rate is different from the third output data rate, and the second output data rate is different from the fourth output data rate. Note that if the first output data rate is greater than the third output data rate, the second output data rate may be less than the fourth output data rate, and if the first output data rate is less than the third output data rate, the second output data rate may be greater than the fourth output data rate.

[0064] As described above, in this embodiment, the output data rates of the grayscale signal and the event signal are changed to different values ​​according to the pattern, so that optimal control can be performed when outputting the grayscale signal and the event signal. In addition, since this embodiment can acquire both the grayscale signal and the event signal, more information can be acquired compared to a photoelectric conversion device that can acquire only the grayscale signal or only the event signal. In addition, by switching the above-mentioned patterns for each set mode and divided pixel region according to the purpose or scene, the output control of the grayscale signal and the event signal can be optimized according to the purpose or scene.

[0065] In addition, when the grayscale signal and the event signal are outputted to the outside of the photoelectric conversion device 100 using the same output unit, if the amount of the outputted signal exceeds the maximum output data amount of the output unit, there is a possibility that a part of the outputted signal will be lost. Here, at least one of the output data rate of the grayscale signal and the output data rate of the event signal is lowered and at least one of the output data rate of the grayscale signal and the output data rate of the event signal is increased according to the switching pattern. By performing such control, it is possible to appropriately allocate the output data rate of the grayscale signal and the output data rate of the event signal within the maximum output data amount of the output unit. For example, when the event signal is more necessary than the grayscale signal, the output data rate of the grayscale signal is lowered and the output data rate of the event signal is increased, thereby making it possible to appropriately control the output data rate. Similarly, when the grayscale signal and the event signal are read out using the same readout unit 102, if the amount of the readout signal exceeds the maximum readout data amount of the readout unit 102, there is a possibility that a part of the readout signal will be lost. Here, depending on the switching pattern, at least one of the output data rate of the gradation signal and the output data rate of the event signal is lowered, and at least one of the output data rate of the gradation signal and the output data rate of the event signal is increased. By performing such control, it is possible to appropriately allocate the output data rate of the gradation signal and the output data rate of the event signal within the maximum read data amount of the reading unit 102. For example, when the event signal is more necessary than the gradation signal, the output data rate can be appropriately controlled by lowering the output data rate of the gradation signal and increasing the output data rate of the event signal.

[0066] The numerical values ​​used in the configuration and processing method shown in this embodiment are merely examples and do not limit the embodiment. The range and read cycle for performing the binning process on the gradation pixels 110 and the event pixels 120 can take any value. The pixel region setting method shown in FIG. 7(a) is also merely an example, and does not limit the positions, areas, shapes, etc. of the important region and the region D701. The important region set by the region setting unit 600 is one region that changes from region C700a to region C'700b, but it may be multiple regions. Furthermore, the region setting unit 600 may set the importance of each important region for multiple important regions. In this case, a different binning process method or read cycle may be set for each important region so that the output data rate corresponding to the important region with a higher importance is higher. In the example shown in FIG. 7, the event pixels 120 arranged in the region D701 are not binned in the moving object tracking mode, but there may be an example in which the binning process is performed. For example, by lowering the spatial resolution by setting the event pixels 120 in region D701 to "binned (2x2)", it is possible to further shorten the readout period of the gradation pixels 110 in the important region and increase the temporal resolution. The sizes of the binning target regions of the gradation pixels 110 and the event pixels 120 may be different from each other. For example, the event pixels 120 may be set to "binned (4x4)" and the gradation pixels 110 to "binned (2x2)", and so on.

[0067] Furthermore, although the present embodiment has been described as reading out the event pixels 120 at a constant cycle, it may be configured to output only the event signal corresponding to the pixel area where the event occurred asynchronously according to the timing of the event occurrence. In that case, the time resolution for outputting the event signal can be adjusted by adjusting the readout frequency, for example, by setting a minimum readout interval in the time interval for reading out the event signal. In addition, the output data rate of the event signal in this case can be considered as the maximum data rate that can be output. In this case, the control unit 103 may include an arbitration circuit. The arbitration circuit sequentially selects the event pixel 120 in which an event is detected from among the multiple event pixels 120. As a result, the event signals are sequentially read out from the multiple event pixels 120 to the readout unit 102. In other words, the arbitration circuit is a circuit that performs control to arbitrate the readout order from the multiple event pixels 120 in which an event is detected.

[0068] Furthermore, in this embodiment, the grayscale signal and the event signal are generated based on the incident light, but other examples are possible. For example, a combination of grayscale pixels that output grayscale signals to acquire distance information to an imaging target, such as a distance image acquisition sensor, and event pixels that output event signals to acquire changes in the distance information may be used. In this way, if there is a combination of grayscale pixels that output grayscale signals corresponding to an input value and event pixels that output event signals corresponding to changes in the input value, the same effect as this embodiment can be obtained by applying the present invention.

[0069] Third embodiment The third embodiment is also applicable to the first or second embodiment. FIG. 8(a) is a schematic diagram for explaining an apparatus 9191 including the semiconductor device 930 of this embodiment. The photoelectric conversion device 100 of each of the above-mentioned embodiments can be used for the semiconductor device 930. The apparatus 9191 including the semiconductor device 930 will be explained in detail. The semiconductor device 930 can include a package 920 that houses the semiconductor device 910 in addition to the semiconductor device 910. The package 920 can include a base to which the semiconductor device 910 is fixed, and a lid such as glass that faces the semiconductor device 910. The package 920 can further include a bonding member such as a bonding wire or a bump that connects a terminal provided on the base and a terminal provided on the semiconductor device 910.

[0070] The device 9191 can include at least one of an optical device 940, a control device 950, a processing device 960, a display device 970, a storage device 980, and a mechanical device 990. The optical device 940 corresponds to the semiconductor device 930. The optical device 940 is, for example, a lens, a shutter, or a mirror, and includes an optical system that guides light to the semiconductor device 930. The control device 950 controls the semiconductor device 930. The control device 950 is, for example, a semiconductor device such as an ASIC.

[0071] The processing device 960 processes the signal output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or ASIC for configuring an AFE (analog front end) or a DFE (digital front end). The display device 970 is an EL display device or a liquid crystal display device that displays information (images) obtained by the semiconductor device 930. The storage device 980 is a magnetic device or a semiconductor device that stores information (images) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0072] The mechanical device 990 has a moving part or a propulsion part such as a motor or an engine. In the device 9191, the signal output from the semiconductor device 930 is displayed on the display device 970, or transmitted to the outside by a communication device (not shown) included in the device 9191. For this purpose, the device 9191 preferably further includes a memory device 980 and a processing device 960 in addition to the memory circuit and arithmetic circuit included in the semiconductor device 930. The mechanical device 990 may be controlled based on the signal output from the semiconductor device 930.

[0073] The device 9191 is also suitable for electronic devices such as information terminals (e.g., smartphones and wearable devices) with a photographing function and cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). The mechanical device 990 in the camera can drive components of the optical device 940 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 990 in the camera can move the semiconductor device 930 for vibration isolation operation.

[0074] The device 9191 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 990 in the transportation equipment may be used as a moving device. The device 9191 as a transportation equipment is suitable for transporting the semiconductor device 930 or for assisting and / or automating driving (piloting) by using a photographing function. The processing device 960 for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device 990 as a moving device based on information obtained by the semiconductor device 930. Alternatively, the device 9191 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0075] According to the above-described embodiment, it is possible to obtain good pixel characteristics. Therefore, the value of the semiconductor device can be increased. In this case, increasing the value corresponds to at least one of adding functions, improving performance, improving characteristics, improving reliability, improving manufacturing yield, reducing environmental load, reducing costs, reducing size, and reducing weight.

[0076] Therefore, if the semiconductor device 930 according to this embodiment is used in the equipment 9191, the value of the equipment can be improved. For example, by mounting the semiconductor device 930 on a transport equipment, excellent performance can be obtained when photographing the outside of the transport equipment or measuring the external environment. Therefore, in manufacturing and selling the transport equipment, it is advantageous to decide to mount the semiconductor device according to this embodiment on the transport equipment in order to improve the performance of the transport equipment itself. In particular, the semiconductor device 930 is suitable for transport equipment that performs driving assistance and / or automatic driving of the transport equipment using information obtained by the semiconductor device.

[0077] The photoelectric conversion system and the moving object of this embodiment will be described with reference to FIGS. 8(b) and 8(c).

[0078] FIG. 8(b) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 8 has a photoelectric conversion device 100. The photoelectric conversion device 100 is a photoelectric conversion device (imaging device) described in any of the above embodiments. The photoelectric conversion system 8 has an image processing unit 801 that performs image processing on a plurality of image data acquired by the photoelectric conversion device 100, and a parallax acquisition unit 802 that calculates parallax (phase difference of parallax images) from the plurality of image data acquired by the photoelectric conversion system 8. The photoelectric conversion system 8 also has a distance acquisition unit 803 that calculates a distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 802 and the distance acquisition unit 803 are examples of distance information acquisition means that acquire distance information to an object. That is, the distance information is information related to the parallax, the defocus amount, the distance to the object, and the like. The collision determination unit 804 may determine the possibility of collision using any of these distance information. The distance information acquisition means may be realized by dedicated hardware, a software module, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination of these.

[0079] The photoelectric conversion system 8 is connected to a vehicle information acquisition device 810, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 8 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the judgment result of the collision judgment unit 804. The photoelectric conversion system 8 is also connected to an alarm device 830 that issues an alarm to the driver based on the judgment result of the collision judgment unit 804. For example, when the judgment result of the collision judgment unit 804 indicates that there is a high possibility of a collision, the control ECU 820 performs vehicle control to avoid a collision and reduce damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., and vibrating the seat belt or steering wheel.

[0080] In this embodiment, the surroundings of the vehicle, for example the front or rear, are imaged by the photoelectric conversion system 8. Fig. 8(c) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 850). A vehicle information acquisition device 810 sends an instruction to the photoelectric conversion system 8 or the photoelectric conversion device 100. This configuration can further improve the accuracy of distance measurement.

[0081] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from lanes, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as the vehicle itself, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention can be applied not only to moving bodies, but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0082] The above-described embodiments can be modified as appropriate without departing from the technical concept. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached hereto. The disclosure of this specification also includes the complement of the concepts described in this specification. In other words, if this specification contains a statement that "A is greater than B," for example, this specification can be said to disclose that "A is not greater than B" even if the statement that "A is not greater than B" is omitted. This is because when it is stated that "A is greater than B," it is assumed that the case in which "A is not greater than B" is taken into consideration.

[0083] The disclosure of this embodiment includes the following configuration.

[0084] (Configuration 1) 1. A photoelectric conversion device comprising: a pixel array section in which a plurality of first pixels capable of generating a gradation signal representing an amount of incident light and a plurality of second pixels capable of generating an event signal are arranged in a two-dimensional array; a readout section that reads out the gradation signals and the event signals from the pixel array section; and a control section that changes an output data rate of the gradation signals and an output data rate of the event signals, wherein the photoelectric conversion device switches between a first pattern in which the gradation signals are output at a first output data rate and the event signals are output at a second output data rate, and a second pattern in which the gradation signals are output at a third output data rate different from the first output data rate and the event signals are output at a fourth output data rate different from the second output data rate.

[0085] (Configuration 2) The photoelectric conversion device of configuration 1, characterized in that when the first output data rate is greater than the third output data rate, the second output data rate is smaller than the fourth output data rate, and when the first output data rate is smaller than the third output data rate, the second output data rate is greater than the fourth output data rate.

[0086] (Configuration 3) The photoelectric conversion device according to configuration 1 or 2, wherein the control unit changes at least one of an output data rate of the gradation signal and an output data rate of the event signal by controlling the pixel array unit.

[0087] (Configuration 4) The photoelectric conversion device according to any one of configurations 1 to 3, characterized in that the control unit changes at least one of the output data rate of the gradation signal and the output data rate of the event signal by controlling the readout unit.

[0088] (Configuration 5) The photoelectric conversion device according to any one of configurations 1 to 4, characterized in that the control unit controls the pixel array unit to change the number of times at least one of the gradation signal and the event signal is read from the pixel array unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

[0089] (Configuration 6) The photoelectric conversion device according to any one of configurations 1 to 5, characterized in that the control unit controls the pixel array unit to change a period for reading out at least one of the gradation signal and the event signal from the pixel array unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

[0090] (Configuration 7) The photoelectric conversion device according to any one of configurations 1 to 6, characterized in that the control unit changes at least one of the output data rate of the gradation signal and the output data rate of the event signal by controlling the readout unit.

[0091] (Configuration 8) The photoelectric conversion device according to any one of configurations 1 to 7, characterized in that the control unit controls the readout unit to reduce the number of signals output from the readout unit compared to the number of signals corresponding to at least one of the gradation signals and the event signals input to the readout unit over a predetermined length of time, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

[0092] (Configuration 9) The photoelectric conversion device according to any one of configurations 1 to 8, characterized in that the control unit controls the readout unit to change a period for outputting at least one of the gradation signal and the event signal from the readout unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

[0093] (Configuration 10) 10. The photoelectric conversion device according to any one of configurations 1 to 9, wherein the control unit switches between the first pattern and the second pattern depending on a set mode.

[0094] (Configuration 11) The photoelectric conversion device described in any one of configurations 1 to 10, characterized in that the pixel array section can be divided into a plurality of pixel regions, each of which includes at least one of the first pixels and at least one of the second pixels, and the control section controls one of the plurality of pixel regions with the first pattern and controls the other pixel region with the second pattern.

[0095] (Configuration 12) 12. The photoelectric conversion device according to any one of configurations 1 to 11, further comprising a region setting unit that sets the plurality of pixel regions by using at least one of the grayscale signal and the event signal.

[0096] (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 12, characterized in that the region setting unit has a spiking neural network connected to the second pixel, and sets the plurality of pixel regions using the spiking neural network.

[0097] (Configuration 14) The photoelectric conversion device according to any one of configurations 1 to 13, characterized in that the region setting unit has a convolutional neural network connected to the second pixel, and sets the multiple pixel regions using the convolutional neural network.

[0098] (Configuration 15) 15. The photoelectric conversion device according to any one of configurations 1 to 14, wherein the first pixel is capable of generating the event signal, and the second pixel is capable of generating the gradation signal.

[0099] (Configuration 16) 16. The photoelectric conversion device according to any one of configurations 1 to 15, wherein distance information is obtained by using at least one of the grayscale signal and the event signal.

[0100] (Configuration 17) The photoelectric conversion device described in any one of configurations 1 to 16, characterized in that the output data rate of the gradation signal is a data rate at which the gradation signal is output from the pixel array unit via the readout unit to outside the photoelectric conversion device, and the output data rate of the event signal is a data rate at which the event signal is output from the pixel array unit via the readout unit to outside the photoelectric conversion device.

[0101] (Configuration 18) 18. The photoelectric conversion device according to any one of configurations 1 to 17, wherein the event signal represents a change in the amount of incident light.

[0102] (Configuration 19) 19. A moving body comprising the photoelectric conversion device according to any one of configurations 1 to 18, further comprising a control device for controlling the movement of the moving body.

[0103] (Configuration 20) An apparatus including a photoelectric conversion device according to any one of configurations 1 to 19, further including at least one of an optical device that guides light to the photoelectric conversion device, a control device that controls the photoelectric conversion device, a processing device that processes a signal output from the photoelectric conversion device, a display device that displays information obtained by the photoelectric conversion device, a memory device that stores information obtained by the photoelectric conversion device, and a mechanical device that operates based on the information obtained by the photoelectric conversion device. [Explanation of symbols]

[0104] 100 Photoelectric conversion device 101 Pixel array section 102 Readout section 103 Control section 110 1st pixel 120 2nd pixel

Claims

1. a pixel array unit in which a plurality of first pixels capable of generating a gradation signal representing an amount of incident light and a plurality of second pixels capable of generating an event signal are arranged in a two-dimensional manner; a readout unit that reads out the grayscale signal and the event signal from the pixel array unit; a control unit that changes an output data rate of the gradation signal and an output data rate of the event signal; A photoelectric conversion device comprising: and switching between a first pattern in which the gradation signal is output at a first output data rate and the event signal is output at a second output data rate and a second pattern in which the gradation signal is output at a third output data rate different from the first output data rate and the event signal is output at a fourth output data rate different from the second output data rate. A photoelectric conversion device comprising:

2. 2. The photoelectric conversion device according to claim 1, wherein when the first output data rate is greater than the third output data rate, the second output data rate is smaller than the fourth output data rate, and when the first output data rate is smaller than the third output data rate, the second output data rate is greater than the fourth output data rate.

3. 2 . The photoelectric conversion device according to claim 1 , wherein the control unit changes at least one of an output data rate of the grayscale signal and an output data rate of the event signal by controlling the pixel array unit.

4. 4. The photoelectric conversion device according to claim 3, wherein the control unit changes at least one of an output data rate of the grayscale signal and an output data rate of the event signal by controlling the readout unit.

5. The photoelectric conversion device according to claim 3, characterized in that the control unit controls the pixel array unit to change the number of times that at least one of the gradation signal and the event signal is read from the pixel array unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

6. The photoelectric conversion device according to claim 3, characterized in that the control unit controls the pixel array unit to change a period for reading out at least one of the gradation signal and the event signal from the pixel array unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

7. 2. The photoelectric conversion device according to claim 1, wherein the control unit changes at least one of an output data rate of the grayscale signal and an output data rate of the event signal by controlling the readout unit.

8. The control unit controls the readout unit to reduce the number of signals output from the readout unit compared to the number of signals corresponding to at least one of the gradation signal and the event signal input to the readout unit over a predetermined period of time, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

9. The photoelectric conversion device according to claim 7, characterized in that the control unit controls the readout unit to change a period for outputting at least one of the gradation signal and the event signal from the readout unit, thereby changing at least one of the output data rate of the corresponding gradation signal and the output data rate of the corresponding event signal.

10. The photoelectric conversion device according to claim 1 , wherein the control unit switches between the first pattern and the second pattern depending on a set mode.

11. The photoelectric conversion device according to claim 1, characterized in that the pixel array portion can be divided into a plurality of pixel regions, each including at least one of the first pixels and at least one of the second pixels, and the control portion controls one of the plurality of pixel regions with the first pattern and controls the other pixel region with the second pattern.

12. 12. The photoelectric conversion device according to claim 11, further comprising a region setting unit that sets the plurality of pixel regions by using at least one of the grayscale signal and the event signal.

13. The photoelectric conversion device according to claim 12 , wherein the region setting unit has a spiking neural network connected to the second pixels, and sets the plurality of pixel regions using the spiking neural network.

14. The photoelectric conversion device according to claim 12 , wherein the region setting section has a convolutional neural network connected to the second pixels, and sets the plurality of pixel regions using the convolutional neural network.

15. The photoelectric conversion device according to claim 1 , wherein the first pixel is capable of generating the event signal, and the second pixel is capable of generating the gradation signal.

16. 2. The photoelectric conversion device according to claim 1, wherein distance information is obtained by using at least one of the grayscale signal and the event signal.

17. 2. The photoelectric conversion device according to claim 1, wherein the output data rate of the gradation signal is a data rate at which the gradation signal is output from the pixel array unit via the readout unit to an outside of the photoelectric conversion device, and the output data rate of the event signal is a data rate at which the event signal is output from the pixel array unit via the readout unit to an outside of the photoelectric conversion device.

18. 2. The photoelectric conversion device according to claim 1, wherein the event signal represents a change in the amount of incident light.

19. A moving object comprising the photoelectric conversion device according to any one of claims 1 to 18, The moving body further comprises a control device for controlling the movement of the moving body.

20. An apparatus comprising the photoelectric conversion device according to any one of claims 1 to 18, an optical device that guides light to the photoelectric conversion device; A control device for controlling the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

Images