Solid-state imaging device

Abstract

[Problem] To correct distortion in event detection pixels. [Solution] This solid-state imaging device is provided with: an optical system; a first pixel; a second pixel; one or more pixel arrays; and a signal processing circuit. The first pixel obtains image information based on luminance information obtained through the optical system. The second pixel obtains event detection information based on a change in luminance information obtained by the optical system. In one or more pixel arrays, first pixels and second pixels are arranged in a two-dimensional array. The signal processing circuit selects whether to perform distortion correction and performs signal processing on event detection information obtained by the second pixel.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device. Background Art

[0002] With the improvement of semiconductor technology for camera modules, camera modules have been developing at a remarkable pace in terms of reducing height and widening the angle. In addition to pixels for obtaining color information such as RGB information, pixels for event detection can be configured in the pixel array in a mixed manner, and event detection and acquisition of color information can be performed in parallel on the same coordinate axis.

[0003] However, there is no discussion on distortion correction for a sensor that can obtain two different types of data in a camera module. That is, distortion correction is performed on an image of color information such as RGB, but distortion correction is not performed on the output from event detection pixels. For this reason, there is a possibility that the positions of points representing the same coordinates are different between event detection pixels and pixels for obtaining color information.

[0004] Citation List

[0005] Patent Document

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-521093 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] Therefore, one of the non-limiting problems to be solved by embodiments of the present disclosure is to correct the distortion of event detection pixels. The problems to be solved by embodiments of the present disclosure may also be problems corresponding to the effects described in the embodiments as some further non-limiting examples. That is, problems corresponding to at least any one of the effects described in the description of the embodiments of the present disclosure may be problems to be solved in the present disclosure.

[0009] Solutions to the Problems

[0010] According to one embodiment, a solid-state imaging device includes an optical system, a first pixel, a second pixel, one or more pixel arrays, and a signal processing circuit.

[0011] The first pixel obtains image information based on luminance information via the optical system.

[0012] The second pixel obtains event detection information based on a change in luminance information via the optical system.

[0013] One or more pixel arrays include the first pixel and the second pixel arranged in a two-dimensional array.

[0014] The signal processing circuit selects whether to perform distortion correction on the event detection information obtained from the second pixel and performs signal processing.

[0015] One or more pixel arrays may include a pixel array, and

[0016] The second pixel may be arranged in a predetermined region in the pixel array at a predetermined ratio relative to the first pixel.

[0017] The predetermined region may include the entire region of the pixel array.

[0018] The second pixels may be arranged at equal intervals so as to cover the entire region of the predetermined region.

[0019] The second pixel can detect events for each frame.

[0020] The second pixel can detect events by an arbiter method.

[0021] The signal processing circuit can perform distortion correction on the image information.

[0022] The signal processing circuit can:

[0023] Convert the resolution of the event detection information into the resolution of the image information, and

[0024] Perform distortion correction on the converted event detection information.

[0025] The signal processing circuit can:

[0026] Perform distortion correction on the event detection information based on the difference between the resolution of the event detection information and the resolution of the image information.

[0027] The signal processing circuit can:

[0028] Set a region of interest based on the image information, and

[0029] Obtain the event detection information corresponding to the region of interest.

[0030] The signal processing circuit can:

[0031] Perform tracking based on the event detection information that has undergone distortion correction corresponding to the region of interest.

[0032] The signal processing circuit can:

[0033] Compare the value obtained after performing distortion correction on the event detection information with the coordinate threshold after performing true correction on the event detection information, and obtain the event detection information at the coordinates after performing distortion correction.

[0034] The signal processing circuit can:

[0035] Store the event detection information at the coordinates after performing distortion correction as 1.5-bit information.

[0036] One or more pixel arrays may include:

[0037] A first pixel array, wherein the first pixels are arranged in a two-dimensional array; and

[0038] A second pixel array, wherein the second pixels are arranged in a two-dimensional array, and the second pixel array obtains event detection information for the same target as the first pixel array.

[0039] The optical system may include:

[0040] A first optical system that focuses light on the first pixel array; and

[0041] A second optical system that focuses light on the second pixel array.

[0042] The signal processing circuit may include:

[0043] A first signal processing circuit that performs signal processing on the output from the first pixels; and

[0044] A second signal processing circuit that performs signal processing on the output from the second pixels.

[0045] The signal processing circuit may:

[0046] Perform image processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction.

[0047] The signal processing circuit may:

[0048] Perform deblurring processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction.

[0049] The signal processing circuit may:

[0050] Perform high frame rate processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction.

[0051] The signal processing circuit may:

[0052] Perform tracking processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a block diagram schematically showing a solid-state imaging device according to an embodiment.

[0054] Figure 2is a block diagram schematically showing an imaging unit according to an embodiment.

[0055] Figure 3 is a diagram showing an example of the arrangement of pixels in a pixel array according to an embodiment.

[0056] Figure 4 is a block diagram schematically showing an imaging unit according to an embodiment.

[0057] Figure 5 is a diagram showing an example of data conversion according to an embodiment.

[0058] Figure 6 is a flowchart showing the processing performed by a solid-state imaging device according to an embodiment.

[0059] Figure 7 is a flowchart showing the processing by a solid-state imaging device according to an embodiment.

[0060] Figure 8 is a flowchart showing the processing performed by a solid-state imaging device according to an embodiment.

[0061] Figure 9 is a diagram showing an example of setting a region of interest according to an embodiment.

[0062] Figure 10 is a flowchart showing the processing by a solid-state imaging device according to an embodiment.

[0063] Figure 11 is a block diagram schematically showing a solid-state imaging device according to an embodiment.

[0064] Figure 12 is a diagram schematically showing a substrate of a solid-state imaging device according to an embodiment.

[0065] Figure 13 is a diagram schematically showing a substrate of a solid-state imaging device according to an embodiment.

[0066] Figure 14 is a diagram schematically showing a substrate of a solid-state imaging device according to an embodiment.

[0067] Figure 15 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0068] Figure 16 is a diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit. Detailed Description

[0069] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. The accompanying drawings are for description purposes, and the shapes and sizes of the configurations of each component in the actual device, the ratios of the sizes to other components, etc. do not necessarily have to be as shown in the drawings. In addition, since the accompanying drawings are shown in a simplified manner, it is assumed that configurations necessary for implementation other than those shown in the drawings are appropriately provided.

[0070] (First Embodiment)

[0071] Figure 1 FIG. 1 is a block diagram schematically showing a solid-state imaging device according to an embodiment. The solid-state imaging device 1 includes an optical system 10, an imaging unit 12, a storage unit 14, a signal processing unit 16, and an input / output interface (hereinafter referred to as input / output I / F 18). The solid-state imaging device 1 is a device including at least a first pixel and a second pixel. The first pixel obtains image information from luminance information, and the second pixel obtains event detection information from a change in luminance information, obtains image information, and also obtains event detection information in the image.

[0072] The optical system 10 is an optical system that appropriately focuses light on the imaging unit 12.

[0073] The imaging unit 12 includes a first pixel and a second pixel, and obtains image information and event detection information. For example, the imaging unit 12 obtains information about a region including the same target in the first pixel and the second pixel.

[0074] The storage unit 14 stores data obtained by the imaging unit 12, or data being processed or data processed by the signal processing unit 16. The storage unit 14 may include, for example, a storage circuit that stores data temporarily or non-temporarily, such as various memories, various memories, etc. At least a part of the storage unit 14 may be provided outside the solid-state imaging device 1.

[0075] The signal processing unit 16 performs various processes on the data obtained by the imaging unit 12 and outputs the data. The processes may include, for example, a process of converting an analog signal output from a pixel circuit into a digital signal, a process of obtaining a pixel value from the digital signal, a process of performing image processing, etc. Incidentally, at least a part of the above processes may be performed in the pixel circuit of the imaging unit 12.

[0076] The signal processing unit 16 may include, in at least a part thereof, a dedicated processing circuit such as an application specific integrated circuit (ASIC), a general-purpose processing circuit such as a central processing unit (CPU), or a programmable circuit such as a field programmable gate array (FPGA). In the case where at least a part of the signal processing unit 16 includes a general-purpose circuit, the processing performed by at least a part of the signal processing unit 16 may be in a form in which information processing is specifically implemented by software using hardware resources. In this case, programs, executable files, etc. related to the software are stored in the storage unit 14, and the general-purpose circuit of the signal processing unit 16 can obtain the programs stored in the storage unit 14 to implement information processing.

[0077] The input / output I / F 18 is an interface that connects the inside and outside of the solid-state imaging device 1. The solid-state imaging device 1 can receive user control directly or indirectly from the input / output I / F 18. In addition, the solid-state imaging device 1 can achieve the transmission and reception of data with the inside and outside via the input / output I / F 18.

[0078] In addition to the above configuration, the solid-state imaging device 1 appropriately includes, as needed, for example, a control unit for controlling each component of the solid-state imaging device 1, a power supply unit for supplying power to each component, and the like.

[0079] Figure 2 is a diagram schematically showing a non-limiting example of the imaging unit 12. As an example, the imaging unit 12 includes a pixel array 120, a horizontal drive circuit 124, a vertical drive circuit 126, and a processing circuit 128. The imaging unit 12 appropriately processes and outputs signals obtained by pixels arranged in the pixel array 120.

[0080] The pixel array 120 is formed by arranging pixels in a two-dimensional array. The pixel array 120 includes first pixels that obtain luminance information and second pixels that obtain changes in luminance information.

[0081] Figure 3 is a diagram showing an example of the arrangement of pixels in the pixel array 120. The pixel array 120 is formed by, for example, periodically arranging first pixels 121 and second pixels 122.

[0082] The first pixel 121 may include a light-receiving element that obtains luminance information, and each light-receiving element may be configured to obtain a pixel of any one of the RGB colors. The color can be selected by a color filter included in the light-receiving element, or the light-receiving element may include an organic photoelectric conversion film or the like.

[0083] The second pixel 122 includes a light-receiving element that obtains a change in luminance information as event detection information.

[0084] As Figure 3As shown, the first pixels 121R, 121G, and 121B and the second pixel 122 may form a pixel group as a group, and the first pixels 121 that obtain signals representing RGB colors and the second pixels 122 that obtain signals representing event information may be periodically arranged with respect to the same coordinates in the image coordinate system.

[0085] These light-receiving elements may be in the form of divided pixels formed to include divided light-receiving elements in the same pixel, that is, in a form formed by setting a group of a first divided pixel for obtaining luminance information and a second divided pixel for obtaining a change in luminance information as one pixel.

[0086] In addition, the arrangement of the pixels is not limited to the arrangement in this figure. For example, it may also include a first pixel 121W that obtains light representing white, and may also include first pixels 121Mg, 121Cy, 121Ye, etc. that represent light of complementary colors. In addition, the group of the first pixels 121 and the second pixels 122 may include another number of pixels, such as 2×3 pixels, 3×3 pixels, etc., rather than 2×2 pixels. That is, the pixel group may be in a form in which the second pixels 122 are appropriately arranged with respect to the first pixels 121 at a predetermined ratio.

[0087] In addition, in Figure 3 the second pixels 122 are arranged at a predetermined ratio over the entire area of the pixel array 120, but the form of the present disclosure is not limited thereto. For example, the second pixels 122 may be in a form arranged in a predetermined area of the pixel array 120. For example, in a case where it is desired to obtain the movement of a target in a predetermined area, a form may be adopted in which the second pixels 122 are included in a predetermined area in the image area and the second pixels 122 are not arranged in other areas of the pixel array 120.

[0088] Return Figure 2 , the horizontal drive circuit 124 and the vertical drive circuit 126 select and drive the first pixels 121 and the second pixels 122 in the pixel array 120, and send the signals from the driven pixels to the processing circuit 128. For example, in the row selected by the horizontal drive circuit 124, the vertical drive circuit 126 drives the pixels in the column so that the signals from the selected and driven pixels are sent to the processing circuit 128.

[0089] The processing circuit 128 processes the received signal appropriately and outputs it to the storage unit 14 or the signal processing unit 16. Incidentally, the processing circuit 128 may be formed as part of the signal processing unit 16 instead of the imaging unit 12. For example, in the case where the signal output from the pixel array 120 is an analog signal and the processing circuit 128 includes an analog-to-digital converter (ADC) that converts the analog signal into a digital signal, the ADC may be included as part of the imaging unit 12, or the ADC may be included as part of the signal processing unit 16.

[0090] Although only one horizontal drive circuit 124 is shown, each of the horizontal drive circuit that selects the first pixel 121 and the horizontal drive circuit that selects the second pixel 122 may be included. Similarly, for the vertical drive circuit 126, each of the vertical drive circuit that selects the column of the first pixel 121 and the vertical drive circuit that selects the column of the second pixel 122 may be included.

[0091] Similarly, for the processing circuit 128, each of the processing circuit that receives and processes the signal from the first pixel 121 and the processing circuit that receives and processes the signal from the second pixel 122 may be included.

[0092] In this way, the imaging unit 12 can obtain image information and event detection information by driving and processing the first pixel 121 and the second pixel 122 that form the pixel array 120 for each frame. That is, the imaging unit 12 can obtain image information and event detection information as a frame image.

[0093] On the other hand, a form may be adopted in which the signal from the first pixel 121 is obtained for each frame, and the signal from the second pixel 122 is obtained at the timing when an event is detected.

[0094] Figure 4 is a block diagram schematically showing another example of the imaging unit according to an embodiment. As Figure 4 shown, the output of the second pixel 122 can be processed by the horizontal arbiter 130 and the vertical arbiter 132. As an example of the path followed by the signal from the second pixel 122, the path indicated by the dotted line is shown.

[0095] In this way, a form may be adopted in which the imaging unit 12 can process the signal from the first pixel 121 as a frame image and obtain the output from the second pixel 122 at the timing when an event is detected in the second pixel 122.

[0096] Processed by the signal processing unit 16 from Figure 2 or Figure 4The image information and event detection information output by the imaging unit 12 in this form. Generally, the aberration caused by the optical system 100 occurs in the light obtained via the optical system 100. As an aberration that can occur as an error as large as an image, there is a distortion aberration. In the present disclosure, the correction of the distortion aberration will be described, but for the processing that can perform position correction through a so-called image and for other aberrations, similar processing can also be performed for signal processing.

[0097] The signal processing unit 16 performs signal processing for correcting the distortion aberration of the image information obtained from the first pixel 121. In addition, the signal processing unit 16 can selectively perform signal processing for correcting the distortion aberration of the event detection information obtained from the second pixel 122 in a manner similar to correcting the image information obtained from the first pixel 121. This selection can be a setting that always performs the distortion correction of the event detection information, or can be a setting that can be switched on / off by the user. In addition, a form in which the solid-state imaging device 1 determines conditions and determines whether to perform distortion correction on the event detection information can be adopted.

[0098] The processing of the signal processing unit 16 will be described in detail.

[0099] Figure 5 It is a diagram schematically showing a series of processes from light reception to data processing in the solid-state imaging device 1. The solid-state imaging device 1 performs processing related to image information and processing related to event detection information based on the signals respectively output from the first pixel 121 and the second pixel 122 included in the pixel array 120.

[0100] The signal processing unit 16 (a part of its processing can be implemented by the imaging unit 12) can, for example, convert the analog signal obtained through the first pixel 121 into a digital signal and interpolate defects at the position of the second pixel 122. Thereafter, the signal processing unit 16 obtains multiple pieces of image information for each color by performing demosaicing processing, color matrix processing, etc. As needed, the signal processing unit 16 performs distortion correction on this image information.

[0101] On the other hand, the signal processing unit 16 processes the signal output from the second pixel 122 in parallel. In the figure, ○ represents event information in the positive direction, and × represents event information in the negative direction.

[0102] In the case of obtaining the output of the second pixel 122 by the arbiter method, the signal processing unit 16 can perform conversion to frame format data (event grid) at the same timing as the frame for obtaining the image information via the first pixel 121. In the case of obtaining the output of the second pixel 122 by the frame method indicated by the dashed arrow, the signal processing unit 16 can set the data obtained for each frame as event detection information (event grid).

[0103] In addition, as indicated by the dashed arrow, the signal processing unit 16 can process, in the next stage, the coordinate information of the signal for determining whether the event is in the positive direction or the negative direction in the event detection information obtained by the arbiter method, without generating an event grid.

[0104] The signal processing unit 16 performs distortion correction on the event detection information of the frame method or the event detection information of the arbiter method as needed.

[0105] The signal processing unit 16 can implement distortion correction of the image information or the event detection information based on, for example, the following expression.

[0106] [Mathematical Expression 1]

[0107]

[0108] Here, (x1, y1) are the image coordinates after distortion correction, and (x2, y2) are the image coordinates before distortion correction. (x1, y1) can also be the coordinates in a coordinate system that is converted from world coordinates to camera coordinates using external parameters and then to image coordinates using internal parameters. The radial distortion coefficients of the lens are represented by k1 and k2, and the circumferential distortion coefficients of the lens are represented by p1 and p2. Incidentally, the distortion can be barrel distortion, pincushion distortion, or a distortion obtained by combining these distortions.

[0109] Thereafter, the signal processing unit 16 performs appropriate signal processing using the image information that has undergone distortion correction or not undergone distortion correction and the event detection information that has undergone distortion correction or not undergone distortion correction. This signal processing can be processing using, for example, a spiking neural network (SNN). This processing is not limited to processing using an SNN, and as needed, the signal processing unit 16 can perform appropriate processing on the image information using the event detection information that has undergone distortion correction.

[0110] As described above, according to this embodiment, the solid-state imaging device 1 can implement signal processing including distortion correction of the event detection information obtained as necessary. By performing distortion correction on the event detection information, it is possible to implement signal processing and image processing that add the event detection information using the same coordinate system to the coordinate information in the image information.

[0111] (Second Embodiment)

[0112] In the above embodiment, the overall process of the solid-state imaging device 1 has been described. In the second embodiment, the distortion processing of the obtained event detection information will be described in detail.

[0113] Figure 6It is a flowchart showing the processing performed by the solid-state imaging device 1 according to an embodiment. This flowchart shows the process in the case where floating-point operations (floating-point operations) are possible in the processing of event detection information. In addition, the following flowchart can also handle the case where fixed-point execution processing is possible.

[0114] The solid-state imaging device 1 obtains information on reflected light, transmitted light, or emitted light from various targets in the imaging area in the imaging unit 12 through the first pixel 121 and the second pixel 122 in the pixel array 120 (S100).

[0115] The signal processing unit 16 appropriately preprocesses the output obtained from the first pixel 121 (S102). As described above, the preprocessing may include conversion from an analog signal to a digital signal, defect correction, demosaicing, color matrix processing, etc. Incidentally, a form in which the imaging unit 12 performs part of the processing to send the processed data to the signal processing unit 16 can be adopted.

[0116] The signal processing unit 16 determines whether distortion correction is set to be performed as specified by the user or by the solid-state imaging device 1 or is not set to be performed as specified by the user or by the solid-state imaging device 1 (S104). The setting of distortion correction can be specified by the user, or can be determined and specified by the solid-state imaging device 1 based on information such as a video signal.

[0117] In the case where distortion correction is set to be performed (S104: Yes), the signal processing unit 16 performs distortion correction on the preprocessed image information and performs distortion correction on the image obtained from the luminance information (S106). For example, distortion correction is performed based on the above expressions (1) to (3).

[0118] On the other hand, signal processing is also performed on the event detection information. The signal processing unit 16 appropriately preprocesses the output obtained from the second pixel 122 (S108). This preprocessing may include, for example, processing of converting the information obtained in the arbiter format into an event grid for each frame.

[0119] The signal processing unit 16 determines whether distortion correction is set to be performed as specified by the user or by the solid-state imaging device 1 or is not set to be performed as specified by the user or by the solid-state imaging device 1 (S110). The setting of distortion correction can be specified by the user, or can be determined and specified by the solid-state imaging device 1 based on information such as a video signal. In addition, in the case where distortion correction is performed on the image information, the signal processing unit 16 may be set to automatically perform distortion correction on the event detection signal.

[0120] In the case where distortion correction is set to be performed (S110: Yes), the signal processing unit 16 performs distortion correction on the preprocessed event detection signal and performs distortion correction on the event detection information obtained from the change information regarding the luminance (S112).

[0121] Figure 7 FIG. is a flowchart showing an example of the process in S112 according to one embodiment.

[0122] The signal processing unit 16 determines whether to adjust the size of the event detection information (S1120). The determination of adjusting the size can be preset. In this case, according to the setting, the process of S1122 or the following S1126 can be performed by a software implementation method or a hardware implementation method without performing the process of S1120.

[0123] In the case where size adjustment is performed (S1120: Yes), the signal processing unit 16 performs a process of adjusting the resolution of the data preprocessed as event detection information to match the coordinates in the image information. A general method can be used to convert the resolution. Through this conversion of the resolution, the event detection information can be arranged on the same coordinate system as the image information at the same scaling ratio. The event detection information can use floating-point representation.

[0124] The signal processing unit 16 performs distortion correction on the event detection information with the resolution converted (S1124). The signal processing unit 16 can perform this distortion correction according to expressions (1) to (3) having coefficients common to the coefficients of the image information. In this case, the signal processing unit 16 can obtain event detection information that has undergone distortion correction in the same coordinate system as the image information by using high-resolution event detection information.

[0125] In this way, the signal processing unit 16 can convert the resolution of the event detection information into the resolution of the image information and perform distortion correction on the converted event detection information.

[0126] In the case where readjustment of the size is not performed (S1120: No), the signal processing unit 16 performs distortion correction considering the scaling ratio of the event detection information (S1126). For example, the signal processing unit 16 can set the image coordinates before distortion correction to (x'2, y'2) and apply expressions (1) to (3) to the coordinates (x2, y2) scaled by the following conversion expression. Incidentally, the same distortion coefficient as the distortion coefficient of the distortion correction of the above image information can be used similarly.

[0127] [Mathematical Expression 2]

[0128] x2 = Nx'2 (4)

[0129] y2 = Ny′2 (5)

[0130] Also in this case, the event detection information can be represented using floating-point numbers.

[0131] In this way, the signal processing unit 16 can perform distortion correction while taking into account the scaling ratio that considers the difference between the resolution based on the event detection information and the resolution of the image information. For example, N in expressions (4) and (5) can be determined based on the above-mentioned predetermined ratio.

[0132] Return Figure 6 , after performing distortion correction or when distortion correction is not set (S104: No, S110: No), the signal processing unit 16 can perform arbitrary post-processing (S114) based on the obtained image information and event detection information. The post-processing can be, for example, processing using the above-mentioned SNN or the like, processing using another learning model, or any processing without using a learning model.

[0133] When both the image information and the event detection information are subjected to distortion correction, based on these pieces of information that have undergone distortion correction, the coordinates of the event detection information are matched with the coordinates of the image information, and then, processing using various learning models such as SNN, tracking processing, etc. can be realized.

[0134] The signal processing unit 16 can appropriately output data (S116) after performing the post-processing. This output can be in the form of outputting to the outside of the solid-state imaging device 1 via the input / output I / F 18, or can be in the form of storing the data in the storage unit 14.

[0135] As described above, according to the present embodiment, distortion correction can be appropriately performed on the event detection information, and processing for more precisely matching the coordinates with the image information can be performed.

[0136] (Third Embodiment)

[0137] In the above embodiment, the case where the event detection information can be represented by floating-point numbers has been described, but in some cases, the event detection information may not be represented by floating-point numbers. In addition, even when the event detection information can be represented as floating-point numbers, it may be desirable to obtain only the flag of the event detection information in subsequent processing. In this case, there is a possibility of problems such as time and arithmetic costs occurring when performing operations using floating-point numbers as in the second embodiment. In the third embodiment, an implementation method that can omit such floating-point operations will be described.

[0138] The signal processing unit 16 can set a region of interest on the image by, for example, processing such as object detection in the signal processing unit 16 or being specified by the user. The signal processing unit 16 can implement event detection for this region of interest.

[0139] Figure 8 It is a flowchart showing the processing of the solid-state imaging device 1 according to an embodiment.

[0140] The signal processing unit 16 sets a region of interest in the image information (S200). The signal processing unit 16 may set, as the region of interest, the region designated by the user via the input / output I / F 18. The signal processing unit 16 may set, as the region of interest, the region including the target designated by the user via the input / output I / F 18. In addition, the signal processing unit 16 may also detect the target by any process and set the region of interest based on the detection result.

[0141] The signal processing unit 16 may obtain the region of interest in the coordinate system of the event detection information that has not undergone distortion correction by performing inverse distortion correction on the region of interest in the image information (S202). Incidentally, at the timing of this inverse distortion correction, the signal processing unit 16 may also perform coordinate conversion in consideration of the resizing and scaling described in the above second embodiment as needed.

[0142] Based on the result, the signal processing unit 16 obtains the region of interest in the event detection information (S204).

[0143] In this way, the solid-state imaging device 1 can also convert the region of interest in the image that has undergone distortion correction into the region of interest in the event detection information that has not undergone distortion correction.

[0144] Figure 9 It is a diagram showing an example of setting a region of interest according to an embodiment. For example, the solid-state imaging device 1 obtains video information including a person or image information continuous in time series as a target. The signal processing unit 16 automatically extracts the eye region from the image that has undergone distortion correction and sets the region of interest ROIi. As another example, the user may specify an arbitrary position in the image, and the signal processing unit 16 may set the region of interest ROIi based on the specification.

[0145] The signal processing unit 16 can obtain information about the region of interest ROIe in the coordinate system of the event detection information that has not undergone distortion correction by performing inverse distortion conversion on the region of interest ROIi. In addition, in the figure, the region of interest ROIe is represented by a rectangle, but it is not limited thereto. In the case where there is distortion in the barrel type, bobbin type, or a combination thereof, information about the region of interest ROIe can also be obtained according to the shape of the distortion.

[0146] The signal processing unit 16 can perform tracking of the target included in the region of interest ROIi set in the image by obtaining the region of interest ROIe in the coordinate system of the event detection information and the event information (not undergoing distortion correction) around the region of interest ROIe.

[0147] Figure 10 It is a flowchart showing an example of processing in the above-described tracking example.

[0148] The signal processing unit 16 determines whether to perform tracking (S300). Whether tracking exists can be specified by the user or automatically specified by the solid-state imaging device 1. In the case where tracking is not performed (S300: No), the following processing does not need to be performed, and a transition to the standby state up to the determination in S300 can be executed.

[0149] In the case where tracking is performed (S300: Yes), the signal processing unit 16 obtains, through the processing of S200 to S204 in Figure 8 the region of interest in the coordinates of the event detection information that has not undergone distortion correction.

[0150] The signal processing unit 16 starts tracking by using the event detection information (S302). For example, in the following processing, the signal processing unit 16 performs tracking on the Figure 9 region of interest ROIe in

[0151] The signal processing unit 16 detects event information inside and around the region of interest ROIe, and calculates the amount of movement from the previous frame to the current frame based on the change in the luminance information in the region of interest ROIe (S304). For example, the signal processing unit 16 performs tracking of the region of interest ROIe from the event detection information in the lower left to the event detection information in the lower right in Figure 9 and obtains the amount of movement or coordinates (with distortion) of the moving target.

[0152] The signal processing unit 16 sets, for the image information before distortion correction, the same region as the region of interest ROIe obtained from the event detection information (in the case of scale adjustment, considering the scale adjustment) as the region of interest, performs distortion correction on the image information including this region of interest, and performs tracking in the image (S306). Thus, the signal processing unit 16 can obtain the region of interest ROIi in the coordinate system that has undergone distortion correction as shown in the upper right of Figure 9

[0153] The signal processing unit 16 continues the processing of S304 and S306 until the tracking ends (S308: No). The end of tracking can be determined according to conditions such as imaging has ended, the user has commanded the end of tracking, etc.

[0154] ​As described above, it is possible to set a region of interest in the image information and implement processing in the set region of interest without performing floating-point operations in the event detection information. For example, in the case of the arbiter format, event detection information can be obtained faster than luminance information (image information), and the computational cost can be lower than the computational cost of luminance information. Therefore, processing related to event detection, such as tracking, can be achieved faster than processing using image information.

[0155] The solid-state imaging device 1 can perform the above-described implementation, for example, in autofocus processing using eye tracking. That is, for example, the solid-state imaging device 1 can perform eye tracking based on event detection information and obtain an image, where the optical system is controlled such that the focus is on the region of interest obtained in the image information. Furthermore, the present disclosure is not limited thereto, and processing that can obtain event detection information faster and apply the event detection information to a frame of image information can be implemented.

[0156] (Fourth Embodiment)

[0157] As described above, the processing after obtaining the event detection information may not correspond to a floating-point operation, for example, using the result as an input to an SNN. In such a case, a form in which data obtained after distortion correction is output without using a floating-point representation can be used.

[0158] For example, the signal processing unit 16 controls and outputs a result that is not floating-point after distortion correction. As an example, after distortion correction, the signal processing unit 16 can select the nearest (x, y) coordinates as the coordinates of the event detection information, obtain the value after converting the event detection information at the coordinates (x, y), and compare the obtained value with a threshold to obtain the event detection result at the coordinates (x, y).

[0159] The signal processing unit 16 can determine that a positive event has occurred when the value of the event detection at the coordinates (x, y) after distortion correction is above a predetermined positive threshold, can determine that a negative event has occurred when the value is below a predetermined negative threshold, and can otherwise determine that no event has occurred.

[0160] The signal processing unit 16 can also output the result as a 1.5-bit result.

[0161] As described above, the signal processing unit 16 can convert the result obtained as a floating point into 1.5-bit data and output the 1.5-bit data. By performing the conversion to such data, the output data can be used as it is when the output data is used.

[0162] (Fifth Embodiment)

[0163] In each of the above embodiments, the form includes hybrid pixels in which the first pixel 121 and the second pixel 122 are arranged in the same pixel array 120. The embodiments in the present disclosure are not limited thereto, and may be in the form of so-called sensor fusion.

[0164] Figure 11 FIG. is a block diagram schematically showing a solid-state imaging device according to an embodiment. The solid-state imaging device 1 includes a first optical system 10A and a second optical system 10B as optical systems, and a first imaging unit 12A and a second imaging unit 12B as imaging units.

[0165] The first imaging unit 12A includes a first pixel array in which first pixels 121 for obtaining luminance information are arranged in a two-dimensional array. The first optical system 10A is arranged such that light is focused on the first pixel array.

[0166] The second imaging unit 12B includes a second pixel array in which second pixels 121 for obtaining information on changes in luminance are arranged in a two-dimensional array. The second optical system 10B is arranged such that light is focused on the second pixel array.

[0167] For example, the signal processing unit 16 can perform processing to adapt the event detection information obtained in the second pixel array to the image information obtained in the first pixel array. With this processing, the signal processing unit 16 can perform processing to match the coordinates of the event detection information with the coordinates of the image information before or after distortion correction.

[0168] In this way, in the form of sensor fusion as well, processing similar to that of the solid-state imaging device 1 described above can be implemented. According to this form, even in an existing device including two imaging systems and performing event detection and image acquisition, distortion correction of the event detection information can be achieved by changing the processing of the signal processing unit 16.

[0169] Incidentally, the solid-state imaging device 1 is not limited to the Figure 11 form. For example, the solid-state imaging device 1 may have a form with one optical system for the first imaging unit 12A and the second imaging unit 12B.

[0170] In these cases, a distortion coefficient can be set for each pixel array as needed.

[0171] Furthermore, the solid-state imaging device 1 may have a form including, for example, a first signal processing unit that processes signals from the first imaging unit 12A and a second signal processing unit that processes signals from the second imaging unit 12B. In this case, a form in which the first signal processing unit and the second signal processing unit can share data as needed can be adopted.

[0172] In any case, each imaging unit may be formed on another semiconductor chip or may be formed on the same semiconductor chip.

[0173] (Sixth Embodiment)

[0174] In each of the above embodiments, for example, the form of performing tracking has been described, but the present disclosure is not limited thereto. The solid-state imaging device 1 may implement, for example, deblurring processing, optical flow processing, motion blur removal processing, etc. based on the image information subjected to distortion correction and the event detection information subjected to distortion correction.

[0175] In addition, of course, the tracking processing described as an example of the region of interest in the above embodiments can be implemented.

[0176] In addition to the above blur correction and tracking processing, the solid-state imaging device 1 may perform arbitrary image processing on the image information by using the event detection information subjected to distortion correction. As another non-limiting example, the solid-state imaging device 1 may also use the event detection information subjected to distortion correction to achieve a high frame rate of the image information, etc.

[0177] Figure 12 is an implementation example of the chip 20 in the solid-state imaging device 1. The chip 20 includes a pixel array region 200, a storage circuit region 202, and a processing circuit region 204 located on the same semiconductor substrate 30. In this way, the pixel array region 200, the storage circuit region 202, and the processing circuit region 204 can be provided on one semiconductor substrate 30. Each component is connected by appropriate wires, etc.

[0178] The pixel array region 200 is a region in which the pixel array 120 is arranged. The storage circuit region 202 is a region in which at least a part of the storage unit 14 is arranged. The processing circuit region 204 is a region in which at least the signal processing unit 16 is configured.

[0179] Figure 13 is another implementation example different from the above. The chip 20 may be implemented on a first semiconductor layer 31 and a second semiconductor layer 32 that are different semiconductor layers. The first semiconductor layer 31 has the pixel array region 200, and the second semiconductor layer 32 has the storage circuit region 202 and the processing circuit region 204. The first semiconductor layer 31 and the second semiconductor layer 32 are stacked to form an integrated semiconductor device and operate. For example, the first semiconductor layer 31 is arranged closer to the optical system 100 than the second semiconductor layer 32, the light passing through the optical system 100 is received by the first semiconductor layer 31, and the signal is output to the second semiconductor layer 32.

[0180] Figure 14This is another implementation example different from the above. The chip 20 can be implemented on a first semiconductor layer 31, a second semiconductor layer 32, and a third semiconductor layer 33 that are different semiconductor layers. The first semiconductor layer 31 has a pixel array region 200, the second semiconductor layer 32 has a storage circuit region 202, and the third semiconductor layer 33 has a processing circuit region 204. The first semiconductor layer 31, the second semiconductor layer 32, and the third semiconductor layer 33 are stacked to form an integrated semiconductor device and operate. For example, the first semiconductor layer 31 is arranged closest to the optical system 100, the light passing through the optical system 100 is received by the first semiconductor layer 31, and the signal is output to at least one of the second semiconductor layer 32 or the third semiconductor layer 33.

[0181] In Figure 3 and Figure 4 In the case of the form shown, for example, the chip - on - chip (CoC) method can be adopted, in which the semiconductor layers are cut out from a wafer, the semiconductor layers are divided into individual chips, and then stacked and vertically joined to each other. In addition, the chip - on - wafer (CoW) method can be adopted, in which any layer is cut and divided into individual chips and then joined to the wafer. Alternatively, the wafer - on - wafer (WoW) method of joining the wafers to each other and then dividing them into individual chips can also be adopted.

[0182] For joining the semiconductor layers, through - holes, micro - bumps, micro - particles, plasma bonding, etc. can be used as non - limiting examples. By this method, the individual semiconductor layers are properly electrically connected and formed to be able to send and receive signals.

[0183] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can also be implemented as a device included in any type of mobile body (such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.).

[0184] Figure 15 This is a block diagram showing an example of the schematic configuration of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. In Figure 15In the illustrated example, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 that connects the multiple control units to each other can be an in-vehicle communication network conforming to any standard, such as Controller Area Network (CAN), Local Interconnect Network (LIN), Local Area Network (LAN), FlexRay (registered trademark), etc.

[0185] Each control unit includes: a microcomputer that performs arithmetic processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control target devices. Each control unit further includes: a network interface (I / F) for performing communication with other control units via the communication network 7010; and a communication I / F for performing communication with devices, sensors, etc. inside and outside the vehicle through wired communication or radio communication. Figure 15 The functional configuration of the illustrated integrated control unit 7600 includes a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication I / F, a storage unit, etc.

[0186] The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 serves as a control device to control: a driving force generation device for generating the driving force of the vehicle, such as an internal combustion engine, a drive motor, etc.; a driving force transmission mechanism for transmitting the driving force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; a braking device for generating the braking force of the vehicle, etc. The drive system control unit 7100 can have the functions of control devices such as an Anti-lock Braking System (ABS) and Electronic Stability Control (ESC).

[0187] The drive system control unit 7100 is connected to a vehicle state detection unit 7110. The vehicle state detection unit 7110 includes, for example, at least one of the following: a gyro sensor that detects the angular velocity of the axial rotational movement of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, and sensors for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle of the steering wheel, the engine speed, or the rotational speed of the wheels, etc. The drive system control unit 7100 performs arithmetic processing using the signals input from the vehicle state detection unit 7110 to control the internal combustion engine, the drive motor, the electric power steering device, the braking device, etc.

[0188] The vehicle body system control unit 7200 controls the operations of various devices assembled on the vehicle body according to various programs. For example, the vehicle body system control unit 7200 serves as a control device to control: a keyless entry system, a smart key system, an electric window device, or various lights such as headlamps, reverse lamps, brake lamps, turn signal lamps, and fog lamps. In this case, the vehicle body system control unit 12020 can receive radio waves or signals from various switches sent by a mobile device of a substitute key as inputs. The vehicle body system control unit 7200 receives these input radio waves or signals to control the vehicle door lock device, the electric window device, the lights, etc.

[0189] The battery control unit 7300 controls the secondary battery 7310 that serves as the power source for the drive motor according to various programs. For example, the battery control unit 7300 receives information regarding battery temperature, battery output voltage, remaining battery charge, etc. from a battery device including the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, executes temperature regulation control of the secondary battery 7310, or controls a cooling device of the battery device, etc.

[0190] The vehicle exterior information detection unit 7400 detects information on the exterior of the vehicle including the vehicle control system 7000. For example, the vehicle exterior information detection unit 7400 is connected to at least one of an imaging unit 7410 and a vehicle exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The vehicle exterior information detection unit 7420 may include at least one of the following: an environmental sensor for detecting the current atmospheric conditions or weather conditions, a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle including the vehicle control system 7000.

[0191] The environmental sensor may be, for example, at least one of the following: a rain drop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunshine, and a snow sensor for detecting snowfall. The surrounding information detection sensor may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR device (light detection and ranging device, or laser imaging detection and ranging device). Each of the imaging unit 7410 and the vehicle exterior information detection unit 7420 may be set as an independent sensor or device, or may be set as a device in which multiple sensors or devices are integrated.

[0192] Figure 16An example of the installation positions of the imaging unit 7410 and the out-of-vehicle information detection unit 7420 is shown. The imaging units 7910, 7912, 7914, 7916, and 7918 can be arranged at positions such as the front nose of the vehicle 7900, the side mirrors, the rear bumper, the rear door, and the upper part of the windshield inside the vehicle. The imaging unit 7910 arranged at the front nose and the imaging unit 7918 arranged at the upper part of the windshield inside the vehicle mainly obtain images in front of the vehicle 7900. The imaging units 7912 and 7914 arranged at the side mirrors mainly obtain images on the sides of the vehicle 7900. The imaging unit 7916 arranged at the rear bumper or the rear door mainly obtains images behind the vehicle 7900. The imaging unit 7918 arranged at the upper part of the windshield inside the vehicle is mainly used to detect preceding vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0193] Incidentally, Figure 16 An example of the imaging ranges of the respective imaging units 7910, 7912, 7914, and 7916 is shown. The imaging range a represents the imaging range of the imaging unit 7910 arranged at the front nose. The imaging ranges b and c respectively represent the imaging ranges of the imaging units 7912 and 7914 arranged at the side mirrors. The imaging range d represents the imaging range of the imaging unit 7916 arranged at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 7900 observed from above can be obtained by superimposing the image data imaged by the imaging units 7910, 7912, 7914, and 7916.

[0194] The out-of-vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 arranged at the front, rear, sides, corners of the vehicle 7900, and the upper part of the windshield inside the vehicle can be ultrasonic sensors or radar devices. The out-of-vehicle information detection units 7920, 7926, and 7930 arranged at the front nose of the vehicle 7900, the rear bumper of the vehicle 7900, the rear door, and the upper part of the windshield inside the vehicle can be LIDAR devices. These out-of-vehicle information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0195] Return to Figure 15, continue the description. The vehicle exterior information detection unit 7400 causes the imaging unit 7410 to image the exterior of the vehicle and receives the imaged image data. In addition, the vehicle exterior information detection unit 7400 receives detection information from the vehicle exterior information detection unit 7420 connected to the vehicle exterior information detection unit 7400. When the vehicle exterior information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. Based on the received information, the vehicle exterior information detection unit 7400 can perform processing of detection objects (such as people, vehicles, obstacles, signs, symbols, etc. on the road surface), or perform processing of the distance to the detected object. The vehicle exterior information detection unit 7400 can perform environment recognition processing based on the received information to recognize rainfall, fog, road surface conditions, etc. The vehicle exterior information detection unit 7400 can calculate the distance to an object outside the vehicle based on the received information.

[0196] In addition, based on the received image data, the vehicle exterior information detection unit 7400 can perform image recognition processing for identifying objects (such as people, vehicles, obstacles, signs, symbols, etc. on the road surface), or perform processing of the distance to the detected object. The vehicle exterior information detection unit 7400 can perform processing such as distortion correction and alignment on the received image data, and generate a bird's-eye view image or a panoramic image by combining the image data imaged by multiple different imaging units 7410. The vehicle exterior information detection unit 7400 can perform viewpoint conversion processing using the image data imaged by different imaging units 7410.

[0197] The vehicle interior information detection unit 7500 detects information inside the vehicle. The vehicle interior information detection unit 7500 may be connected to a driver state detection unit 7510 that detects the state of the driver. The driver state detection unit 7510 may include a camera that photographs the driver, a biosensor that detects the biological information of the driver, a microphone that collects the sound inside the vehicle, etc. The biosensor may be arranged on the seat surface, the steering wheel, etc., and detects the biological information of the passenger sitting in the seat or the driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the vehicle interior information detection unit 7500 can calculate the fatigue level of the driver or the degree of concentration of the driver's attention, or can distinguish whether the driver is dozing off. The vehicle interior information detection unit 7500 can perform processing such as noise cancellation processing on the audio signal obtained by sound collection.

[0198] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is a device that can be operated by a passenger for input, such as a touch panel, buttons, a microphone, switches, a control lever, etc. The integrated control unit 7600 can receive data obtained by performing speech recognition on the speech input via the microphone. The input unit 7800 can be a remote control device using infrared rays or other radio waves, or can be an external connection device such as a mobile phone, a personal digital assistant (PDA), etc. that supports the operation of the vehicle control system 7000. The input unit 7800 can be a camera. In this case, the passenger can input information through gestures. Or, data obtained by detecting the movement of a wearable device worn by the passenger can be input. In addition, the input unit 7800 may include an input control circuit, etc., which generates an input signal based on the information input using the above input unit 7800 by a passenger, etc., and outputs the generated input signal to the integrated control unit 7600. A passenger, etc., can input various data and processing operation instructions to the vehicle control system 7000 by operating the input unit 7800.

[0199] The storage unit 7690 may include a read-only memory (ROM) that stores various programs executed by the microcomputer and a random access memory (RAM) that stores various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 can be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc.

[0200] The general communication I / F 7620 is a widely used communication I / F that mediates communication with various devices existing in the external environment 7750. The general communication I / F 7620 can implement: cellular communication protocols such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), LTE-Advanced (LTE-A), etc., or other wireless communication protocols such as Wireless LAN (also known as Wi-Fi (registered trademark)), Bluetooth (registered trademark), etc. The general communication I / F 7620 can be connected to devices (such as an application server or a control server) existing on an external network (such as the Internet, a cloud network, or a company-specific network) via a base station or an access point. In addition, the general communication I / F 7620 can use peer-to-peer (P2P) technology to connect to a terminal existing near the vehicle (the terminal is, for example, a driver, a pedestrian, or a terminal of a store, or a machine type communication (MTC) terminal).

[0201] The dedicated communication I / F 7630 is a communication I / F that supports communication protocols developed for vehicle use. The dedicated communication I / F 7630 can implement: standard protocols, such as wireless access in vehicle environments (WAVE) (which is a combination of Institute of Electrical and Electronics Engineers (IEEE) 802.11p as the lower layer and IEEE 1609 as the upper layer), dedicated short-range communication (DSRC), or cellular communication protocols. The dedicated communication I / F 7630 generally performs V2X communication including one or more of the following concepts: communication between vehicle and vehicle (vehicle-to-vehicle), communication between road and vehicle (vehicle-to-infrastructure), communication between vehicle and home (vehicle-to-home), and communication between pedestrian and vehicle (vehicle-to-pedestrian).

[0202] The positioning unit 7640 can perform positioning by receiving global navigation satellite system (GNSS) signals from GNSS satellites (for example, GPS signals from Global Positioning System (GPS) satellites) and generating position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 can identify the current position by exchanging signals with a wireless access point, or can obtain position information from a terminal such as a mobile phone, a personal handyphone system (PHS), or a smart phone with a positioning function.

[0203] The beacon receiving unit 7650 can receive radio waves or electromagnetic waves transmitted from radio stations installed on roads or the like, thereby obtaining information about the current position, traffic jams, road closures, required time, etc. Incidentally, the function of the beacon receiving unit 7650 can be included in the above-mentioned dedicated communication I / F 7630.

[0204] The in-vehicle device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using wireless communication protocols such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC), or wireless universal serial bus (WUSB). In addition, the in-vehicle device I / F 7660 can establish a wired connection via connection terminals (and cables if necessary) not shown in the figure through universal serial bus (USB), high-definition multimedia interface (HDMI (registered trademark)), mobile high-definition link (MHL), etc. The in-vehicle devices 7760 can include at least one of the following: mobile devices and wearable devices owned by passengers and information devices loaded into or attached to the vehicle. The in-vehicle devices 7760 can also include a navigation device that searches for a path to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0205] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals, etc. in accordance with a predetermined protocol supported by the communication network 7010.

[0206] Based on the information obtained via at least one of the general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon reception unit 7650, in-vehicle device I / F 7660, and in-vehicle network I / F 7680, the microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs. For example, based on the obtained information related to the inside or outside of the vehicle, the microcomputer 7610 can calculate control target values for the driving force generation device, steering mechanism, or braking device, and output control commands to the drive system control unit 7100. For example, the microcomputer 7610 can execute cooperative control for functions aimed at implementing an advanced driver assistance system (ADAS), which includes collision avoidance or impact buffering for the vehicle, following driving based on the inter-vehicle distance, vehicle speed holding driving, vehicle collision alarm, alarm for the vehicle deviating from the lane, etc. In addition, based on the obtained information about the vehicle surrounding environment, the microcomputer 7610 can control the driving force generation device, steering mechanism, and braking device to execute cooperative control for automatic driving, etc. that does not rely on the driver's operation.

[0207] Based on the information obtained via at least one of the general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon reception unit 7650, in-vehicle device I / F 7660, and in-vehicle network I / F 7680, the microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures, people, etc., and generate local map information including information about the surrounding environment where the vehicle is currently located. In addition, based on the obtained information, the microcomputer 7610 can predict dangers such as vehicle collisions, approach of pedestrians, etc., and entry into a closed road, and generate an alarm signal. This alarm signal can be a signal for generating a warning sound or lighting an alarm lamp.

[0208] The sound / image output unit 7670 sends an output signal of at least one of sound and image to an output device that can notify information visually or auditorily to the passengers of the vehicle or to the outside of the vehicle. Figure 15In the example, the audio speaker 7710, the display unit 7720, and the instrument panel 7730 are shown as output devices. The display unit 7720 may include at least one of an in-vehicle display and a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be other devices other than these devices, such as headphones, wearable devices such as glasses-type displays worn by passengers, projectors, lights, etc. When the output device is a display device, the display device visually displays the results obtained by various processes executed by the microcomputer 7610, or displays information in various forms (such as text, images, tables, curves, etc.) received from other control units. In addition, when the output device is an audio output device, the audio output device converts an audio signal composed of played audio data or sound data, etc. into an analog signal and outputs the analog signal auditorily.

[0209] Incidentally, in Figure 15 the example shown, at least two control units connected to each other via the communication network 7010 may be integrated into one control unit. Alternatively, each individual control unit may include a plurality of control units. In addition, the vehicle control system 7000 may include other control units not shown in the figure. In addition, part or all of the functions executed by one of the control units in the above description may be assigned to another control unit. That is, predetermined arithmetic processing may be executed by any one of the control units as long as information is transmitted and received via the communication network 7010. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and the plurality of control units may transmit and receive detection information to and from each other via the communication network 7010.

[0210] Incidentally, a computer program for implementing each function (specifically, the signal processing unit 16) of the solid-state imaging device 1 according to the present embodiment described with reference to Figures 1 to 14 can be implemented on any control unit or the like. In addition, a computer-readable recording medium storing such a computer program may also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed via, for example, a network without using a recording medium.

[0211] The above embodiment may have the following forms. (1)

[0213] A solid-state imaging device, comprising:

[0214] an optical system;

[0215] a first pixel that obtains image information based on luminance information via the optical system;

[0216] A second pixel that obtains event detection information based on a change in luminance information via an optical system;

[0217] One or more pixel arrays, wherein the first pixel and the second pixel are arranged in a two-dimensional array; and

[0218] A signal processing circuit that selects whether to perform distortion correction on the event detection information obtained by the second pixel and performs signal processing. (2)

[0220] The solid-state imaging device according to (1), wherein

[0221] One or more pixel arrays include one pixel array, and

[0222] The second pixel is arranged in a predetermined region in the pixel array at a predetermined ratio with respect to the first pixel. (3)

[0224] The solid-state imaging device according to (2), wherein

[0225] The predetermined region includes the entire region of the pixel array. (4)

[0227] The solid-state imaging device according to (2) or (3), wherein

[0228] The second pixels are arranged at equal intervals so as to cover the entire region of the predetermined region. (5)

[0230] The solid-state imaging device according to any one of (1) to (4), wherein

[0231] The second pixel detects events for each frame. (6)

[0233] The solid-state imaging device according to any one of (1) to (4), wherein

[0234] The second pixel detects events by an arbiter method. (7)

[0236] The solid-state imaging device according to any one of (1) to (6), wherein

[0237] The signal processing circuit performs distortion correction on the image information. (8)

[0239] The solid-state imaging device according to (7), wherein

[0240] The signal processing circuit

[0241] Converts the resolution of the event detection information into the resolution of the image information, and

[0242] Perform distortion correction on the converted event detection information. (9)

[0244] The solid-state imaging device according to (7), wherein

[0245] Signal processing circuit

[0246] Perform distortion correction on the event detection information based on the difference between the resolution of the event detection information and the resolution of the image information. (10)

[0248] The solid-state imaging device according to any one of (1) to (9), wherein

[0249] Signal processing circuit

[0250] Set a region of interest based on the image information, and

[0251] Obtain event detection information corresponding to the region of interest. (11)

[0253] The solid-state imaging device according to (10), wherein

[0254] Signal processing circuit

[0255] Perform tracking based on the event detection information that has undergone distortion correction corresponding to the region of interest. (12)

[0257] The solid-state imaging device according to any one of (1) to (11), wherein

[0258] Signal processing circuit

[0259] Compare the value obtained after performing distortion correction on the event detection information with the coordinate threshold after performing distortion correction on the event detection information, and obtain the event detection information at the coordinates after performing distortion correction. (13)

[0261] The solid-state imaging device according to (12), wherein

[0262] Signal processing circuit

[0263] Store the event detection information at the coordinates after performing distortion correction as 1.5-bit information. (14)

[0265] The solid-state imaging device according to (1), wherein

[0266] One or more pixel arrays include:

[0267] A first pixel array, wherein the first pixels are arranged in a two-dimensional array; and

[0268] A second pixel array, wherein the second pixels are arranged in a two-dimensional array, and the second pixel array obtains event detection information of the same target as that of the first pixel array. (15)

[0270] The solid-state imaging device according to (14), wherein

[0271] The optical system includes:

[0272] A first optical system that focuses light on the first pixel array; and

[0273] A second optical system that focuses light on the second pixel array. (16)

[0275] The solid-state imaging device according to any one of (1) to (15), wherein

[0276] The signal processing circuit includes:

[0277] A first signal processing circuit that performs signal processing on the output from the first pixels; and

[0278] A second signal processing circuit that performs signal processing on the output from the second pixels. (17)

[0280] The solid-state imaging device according to any one of (1) to (16), wherein

[0281] The signal processing circuit

[0282] Performs image processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction. (18)

[0284] The solid-state imaging device according to (17), wherein

[0285] The signal processing circuit

[0286] Performs deblurring processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction. (19)

[0288] The solid-state imaging device according to (17) or (18), wherein

[0289] The signal processing circuit

[0290] Performs high frame rate processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction. (20)

[0292] The solid-state imaging device according to any one of (1) to (19), wherein,

[0293] Signal processing circuit

[0294] Performs tracking processing based on the image information subjected to distortion correction and the event detection information subjected to distortion correction.

[0295] Aspects of the present disclosure are not limited to the above embodiments and include various conceivable variations. The effects of the present disclosure are not limited to the above either. Components in each embodiment can be appropriately combined and applied. That is, various additions, modifications, and partial deletions can be made without departing from the conceptual idea and gist of the present disclosure derived from the content defined in the claims and their equivalents.

[0296] List of reference symbols

[0297] 1 Solid-state imaging device

[0298] 10 Optical system

[0299] 12 Imaging unit

[0300] 120 Pixel array

[0301] 121 First pixel

[0302] 122 Second pixel

[0303] 124 Horizontal drive circuit

[0304] 126 Vertical drive circuit

[0305] 128 Processing circuit

[0306] 14 Storage unit

[0307] 16 Signal processing unit

[0308] 18 Input / output I / F

[0309] 20 Chip

[0310] 200 Pixel array region

[0311] 202 Storage circuit region

[0312] 204 Processing circuit region

[0313] 31 First semiconductor layer

[0314] 32 Second semiconductor layer

[0315] 33 Third semiconductor layer.

Claims

1. A solid-state imaging device, comprising: an optical system; a first pixel that obtains image information based on luminance information via the optical system; a second pixel that obtains event detection information based on a change in the luminance information via the optical system; one or more pixel arrays, wherein the first pixel and the second pixel are arranged in a two-dimensional array; and a signal processing circuit that selects whether to perform distortion correction on the event detection information obtained by the second pixel and performs signal processing.

2. The solid-state imaging device according to claim 1, wherein the one or more pixel arrays include one pixel array, and the second pixel is arranged in a predetermined area in the pixel array at a predetermined ratio with respect to the first pixel.

3. The solid-state imaging device according to claim 2, wherein the predetermined area includes the entire area of the pixel array.

4. The solid-state imaging device according to claim 2, wherein the second pixels are arranged at equal intervals so as to cover the entire area of the predetermined area.

5. The solid-state imaging device according to claim 1, wherein the second pixel detects events in each frame.

6. The solid-state imaging device according to claim 1, wherein the second pixel detects events by an arbiter method.

7. The solid-state imaging device according to claim 1, wherein the signal processing circuit performs distortion correction on the image information.

8. The solid-state imaging device according to claim 7, wherein the signal processing circuit converts the resolution of the event detection information into the resolution of the image information, and performs distortion correction on the converted event detection information.

9. The solid-state imaging device according to claim 7, wherein the signal processing circuit performs the distortion correction on the event detection information based on a difference between the resolution of the event detection information and the resolution of the image information.

10. The solid-state imaging device according to claim 1, wherein the signal processing circuit sets a region of interest based on the image information, and obtains the event detection information corresponding to the region of interest.

11. The solid-state imaging device according to claim 10, wherein the signal processing circuit performs tracking based on the event detection information that has undergone distortion correction corresponding to the region of interest.

12. The solid-state imaging device according to claim 1, wherein the signal processing circuit compares a value obtained by performing the distortion correction on the event detection information with a coordinate threshold after performing distortion correction on the event detection information, and obtains the event detection information at the coordinates after performing the distortion correction.

13. The solid-state imaging device according to claim 12, wherein the signal processing circuit stores the event detection information at the coordinates after performing the distortion correction as 1.5-bit information.

14. The solid-state imaging device according to claim 1, wherein the one or more pixel arrays include: a first pixel array, wherein the first pixels are arranged in a two-dimensional array; and A second pixel array, wherein the second pixels are arranged in a two-dimensional array, and the second pixel array obtains the event detection information of the same target as the target of the first pixel array.

15. The solid-state imaging device according to claim 14, wherein, The optical system includes: A first optical system that focuses light on the first pixel array; and A second optical system that focuses light on the second pixel array.

16. The solid-state imaging device according to claim 1, wherein, The signal processing circuit includes: A first signal processing circuit that performs signal processing on the output from the first pixel; and A second signal processing circuit that performs signal processing on the output from the second pixel.

17. The solid-state imaging device according to claim 1, wherein, The signal processing circuit Performs image processing based on the image information subjected to the distortion correction and the event detection information subjected to the distortion correction.

18. The solid-state imaging device according to claim 17, wherein, The signal processing circuit Performs deblurring processing based on the image information subjected to the distortion correction and the event detection information subjected to the distortion correction.

19. The solid-state imaging device according to claim 1, wherein, The signal processing circuit Performs tracking processing based on the image information subjected to the distortion correction and the event detection information subjected to the distortion correction.

Images