Driving assistance system

Abstract

To provide a travel support system suitable for a vehicle which performs semiautomatic driving, and an electronic apparatus for vehicles.SOLUTION: A travel support system comprises: an imaging device capable of picking up a first black-and-white image in a direction of travel of a vehicle; a first neural network for segmentation processing; a second neural network for depth estimation processing; a determination section for determining a center of a portion to be segmented from the first black-and-white image on the basis of the segmentation processing and the depth estimation processing; a third neural network for performing coloring processing only on a segmented second black-and-white image; and a display device for performing enlarged display of the second black-and-white image on which the coloring processing is performed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] One aspect of the present invention relates to a neural network and an imaging system using the same. Another aspect of the present invention relates to an electronic device using a neural network. Another aspect of the present invention relates to a vehicle using a neural network. Another aspect of the present invention relates to an imaging system that obtains a color image from a black and white image obtained with a solid-state image sensor using image processing technology. Another aspect of the present invention relates to a video surveillance system, security system, safety information provision system, or driving support system using the imaging system.

[0002] Furthermore, one aspect of the present invention is not limited to the above-mentioned technical fields. One aspect of the invention disclosed herein relates to a product, a method, or a method of manufacture. One aspect of the present invention relates to a process, a machine, a manufacture, or a composition of matter. More specifically, examples of technical fields of one aspect of the present invention disclosed herein include semiconductor devices, display devices, light-emitting devices, energy storage devices, memory devices, electronic devices, lighting devices, input devices, input / output devices, methods for driving them, or methods for manufacturing them. One aspect of the present invention relates to a vehicle or electronic equipment for vehicles installed in a vehicle.

[0003] In this specification, the term "semiconductor device" refers to any device that can function by utilizing semiconductor properties. Transistors and semiconductor circuits are examples of semiconductor devices. Furthermore, memory devices, display devices, imaging devices, and electronic devices may also contain semiconductor devices.

[0004] Furthermore, one aspect of the present invention relates to a program that uses a neural network.

[0005] Furthermore, one aspect of the present invention relates to a driving system that allows a vehicle, such as an automobile, to freely switch between a safety-assisted driving state, a semi-autonomous driving state, or an autonomous driving state. [Background technology]

[0006] A technology for constructing transistors using oxide semiconductor thin films formed on a substrate is attracting attention. For example, Patent Document 1 discloses an imaging device in which an oxide semiconductor transistor with an extremely low off-current is used in the pixel circuit.

[0007] Furthermore, a technology for adding a calculation function to an imaging device is disclosed in Patent Document 2. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2011-119711 [Patent Document 2] Japanese Patent Publication No. 2016-123087 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] With imaging devices equipped with solid-state image sensors such as CMOS image sensors, technological advancements have made it easy to capture high-quality images. In the next generation, there is a demand for imaging devices to incorporate even more intelligent functions.

[0010] One aspect of the present invention aims to provide an imaging device capable of image processing. Alternatively, it aims to provide an imaging device capable of high-speed operation. Alternatively, it aims to provide an imaging device with low power consumption. Alternatively, it aims to provide a highly reliable imaging device. Alternatively, it aims to provide a novel imaging device or the like. Alternatively, it aims to provide a method for driving the above-mentioned imaging device. Alternatively, it aims to provide a novel semiconductor device or the like.

[0011] Furthermore, one aspect of the present invention aims to provide a driving support system and vehicle electronic equipment suitable for vehicles performing semi-autonomous driving.

[0012] Note that the description of these problems does not preclude the existence of other problems. Note that one aspect of the present invention does not necessarily have to solve all of these problems. Note that other problems will become apparent from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc.

Means for Solving the Problems

[0013] The technology of imaging in black and white and colorizing is particularly suitable for imaging at night. For example, at night, there is less light than during the day, and the colors are different from those during the day. Also, in a dark environment where infrared imaging is required, when imaging with infrared rays, a monochrome image is obtained. In addition, in infrared imaging, there is a problem regarding depth in that the brightness is significantly different between a material that reflects infrared rays and a material that absorbs infrared rays, and even when arranged at the same position, the reflecting material appears closer and the absorbing material appears farther away.

[0014] Also, when assuming driving in a dark environment, when the oncoming vehicle's light shines, the visibility of the surroundings becomes poor. In particular, the human eye's pupil becomes small and the surroundings cannot be seen. Therefore, by using the image display by an imaging device with an arbitrarily set exposure as an auxiliary, it becomes easier to ensure the safety of the surroundings. This image is preferably highly visible.

[0015] Such an image is preferably subjected to segmentation processing to colorize and highlight a human or a vehicle in addition to colorization, but since the image in the distance is small and has a small amount of information, the accuracy decreases. In particular, in the case of in-vehicle use, the image may be blurred due to vibration or the like. Particularly when used as an in-vehicle camera, as the vehicle speed increases, information in the distance becomes necessary.

[0016] As the vehicle speed increases, the driver's viewing angle becomes narrower, but there is a contradiction in that a wide viewing angle is required to perform safe driving. Also, it is not practical to reduce the vehicle speed to ensure a wide viewing angle. An environment where safe driving can be performed regardless of whether the speed is low or high is desired.

[0017] Although the driver's viewing angle becomes narrower as the speed increases, in order to assist the driver, a camera is used to automatically image the areas that need attention and display them to the driver, thereby enhancing the safety during vehicle travel.

[0018] On the other hand, it is also possible to enlarge by adjusting the camera lens, but it is difficult for the driver during driving to perform adjustments of the optical system such as enlargement and reduction. Also, it is difficult for the driver to change the lens orientation to move the lens focus and provide a mechanism to change the imaging direction. Further, even when imaging a distant location with a telephoto lens, the object may be lost due to vibrations during travel. It is desired to automatically extract and display only important areas without depending on the driver and without changing the focal length or orientation of the camera. Important areas are, for example, areas where there are moving vehicles or pedestrians on a distant road.

[0019] During vehicle driving, a driving support system that pinpoints and displays the areas that should arouse attention and supports the driver's burden is desired. Also, a display system that enlarges when the area that should arouse attention is distant is desired.

[0020] The configuration of the invention disclosed in this specification is a driving support system having an imaging device that can image a first black-and-white image in the traveling direction of the vehicle, a first neural network for segmentation processing, a second neural network for depth estimation processing, a determination unit that determines the center of the portion to be cut out from the first black-and-white image based on the segmentation processing and the depth estimation processing, a third neural network for colorizing only the cut-out second black-and-white image, and a display device for enlargedly displaying the colorized second black-and-white image.

[0021] In the above configuration, the driving assistance system is preferably equipped in a vehicle and therefore uses multiple trained neural networks, and preferably has one or more memory units that store programs for executing them. One or more processors are installed in the vehicle and execute these neural networks.

[0022] Furthermore, the driving support system includes the steps of: driving a vehicle equipped with an imaging device; capturing a grayscale image of the area in front of the moving vehicle using the imaging device; performing segmentation processing on the grayscale image including the distant region to infer the regions of at least the sky, vehicles, and roads; performing depth estimation processing on the grayscale image including the distant region to infer a specific distant region; determining the center of the portion to be cut from the grayscale image based on the segmentation processing and depth estimation processing; extracting a rectangular region with the center as the central part, inputting the extracted data, and performing super-resolution processing; inputting the output result of the super-resolution processing and performing colorization processing to accurately highlight objects included in the distant region; and enlarging and displaying the colorized distant region. The specific distant region refers to the region including at least the edge of the road in the direction of travel.

[0023] In addition to the above steps, the system may also include a step of measuring the vehicle's speed. The size of the image to be cropped can be changed according to the vehicle's speed. For example, the size of the rectangular area centered on the part to be cropped from the black and white image can be determined by the vehicle's speed. The cropped area should be larger at higher speeds than at lower speeds. This helps to compensate for the narrowing of the driver's field of view due to speed.

[0024] One or more processors can read and execute a program that includes one or all of the above steps. The program that causes the computer to execute each step is stored in memory beforehand. Furthermore, it is not limited to processors, but can also be executed by circuits (e.g., FPGA circuits, ASIC circuits) that implement the functionality to execute one or all of the above steps.

[0025] In the above configuration, segmentation processing uses the first neural network processing, depth estimation processing uses the second neural network processing, super-resolution processing uses the third neural network processing, and colorization processing uses the fourth neural network processing. For training the segmentation processing, MSCOCO, Cityscapes, etc., can be used as the training dataset. For training the depth estimation processing, KITTI, etc., can be used as the training dataset. The training dataset for training the super-resolution processing is not particularly limited and may include not only photographs but also illustrations. The training dataset for training the colorization processing is not particularly limited as long as it is in color; ImageNet, processed color images from a dashcam, etc., can be used.

[0026] To explain the aforementioned driving assistance system in more detail, it reduces color information, performs neural network processing on the reduced amount of captured information, and further reduces the amount of data by cropping the distant region, colorizing only the distant region, and displaying it enlarged. Reducing color information reduces the amount of data, and also simplifies the computational processing in the neural network. Furthermore, if the amount of data can be reduced, the hardware capable of performing neural network processing can be made smaller. Imaging devices without color filters not only reduce color information, but also have no reduction in light due to color filters, making it easier to ensure sufficient light reaching the light receiving sensor, thus expanding the dynamic range.

[0027] Furthermore, the imaging system and driving support system disclosed herein, along with the device or vehicle, may also be referred to as an image generation device. The image generation device selectively colorizes and enlarges a portion of a wide dynamic range black and white image captured by an imaging device without a color filter.

[0028] Furthermore, images colorized through colorization often do not have natural colors, which acts as an accent, making them easier for drivers to recognize.

[0029] Furthermore, the system is not limited to imaging devices that do not use color filters. In addition to imaging devices that do not use color filters, the system may also be constructed in combination with imaging devices that have color filters, or with other environmental recognition units, such as stereo cameras, sonar, multifocal multi-camera systems, LIDAR, millimeter-wave radar, and infrared sensors (TOF method). TOF distance measurement consists of a light source and a photodetector (sensor or camera). The camera used in this TOF method is called a time-of-flight camera, or simply a TOF camera. A TOF camera can obtain distance information from a light-emitting light source to an object based on the time of flight of the reflected light from the object (time of flight). [Effects of the Invention]

[0030] By using multiple neural networks, it is possible to enlarge and display areas that require attention for the driver. This primarily provides image displays to assist the driver.

[0031] Furthermore, it has a particularly remarkable effect in environments with insufficient light, such as during twilight, nighttime, early morning hours, and when passing through long tunnels, as it can provide the driver with a clear color display of the distant area in the direction of travel. [Brief explanation of the drawing]

[0032] [Figure 1] Figure 1 is a diagram showing an example of a flowchart illustrating one aspect of the present invention. [Figure 2] Figure 2 shows an example of a flowchart illustrating one aspect of the present invention. [Figure 3] Figure 3 shows an example of a flowchart illustrating one aspect of the present invention. [Figure 4] Figure 4 is a block diagram showing one embodiment of the present invention. [Figure 5] Figure 5 is a block diagram showing one embodiment of the present invention. [Figure 6] Figure 6 is a block diagram showing an example of the configuration of the imaging unit. [Figure 7] Figure 7 shows an example configuration of the pixel block 200 and circuit 201. [Figure 8] Figure 8 shows an example of a pixel configuration. [Figure 9] Figures 9A to 9C show filters. [Figure 10] Figure 10A shows an example of a pixel configuration. Figures 10B to 10D show examples of a photoelectric conversion device configuration. [Figure 11] Figure 11 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 12] Figures 12A to 12C are cross-sectional views showing examples of transistor configurations. [Figure 13] Figure 13 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 14] Figure 14 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 15] Figures 15A1 to 15A3 are perspective views of the package containing the imaging device, and Figures 15B1 to 15B3 are perspective views of the module. [Figure 16] Figure 16A shows the exterior of the vehicle, and Figure 16B is a schematic diagram showing the driver's field of view from inside the vehicle. [Figure 17] Figure 17 shows an example of an application product to which an imaging system according to one embodiment of the present invention can be applied. [Modes for carrying out the invention]

[0033] Embodiments of the present invention will be described in detail below with reference to the drawings. However, it will be readily apparent to those skilled in the art that the present invention is not limited to the following description, and its form and details can be modified in various ways. Furthermore, the present invention is not to be interpreted as being limited to the embodiments described below.

[0034] (Embodiment 1) In this embodiment, Figure 1 shows an example of the flow of a driving assistance system that selectively extracts distant regions that the driver should pay attention to from a black and white image obtained by a solid-state image sensor mounted in the vehicle, colors a portion of those regions, and provides the driver with an enlarged image.

[0035] An imaging system, which includes a solid-state image sensor and other hardware components, is installed on a part of the vehicle (such as the hood, interior, or roof) that can capture images in the direction of travel (including the distant area). The system is then activated and continuous shooting begins.

[0036] The in-vehicle hardware has a configuration in which one or more processors primarily control the operation at each step. When performing neural network processing, hardware is required that has sufficient memory or other storage to store training data and sufficient computational processing capabilities. The storage unit refers to a small, high-capacity storage device (e.g., SSD or hard disk) that can be installed in the vehicle. By executing a program stored in the small, high-capacity storage device, the processing shown in the flowchart in Figure 1 can be realized.

[0037] First, start acquiring data (S1).

[0038] Black and white image data is acquired using a solid-state image sensor without a color filter (S2). Note that a set of multiple solid-state image sensors arranged in a matrix direction is sometimes called a pixel array. Also, the left side of Figure 1 shows an example display of the captured image immediately after step S2 (21). However, this example is illustrated for illustrative purposes only and is not actually displayed. In reality, it is black and white image data converted to a signal format (such as JPEG®).

[0039] Next, the distant region, which is part of the captured image data, is extracted and cropped (S3). The distant region in the direction of the vehicle's movement is the region that is most difficult for the driver to recognize when driving in dark conditions, such as in the evening, at night, or in the early morning. A solid-state image sensor without a color filter can capture images with a wider dynamic range than a solid-state image sensor with a color filter, so objects in the distant region can be captured as images.

[0040] In this embodiment, the distant region in the direction of vehicle travel is selectively extracted. The extraction method involves identifying the location using depth inference or data from a depth sensor module (such as a TOF camera) that measures depth. When depth inference is used, neural network processing is performed. As an example of extracting the distant region in step S3, the area enclosed by the dotted line in Figure 1 is shown as the extracted distant region 22. By performing cropping in step S3, the amount of data used can be reduced.

[0041] Furthermore, in step S3, the size to be cut may be determined using speedometer data. The speedometer is a device that can acquire signals from GPS and GLONASS satellites. The speed, position, or distance traveled of a moving vehicle can be measured using GPS or the like.

[0042] Next, the data is reduced to only the data in the far region 22 (S4). In step S4, data other than the data extracted in S3 is deleted. The original data may be stored in a dedicated storage device.

[0043] Subsequently, colorization inference is performed on the data of the far region 22 only (S5). In this embodiment, the data reduced in step S4 is used as input data, and a convolution process is performed using a CPU or the like to infer contours, colors, etc., and then colorize the data.

[0044] Furthermore, when colorizing a black and white image using software, the software components may be installed from a computer with built-in hardware, or from a network or recording medium. The program is installed from a recording medium such as a computer-readable CD-ROM (Compact Disk Read Only Memory), and the program for colorizing the black and white image is executed. The processing performed by the program is not limited to sequential processing; it does not have to be chronological, and may, for example, be performed in parallel.

[0045] Furthermore, the software program that executes the inference program for performing neural network processing used for depth inference or colorization inference can be written in various programming languages ​​such as Python, Go, Perl, Ruby, Prolog, Visual Basic, C, C++, Swift, Java (registered trademark), and .NET. Applications may also be created using frameworks such as Chainer (available in Python), Caffe (available in Python and C++), and TensorFlow (available in C, C++, and Python). For example, the LSTM algorithm can be programmed in Python and used with a CPU (Central Processor Unit) or GPU (Graphics Processing Unit). A chip that integrates both a CPU and GPU is sometimes called an APU (Accelerated Processing Unit), and this APU chip can also be used. Alternatively, an IC (also called an inference chip) incorporating an AI system may be used. An IC incorporating an AI system is sometimes called a circuit (microprocessor) that performs neural network calculations.

[0046] In this embodiment, since inference is performed inside the vehicle, pre-trained features (colorization weights) are used. By storing the pre-trained features in memory and performing calculations, data output at a level comparable to that without using pre-trained features can be achieved. Because pre-reduced data is used, the burden of computation is reduced. Figure 1 shows an example of the display after colorization as colorized image 23.

[0047] The resulting colorized image 23 is based on a wide dynamic range black and white image obtained using an image sensor without a color filter. Therefore, even in cases where the light level is low and the image is indistinguishable with conventional image sensors that have a color filter, it is possible to obtain identifiable colorized image data.

[0048] Finally, the colorized image 23 is enlarged and displayed as an enhanced image 24 on the display unit of the display device (S6). While it can be called an enhanced image simply by enlarging it, in this embodiment, further colorization is performed using colorization inference, resulting in an image that is close to natural colors but different from the actual video, thus creating an enhanced image. Note that the order of steps S6 and S5 does not matter, as the result is the same, so either can be performed first.

[0049] The above enhancement image acquisition process is repeated. By repeating this process, it is possible to display only the distant region as the enhancement image in real time.

[0050] The above-described driving assistance system can clearly capture images even in relatively dark places and provide the driver with highlighted images of areas that the driver should pay attention to. This driving assistance system can help prevent accidents, especially in dimly lit areas such as evenings and nights.

[0051] Furthermore, the aforementioned driving assistance system can also be applied to vehicles capable of semi-autonomous driving or fully autonomous driving by combining a camera or radar that captures images of the area around the vehicle with an ECU (Electronic Control Unit) that performs image processing. Vehicles using electric motors have multiple ECUs, which control the engine and other functions. ECUs include microcomputers. ECUs are connected to a CAN (Controller Area Network) installed in the electric vehicle. CAN is one of the serial communication standards used as an in-vehicle LAN. ECUs use a CPU or GPU. For example, one of the multiple cameras installed in an electric vehicle (such as a drive recorder camera or rear camera) can be a solid-state image sensor without a color filter. A portion of the obtained monochrome image can be extracted, inferred by the ECU via CAN, a colorized image can be created, and the enhanced image can be displayed on the display screen of the in-vehicle display device after enlargement.

[0052] Furthermore, in this embodiment, when using data from a depth sensor module (such as a TOF camera) to measure depth to identify the area to be extracted as a method for selectively extracting distant regions, the colorization inference will only require one neural network processing step. In this case, since only one neural network processing step is performed, the computational processing is small, which has the advantage of reducing the load on the CPU.

[0053] Furthermore, when using depth inference as a method for selectively extracting distant regions, the first neural network processing is performed by depth inference, and colorization inference is performed by the second neural network processing. In this case, the installation of a depth sensor module to measure depth can be eliminated. Although there are two neural network processing steps, the computational burden is reduced by reducing the input data for the colorization neural network processing.

[0054] (Embodiment 2) This embodiment shows an example where the method for extracting the distant region differs from that of Embodiment 1. Since many other parts are the same as those of Embodiment 1, a detailed explanation will be omitted here.

[0055] Figure 2 shows an example of the flow of the driving support system. Note that the same reference numerals are used for the same steps as in the flowchart shown in Figure 1 of Embodiment 1.

[0056] First, start acquiring data (S1).

[0057] Black and white image data is acquired using a solid-state image sensor without a color filter (S2).

[0058] Next, depth inference is performed using the grayscale image data to select the far region, which is part of the captured image data (S21A). Segmentation inference is also performed using the grayscale image data (21B).

[0059] Regarding depth inference, an example is also shown in Embodiment 1. In this embodiment, since inference is performed inside the vehicle, pre-trained features (depth weights) are used.

[0060] Segmented inference is also called segmentation. Segmentation refers to the process of identifying what object each pixel in an input image represents. It is also called semantic segmentation. Software that generates multiple image segments for use in image analysis is executed using neural network processing. Specifically, segmentation is performed based on learned data using image processing and a type of convolutional neural network (CNN), such as U-net or FCRN (Fully Convolutional Residual Networks). The segmentation labels are distinguished by terms such as vehicle, sky, plant, ground, human, and building. In this embodiment, since inference is performed inside a vehicle, pre-trained features (segmentation weights) are used.

[0061] Steps S21A and S21B may be performed sequentially or in parallel.

[0062] Next, focusing on the sky and road regions obtained by segmentation inference, we extract the coordinates of the upper edge of the road where the distance between the sky and the road (the gap between the lower edge of the sky and the upper edge of the road) is shortest within the segmented image. This is because, as shown in display example 21 in Figure 1, the area around the leading edge of the driving lane corresponds to the point where the gap between the sky and the road is shortest within the image.

[0063] Within a segmented image, there may be more than one coordinate on the road where the distance between the sky and the road is shortest.

[0064] If there are multiple coordinates on the road within the segmented image that result in the shortest distance between the sky and the road, the coordinate furthest from them is selected (S22).

[0065] From the ground region obtained in step S21B, the location of the maximum depth in the depth image within that region is extracted from the results of step 21A. By identifying the leading edge of the driving lane in the grayscale image, the central part of the distant region to be cropped later can be determined.

[0066] Next, the data is reduced to include only the distant region (S23).

[0067] Subsequently, colorized inference is performed on data only from the far-field region (S24). In this embodiment, since inference is performed inside the vehicle, pre-trained features (colorization weights) are used.

[0068] Finally, the colorized image is enlarged and displayed as an enhanced image on the display unit of the device (S25).

[0069] The process of acquiring the enhanced image described above is repeated.

[0070] In this embodiment, as a method for selectively extracting distant regions, the first neural network processing is performed using depth inference, the second neural network processing is performed using segmentation inference, and the third neural network processing is performed using colorization inference. Since all of these inferences use pre-trained features, the calculations can be performed inside the vehicle.

[0071] This embodiment can be freely combined with Embodiment 1.

[0072] (Embodiment 3) This embodiment shows an example where the method for extracting the distant region differs from that of Embodiment 2. Since many other parts are the same as those of Embodiment 1 or Embodiment 2, a detailed explanation will be omitted here.

[0073] Figure 3 shows an example of the flow of the driving support system. Note that the same reference numerals are used for the same steps as in the flowcharts shown in Figure 1 of Embodiment 1 and Figure 2 of Embodiment 2.

[0074] Black and white image data is acquired using a solid-state image sensor without a color filter (S2).

[0075] Next, depth inference is performed using the grayscale image data to select the far region, which is part of the captured image data (S21A). Segmentation inference is also performed using the grayscale image data (21B).

[0076] Next, we select the far region (S22).

[0077] Furthermore, vehicle speed data is acquired (S26). The speedometer is a device that can acquire signals from GPS and GLONASS satellites. The speed, position, or distance traveled of a moving vehicle can be measured using GPS, etc. The vehicle speed data may also be a value obtained from a general speedometer. Note that the timing of step S26 is not particularly limited and can be anytime between the start of data acquisition and step S22.

[0078] Next, the cropping size for the distant area is determined (S27). The size of the cropping is determined using the speedometer data obtained in step S26. For example, a configuration is used in which a wider area is cropped when the speed is higher than when the speed is lower.

[0079] Next, the data is reduced to include only the data from the far region (S28).

[0080] Subsequently, super-resolution inference is performed on data only from the distant region (S29). Super-resolution processing refers to image processing that generates high-resolution images from low-resolution images. Super-resolution processing may be repeated multiple times. By mixing not only color photographic images but also color illustration (animation) images as training data to create a learning model for discriminating color boundaries, it is possible to obtain colorized image data with clearly defined color boundaries. Therefore, it is preferable to mix illustration (animation) images as training data during learning, and super-resolution weights, which will be the weight coefficients for neural network processing, are calculated in advance. In this embodiment, since inference is performed inside the vehicle, pre-trained features (super-resolution weights) are used.

[0081] Next, colorization inference is performed (S30). In this embodiment, since inference is performed inside the vehicle, pre-trained features (colorization weights) are used.

[0082] The order in which super-resolution inference is performed followed by colorization inference is also important. It is preferable to perform calculations using grayscale image data and then perform colorization as the final image processing step. Grayscale image data has less data volume than image data with color information, which reduces the burden on the processing power of the computing device.

[0083] Finally, the colorized image is enlarged and displayed as an enhanced image on the display unit of the display device (S31).

[0084] The process of acquiring the enhanced image described above is repeated.

[0085] In this embodiment, as a method for selectively extracting distant regions, the first neural network processing is performed using depth inference, the second neural network processing is performed using segmentation inference, the third neural network processing is performed using super-resolution inference, and the fourth neural network processing is performed using colorization inference. Since all of these inferences use pre-trained features, calculations can be performed inside the vehicle. When the speed is high, a wide range of areas is magnified and super-resolution is applied, so clear highlighting can be obtained.

[0086] This embodiment can be freely combined with Embodiment 1 or Embodiment 2.

[0087] (Embodiment 4) This embodiment shows an example of a block diagram of the driving support system 31 that executes the flowchart of Embodiment 2. Figure 4 is a block diagram of the driving support system 31 and will be described below with reference to it.

[0088] The data acquisition device 10 is a semiconductor chip including a solid-state image sensor 11 and a memory unit 14, and does not have a color filter. The data acquisition device 10 has an optical system such as a lens. The optical system can have any configuration as long as its imaging characteristics are known, and is not particularly limited.

[0089] For example, the data acquisition device 10 may use a single semiconductor chip formed by stacking a back-illuminated CMOS image sensor chip, a DRAM chip, and a logic circuit chip. Alternatively, it may use a single semiconductor chip formed by stacking a back-illuminated CMOS image sensor chip and a logic circuit chip including an analog-to-digital conversion circuit, in which case the memory section 14 is SRAM. Furthermore, the stacked chips can be stacked and electrically connected using known bonding techniques.

[0090] The memory section 14 is a circuit that stores the converted digital data, and is configured to store the data before inputting it to the neural network sections 16a, 16b, and 16c, but is not limited to this configuration.

[0091] The neural network sections 16a, 16b, and 16c are implemented by software computations performed by a microcontroller. A microcontroller is a computer system integrated into a single integrated circuit (IC). If the computational scale or the amount of data to be handled is large, multiple ICs may be combined to configure the neural network sections 16a, 16b, and 16c. These multiple ICs are included in the learning device at a minimum. Furthermore, a microcontroller equipped with Linux® is preferable because it allows the use of free software, thereby reducing the total cost for configuring the neural network sections 16a, 16b, and 16c. However, it is not limited to Linux®, and other operating systems (OS) may also be used.

[0092] The learning process for the neural network units 16a, 16b, and 16c shown in Figure 4 is described below. In this embodiment, the neural network is pre-trained and then processed using the weights. The training data can also be stored in the memory units 18a, 18b, and 18c and trained as needed.

[0093] In Embodiment 2, data reduction is performed by selecting only the far-field region based on the output results of the neural network units 16a and 16b. The data extraction unit 17 can also be described as a data control unit that selects only the far-field region and performs data reduction. The data extracted by the data extraction unit 17 is input to the neural network unit 16c and colorized.

[0094] The data output from the neural network unit 16c contains contour and color information of the subject only in the distant region, and is input to the display device 15. The display device 15 has a display unit 19 and forms a signal representing an image, including enlarged display, according to the number of displayable gray levels and screen size.

[0095] Alternatively, a passenger's personal information terminal (such as a smartphone) can be used as the display device 15, and the display unit of the personal information terminal can be used as the display unit 19. In this case, a transceiver unit for transmitting the output of the neural network unit 16c to the passenger's personal information terminal must be installed in the vehicle. Of course, it is not limited to passengers; the driver's personal information terminal (such as a smartphone) can also be installed on the hood or elsewhere so that the driver can view the image.

[0096] The above-mentioned driving support system 31 can prevent accidents, especially in dark places with little lighting, such as in the evening or at night.

[0097] This embodiment can be freely combined with Embodiment 1, Embodiment 2, or Embodiment 3.

[0098] (Embodiment 5) This embodiment shows a modified version that changes a part of Embodiment 4. The same reference numerals are used to describe parts identical to those in Embodiment 4. The configuration from the memory section 14 onwards is the same as in Embodiment 4, and is therefore omitted here.

[0099] An example of the configuration of the imaging system 41 will be explained with reference to the block diagram shown in Figure 5.

[0100] The data acquisition device 10 is a semiconductor chip including a solid-state image sensor 11 and an analog arithmetic circuit 12, and does not have a color filter. The data acquisition device 10 has an optical system such as a lens. The optical system can have any configuration as long as its imaging characteristics are known, and is not particularly limited.

[0101] Furthermore, the analog arithmetic circuit 12 can use a transistor (hereinafter referred to as an OS transistor) made of a metal oxide formed on the silicon chip of a solid-state image sensor formed using a silicon substrate.

[0102] The A / D circuit 13 (also called an A / D converter) is an analog-to-digital conversion circuit that converts the analog data output from the data acquisition device 10 into digital data. If necessary, an amplification circuit may be provided between the data acquisition device 10 and the A / D circuit 13 to amplify the analog signal before conversion to digital data.

[0103] The memory unit 14 is a circuit that stores the converted digital data, and is configured to store the data before inputting it to the neural network units 16a, 16b, and 16c, but is not limited to this configuration. Depending on the amount of data output from the data acquisition device or the data processing capability of the image processing device, if the data is small in size, the output from the A / D circuit 13 may be input directly to the neural network units 16a, 16b, and 16c without storing it in the memory unit 14.

[0104] In this embodiment, the analog arithmetic circuit shown in Figure 5 allows some of the calculations performed in common by the neural network sections 16a, 16b, and 16c to be performed in advance. By using the imaging system 41 shown in Figure 5, the number of calculations performed by the neural network sections 16a, 16b, and 16c can be reduced.

[0105] This embodiment can be freely combined with Embodiment 1, Embodiment 2, Embodiment 3, or Embodiment 4.

[0106] (Embodiment 6) In this embodiment, a configuration example in which the data acquisition device 10 is part of the imaging system 41 is described below. Figure 6 is a block diagram illustrating the imaging system 41.

[0107] The imaging system 41 includes a pixel array 300, and circuits 201, 301, 302, 303, 304, 305, and 306. Note that each of circuits 201 and 301 through 306 is not limited to a single circuit configuration, but may be composed of a combination of multiple circuits. Alternatively, any multiple of the above circuits may be integrated. Furthermore, other circuits may be connected.

[0108] The pixel array 300 has imaging and calculation functions. Circuits 201 and 301 have calculation functions. Circuit 302 has either a calculation function or a data conversion function. Circuits 303, 304, and 306 have selection functions. Circuit 303 is electrically connected to the pixel block 200 via wiring 424. Circuit 304 is electrically connected to the pixel block 200 via wiring 423. Circuit 305 has the function of supplying potential for multiply-accumulate calculations to the pixels. A shift register or decoder can be used for the circuit with the selection function. Circuit 306 is electrically connected to the pixel block 200 via wiring 413. Note that circuits 301 and 302 may be provided externally.

[0109] The pixel array 300 has a plurality of pixel blocks 200. As shown in Figure 7, each pixel block 200 has a plurality of pixels 400 arranged in a matrix, and each pixel 400 is electrically connected to a circuit 201 via wiring 412. The circuit 201 can also be provided within the pixel block 200.

[0110] Furthermore, each pixel 400 is electrically connected to an adjacent pixel 400 via a transistor 450 (transistor 450a to transistor 450f). The function of transistor 450 will be described later.

[0111] Pixel 400 can acquire image data and generate data by adding the image data to a weight coefficient. In Figure 7, the number of pixels in pixel block 200 is shown as 3x3 as an example, but it is not limited to this. For example, it can be 2x2, 4x4, etc. Alternatively, the number of pixels in the horizontal and vertical directions may be different. In addition, some pixels may be shared by adjacent pixel blocks. In Figure 7, an example is shown in which 10 transistors 450 (transistors 450a to 450j) are provided between pixels 400, but the number of transistors 450 may be increased further. Also, in transistors 450g to 450j, some transistors may be omitted to eliminate parallel paths. Wirings 413g to 413j are connected as gates to transistors 450g to 450j, respectively.

[0112] The pixel block 200 and circuit 201 can be operated as a multiply-accumulate circuit.

[0113] As shown in Figure 8, the pixel 400 may have a photoelectric conversion device 401, a transistor 402, a transistor 403, a transistor 404, a transistor 405, a transistor 406, and a capacitor 407.

[0114] One electrode of the photoelectric conversion device 401 is electrically connected to either the source or the drain of transistor 402. The other electrode of the source or drain of transistor 402 is electrically connected to either the source or the drain of transistor 403, the gate of transistor 404, and one electrode of capacitor 407. One electrode of the source or drain of transistor 404 is electrically connected to either the source or the drain of transistor 405. The other electrode of capacitor 407 is electrically connected to either the source or the drain of transistor 406.

[0115] The other electrode of the photoelectric conversion device 401 is electrically connected to the wiring 414. The other source or drain of transistor 403 is electrically connected to the wiring 415. The other source or drain of transistor 405 is electrically connected to the wiring 412. The other source or drain of transistor 404 is electrically connected to the GND wiring, etc. The other source or drain of transistor 406 is electrically connected to the wiring 411. The other electrode of capacitor 407 is electrically connected to the wiring 417.

[0116] The gate of transistor 402 is electrically connected to wire 421. The gate of transistor 403 is electrically connected to wire 422. The gate of transistor 405 is electrically connected to wire 423. The gate of transistor 406 is electrically connected to wire 424.

[0117] Here, node FD is defined as the electrical connection point between the other source or drain of transistor 402, one source or drain of transistor 403, one electrode of capacitor 407, and the gate of transistor 404. Also, node FDW is defined as the electrical connection point between the other electrode of capacitor 407 and one source or drain of transistor 406.

[0118] Wires 414 and 415 can function as power lines. For example, wire 414 can function as a high-potential power line, and wire 415 can function as a low-potential power line. Wires 421, 422, 423, and 424 can function as signal lines that control the conduction of each transistor. Wire 411 can function as a wire that supplies a potential corresponding to a weighting coefficient to pixel 400. Wire 412 can function as a wire that electrically connects pixel 400 and circuit 201. Wire 417 can function as a wire that electrically connects the other electrode of capacitor 407 of one pixel to the other electrode of capacitor 407 of another pixel via transistor 450 (see Figure 7).

[0119] An amplification circuit or a gain adjustment circuit may be electrically connected to wiring 412.

[0120] A photodiode can be used as the photoelectric conversion device 401. Any type of photodiode is acceptable, including Si photodiodes with silicon as the photoelectric conversion layer, and organic photodiodes with an organic photoconductive film as the photoelectric conversion layer. Furthermore, if you wish to improve light detection sensitivity at low light levels, it is preferable to use an avalanche photodiode.

[0121] Transistor 402 may have the function of controlling the potential of node FD. Transistor 403 may have the function of initializing the potential of node FD. Transistor 404 may have the function of controlling the current that circuit 201 flows according to the potential of node FD. Transistor 405 may have the function of selecting pixels. Transistor 406 may have the function of supplying a potential corresponding to a weighting coefficient to node FDW.

[0122] When an avalanche photodiode is used in the photoelectric conversion device 401, a high voltage may be applied, and it is preferable to use a high-voltage transistor for the transistor connected to the photoelectric conversion device 401. For example, a transistor using a metal oxide in the channel formation region (hereinafter referred to as an OS transistor) can be used as the high-voltage transistor. Specifically, it is preferable to apply an OS transistor to transistor 402.

[0123] Furthermore, OS transistors also possess the characteristic of extremely low off-current. By using OS transistors for transistors 402, 403, and 406, the period during which charge can be held at node FD and node FDW can be made extremely long. Therefore, a global shutter method that performs charge accumulation operation simultaneously at all pixels can be applied without complicating the circuit configuration or operating method. In addition, it is possible to hold image data at node FD and perform multiple calculations using that image data.

[0124] On the other hand, it is sometimes desirable for transistor 404 to have excellent amplification characteristics. Also, it is sometimes preferable to use a transistor with high mobility that can operate at high speed for transistor 406. Therefore, transistors 404 and 406 may be transistors that use silicon in the channel formation region (hereinafter referred to as Si transistors).

[0125] Furthermore, OS transistors and Si transistors may be applied in any combination, not limited to the above. Alternatively, all transistors may be OS transistors, or all transistors may be Si transistors. Examples of Si transistors include transistors made of amorphous silicon, and transistors made of crystalline silicon (microcrystalline silicon, low-temperature polysilicon, single-crystal silicon).

[0126] The potential of node FD at pixel 400 is determined by the sum of the reset potential supplied from wiring 415 and the potential (image data) generated by photoelectric conversion by photoelectric conversion device 401. Alternatively, it is determined by capacitive coupling with a potential corresponding to a weighting coefficient supplied from wiring 411. Therefore, transistor 405 can supply a current corresponding to data obtained by adding an arbitrary weighting coefficient to the image data.

[0127] Note that the above is just one example of a circuit configuration for 400 pixels, and the photoelectric conversion operation can also be performed with other circuit configurations.

[0128] As shown in Figure 7, each pixel 400 is electrically connected to one another by wiring 412. Circuit 201 can perform calculations using the sum of the currents flowing through the transistors 404 of each pixel 400.

[0129] Circuit 201 includes a capacitor 202, transistors 203, 204, 205, 206, and a resistor 207.

[0130] One electrode of capacitor 202 is electrically connected to either the source or the drain of transistor 203. One of the source or the drain of transistor 203 is electrically connected to the gate of transistor 204. One of the source or the drain of transistor 204 is electrically connected to either the source or the drain of transistor 205. One of the source or the drain of transistor 205 is electrically connected to either the source or the drain of transistor 206. One electrode of resistor 207 is electrically connected to the other electrode of capacitor 202.

[0131] The other electrode of capacitor 202 is electrically connected to wiring 412. The other source or drain of transistor 203 is electrically connected to wiring 218. The other source or drain of transistor 204 is electrically connected to wiring 219. The other source or drain of transistor 205 is electrically connected to a reference power line such as a GND wire. The other source or drain of transistor 206 is electrically connected to wiring 212. The other electrode of resistor 207 is electrically connected to wiring 217.

[0132] Wires 217, 218, and 219 can function as power lines. For example, wire 218 can function as a wire supplying a dedicated potential for reading. Wires 217 and 219 can function as high-potential power lines. Wires 213, 215, and 216 can function as signal lines controlling the conduction of each transistor. Wire 212 is an output line and can be electrically connected to, for example, the circuit 301 shown in Figure 6.

[0133] The transistor 203 can have a function of resetting the potential of the wiring 211 to the potential of the wiring 218. The wiring 211 is a wiring connected to one electrode of the capacitor 202, one of the source or drain of the transistor 203, and the gate of the transistor 204. The transistors 204 and 205 can have a function as a source follower circuit. The transistor 206 can have a function of controlling reading. Note that the circuit 201 has a function as a correlated double sampling circuit (CDS circuit), and can be replaced with a circuit having the same function and other configurations.

[0134] In one aspect of the present invention, an offset component other than the product of the image data (X) and the weight coefficient (W) is removed, and the target WX is extracted. WX can be calculated using data with and without imaging for the same pixel, and data when weights are added to each of them.

[0135] The total current (I p ) flowing through the pixel 400 when imaging is kΣ(X - V th ) 2 , and the total current (I p ) flowing through the pixel 400 when weights are added is kΣ(W + X - V th ) 2 . Also, the total current (I ref ) flowing through the pixel 400 when not imaging is kΣ(0 - V th ) 2 , and the total current (I ref ) flowing through the pixel 400 when weights are added is kΣ(W - V th ) 2 . Here, k is a constant, and V th is the threshold voltage of the transistor 405.

[0136] First, the difference (data A) between the data with imaging and the data with weights added to the data is calculated. kΣ((X - V th ) 2 - (W + X - V th ) 2 ) = kΣ(-W 2 - 2W·X + 2W·Vth )

[0137] Next, we calculate the difference (data B) between the data without imaging and the data with weights added to it. kΣ((0-V th ) 2 -(WV th ) 2 )=kΣ(-W 2 +2W·V th )

[0138] Then, we take the difference between data A and data B. kΣ(-W 2 -2W·X+2W·V th -(-W 2 +2W·V th )) = kΣ(-2W·X). In other words, offset components other than the product of the image data (X) and the weight coefficient (W) can be removed.

[0139] Circuit 201 can read out data A and data B. The difference calculation between data A and data B can be performed, for example, by circuit 301.

[0140] Here, the weights supplied to the entire pixel block 200 function as a filter. As this filter, for example, a convolutional filter of a convolutional neural network (CNN) can be used. Alternatively, an image processing filter such as an edge detection filter can be used. Examples of edge detection filters include the Laplacian filter shown in Figure 9A, the Prewitt filter shown in Figure 9B, and the Sobel filter shown in Figure 9C.

[0141] If the pixel block 200 has 3x3 pixels 400, the elements of the edge extraction filter can be assigned as weights to each pixel 400 and supplied. As mentioned above, data A and data B can be calculated using data with and without imaging, and the data with weights added to each of them. Here, the data with and without imaging is data without weights, which can also be rephrased as data with a weight of 0 added to all pixels 400.

[0142] The edge extraction filters illustrated in Figures 9A to 9C are filters in which the sum of the filter elements (weights: ΔW) (ΣΔW / N, where N is the number of elements) is 0. Therefore, without having to supply ΔW=0 from another circuit, by performing the operation to acquire ΣΔW / N, it is possible to obtain data in which ΔW=0 is added to all 400 pixels.

[0143] This operation is equivalent to making the transistors 450 (transistors 450a to 450f) located between the pixels 400 conduct (see Figure 7). By making the transistors 450 conduct, all node FDWs of each pixel 400 are short-circuited via the wiring 417. At this time, the charge accumulated in the node FDWs of each pixel 400 is redistributed, and when using the edge extraction filters exemplified in Figures 9A to 9C, the potential (ΔW) of the node FDW becomes 0 or approximately 0. Therefore, data equivalent to ΔW=0 can be obtained.

[0144] Furthermore, when supplying charge from a circuit outside the pixel array 300 to rewrite the weight (ΔW), the capacitance of the long wiring 411 and other factors cause the rewriting to take time to complete. On the other hand, the pixel block 200 is a tiny area, and the wiring 417 is short and has small capacitance. Therefore, the operation of redistributing the charge accumulated in node FDW within the pixel block 200 can rewrite the weight (ΔW) at high speed.

[0145] In the pixel block 200 shown in Figure 7, transistors 450a to 450f are electrically connected to different gate lines (wirings 413a to 413f). In this configuration, the conduction of transistors 450a to 450f can be controlled independently, and the operation to acquire ΣΔW / N can be selectively performed.

[0146] For example, when using the filter shown in Figure 9B or Figure 9C, there are pixels that are initially supplied with ΔW=0. Assuming that ΣΔW / N=0, pixels supplied with ΔW=0 may be excluded from the pixels subject to summation. By excluding these pixels, it becomes unnecessary to supply the potential required to operate some of transistors 450a to 450f, thus reducing power consumption.

[0147] The data resulting from the sum-of-accumulate operation output from circuit 201 is sequentially input to circuit 301. In addition to the function of calculating the difference between data A and data B as described above, circuit 301 may have various other calculation functions. For example, circuit 301 can have the same configuration as circuit 201. Alternatively, the functions of circuit 301 may be replaced by software processing.

[0148] Furthermore, circuit 301 may have a circuit that performs calculations on the activation function. For example, a comparator circuit can be used for this circuit. In a comparator circuit, the result of comparing the input data with a set threshold value is output as binary data. That is, the pixel block 200 and circuit 301 can act as elements of a neural network.

[0149] The data output from circuit 301 is sequentially input to circuit 302. Circuit 302 can be configured to include, for example, a latch circuit and a shift register. This configuration allows for parallel-to-serial conversion, and the data input in parallel can be output as serial data to wiring 311.

[0150] [Example of pixel configuration] Figure 10A shows an example of the configuration of pixel 400. Pixel 400 can have a stacked structure of layer 561 and layer 563.

[0151] Layer 561 contains a photoelectric conversion device 401. The photoelectric conversion device 401 may have layers 565a and 565b, as shown in Figure 10B. Note that, depending on the context, "layer" may be replaced with "region."

[0152] The photoelectric conversion device 401 shown in Figure 10B is a pn junction type photodiode, and for example, a p-type semiconductor can be used for layer 565a and an n-type semiconductor for layer 565b. Alternatively, an n-type semiconductor may be used for layer 565a and a p-type semiconductor for layer 565b.

[0153] The above-mentioned pn junction photodiode can typically be formed using single-crystal silicon.

[0154] Furthermore, the photoelectric conversion device 401 in layer 561 may be a stack of layers 566a, 566b, 566c, and 566d, as shown in Figure 10C. The photoelectric conversion device 401 shown in Figure 10C is an example of an avalanche photodiode, where layers 566a and 566d correspond to electrodes, and layers 566b and 566c correspond to the photoelectric conversion section.

[0155] Layer 566a is preferably a low-resistance metal layer or the like. For example, aluminum, titanium, tungsten, tantalum, silver, or a lamination thereof can be used.

[0156] It is preferable to use a conductive layer with high light transmittance to visible light for layer 566d. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, or graphene can be used. It is also possible to omit layer 566d.

[0157] The photoelectric conversion layers 566b and 566c can be configured as a pn junction type photodiode, for example, with a selenium-based material as the photoelectric conversion layer. It is preferable to use a p-type semiconductor selenium-based material for layer 566b and an n-type semiconductor such as gallium oxide for layer 566c.

[0158] Photoelectric conversion devices using selenium-based materials exhibit high external quantum efficiency for visible light. These photoelectric conversion devices utilize avalanche multiplication to significantly amplify electrons in relation to the amount of incident light. Furthermore, selenium-based materials have a high light absorption coefficient, offering production advantages such as the ability to fabricate the photoelectric conversion layer as a thin film. Thin films of selenium-based materials can be formed using methods such as vacuum deposition or sputtering.

[0159] As selenium-based materials, crystalline selenium such as single-crystal selenium or polycrystalline selenium, amorphous selenium, compounds of copper, indium, and selenium (CIS), or compounds of copper, indium, gallium, and selenium (CIGS) can be used.

[0160] n-type semiconductors are preferably formed from materials with a wide bandgap and transparency to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or oxides containing a mixture of these materials can be used. These materials also function as hole injection blocking layers, which can reduce dark current.

[0161] Furthermore, the photoelectric conversion device 401 in layer 561 may be a laminate of layers 567a, 567b, 567c, 567d, and 567e, as shown in Figure 10D. The photoelectric conversion device 401 shown in Figure 10D is an example of an organic photoconductive film, where layer 567a is the lower electrode, layer 567e is the light-transmitting upper electrode, and layers 567b, 567c, and 567d correspond to the photoelectric conversion section.

[0162] One of the layers 567b or 567d in the photoelectric conversion section can be a hole transport layer. The other of the layers 567b or 567d can be an electron transport layer. Layer 567c can be a photoelectric conversion layer.

[0163] For the hole transport layer, for example, molybdenum oxide can be used. For the electron transport layer, for example, C 60 , C 70 Fullerenes such as the above, or their derivatives, can be used.

[0164] As the photoelectric conversion layer, a mixed layer of n-type organic semiconductors and p-type organic semiconductors (bulk heterojunction structure) can be used.

[0165] The layer 563 shown in Figure 10A includes, for example, a silicon substrate. Si transistors and the like are provided on the silicon substrate. Pixels 400 can be formed using these Si transistors. Circuits 201 and 301 to 306 shown in Figure 6 can also be formed.

[0166] Next, the laminated structure of the imaging device will be explained using a cross-sectional view. Note that the insulating layer, conductive layer, and other elements shown below are examples, and other elements may be included. Alternatively, some of the elements shown below may be omitted. Furthermore, the laminated structure shown below can be formed using processes such as bonding and polishing, as needed.

[0167] The imaging device with the configuration shown in Figure 11 has layers 560, 561, and 563. In Figure 11, transistors 402 and 403 are shown as elements provided in layer 563, but other elements such as transistors 404 to 406 can also be provided in layer 563.

[0168] Layer 563 is provided with a silicon substrate 632, insulating layers 633, 634, 635, and 637. A conductive layer 636 is also provided.

[0169] Insulating layers 634, 635, and 637 function as interlayer insulating films and planarizing films. Insulating layer 633 functions as a protective film. Conductive layer 636 is electrically connected to the wiring 414 shown in Figure 8.

[0170] As the interlayer insulating film and planarizing film, for example, inorganic insulating films such as silicon oxide films, or organic insulating films such as acrylic resins and polyimide resins can be used. As the protective film, for example, silicon nitride films, silicon oxide films, aluminum oxide films, etc. can be used.

[0171] The conductive layer may be made from a metal element selected from aluminum, chromium, copper, silver, gold, platinum, tantalum, nickel, titanium, molybdenum, tungsten, hafnium, vanadium, niobium, manganese, magnesium, zirconium, beryllium, indium, ruthenium, iridium, strontium, lanthanum, etc., or from an alloy composed of the above-mentioned metal elements, or from an alloy combining the above-mentioned metal elements, as appropriate. The conductive layer is not limited to a single layer, but may consist of multiple layers made of different materials.

[0172] The Si transistor shown in Figure 11 is a fin-type transistor with a channel formation region on the silicon substrate. A cross-section in the channel width direction (cross-section of A1-A2 shown in layer 563 of Figure 11) is shown in Figure 12A. Note that the Si transistor may also be of the planar type, as shown in Figure 12B.

[0173] Alternatively, as shown in Figure 12C, the transistor may have a semiconductor layer 545 made of a silicon thin film. The semiconductor layer 545 can be, for example, single-crystal silicon (SOI: Silicon on Insulator) formed on an insulating layer 546 on a silicon substrate 632.

[0174] A photoelectric conversion device 401 is provided in layer 561. The photoelectric conversion device 401 can be formed on layer 563. Figure 11 shows a configuration in which the organic photoconductive film shown in Figure 10D is used as the photoelectric conversion layer for the photoelectric conversion device 401. Here, layer 567a is the cathode and layer 567e is the anode.

[0175] Layer 561 is provided with an insulating layer 651, an insulating layer 652, an insulating layer 653, an insulating layer 654, and a conductive layer 655.

[0176] Insulating layers 651, 653, and 654 function as interlayer insulating films and planarizing films. In addition, insulating layer 654 is provided covering the edges of the photoelectric conversion device 401 and also has the function of preventing short circuits between layer 567e and layer 567a. Insulating layer 652 functions as an element isolation layer. It is preferable to use an organic insulating film or the like as the element isolation layer.

[0177] The layer 567a, which corresponds to the cathode of the photoelectric conversion device 401, is electrically connected to either the source or the drain of the transistor 402 in the layer 563. The layer 567e, which corresponds to the anode of the photoelectric conversion device 401, is electrically connected to the conductive layer 636 provided in the layer 563 via the conductive layer 655.

[0178] Layer 560 is formed on layer 561. Layer 560 has a light-shielding layer 671 and a microlens array 673.

[0179] The light-shielding layer 671 can suppress the inflow of light to adjacent pixels. The light-shielding layer 671 can be made of a metal layer such as aluminum or tungsten. Alternatively, a dielectric film having the function of an anti-reflective coating may be laminated with the metal layer.

[0180] A microlens array 673 is provided on the photoelectric conversion device 401. Light passing through each lens of the microlens array 673 is directed onto the photoelectric conversion device 401 directly below it. By providing the microlens array 673, focused light can be incident on the photoelectric conversion device 401, thereby enabling efficient photoelectric conversion. The microlens array 673 is preferably formed of a resin or glass with high light transmittance to the wavelength of light of the object to be imaged.

[0181] Figure 13 shows a modified version of the laminated structure shown in Figure 11, with differences in the configuration of the photoelectric conversion device 401 in layer 561 and a partial configuration of layer 563. In the configuration shown in Figure 13, there is a bonding surface between layer 561 and layer 563.

[0182] Layer 561 includes a photoelectric conversion device 401, insulating layers 661, 662, 664, and 665, as well as conductive layers 685 and 686.

[0183] The photoelectric conversion device 401 is a pn junction type photodiode formed on a silicon substrate, and has a layer 565b corresponding to the p-type region and a layer 565a corresponding to the n-type region. The photoelectric conversion device 401 is an embedded photodiode, and the dark current can be suppressed and noise reduced by a thin p-type region (part of layer 565b) provided on the surface side (current extraction side) of layer 565a.

[0184] The insulating layer 661, and the conductive layers 685 and 686, function as bonding layers. The insulating layer 662 functions as an interlayer insulating film and a planarizing film. The insulating layer 664 functions as an element isolation layer. The insulating layer 665 has the function of suppressing carrier outflow.

[0185] The silicon substrate is provided with grooves to separate pixels, and the insulating layer 665 is provided on the upper surface of the silicon substrate and in these grooves. The provision of the insulating layer 665 prevents carriers generated in the photoelectric conversion device 401 from flowing out to adjacent pixels. The insulating layer 665 also has the function of suppressing the intrusion of stray light. Therefore, the insulating layer 665 can suppress color mixing. An anti-reflective film may be provided between the upper surface of the silicon substrate and the insulating layer 665.

[0186] The element isolation layer can be formed using the LOCOS (LOCal Oxidation of Silicon) method, or it may be formed using the STI (Shallow Trench Isolation) method, etc. For example, the insulating layer 665 can be an inorganic insulating film such as silicon oxide or silicon nitride, or an organic insulating film such as polyimide resin or acrylic resin. The insulating layer 665 may also be a multilayer structure. Furthermore, a configuration without an element isolation layer is also possible.

[0187] Layer 565a (n-type region, corresponding to the cathode) of the photoelectric conversion device 401 is electrically connected to the conductive layer 685. Layer 565b (p-type region, corresponding to the anode) is electrically connected to the conductive layer 686. Conductive layers 685 and 686 have regions embedded in the insulating layer 661. Furthermore, the surfaces of the insulating layer 661, as well as the conductive layers 685 and 686, are flattened so that their heights are the same.

[0188] In layer 563, an insulating layer 638 is formed on the insulating layer 637. In addition, a conductive layer 683 is formed which is electrically connected to either the source or the drain of the transistor 402, and a conductive layer 684 is formed which is electrically connected to the conductive layer 636.

[0189] The insulating layer 638, and the conductive layers 683 and 684, function as bonding layers. The conductive layers 683 and 684 have regions embedded in the insulating layer 638. Furthermore, the surfaces of the insulating layer 638, and the conductive layers 683 and 684 are flattened so that their heights are the same.

[0190] Here, it is preferable that conductive layer 683 and conductive layer 685 are composed of the same metal element as their main component, and it is preferable that conductive layer 684 and conductive layer 686 are composed of the same metal element as their main component. Furthermore, it is preferable that insulating layer 638 and insulating layer 661 have the same main component.

[0191] For example, conductive layers 683 to 686 can be made of Cu, Al, Sn, Zn, W, Ag, Pt, or Au. Due to their ease of bonding, Cu, Al, W, or Au are particularly preferred. In addition, insulating layers 638 and 661 can be made of silicon oxide, silicon oxide nitride, silicon oxide nitride, silicon nitride, titanium nitride, or the like.

[0192] In other words, it is preferable to use the same metal material described above for each of the conductive layers 683 to 686. It is also preferable to use the same insulating material described above for each of the insulating layers 638 and 661. With this configuration, bonding can be performed with the boundary between layer 563 and layer 561 as the bonding position.

[0193] Furthermore, conductive layers 683 to 686 may have a multilayer structure consisting of multiple layers, in which case the surface layer (bonding surface) may be made of the same metal material. Also, insulating layers 638 and 661 may have a multilayer structure consisting of multiple layers, in which case the surface layer (bonding surface) may be made of the same insulating material.

[0194] This bonding allows conductive layer 683 and conductive layer 685 to be electrically connected to each other, and conductive layer 684 and conductive layer 686 to be electrically connected to each other. Furthermore, a mechanically strong connection can be obtained between insulating layer 661 and insulating layer 638.

[0195] For joining metal layers, a surface activation bonding method can be used, in which the oxide film and impurity adsorption layers on the surface are removed by sputtering or other methods, and the cleaned and activated surfaces are brought into contact for bonding. Alternatively, a diffusion bonding method can be used, which uses a combination of temperature and pressure to bond the surfaces. In both cases, bonding occurs at the atomic level, resulting in a bond that is excellent not only electrically but also mechanically.

[0196] Furthermore, for joining insulating layers, a hydrophilic joining method can be used, in which highly flat surfaces are obtained by polishing, then hydrophilic treated with oxygen plasma or the like, and the surfaces are brought into contact for temporary joining, followed by dehydration by heat treatment to perform the final joining. Since bonding occurs at the atomic level in the hydrophilic joining method, a mechanically superior bond can be obtained.

[0197] When bonding layer 563 and layer 561, since the bonding surfaces of each layer contain both metal and insulating layers, a combination of methods such as surface activation bonding and hydrophilic bonding can be used.

[0198] For example, a method can be used in which the surface is cleaned after polishing, an anti-oxidation treatment is applied to the surface of the metal layer, and then a hydrophilic treatment is performed before joining. Alternatively, the surface of the metal layer may be made of a metal that is difficult to oxidize, such as Au, and then a hydrophilic treatment is performed. In addition, joining methods other than those described above may also be used.

[0199] The above bonding process allows for the electrical connection between the elements of layer 563 and the elements of layer 561.

[0200] Figure 14 shows a modified version of the laminated structure shown in Figure 13, with some differences in the configuration of layers 561 and 563.

[0201] This modified configuration has a transistor 402 of the pixel 400 provided in layer 561. In layer 561, the transistor 402, covered with an insulating layer 663, is formed of a Si transistor. One of the sources or drains of the transistor 402 is directly connected to one electrode of the photoelectric conversion device 401. The other of the sources or drains of the transistor 402 is electrically connected to node FD.

[0202] In the imaging device shown in Figure 14, layer 563 is provided with at least the transistors constituting the imaging device, excluding transistor 402. In Figure 14, transistors 404 and 405 are shown as elements provided in layer 563, but other elements such as transistors 403 and 406 can also be provided in layer 563. In addition, in layer 563 of the imaging device shown in Figure 14, an insulating layer 647 is provided between insulating layers 635 and 637. The insulating layer 647 functions as an interlayer insulating film and a planarization film.

[0203] (Embodiment 7) In this embodiment, the package containing the imaging unit, or so-called image sensor chip, will be described below.

[0204] Figure 15A1 is a perspective view of the top surface of a package containing an image sensor chip. The package includes a package substrate 410 for fixing the image sensor chip 452 (see Figure 15A3), a cover glass 420, and an adhesive 430 for bonding the two together.

[0205] Figure 15A2 is a perspective view of the bottom surface of the package. The bottom surface of the package has a BGA (Ball grid array) with solder balls as bumps 440. Note that it may also have an LGA (Land grid array) or PGA (Pin Grid Array), etc., instead of a BGA.

[0206] Figure 15A3 is a perspective view of the package, with the cover glass 420 and part of the adhesive 430 omitted. Electrode pads 460 are formed on the package substrate 410, and the electrode pads 460 and bumps 440 are electrically connected via through-holes. The electrode pads 460 are electrically connected to the image sensor chip 452 by wires 470.

[0207] Figure 15B1 is a perspective view of the top side of a camera module in which an image sensor chip is housed in a lens-integrated package. The camera module includes a package substrate 431 for fixing the image sensor chip 451 (Figure 15B3), a lens cover 432, and a lens 435, etc. An IC chip 490 (Figure 15B3) having functions such as a drive circuit and a signal conversion circuit for the imaging device is also provided between the package substrate 431 and the image sensor chip 451, thus having a System in Package (SiP) configuration.

[0208] Figure 15B2 is a perspective view of the lower side of the camera module. The package substrate 431 has a QFN (Quad flat no-lead package) configuration with mounting lands 441 on the lower and side surfaces. Note that this configuration is just an example, and a QFP (Quad flat package) or the aforementioned BGA may also be provided.

[0209] Figure 15B3 is a perspective view of the module, with the lens cover 432 and part of the lens 435 omitted. Land 441 is electrically connected to electrode pad 461, and electrode pad 461 is electrically connected to image sensor chip 451 or IC chip 490 by wire 471.

[0210] By housing the image sensor chip in the type of package described above, mounting it onto printed circuit boards and other devices becomes easier, allowing the image sensor chip to be incorporated into various semiconductor devices and electronic equipment.

[0211] This embodiment can be appropriately combined with descriptions of other embodiments.

[0212] (Embodiment 8) Using the driving support system according to the above-described embodiment, a driving support device suitable for a vehicle performing semi-autonomous driving is provided.

[0213] In Japan, the automation levels of vehicle driving assistance systems are defined in four stages, from Level 1 to Level 4. Level 1 refers to the automation of either acceleration, steering, or braking, and is called a safe driving assistance system. Level 2 automates multiple operations from acceleration, steering, and braking simultaneously and is called a semi-automated driving system (also called semi-autonomous driving). Level 3 is a system in which acceleration, steering, and braking are all automated, with the driver only intervening in emergencies, and is also called a semi-automated driving system (also called semi-autonomous driving). Level 4 is a fully automated driving system in which acceleration, steering, and braking are all automated, with little to no driver involvement.

[0214] In this specification, we propose new configurations or new driving assistance systems, primarily based on semi-autonomous driving at Level 2 or Level 3.

[0215] In order to display warnings to the driver of danger based on the information obtained from various cameras or sensors, the display area must be commensurate with the number of cameras or sensors.

[0216] Figure 16A shows an external view of the vehicle 120. Figure 16A shows an example of the installation locations of the forward image sensor 114a and the left-side image sensor 114L. Figure 16B is a schematic diagram showing the driver's field of view from inside the vehicle. The upper part of the driver's field of view is the windshield 110, and the lower part of the field of view is where the display device 111 with a display screen is installed.

[0217] The upper part of the driver's field of vision is the windshield 110, which is sandwiched between pillars 112. Figure 16A shows an example in which the forward-facing image sensor 114a is installed in a position close to the driver's line of sight, but it is not particularly limited and may be installed in the front grille or front bumper. Also, this embodiment shows a right-hand drive vehicle as an example, but it is not particularly limited and for left-hand drive vehicles, it should be installed according to the driver's position.

[0218] It is preferable to use the image sensor chip shown in Embodiment 7 for at least one of these image sensors.

[0219] The driver primarily uses the display device 111 to accelerate, steer, and brake, and secondarily checks the outside of the vehicle through the windshield. The display device 111 can be any one of the following: a liquid crystal display device, an EL (Electro Luminescence) display device, or a micro-LED (Light Emitting Diode) display device. Here, LEDs with a side length exceeding 1 mm are called macro-LEDs, those greater than 100 μm but less than or equal to 1 mm are called mini-LEDs, and those less than or equal to 100 μm are called micro-LEDs. It is particularly preferable to use micro-LEDs as the LED elements applied to the pixels. By using micro-LEDs, an extremely high-resolution display device can be realized. The higher the resolution of the display device 111, the better. The pixel density of the display device 111 can be 100 ppi or more and 5000 ppi or less, preferably 200 ppi or more and 2000 ppi or less.

[0220] For example, the central part 111a of the display screen of the display device displays an image acquired from an imaging device installed in front of the vehicle. In addition, parts 111b and 111c of the display screen display meters such as speed, estimated remaining driving distance, and abnormal warning indicators. Furthermore, the lower left part 111L of the display screen displays an image of the left side outside the vehicle, and the lower right part 111R of the display screen displays an image of the right side outside the vehicle.

[0221] The 111L (lower left of the display screen) and 111R (lower right of the display screen) indicate that the side mirrors (also called door mirrors) can be digitized, eliminating the large, protruding parts of the side mirrors that extend outside the vehicle.

[0222] The display screen of the display device 111 may be configured to allow touch input operation, enabling functions such as enlarging or shrinking a portion of the image, changing its display position, or expanding the area of ​​the display region.

[0223] The image on the display screen of the display device 111 is created using an image signal processing device such as a GPU, as it is a composite of data from multiple imaging devices or sensors.

[0224] By using the driving support system shown in Embodiment 1, it is possible to acquire grayscale image data with a wide dynamic range, extract only the distant region, perform inference to colorize it, and then enlarge and output the highlighted image to the display device 111.

[0225] By appropriately utilizing AI, drivers can operate the vehicle primarily by viewing images displayed on the display device, i.e., images generated by image sensors and AI, with their view of the windshield serving as a supplement. Operating the vehicle by viewing AI-generated images can lead to safer driving than driving solely by the driver's eyes. Furthermore, drivers can operate the vehicle with a greater sense of security.

[0226] Furthermore, the display device 111 can be applied to the cockpit area (also called the driver's seat area) of various types of vehicles, including large, medium, and small vehicles. It can also be applied to the cockpit area of ​​vehicles such as aircraft and ships.

[0227] Furthermore, although this embodiment shows an example in which the forward-facing image sensor 114a is installed below the windshield, it is not particularly limited, and the imaging camera shown in Figure 17 may be installed on the hood or around the rearview mirror inside the vehicle.

[0228] The imaging camera shown in Figure 17 can also be called a drive recorder and has a housing 961, a lens 962, a support part 963, etc. By attaching double-sided tape or the like to the support part 963, it can be installed on the windshield, hood, rearview mirror support, etc.

[0229] The imaging camera in Figure 17 is equipped with an image sensor, and can record and store driving footage either inside the imaging camera or in a storage device mounted on the vehicle.

[0230] This embodiment can be freely combined with other embodiments. [Explanation of Symbols]

[0231] 10: Data acquisition device, 11: Solid-state image sensor, 12: Analog arithmetic circuit, 13: A / D circuit, 14: Memory unit, 15: Display device, 16a: Neural network unit, 16b: Neural network unit, 16c: Neural network unit, 17: Data extraction unit, 18a: Storage unit, 18b: Storage unit, 18c: Storage unit, 19: Display unit, 21: Display example, 22: Distant area, 23: Colorized image, 24: Enhanced image, 31: Driving support system, 41: Imaging system, 110: Windshield, 111: Display device, 111a: Central part, 111b: Part, 111c: Some parts: 111L: lower left, 111R: lower right, 112: pillar, 114a: front image sensor, 114L: left side image sensor, 120: vehicle, 200: pixel block, 201: circuit, 202: capacitor, 203: transistor, 204: transistor, 205: transistor, 206: transistor, 207: resistor, 211: wiring, 212: wiring, 213: wiring, 215: wiring, 216: wiring, 217: wiring, 218: wiring, 219: wiring, 300: pixel array, 301: circuit, 302: circuit, 303: circuit, 304: circuit, 305: circuit, 306: Circuit, 311: Wiring, 400: Pixel, 401: Photoelectric conversion device, 402: Transistor, 403: Transistor, 404: Transistor, 405: Transistor, 406: Transistor, 407: Capacitor, 410: Package substrate, 411: Wiring, 412: Wiring, 413: Wiring, 413a: Wiring, 413b: Wiring, 413c: Wiring, 413d: Wiring, 413e: Wiring, 413f: Wiring, 413g: Wiring, 413h: Wiring, 413i: Wiring, 413j: Wiring, 414: Wiring, 415: Wiring, 417: Wiring, 420: Cover glass, 421: Distribution Wire, 422: Wiring, 423: Wiring, 424: Wiring, 430: Adhesive, 431: Package substrate, 432: Lens cover, 435: Lens, 440: Bump, 441: Land, 450: Transistor, 450a: Transistor, 450b: Transistor, 450c: Transistor, 450d: Transistor, 450e: Transistor, 450f: Transistor, 450g: Transistor, 450h: Transistor, 450i: Transistor, 450j: Transistor, 451: Image sensor chip, 452: Image sensor chip, 460: Electrode pad,461: Electrode pad, 470: Wire, 471: Wire, 490: IC chip, 545: Semiconductor layer, 546: Insulating layer, 560: Layer, 561: Layer, 563: Layer, 565a: Layer, 565b: Layer, 566a: Layer, 566b: Layer, 566c: Layer, 566d: Layer, 567a: Layer, 567b: Layer, 567c: Layer, 567d: Layer, 567e: Layer, 632: Silicon substrate, 633: Insulating layer, 634: Insulating layer, 635: Insulating layer 636: conductive layer, 637: insulating layer, 638: insulating layer, 647: insulating layer, 651: insulating layer, 652: insulating layer, 653: insulating layer, 654: insulating layer, 655: conductive layer, 661: insulating layer, 662: insulating layer, 664: insulating layer, 665: insulating layer, 671: light-shielding layer, 673: microlens array, 683: conductive layer, 684: conductive layer, 685: conductive layer, 686: conductive layer, 961: housing, 962: lens, 963: support part,

Claims

[Claim 1] Steps include: driving a vehicle equipped with an imaging device, The steps include: capturing a black and white image of the area in front of a moving vehicle using the imaging device; The process involves performing segmentation on a grayscale image including the far region to infer at least the areas of the sky, cars, and roads, and The steps include performing depth estimation processing on a grayscale image including the aforementioned far-field region to infer a specific far-field region, Based on the segmentation process and the depth estimation process, the step of determining the center of the portion to be cut from the grayscale image, The steps include extracting a rectangular region with the aforementioned center as the central part, inputting the extracted data, and performing super-resolution processing, The steps include: inputting the output result of the super-resolution processing and performing a colorization process to accurately highlight objects included in the distant region; The process includes the step of enlarging and displaying a colorized distant region, In this driving assistance system, the size of the rectangular region with the aforementioned center as its central part is larger when the vehicle is traveling at a higher speed than when it is traveling at a lower speed.

Images