Travel support system

Abstract

To provide a travel support system suitable for a vehicle which performs semiautomatic driving, and an electronic apparatus for vehicles.SOLUTION: A travel support system comprises: an imaging device capable of picking up a first black-and-white image in a direction of travel of a vehicle; a first neural network for segmentation processing; a second neural network for depth estimation processing; a determination section for determining a center of a portion to be segmented from the first black-and-white image on the basis of the segmentation processing and the depth estimation processing; a third neural network for performing coloring processing only on a segmented second black-and-white image; and a display device for performing enlarged display of the second black-and-white image on which the coloring processing is performed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] One aspect of the present invention relates to a neural network and an imaging system using the same. One aspect of the present invention also relates to an electronic device using a neural network. One aspect of the present invention further relates to a vehicle using a neural network. One aspect of the present invention also relates to an imaging system that obtains a color image from a black-and-white image obtained by a solid-state imaging device using image processing technology. One aspect of the present invention also relates to a video surveillance system, a security system, a safety information providing system, or a driving support system using the imaging system.

[0002] Note that one aspect of the present invention is not limited to the above technical field. One aspect of the invention disclosed in this specification or the like relates to an article, a method, or a manufacturing method. One aspect of the present invention relates to a process, a machine, a manufacture, or a composition of matter. Therefore, more specifically, examples of the technical field of one aspect of the present invention disclosed in this specification or the like include semiconductor devices, display devices, light-emitting devices, power storage devices, storage devices, electronic devices, lighting devices, input devices, input / output devices, their driving methods, or their manufacturing methods. One aspect of the present invention relates to a vehicle or vehicle-mounted electronic devices provided in a vehicle.

[0003] Note that in this specification or the like, the semiconductor device refers to all devices that can function by utilizing semiconductor characteristics. Transistors and semiconductor circuits are one aspect of semiconductor devices. In addition, storage devices, display devices, imaging devices, and electronic devices may have semiconductor devices.

[0004] One aspect of the present invention also relates to a program using a neural network.

[0005] One aspect of the present invention also relates to a driving system in which a vehicle such as an automobile can freely switch between a safe support driving state, a semi-automatic driving state, and an automatic driving state.

Background Art

[0006] Techniques for constructing transistors using oxide semiconductor thin films formed on a substrate have attracted attention. For example, Patent Document 1 discloses an imaging device configured to use a transistor having an extremely low off-current with an oxide semiconductor in a pixel circuit.

[0007] In addition, a technique for adding an arithmetic function to an imaging device is disclosed in Patent Document 2.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0009] In imaging devices equipped with solid-state imaging devices such as CMOS image sensors, high-quality images can be easily captured due to technological development. In the next generation, it is required to further equip imaging devices with intelligent functions.

[0010] One aspect of the present invention aims to provide an imaging device capable of performing image processing. Or, one aspect of the present invention aims to provide an imaging device capable of performing high-speed operation. Or, one aspect of the present invention aims to provide an imaging device with low power consumption. Or, one aspect of the present invention aims to provide an imaging device with high reliability. Or, one aspect of the present invention aims to provide a novel imaging device or the like. Or, one aspect of the present invention aims to provide a driving method for the above imaging device. Or, one aspect of the present invention aims to provide a novel semiconductor device or the like.

[0011] In addition, one aspect of the present invention aims to provide a driving support system suitable for a vehicle performing semi-automatic driving and vehicle electronic equipment.

[0012] Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will become apparent from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc.

Means for Solving the Problems

[0013] The technology of imaging in black-and-white images and colorizing them is particularly suitable for night imaging. For example, at night, there is less light than during the day, and the colors are different from those during the day. Also, in a dark environment where infrared imaging is required, when imaging with infrared rays, a monochrome image is obtained. Further, in infrared imaging, there is a problem regarding depth in that the brightness is significantly different between a material that reflects infrared rays and a material that absorbs infrared rays, and even when placed at the same position, the reflecting material appears closer and the absorbing material appears farther away.

[0014] Also, when assuming driving in a dark environment, when the headlights of an oncoming vehicle shine, the visibility of the surroundings deteriorates. In particular, the human eye has a smaller pupil and the surroundings become invisible. Therefore, by using the image display by an imaging device whose exposure can be arbitrarily set as an auxiliary, it becomes easier to ensure the safety of the surroundings. This image is preferably highly visible.

[0015] It is desirable to perform segmentation processing on such an image to colorize and highlight a human or a vehicle in addition to colorizing them, but since the image in the distance is small and has a small amount of information, the accuracy decreases. In particular, in the case of in-vehicle use, the image may be blurred due to vibration or the like. In particular, when used as an in-vehicle camera, as the vehicle speed increases, information in the distance becomes necessary.

[0016] As the vehicle speed increases, the driver's viewing angle becomes narrower, but there is a contradiction in that a wide viewing angle is required to perform safe driving. Also, it is not practical to lower the vehicle speed to ensure a wide viewing angle. An environment where safe driving can be performed whether at low speed or high speed is desired.

[0017] Increasing the speed narrows the driver's viewing angle, but to assist the driver, a camera is used to automatically image areas that need attention and display them to the driver, thereby enhancing the safety during vehicle travel.

[0018] On the other hand, it is also possible to enlarge by adjusting the camera lens, but it is difficult for the driver during driving to perform adjustments of the optical system such as enlargement and reduction, and it is also difficult for the driver to provide a mechanism for changing the lens orientation to move the lens focus and change the imaging direction. Also, even when imaging a distant location with a telephoto lens, the object may be lost due to vibrations during travel. It is desired to automatically extract and display only important areas without depending on the driver and without changing the focal length or orientation of the camera. Important areas are, for example, areas where there are moving vehicles or pedestrians on a distant road.

[0019] During vehicle operation, a driving support system that pinpoints and displays areas that need to arouse attention and supports the driver's burden is desired. Also, a display system that enlarges when the area that needs to arouse attention is far away is desired.

[0020] The configuration of the invention disclosed in this specification is a driving support system having an imaging device capable of imaging a first black-and-white image in the traveling direction of the vehicle, a first neural network for segmentation processing, a second neural network for depth estimation processing, a determination unit for determining the center of the portion to be cut out from the first black-and-white image based on the segmentation processing and the depth estimation processing, a third neural network for colorizing only the cut-out second black-and-white image, and a display device for enlargedly displaying the colorized second black-and-white image.

[0021] In the above configuration, for in-vehicle use, it is preferable to use a plurality of learned neural networks, and it is preferable to have one or more storage units that store programs for executing them. One or more processors are in-vehicle and execute these neural networks.

[0022] Further, the driving support system includes steps of driving a vehicle equipped with an imaging device, imaging the front of the driving vehicle in black and white by the imaging device, performing segmentation processing on the black and white image including the distant area to infer at least the areas of sky, vehicle, and road, performing depth estimation processing on the black and white image including the distant area to infer a specific distant area, determining the center of the portion to be cut out from the black and white image based on the segmentation processing and the depth estimation processing, extracting a rectangular area with the center as the central part, inputting the extracted data, and performing super-resolution processing, inputting the output result of the super-resolution processing, performing a colorization process to accurately highlight the object included in the distant area, and enlarging and displaying the colorized distant area. Note that the specific distant area refers to an area including at least the road end in the traveling direction.

[0023] In addition to the above steps, it may also include a step of measuring the traveling speed of the vehicle. The image size to be cut out can be changed according to the traveling speed of the vehicle. For example, the size of the rectangular area with the center of the portion cut out from the black and white image as the central part can also be determined by the traveling speed of the vehicle. The cutting area is made wider when the vehicle speed is higher compared to when the vehicle speed is low. By doing so, it is possible to cover the narrowing of the driver's field of view due to the speed.

[0024] One or more processors may perform a process of reading and executing a program including any one or all of the above steps. A program for causing a computer to execute each step is recorded in a storage unit in advance. Also, it is not limited to a processor, and it can also be executed by a circuit (for example, an FPGA circuit or an ASIC circuit) that realizes a function for executing any one or all of the above steps.

[0025] In the above configuration, the segmentation process uses the first neural network process, the depth estimation process uses the second neural network process, the super-resolution process uses the third neural network process, and the colorization process uses the fourth neural network process. As teacher datasets for learning the segmentation process, MSCOCO, Cityscapes, etc. are used. Also, as a teacher dataset for learning the depth estimation process, KITTI, etc. may be used. The teacher dataset for learning the super-resolution process is not particularly limited, and not only photos but also illustrations may be used. The teacher dataset for learning the colorization process is not particularly limited as long as it is in color, and ImageNet, color images of a recorded drive recorder processed and used can be used.

[0026] Specifically explaining the above-described driving support system, neural network processing is performed in a state where color information is reduced and the amount of imaged information is reduced, and further, in order to reduce the data amount, a distant area is cut out, only the distant area is colorized, and enlarged display is performed. By reducing color information, the data amount can be reduced, and the arithmetic processing in the neural network process can also be made simple. Also, if the data amount can be reduced, miniaturization of the hardware capable of executing the neural network process can be achieved. An imaging device without a color filter not only reduces color information but also has no light reduction due to the color filter, and it is easy to ensure the amount of light reaching the light receiving sensor, so the dynamic range is expanded.

[0027] In addition, an apparatus or vehicle having the imaging system and the driving support system disclosed in this specification can also be referred to as an image generation device. The image generation device selectively colorizes and enlarges a part of a black-and-white image with a wide dynamic range captured by an imaging device without a color filter for enlarged display.

[0028] In addition, the image colored by the coloring process often does not have a natural color tone, which becomes an emphasized display and is an image that is easy for the driver to recognize.

[0029] Moreover, it is not limited to an imaging device without a color filter. In addition to an imaging device without a color filter, it can also be combined with an imaging device having a color filter, or with other environmental recognition units such as a stereo camera, a sonar, a multi-focus multi-eye camera system, a LIDAR, a millimeter-wave radar, an infrared sensor (TOF method), etc. to construct a driving support system. The distance measurement by the TOF method is composed of a light source and a light detector (sensor or camera). The camera used in this TOF method is called a time-of-flight camera, or also a TOF camera. The TOF camera can obtain distance information from a light source that emits light to an object based on the flight time (time of flight) of the reflected light of the light irradiated on the object.

Advantages of the Invention

[0030] Using a plurality of neural networks, it is possible to enlarge and display an area that should arouse attention and provide it to the driver. Mainly, it is possible to provide image display for assisting the driver.

[0031] In addition, in an environment where the amount of light is insufficient, at dusk, at night, in the early morning, or when passing through a long tunnel, etc., it is possible to provide the driver with a clear color display of a distant area in the traveling direction of the vehicle, so it has a particularly remarkable effect.

Brief Description of the Drawings

[0032]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

DETAILED DESCRIPTION OF THE INVENTION

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and those skilled in the art can easily understand that its form and details can be variously changed. Further, the present invention is not to be construed as being limited to the description of the embodiments shown below.

[0034] (Embodiment 1) In this embodiment, an example of the flow of a driving support system is shown in FIG. 1, which selectively extracts a distant area that the driver should pay attention to from the black-and-white video obtained by the in-vehicle solid-state image sensor, colorizes a part of it, and provides the driver with an enlarged display video.

[0035] An imaging system having a hardware configuration including a solid-state image sensor is installed in a part of the vehicle (such as the bonnet, inside the vehicle, roof, etc.) that can photograph the traveling direction of the vehicle (including the distant area), activated, and continuous shooting is started.

[0036] The in-vehicle hardware has a configuration in which one or more processors mainly control the operations in each step. When performing neural network processing, a storage unit such as a memory sufficient to store learning data is required, and hardware capable of sufficient arithmetic processing is necessary. The storage unit refers to a small-sized large-capacity storage device (for example, an SSD or a hard disk) that can be mounted on the vehicle. By executing the program stored in the small-sized large-capacity storage device, the processing in the flowchart shown in FIG. 1 can be realized.

[0037] First, data acquisition is started (S1).

[0038] Black-and-white image data is acquired using a solid-state image sensor without a color filter (S2). In some cases, an array of a plurality of solid-state image sensors arranged in a matrix direction is also called a pixel array. Also, on the left side of FIG. 1, a display example 21 of the captured video immediately after step S2 is shown. However, the display example 21 is illustrated for easy explanation and is not actually displayed. Actually, it is black-and-white image data converted into a signal format (such as JPEG (registered trademark)).

[0039] Next, an extraction of a distant region, which is a part of the captured image data, is performed, and cropping is carried out (S3). The distant region in the traveling direction of the vehicle is the region that is most difficult for the driver to recognize in a dark state, for example, during driving at dusk or at night or in the early morning. In the case of a solid-state image sensor without a color filter, an image with a wider dynamic range can be captured than a solid-state image sensor with a color filter, so that an object in the distant region can be captured as an image.

[0040] In the present embodiment, the distant region in the traveling direction of the vehicle is selectively extracted. As a method of extraction, a location is specified using data from depth inference or a depth sensor module (such as a TOF camera) that measures depth. When using depth inference, neural network processing is performed. In step S3, as an example of extracting the distant region, the distant region 22 that extracts the region surrounded by the dotted line in FIG. 1 is illustrated. By performing cropping in step S3, the amount of data to be used can be reduced.

[0041] Also, in step S3, as the size to be cropped, the size may be determined using the data of the speedometer. The speedometer is a device that enables supplementation of signals from GPS and GLONASS satellites. The speed, position, or travel distance of the vehicle during travel can be measured using GPS or the like.

[0042] Next, the data is reduced to only the data of the distant region 22 (S4). In step S4, data other than the data cut out in S3 is deleted. Note that the original data may be stored in a dedicated storage device.

[0043] Thereafter, colorization inference is performed on the data of only the distant region 22 (S5). In the present embodiment, the data reduced in step S4 is used as input data, and convolution processing is performed using a CPU or the like to infer contours, colors, etc., and colorization is performed.

[0044] When colorizing a black-and-white image by software, the program that makes up the software may be installed in a computer with the software incorporated in hardware, or a program or the like may be installed from a network or a recording medium. Install a program recorded on a recording medium such as a computer-readable CD-ROM (Compact Disk Read Only Memory), and execute the program for colorizing the black-and-white image. The processing performed by the program is not limited to performing the processing in order, and may not be in time series. For example, it may be performed in parallel.

[0045] Also, the program of the software that executes the inference program for performing neural network processing used for depth inference or colorization inference, etc. can be described in various programming languages such as Python, Go, Perl, Ruby, Prolog, Visual Basic, C, C++, Swift, Java (registered trademark),.NET. Also, the application may be created using frameworks such as Chainer (available in Python), Caffe (available in Python and C++), TensorFlow (available in C, C++, and Python). For example, the LSTM algorithm is programmed in Python and uses a CPU (Central Processor Unit) or a GPU (Graphics Processing Unit). Also, a chip that integrates a CPU and a GPU into one is sometimes called an APU (Accelerated Processing Unit), and this APU chip can also be used. Also, an IC incorporating an AI system (also called an inference chip) may be used. An IC incorporating an AI system is sometimes called a circuit (microprocessor) that performs neural network operations.

[0046] In this embodiment, in order to perform inference inside the vehicle, pre-trained feature amounts (weights for colorization) are used. By storing the pre-trained feature amounts in a storage device and performing calculations, data output can be achieved at a level comparable to the case where pre-trained feature amounts are not used. Since pre-reduced data is used, the burden of arithmetic processing is reduced. In FIG. 1, a display example after colorization is illustrated as a colorized image 23.

[0047] The colorized image 23 thus obtained uses an image sensor without a color filter and is based on a black-and-white image with a wide dynamic range. Therefore, even in a case where the amount of light is small and identification is impossible with a conventional image sensor with a color filter, colorized image data that can be identified can be obtained.

[0048] Finally, the colorized image 23 is enlarged and displayed as an emphasized image 24 on the display unit of the display device (S6). Although it can be called an emphasized image by enlarging, in this embodiment, further colorization is performed using colorization inference, and although it is close to natural colors, it becomes an image different from the actual video, so an emphasized image can be created. Note that since the results are the same even if the order of step S6 and step S5 is reversed, either one can be executed first.

[0049] The acquisition of the above-mentioned emphasized image is repeated. By repeating, it is also possible to display only the distant area as an emphasized image in real time.

[0050] The above-mentioned driving support system can clearly realize imaging in a relatively dark place and provide the driver with the area to be noted as an emphasized image. With the above-mentioned driving support system, in particular, accidents in dark places with little lighting such as in the evening or at night can be prevented.

[0051] In addition, the above-described driving support system can also be applied to a vehicle capable of semi-automatic driving or a vehicle capable of fully automatic driving by combining a camera or radar that images the surrounding area of the vehicle and an ECU (Electronic Control Unit) that performs image processing and the like. A vehicle using an electric motor has a plurality of ECUs, and the ECUs perform engine control and the like. The ECU includes a microcomputer. The ECU is connected to a CAN (Controller Area Network) provided in the electric vehicle. CAN is one of the serial communication standards used as an in-vehicle LAN. The ECU uses a CPU or a GPU. For example, as one of a plurality of cameras (such as a drive recorder camera and a rear camera) mounted on an electric vehicle, a solid-state imaging device without a color filter is used, a part of the obtained black-and-white image is extracted, inference is performed by the ECU via CAN, a colorized image is created, and it can be configured to be enlarged and a highlighted image is displayed on the display unit of the in-vehicle display device.

[0052] In addition, in the present embodiment, as a method of selectively extracting a distant area, when specifying a location to be extracted using data from a depth sensor module (such as a TOF camera) that measures depth, the colorization inference will perform one neural network process. In this case, since only one neural network process is performed, the arithmetic processing is small-scale, and it has the merit of reducing the burden on the CPU.

[0053] In addition, as a method of selectively extracting a distant area, when using depth inference, the first neural network process is performed for depth inference, and the colorization inference will perform the second neural network process. In this case, the installation of a depth sensor module for measuring depth can be made unnecessary. Also, although it is two neural network processes, the burden of the operation is reduced by reducing the input data of the neural network process for colorization.

[0054] (Embodiment 2) In this embodiment, an example in which the method for extracting the distant region is different from that in Embodiment 1 is shown. Since many other parts are the same as those in Embodiment 1, detailed descriptions will be omitted here.

[0055] This is shown as an example of the flow of the driving support system in FIG. 2. The same reference numerals are used for the same steps as in the flowchart shown in FIG. 1 of Embodiment 1.

[0056] First, data acquisition is started (S1).

[0057] Black-and-white image data is acquired using a solid-state image sensor without a color filter (S2).

[0058] Next, in order to select a distant region which is a part of the captured image data, depth inference is performed using the black-and-white image data (S21A). Also, segmentation inference is performed using the black-and-white image data (21B).

[0059] Regarding the depth inference, an example is also shown in Embodiment 1. In this embodiment, since the inference is performed inside the vehicle, pre-learned feature amounts (weights for depth) are used.

[0060] Segmentation inference is also called segmentation. Note that segmentation refers to the process of identifying which object each pixel of the input image belongs to. It is also called semantic segmentation. Software that generates multiple image segments for use in image analysis is executed by neural network processing. Specifically, segmentation is performed based on the learned content using, for example, U-net or FCRN (Fully Convolutional Residual Networks), which are a type of image processing and convolutional neural network (CNN). Note that the labels for segmentation are distinguished by vehicle (car), sky, plant, ground, human, building, etc. Also, in this embodiment, since inference is performed inside the vehicle, pre-trained feature amounts (weights for segmentation) are used.

[0061] Step S21A and step S21B may be performed sequentially or in parallel.

[0062] Next, focusing on the sky and road regions obtained by segmentation inference, the coordinates of the upper end of the road where the distance between the sky and the road (the interval between the lower end of the sky and the upper end of the road) is the shortest within the segmentation image are extracted. This is because, as shown in the display example 21 of FIG. 1, the vicinity of the tip of the driving lane corresponds to the location where the interval between the sky and the road in the image is the shortest.

[0063] In the segmentation image, the coordinates on the road where the distance between the sky and the road is the shortest are not necessarily in one place, and there may be multiple.

[0064] When there are multiple coordinates on the road in the segmentation image where the distance between the sky and the road is the shortest, the coordinate farthest away is selected among them (S22).

[0065] From the ground area obtained in step S21B, extract the location with the maximum depth in the depth image within the area from the results of step 21A. By identifying the leading edge of the driving lane in the black-and-white image, the central part of the distant area to be cut out later can be determined.

[0066] Next, reduce the data to only the distant area (S23).

[0067] After that, perform colorization inference on the data of only the distant area (S24). In this embodiment, since the inference is performed inside the vehicle, pre-trained feature amounts (weights for colorization) are used.

[0068] Finally, the colorized image is enlarged and displayed as an emphasized image on the display unit of the display device (S25).

[0069] The acquisition of the above emphasized image is repeated.

[0070] In this embodiment, as a method for selectively extracting the distant area, the first neural network process is performed for depth inference, the second neural network process is performed for segmentation inference, and the third neural network process is performed for colorization inference. Since all these inferences use pre-trained feature amounts, calculations can be performed inside the vehicle.

[0071] This embodiment can be freely combined with Embodiment 1.

[0072] (Embodiment 3) In this embodiment, an example is shown in which the method for extracting the distant area is different from that in Embodiment 2. Since many other parts are the same as those in Embodiment 1 or Embodiment 2, detailed descriptions will be omitted here.

[0073] An example of the flow of the driving support system is shown in FIG. 3. The same reference numerals are used for the same steps as those in the flowcharts shown in FIG. 1 of Embodiment 1 and FIG. 2 of Embodiment 2.

[0074] Acquire black-and-white image data using a solid-state image sensor without a color filter (S2).

[0075] Next, perform depth inference using the black-and-white image data in order to select a distant region that is part of the captured image data (S21A). Also, perform segmentation inference using the black-and-white image data (21B).

[0076] Next, select the distant region (S22).

[0077] Also, acquire vehicle speed data (S26). The speedometer shall be a device capable of supplementing signals from GPS and GLONASS satellites. It is possible to measure the speed, position, or travel distance of a vehicle in motion using GPS or the like. Also, the vehicle speed data may use numerical values obtained by a general speedometer. Note that the timing of step S26 is not particularly limited and may be at any time between the start of data acquisition and step S22.

[0078] Next, determine the cropping size of the distant size (S27). As the size to be cropped, determine the size using the data of the speedometer obtained in step S26. For example, a configuration is adopted in which a wider range is cropped when the speed is higher compared to when it is lower.

[0079] Next, reduce the data to only the distant region (S28).

[0080] After that, super-resolution inference is performed on the data of only the distant region (S29). Note that the super-resolution process refers to an image process for generating a high-resolution image from a low-resolution image. The super-resolution process may be repeated multiple times. By mixing not only color photo images but also color illustration (animation) images as teacher data for creating a learning model for discriminating color boundaries, colorized image data with clear color boundaries can be obtained. Therefore, it is preferable to mix illustration (animation) images in the teacher data during learning, and calculate the super-resolution weight that becomes the weight coefficient of the neural network process. In the present embodiment, since inference is performed inside the vehicle, pre-learned feature amounts (super-resolution weights) are used.

[0081] Next, colorization inference is performed (S30). In the present embodiment, since inference is performed inside the vehicle, pre-learned feature amounts (colorization weights) are used.

[0082] The order of performing colorization inference after super-resolution inference is also important. It is preferable to perform calculations using black-and-white image data and perform colorization as the final image process. Black-and-white image data has a smaller data volume compared to image data with color information, and the burden on the processing ability of the arithmetic unit can be reduced.

[0083] Finally, the colorized image is enlarged and displayed as an emphasized image on the display unit of the display device (S31).

[0084] The acquisition of the above-mentioned emphasized image is repeated.

[0085] In this embodiment, as a method for selectively extracting a distant region, the first neural network process is performed by deep inference, the second neural network process is performed by segmentation inference, the third neural network process is performed by super-resolution inference, and the fourth neural network process is performed by colorization inference. Since all of these inferences use pre-learned feature amounts, calculations can be performed inside the vehicle. When the speed is high, a wide range of regions are enlarged and super-resolved, so that a clear highlighting can be obtained.

[0086] This embodiment can be freely combined with Embodiment 1 or Embodiment 2.

[0087] (Embodiment 4) In this embodiment, an example of a block diagram of a driving support system 31 that executes the flowchart of Embodiment 2 is shown. FIG. 4 is a block diagram of the driving support system 31, and will be described below with reference thereto.

[0088] The data acquisition device 10 is a semiconductor chip including a solid-state imaging device 11 and a memory unit 14, and does not have a color filter. The data acquisition device 10 has an optical system such as a lens. Note that the optical system may have any configuration as long as its imaging characteristics are known, and is not particularly limited.

[0089] For example, the data acquisition device 10 may use a device in which a back-illuminated CMOS image sensor chip, a DRAM chip, and a logic circuit chip are stacked to form one semiconductor chip. Alternatively, a device in which a back-illuminated CMOS image sensor chip and a logic circuit chip including an analog-to-digital conversion circuit are stacked to form one semiconductor chip may be used. In that case, the memory unit 14 is an SRAM. Further, the chips to be stacked may be stacked using a known technique for bonding to make an electrical connection.

[0090] The memory unit 14 is a circuit that stores the converted digital data, and is configured to store the data before inputting it to the neural network units 16a, 16b, and 16c, but is not particularly limited to this configuration.

[0091] The neural network units 16a, 16b, and 16c are realized by software operations using a microcontroller. The microcontroller is one that integrates a computer system into a single integrated circuit (IC). When the scale of the operation or the data to be processed is large, the neural network units 16a, 16b, and 16c may be configured by combining a plurality of ICs. The learning device includes at least these plurality of ICs. Also, since free software can be used if it is a microcontroller equipped with Linux (registered trademark), it is preferable because the total cost for configuring the neural network units 16a, 16b, and 16c can be reduced. Also, it is not limited to Linux (registered trademark), and other operating systems (OS) may be used.

[0092] The learning of the neural network units 16a, 16b, and 16c shown in FIG. 4 is shown below. In the present embodiment, learning is performed in advance, and processing by the neural network is performed using weights. The learning teacher data can also be stored in the storage units 18a, 18b, and 18c and learned as appropriate.

[0093] In Embodiment 2, only the remote area is selected based on the output results of the neural network units 16a and 16b, and data reduction is performed. The data extraction unit 17 can also be said to be a data control unit that selects only the remote area and performs data reduction. The data extracted by the data extraction unit 17 is input to the neural network unit 16c for colorization.

[0094] The data output from the neural network unit 16c has the contour and color information of the subject regarding only the area of the distant region, and is input to the display device 15. The display device 15 has a display unit 19 and forms a signal representing a video including enlarged display according to the number of gradations that can be displayed and the screen size.

[0095] Also, the portable information terminal (such as a smartphone) of the passenger can be used as the display device 15, and the display unit of the portable information terminal can be used as the display unit 19. In this case, it is necessary to mount a transmission / reception unit for transmitting the output of the neural network unit 16c to the portable information terminal of the passenger. Of course, not limited to the passenger, the portable information terminal (such as a smartphone) of the driver can be installed on the bonnet or the like so that the driver can view the video.

[0096] The above driving support system 31 can particularly prevent accidents in dark places with little lighting such as in the evening or at night.

[0097] This embodiment can be freely combined with Embodiment 1, Embodiment 2, or Embodiment 3.

[0098] (Embodiment 5) In this embodiment, a modification example in which a part of Embodiment 4 is changed is shown. Note that the same reference numerals are used for the same parts as in Embodiment 4 for explanation. Since the configuration after the memory unit 14 is the same as that in Embodiment 4, it will be omitted here.

[0099] An example of the configuration of the imaging system 41 will be described with reference to the block diagram shown in FIG. 5.

[0100] The data acquisition device 10 is a semiconductor chip including a solid-state imaging device 11 and an analog arithmetic circuit 12, and does not have a color filter. The data acquisition device 10 has an optical system such as a lens. Note that the optical system may have any configuration as long as its imaging characteristics are known, and is not particularly limited.

[0101] Further, the analog arithmetic circuit 12 can use transistors using metal oxides (hereinafter referred to as OS transistors) formed on the silicon chip of the solid-state imaging device formed using a silicon substrate, and the like.

[0102] The A / D circuit 13 (also referred to as an A / D converter) represents an analog-to-digital conversion circuit, and converts the analog data output from the data acquisition device 10 into digital data. If necessary, an amplifier circuit may be provided between the data acquisition device 10 and the A / D circuit 13 to amplify the analog signal before converting it into digital data.

[0103] The memory unit 14 is a circuit that stores the converted digital data, and is configured to store the data before inputting it to the neural network units 16a, 16b, and 16c, but is not particularly limited to this configuration. Depending on the amount of data output from the data acquisition device or the data processing capacity of the image processing device, for small-scale data, the output from the A / D circuit 13 may be directly input to the neural network units 16a, 16b, and 16c without being stored in the memory unit 14.

[0104] In the present embodiment, by the analog arithmetic circuit shown in FIG. 5, a part of the operations commonly performed by the neural network units 16a, 16b, and 16c can be performed first. By using the imaging system 41 shown in FIG. 5, the number of operations performed by the neural network units 16a, 16b, and 16c can be reduced.

[0105] The present embodiment can be freely combined with Embodiment 1 or Embodiment 2 or Embodiment 3 or Embodiment 4.

[0106] (Embodiment 6) In the present embodiment, a configuration example in which the data acquisition device 10 is a part of the configuration of the imaging system 41 will be described below. FIG. 6 is a block diagram for explaining the imaging system 41.

[0107] The imaging system 41 includes a pixel array 300, a circuit 201, a circuit 301, a circuit 302, a circuit 303, a circuit 304, a circuit 305, and a circuit 306. Note that each of the circuit 201 and the circuits 301 to 306 is not limited to a single circuit configuration and may be configured by a combination of a plurality of circuits. Or, any of the above-mentioned plurality of circuits may be integrated. Also, circuits other than the above may be connected.

[0108] The pixel array 300 has an imaging function and an arithmetic function. The circuit 201 and the circuit 301 have an arithmetic function. The circuit 302 has an arithmetic function or a data conversion function. The circuits 303, 304, and 306 have a selection function. The circuit 303 is electrically connected to the pixel block 200 via a wiring 424. The circuit 304 is electrically connected to the pixel block 200 via a wiring 423. The circuit 305 has a function of supplying a potential for a multiplication and accumulation operation to the pixels. For the circuits having a selection function, a shift register, a decoder, or the like can be used. The circuit 306 is electrically connected to the pixel block 200 via a wiring 413. Note that the circuit 301 and the circuit 302 may be provided outside.

[0109] The pixel array 300 includes a plurality of pixel blocks 200. As shown in FIG. 7, the pixel block 200 includes a plurality of pixels 400 arranged in a matrix, and each pixel 400 is electrically connected to the circuit 201 via a wiring 412. Note that the circuit 201 can also be provided inside the pixel block 200.

[0110] Also, the pixel 400 is electrically connected to an adjacent pixel 400 via transistors 450 (transistors 450a to 450f). The function of the transistor 450 will be described later.

[0111] In pixel 400, it is possible to acquire image data and generate data obtained by adding the image data and a weighting coefficient. In FIG. 7, as an example, the number of pixels included in pixel block 200 is set to 3×3, but it is not limited thereto. For example, it can be 2×2, 4×4, etc. Alternatively, the number of pixels in the horizontal direction and the vertical direction may be different. Also, some pixels may be shared by adjacent pixel blocks. In FIG. 7, an example is shown in which ten transistors 450 (transistors 450a to 450j) are provided between pixels 400, but the number of transistors 450 may be further increased. Also, in transistors 450g to 450j, some transistors may be omitted so as to eliminate parallel paths. Wiring wirings 413g to 413j are connected to transistors 450g to 450j as gates, respectively.

[0112] Pixel block 200 and circuit 201 can be operated as a multiplication and accumulation circuit.

[0113] As shown in FIG. 8, pixel 400 can include a photoelectric conversion device 401, a transistor 402, a transistor 403, a transistor 404, a transistor 405, a transistor 406, and a capacitor 407.

[0114] One electrode of the photoelectric conversion device 401 is electrically connected to one of the source or drain of the transistor 402. The other of the source or drain of the transistor 402 is electrically connected to one of the source or drain of the transistor 403, the gate of the transistor 404, and one electrode of the capacitor 407. One of the source or drain of the transistor 404 is electrically connected to one of the source or drain of the transistor 405. The other electrode of the capacitor 407 is electrically connected to one of the source or drain of the transistor 406.

[0115] The other electrode of the photoelectric conversion device 401 is electrically connected to the wiring 414. The other of the source or drain of the transistor 403 is electrically connected to the wiring 415. The other of the source or drain of the transistor 405 is electrically connected to the wiring 412. The other of the source or drain of the transistor 404 is electrically connected to a GND wiring or the like. The other of the source or drain of the transistor 406 is electrically connected to the wiring 411. The other electrode of the capacitor 407 is electrically connected to the wiring 417.

[0116] The gate of the transistor 402 is electrically connected to the wiring 421. The gate of the transistor 403 is electrically connected to the wiring 422. The gate of the transistor 405 is electrically connected to the wiring 423. The gate of the transistor 406 is electrically connected to the wiring 424.

[0117] Here, an electrical connection point between the other of the source or drain of the transistor 402, one of the source or drain of the transistor 403, one electrode of the capacitor 407, and the gate of the transistor 404 is defined as the node FD. Also, an electrical connection point between the other electrode of the capacitor 407 and one of the source or drain of the transistor 406 is defined as the node FDW.

[0118] The wiring 414 and the wiring 415 can function as power supply lines. For example, the wiring 414 can function as a high-potential power supply line, and the wiring 415 can function as a low-potential power supply line. The wiring 421, the wiring 422, the wiring 423, and the wiring 424 can function as signal lines for controlling the conduction of each transistor. The wiring 411 can function as a wiring for supplying a potential corresponding to the weighting factor to the pixel 400. The wiring 412 can function as a wiring for electrically connecting the pixel 400 and the circuit 201. The wiring 417 can function as a wiring for electrically connecting the other electrode of the capacitor 407 of the pixel and the other electrode of the capacitor 407 of another pixel via the transistor 450 (see FIG. 7).

[0119] Note that an amplification circuit or a gain adjustment circuit may be electrically connected to the wiring 412.

[0120] As the photoelectric conversion device 401, a photodiode can be used. Regardless of the type of photodiode, an Si photodiode having silicon in the photoelectric conversion layer, an organic photodiode having an organic photoconductive film in the photoelectric conversion layer, or the like can be used. When it is desired to increase the light detection sensitivity at low illuminance levels, it is preferable to use an avalanche photodiode.

[0121] The transistor 402 can have a function of controlling the potential of the node FD. The transistor 403 can have a function of initializing the potential of the node FD. The transistor 404 can have a function of controlling the current flowing through the circuit 201 according to the potential of the node FD. The transistor 405 can have a function of selecting a pixel. The transistor 406 can have a function of supplying a potential corresponding to the weighting factor to the node FDW.

[0122] When an avalanche photodiode is used for the photoelectric conversion device 401, a high voltage may be applied, and it is preferable to use a high breakdown voltage transistor for the transistor connected to the photoelectric conversion device 401. As the high breakdown voltage transistor, for example, a transistor using a metal oxide in the channel formation region (hereinafter, OS transistor) or the like can be used. Specifically, it is preferable to apply an OS transistor to the transistor 402.

[0123] In addition, the OS transistor also has a characteristic of extremely low off-current. By using the OS transistor for the transistor 402, the transistor 403, and the transistor 406, the period during which charges can be held at the node FD and the node FDW can be made extremely long. Therefore, without complicating the circuit configuration or the operation method, a global shutter method in which the charge accumulation operation is performed simultaneously for all pixels can be applied. Also, while holding the image data at the node FD, a plurality of operations using the image data can be performed.

[0124] On the other hand, the transistor 404 may desirably have excellent amplification characteristics. Also, for the transistor 406, it may be preferable to use a transistor with high mobility capable of high-speed operation. Therefore, for the transistor 404 and the transistor 406, a transistor using silicon in the channel formation region (hereinafter, Si transistor) may be applied.

[0125] Note that, not limited to the above, an OS transistor and an Si transistor may be arbitrarily combined and applied. Also, all the transistors may be OS transistors. Or, all the transistors may be Si transistors. Examples of Si transistors include transistors having amorphous silicon, transistors having crystalline silicon (microcrystalline silicon, low-temperature polysilicon, single-crystalline silicon), and the like.

[0126] The potential of the node FD in the pixel 400 is determined by the potential obtained by adding the reset potential supplied from the wiring 415 and the potential (image data) generated by the photoelectric conversion by the photoelectric conversion device 401. Or, further, the potential corresponding to the weight coefficient supplied from the wiring 411 is capacitively coupled and determined. Therefore, the transistor 405 can pass a current corresponding to the data obtained by adding an arbitrary weight coefficient to the image data.

[0127] Note that the above is an example of the circuit configuration of the pixel 400, and the photoelectric conversion operation can also be performed with other circuit configurations.

[0128] As shown in FIG. 7, the pixels 400 are electrically connected to each other by the wiring 412. The circuit 201 can perform an operation using the sum of the currents flowing through the transistors 404 of the pixels 400.

[0129] The circuit 201 includes a capacitor 202, a transistor 203, a transistor 204, a transistor 205, a transistor 206, and a resistor 207.

[0130] One electrode of the capacitor 202 is electrically connected to one of the source or drain of the transistor 203. One of the source or drain of the transistor 203 is electrically connected to the gate of the transistor 204. One of the source or drain of the transistor 204 is electrically connected to one of the source or drain of the transistor 205. One of the source or drain of the transistor 205 is electrically connected to one of the source or drain of the transistor 206. One electrode of the resistor 207 is electrically connected to the other electrode of the capacitor 202.

[0131] The other electrode of the capacitor 202 is electrically connected to the wiring 412. The other of the source or drain of the transistor 203 is electrically connected to the wiring 218. The other of the source or drain of the transistor 204 is electrically connected to the wiring 219. The other of the source or drain of the transistor 205 is electrically connected to a reference power line such as a GND wiring. The other of the source or drain of the transistor 206 is electrically connected to the wiring 212. The other electrode of the resistor 207 is electrically connected to the wiring 217.

[0132] The wiring 217, the wiring 218, and the wiring 219 can have a function as a power line. For example, the wiring 218 can have a function as a wiring for supplying a dedicated potential for reading. The wiring 217 and the wiring 219 can function as high potential power lines. The wiring 213, the wiring 215, and the wiring 216 can function as signal lines for controlling the conduction of each transistor. The wiring 212 is an output line and can be electrically connected to, for example, the circuit 301 shown in FIG. 6.

[0133] The transistor 203 can have a function of resetting the potential of the wiring 211 to the potential of the wiring 218. The wiring 211 is a wiring connected to one electrode of the capacitor 202, one of the source or drain of the transistor 203, and the gate of the transistor 204. The transistors 204 and 205 can have a function as a source follower circuit. The transistor 206 can have a function of controlling reading. Note that the circuit 201 has a function as a correlated double sampling circuit (CDS circuit), and can be replaced with a circuit of another configuration having the same function.

[0134] In one aspect of the present invention, an offset component other than the product of the image data (X) and the weight coefficient (W) is removed, and the target WX is extracted. WX can be calculated using data with and without imaging for the same pixel, and data when weights are added to each of them.

[0135] The total current (I p ) flowing through the pixel 400 when imaging is performed is kΣ(X - V th ) 2 , and the total current (I p ) flowing through the pixel 400 when weights are added is kΣ(W + X - V th ) 2 . Also, the total current (I ref ) flowing through the pixel 400 when no imaging is performed is kΣ(0 - V th ) 2 , and the total current (I ref ) flowing through the pixel 400 when weights are added is kΣ(W - V th ) 2 . Here, k is a constant, and V th is the threshold voltage of the transistor 405.

[0136] First, a difference (data A) between the data with imaging and the data with weights added to the data is calculated. kΣ((X - V th ) 2 - (W + X - V th ) 2 ) = kΣ(-W 2 - 2W·X + 2W·Vth ) is obtained.

[0137] Next, the difference (data B) between the data without imaging and the data with weights added to the data is calculated. kΣ((0 - V th )) 2 - (W - V th )) 2 ) = kΣ(-W 2 + 2W·V th ) is obtained.

[0138] Then, the difference between data A and data B is taken. kΣ(-W 2 - 2W·X + 2W·V th - (-W 2 + 2W·V th )) = kΣ(-2W·X) is obtained. That is, the offset components other than the product of the image data (X) and the weight coefficient (W) can be removed.

[0139] In circuit 201, data A and data B can be read out. Note that the difference operation between data A and data B can be performed, for example, in circuit 301.

[0140] Here, the weights supplied to the entire pixel block 200 function as a filter. As such a filter, for example, a convolutional filter of a convolutional neural network (CNN) can be used. Or, an image processing filter such as an edge extraction filter can be used. As an edge extraction filter, for example, the Laplacian filter shown in FIG. 9A, the Prewitt filter shown in FIG. 9B, the Sobel filter shown in FIG. 9C, etc. can be cited as an example.

[0141] When the number of pixels 400 included in the pixel block 200 is 3×3, the elements of the edge extraction filter can be supplied to each pixel 400 as weights. As described above, in order to calculate the data A and the data B, it is possible to perform the calculation using the data with and without imaging and the data when weights are added to each of them. Here, the data with and without imaging is data without adding weights, and can also be paraphrased as data obtained by adding a weight of 0 to all the pixels 400.

[0142] The edge extraction filter illustrated in FIGS. 9A to 9C is a filter in which the sum of the elements (weights: ΔW) of the filter (ΣΔW / N, where N is the number of elements) is 0. Therefore, even without performing an operation of newly supplying ΔW = 0 from another circuit, by performing an operation of obtaining ΣΔW / N, it is possible to obtain data obtained by adding an equivalent of ΔW = 0 to all the pixels 400.

[0143] This operation corresponds to turning on the transistors 450 (transistors 450a to 450f) provided between the pixels 400 (see FIG. 7). By turning on the transistors 450, the nodes FDW of each pixel 400 are all short-circuited via the wiring 417. At this time, the charges accumulated in the nodes FDW of each pixel 400 are redistributed, and when the edge extraction filter illustrated in FIGS. 9A to 9C is used, the potential (ΔW) of the node FDW becomes 0 or substantially 0. Therefore, it is possible to obtain data obtained by adding an equivalent of ΔW = 0.

[0144] When rewriting the weight (ΔW) by supplying charges from a circuit outside the pixel array 300, it takes time until the rewriting is completed due to the capacitance of the long wiring 411 and the like. On the other hand, the pixel block 200 is a minute area, and the distance of the wiring 417 is short and the capacitance is small. Therefore, in the operation of redistributing the charges accumulated in the nodes FDW in the pixel block 200, the weight (ΔW) can be rewritten at high speed.

[0145] In the pixel block 200 shown in FIG. 7, transistors 450a to 450f are each electrically connected to different gate lines (wiring 413a to 413f). In this configuration, the conduction of transistors 450a to 450f can be independently controlled, and the operation of obtaining ΣΔW / N can be selectively performed.

[0146] For example, when using the filter shown in FIG. 9B or FIG. 9C, etc., there are pixels where ΔW = 0 is initially supplied. Assuming that ΣΔW / N = 0, the pixels where ΔW = 0 may be excluded from the pixels to be summed. By excluding such pixels, the supply of potential for operating some of transistors 450a to 450f becomes unnecessary, so that the power consumption can be suppressed.

[0147] The data of the sum-of-products operation result output from circuit 201 is sequentially input to circuit 301. Circuit 301 may have various arithmetic functions in addition to the function of calculating the difference between data A and data B described above. For example, circuit 301 can have the same configuration as circuit 201. Or, the function of circuit 301 may be replaced by software processing.

[0148] Also, circuit 301 may have a circuit for performing the operation of an activation function. For example, a comparator circuit can be used for such a circuit. In the comparator circuit, the result of comparing the input data with a set threshold value is output as binary data. That is, pixel block 200 and circuit 301 can act as some elements of a neural network.

[0149] The data output from circuit 301 is sequentially input to circuit 302. Circuit 302 can be configured to have, for example, a latch circuit and a shift register. With this configuration, parallel-serial conversion can be performed, and the data input in parallel can be output as serial data to wiring 311.

[0150] [Configuration Example of Pixel] FIG. 10A is a diagram showing a configuration example of pixel 400. Pixel 400 can have a stacked structure of layer 561 and layer 563.

[0151] Layer 561 has a photoelectric conversion device 401. The photoelectric conversion device 401 can have layer 565a and layer 565b as shown in FIG. 10B. In some cases, the layer may be referred to as a region.

[0152] The photoelectric conversion device 401 shown in FIG. 10B is a pn junction type photodiode. For example, a p-type semiconductor can be used for layer 565a and an n-type semiconductor can be used for layer 565b. Alternatively, an n-type semiconductor can be used for layer 565a and a p-type semiconductor can be used for layer 565b.

[0153] The above pn junction type photodiode can typically be formed using single crystal silicon.

[0154] In addition, the photoelectric conversion device 401 included in layer 561 may be a stack of layer 566a, layer 566b, layer 566c, and layer 566d as shown in FIG. 10C. The photoelectric conversion device 401 shown in FIG. 10C is an example of an avalanche photodiode. Layer 566a and layer 566d correspond to electrodes, and layer 566b and layer 566c correspond to a photoelectric conversion section.

[0155] Layer 566a is preferably a low-resistance metal layer or the like. For example, aluminum, titanium, tungsten, tantalum, silver, or a stack thereof can be used.

[0156] For layer 566d, it is preferable to use a conductive layer having high light transmittance for visible light. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, or graphene can be used. Note that a configuration in which layer 566d is omitted is also possible.

[0157] The layers 566b and 566c of the photoelectric conversion unit can be configured as a pn junction photodiode using, for example, a selenium-based material as the photoelectric conversion layer. As the layer 566b, it is preferable to use a selenium-based material that is a p-type semiconductor, and as the layer 566c, it is preferable to use a gallium oxide or the like that is an n-type semiconductor.

[0158] A photoelectric conversion device using a selenium-based material has a characteristic of high external quantum efficiency for visible light. In this photoelectric conversion device, by utilizing avalanche multiplication, the amplification of electrons with respect to the amount of incident light can be increased. Further, since the selenium-based material has a high light absorption coefficient, it has production advantages such as being able to fabricate the photoelectric conversion layer as a thin film. The thin film of the selenium-based material can be formed using a vacuum evaporation method, a sputtering method, or the like.

[0159] As the selenium-based material, crystalline selenium such as single crystal selenium or polycrystalline selenium, amorphous selenium, a compound of copper, indium, and selenium (CIS), or a compound of copper, indium, gallium, and selenium (CIGS) can be used.

[0160] The n-type semiconductor is preferably formed of a material having a wide bandgap and being transparent to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or an oxide in which they are mixed can be used. Further, these materials also have a function as a hole injection blocking layer and can also reduce the dark current.

[0161] Further, as shown in FIG. 10D, the photoelectric conversion device 401 included in the layer 561 may be a laminate of the layer 567a, the layer 567b, the layer 567c, the layer 567d, and the layer 567e. The photoelectric conversion device 401 shown in FIG. 10D is an example of an organic photoconductive film, the layer 567a is a lower electrode, the layer 567e is a light-transmissive upper electrode, and the layers 567b, 567c, and 567d correspond to the photoelectric conversion unit.

[0162] One of the layers 567b or 567d of the photoelectric conversion unit can be a hole transport layer. Further, the other of the layers 567b or 567d can be an electron transport layer. Further, the layer 567c can be a photoelectric conversion layer.

[0163] As the hole transport layer, for example, molybdenum oxide or the like can be used. As the electron transport layer, for example, C 60 、C 70 fullerenes such as, or derivatives thereof can be used.

[0164] As the photoelectric conversion layer, a mixed layer (bulk heterojunction structure) of an n-type organic semiconductor and a p-type organic semiconductor can be used.

[0165] The layer 563 shown in FIG. 10A includes, for example, a silicon substrate. On the silicon substrate, an Si transistor or the like is provided. Using the Si transistor, the pixel 400 can be formed. Further, the circuit 201 and the circuits 301 to 306 shown in FIG. 6 can be formed.

[0166] Next, the stacked structure of the imaging device will be described using a cross-sectional view. Note that the elements such as the insulating layer and the conductive layer shown below are examples, and other elements may be further included. Or, some of the elements shown below may be omitted. Further, the stacked structure shown below can be formed using a bonding process, a polishing process, or the like as necessary.

[0167] The imaging device having the configuration shown in FIG. 11 includes the layer 560, the layer 561, and the layer 563. In FIG. 11, as the elements provided in the layer 563, the transistor 402 and the transistor 403 are shown, but other elements such as the transistors 404 to 406 can also be provided in the layer 563.

[0168] The layer 563 is provided with a silicon substrate 632, an insulating layer 633, an insulating layer 634, an insulating layer 635, and an insulating layer 637. Further, a conductive layer 636 is provided.

[0169] The insulating layers 634, 635, and 637 have functions as an interlayer insulating film and a planarization film. The insulating layer 633 has a function as a protective film. The conductive layer 636 is electrically connected to the wiring 414 shown in FIG. 8.

[0170] As the interlayer insulating film and the planarization film, for example, an inorganic insulating film such as a silicon oxide film, or an organic insulating film such as an acrylic resin or a polyimide resin can be used. As the protective film, for example, a silicon nitride film, a silicon oxide film, an aluminum oxide film, etc. can be used.

[0171] As the conductive layer, a metal element selected from aluminum, chromium, copper, silver, gold, platinum, tantalum, nickel, titanium, molybdenum, tungsten, hafnium, vanadium, niobium, manganese, magnesium, zirconium, beryllium, indium, ruthenium, iridium, strontium, lanthanum, etc., or an alloy containing the above-mentioned metal element as a component, or an alloy combining the above-mentioned metal elements can be appropriately selected and used. The conductor is not limited to a single layer, and may be a plurality of layers composed of different materials.

[0172] The Si transistor shown in FIG. 11 is a fin type having a channel formation region on a silicon substrate. A cross-section in the channel width direction (the cross-section of A1 - A2 shown in layer 563 of FIG. 11) is shown in FIG. 12A. Note that the Si transistor may be a planar type as shown in FIG. 12B.

[0173] Alternatively, as shown in FIG. 12C, a transistor having a semiconductor layer 545 of a silicon thin film may be used. The semiconductor layer 545 can be, for example, single-crystalline silicon (SOI: Silicon on Insulator) formed on an insulating layer 546 on a silicon substrate 632.

[0174] The photoelectric conversion device 401 is provided on the layer 561. The photoelectric conversion device 401 can be formed on the layer 563. In FIG. 11, as the photoelectric conversion device 401, a configuration using the organic photoconductive film shown in FIG. 10D as the photoelectric conversion layer is shown. Here, the layer 567a is used as the cathode and the layer 567e is used as the anode.

[0175] An insulating layer 651, an insulating layer 652, an insulating layer 653, an insulating layer 654, and a conductive layer 655 are provided on the layer 561.

[0176] The insulating layer 651, the insulating layer 653, and the insulating layer 654 have functions as an interlayer insulating film and a planarizing film. Further, the insulating layer 654 is provided to cover the end portion of the photoelectric conversion device 401 and also has a function of preventing a short circuit between the layer 567e and the layer 567a. The insulating layer 652 has a function as an element isolation layer. As the element isolation layer, an organic insulating film or the like is preferably used.

[0177] The layer 567a corresponding to the cathode of the photoelectric conversion device 401 is electrically connected to one of the source or drain of the transistor 402 included in the layer 563. The layer 567e corresponding to the anode of the photoelectric conversion device 401 is electrically connected to the conductive layer 636 provided in the layer 563 via the conductive layer 655.

[0178] The layer 560 is formed on the layer 561. The layer 560 has a light-shielding layer 671 and a microlens array 673.

[0179] The light-shielding layer 671 can suppress the inflow of light into adjacent pixels. As the light-shielding layer 671, a metal layer such as aluminum or tungsten can be used. Further, a dielectric film having a function as an antireflection film may be laminated on the metal layer.

[0180] A microlens array 673 is provided on the photoelectric conversion device 401. Light passing through the individual lenses of the microlens array 673 irradiates the underlying photoelectric conversion device 401. By providing the microlens array 673, the condensed light can be incident on the photoelectric conversion device 401, so that photoelectric conversion can be performed efficiently. The microlens array 673 is preferably formed of a resin, glass, or the like that is highly transparent to light of the wavelength of the imaging target.

[0181] FIG. 13 is a modified example of the stacked structure shown in FIG. 11, and the configuration of the photoelectric conversion device 401 included in the layer 561 and a part of the configuration of the layer 563 are different. In the configuration shown in FIG. 13, there is a bonding surface between the layer 561 and the layer 563.

[0182] The layer 561 includes a photoelectric conversion device 401, an insulating layer 661, an insulating layer 662, an insulating layer 664, and an insulating layer 665, as well as a conductive layer 685 and a conductive layer 686.

[0183] The photoelectric conversion device 401 is a pn junction type photodiode formed on a silicon substrate, and has a layer 565b corresponding to a p-type region and a layer 565a corresponding to an n-type region. The photoelectric conversion device 401 is an embedded type photodiode, and the dark current can be suppressed and the noise can be reduced by a thin p-type region (a part of the layer 565b) provided on the surface side (current extraction side) of the layer 565a.

[0184] The insulating layer 661, and the conductive layers 685 and 686 have a function as a bonding layer. The insulating layer 662 has a function as an interlayer insulating film and a planarizing film. The insulating layer 664 has a function as an element isolation layer. The insulating layer 665 has a function of suppressing the outflow of carriers.

[0185] The silicon substrate is provided with grooves for separating pixels, and the insulating layer 665 is provided on the upper surface of the silicon substrate and in the grooves. By providing the insulating layer 665, it is possible to suppress carriers generated in the photoelectric conversion device 401 from flowing out to adjacent pixels. In addition, the insulating layer 665 also has a function of suppressing the intrusion of stray light. Therefore, color mixing can be suppressed by the insulating layer 665. Note that an antireflection film may be provided between the upper surface of the silicon substrate and the insulating layer 665.

[0186] The element isolation layer can be formed using the LOCOS (LOCal Oxidation of Silicon) method. Or, it may be formed using the STI (Shallow Trench Isolation) method or the like. As the insulating layer 665, for example, an inorganic insulating film such as silicon oxide or silicon nitride, or an organic insulating film such as polyimide resin or acrylic resin can be used. Note that the insulating layer 665 may have a multilayer structure. Note that a configuration without providing an element isolation layer can also be adopted.

[0187] The layer 565a (n-type region, corresponding to the cathode) of the photoelectric conversion device 401 is electrically connected to the conductive layer 685. The layer 565b (p-type region, corresponding to the anode) is electrically connected to the conductive layer 686. The conductive layer 685 and the conductive layer 686 have regions embedded in the insulating layer 661. In addition, the surfaces of the insulating layer 661, the conductive layer 685, and the conductive layer 686 are flattened so that their heights are the same.

[0188] In the layer 563, an insulating layer 638 is formed on the insulating layer 637. In addition, a conductive layer 683 electrically connected to one of the source or drain of the transistor 402 and a conductive layer 684 electrically connected to the conductive layer 636 are formed.

[0189] The insulating layer 638, and the conductive layers 683 and 684 have the function as a bonding layer. The conductive layer 683 and the conductive layer 684 have regions embedded in the insulating layer 638. Also, the surfaces of the insulating layer 638, the conductive layer 683, and the conductive layer 684 are flattened so that their heights are the same, respectively.

[0190] Here, it is preferable that the main components of the conductive layer 683 and the conductive layer 685 are composed of the same metal element, and it is preferable that the main components of the conductive layer 684 and the conductive layer 686 are composed of the same metal element. Also, it is preferable that the main components of the insulating layer 638 and the insulating layer 661 are the same.

[0191] For example, for the conductive layers 683 to 686, Cu, Al, Sn, Zn, W, Ag, Pt, Au, etc. can be used. From the ease of bonding, it is particularly preferable to use Cu, Al, W, or Au. Also, for the insulating layer 638 and the insulating layer 661, silicon oxide, silicon oxynitride, silicon nitride oxide, silicon nitride, titanium nitride, etc. can be used.

[0192] That is, it is preferable to use the same metal material shown above for each of the conductive layers 683 to 686. Also, it is preferable to use the same insulating material shown above for each of the insulating layer 638 and the insulating layer 661. With such a configuration, bonding can be performed with the boundary between the layer 563 and the layer 561 as the bonding position.

[0193] Note that the conductive layers 683 to 686 may have a multilayer structure composed of a plurality of layers. In that case, it is sufficient that the surface layer (bonding surface) is the same metal material. Also, the insulating layer 638 and the insulating layer 661 may also have a multilayer structure of a plurality of layers. In that case, it is sufficient that the surface layer (bonding surface) is the same insulating material.

[0194] Through this bonding, the conductive layer 683 and the conductive layer 685 can be electrically connected to each other, and the conductive layer 684 and the conductive layer 686 can be electrically connected to each other. Also, a mechanically strong connection between the insulating layer 661 and the insulating layer 638 can be obtained.

[0195] For the bonding between metal layers, a surface activation bonding method can be used, in which the oxide film on the surface and the adsorbed layer of impurities are removed by sputtering or the like, and the cleaned and activated surfaces are brought into contact with each other for bonding. Alternatively, a diffusion bonding method or the like in which the surfaces are bonded together using a combination of temperature and pressure can be used. Since bonding occurs at the atomic level in both cases, excellent bonding can be obtained not only electrically but also mechanically.

[0196] Also, for the bonding between insulating layers, after obtaining high flatness by polishing or the like, a hydrophilic bonding method or the like can be used, in which the surfaces that have been subjected to hydrophilic treatment with oxygen plasma or the like are brought into contact with each other for temporary bonding, and permanent bonding is performed by dehydration through heat treatment. Since the hydrophilic bonding method also involves bonding at the atomic level, excellent mechanical bonding can be obtained.

[0197] When laminating the layer 563 and the layer 561, since the metal layer and the insulating layer are mixed on each bonding surface, for example, a combination of the surface activation bonding method and the hydrophilic bonding method can be used.

[0198] For example, a method can be used in which the surface is cleaned after polishing, an antioxidant treatment is performed on the surface of the metal layer, and then a hydrophilic treatment is performed for bonding. Also, the surface of the metal layer can be made of a metal with low oxidation resistance such as Au, and a hydrophilic treatment can be performed. In addition, a bonding method other than the methods described above may be used.

[0199] Through the above lamination, the elements of the layer 563 and the elements of the layer 561 can be electrically connected.

[0200] FIG. 14 is a modified example of the laminated structure shown in FIG. 13, and a partial configuration of the layer 561 and the layer 563 is different.

[0201] In this modified example, the transistor 402 included in the pixel 400 is provided in the layer 561. In the layer 561, the transistor 402 covered with the insulating layer 663 is formed of an Si transistor. One of the source or drain of the transistor 402 is directly connected to one electrode of the photoelectric conversion device 401. Also, the other of the source or drain of the transistor 402 is electrically connected to the node FD.

[0202] In the imaging device shown in FIG. 14, in the layer 563, transistors other than at least the transistor 402 among the transistors constituting the imaging device are provided. In FIG. 14, as elements provided in the layer 563, the transistors 404 and 405 are shown, but other elements such as the transistor 403 and the transistor 406 can also be provided in the layer 563. Further, in the layer 563 of the imaging device shown in FIG. 14, an insulating layer 647 is provided between the insulating layer 635 and the insulating layer 637. The insulating layer 647 has functions as an interlayer insulating film and a planarization film.

[0203] (Embodiment 7) In the present embodiment, a package containing an imaging unit, a so-called image sensor chip, will be described below.

[0204] FIG. 15A1 is an external perspective view of the upper surface side of a package containing an image sensor chip. The package includes a package substrate 410 for fixing the image sensor chip 452 (see FIG. 15A3), a cover glass 420, an adhesive 430 for bonding the two, and the like.

[0205] FIG. 15A2 is an external perspective view of the lower surface side of the package. The lower surface of the package has a BGA (Ball Grid Array) with solder balls as bumps 440. Note that it may have not only a BGA but also an LGA (Land Grid Array) or a PGA (Pin Grid Array).

[0206] FIG. 15A3 is a perspective view of a package shown with a part of the cover glass 420 and the adhesive 430 omitted. An electrode pad 460 is formed on the package substrate 410, and the electrode pad 460 and the bump 440 are electrically connected via a through hole. The electrode pad 460 is electrically connected to the image sensor chip 452 by a wire 470.

[0207] Also, FIG. 15B1 is an external perspective view of the upper surface side of a camera module in which an image sensor chip is housed in a lens-integrated package. The camera module includes a package substrate 431 (fixing the image sensor chip 451 (FIG. 15B3), a lens cover 432, a lens 435, etc. Further, an IC chip 490 (FIG. 15B3) having functions such as a drive circuit and a signal conversion circuit of the imaging device is provided between the package substrate 431 and the image sensor chip 451, and has a configuration as a SiP (System in package).

[0208] FIG. 15B2 is an external perspective view of the lower surface side of the camera module. The lower surface and side surfaces of the package substrate 431 have a QFN (Quad flat no-lead package) configuration in which mounting lands 441 are provided. Note that this configuration is an example, and a QFP (Quad flat package) or the above-described BGA may be provided.

[0209] FIG. 15B3 is a perspective view of the module shown with a part of the lens cover 432 and the lens 435 omitted. The land 441 is electrically connected to the electrode pad 461, and the electrode pad 461 is electrically connected to the image sensor chip 451 or the IC chip 490 by a wire 471.

[0210] By housing the image sensor chip in a package in the above-described form, mounting on a printed circuit board or the like becomes easy, and the image sensor chip can be incorporated into various semiconductor devices and electronic devices.

[0211] This embodiment can be appropriately combined with the descriptions of other embodiments.

[0212] (Embodiment 8) Provided is a driving support device suitable for a vehicle performing semi-automatic driving, using a driving support system according to the above-described embodiment.

[0213] In Japan, for driving support systems of vehicles such as automobiles, the automation level is defined in four levels from level 1 to level 4. Level 1 means automating any one of acceleration, steering, and braking, and is called a safe driving support system. Level 2 simultaneously automates a plurality of operations among acceleration, steering, and braking, and is called a semi-automatic driving system (also referred to as semi-automatic driving). Level 3 automates all of acceleration, steering, and braking, and the driver responds only in an emergency, and this is also called a semi-automatic driving system (also referred to as semi-automatic driving). Level 4 automates all of acceleration, steering, and braking, and is called fully automatic driving in which the driver is hardly involved.

[0214] In this specification, in level 2 or level 3, a new configuration or a new driving support system is proposed mainly on the premise of semi-automatic driving.

[0215] In order to display a warning to the driver according to the situation obtained from various cameras or sensors, an area of the display region corresponding to the number of each camera or the number of sensors is required.

[0216] Also, FIG. 16A shows an external view of the vehicle 120. FIG. 16A shows an example of the installation locations of the front image sensor 114a and the left-side image sensor 114L. Further, FIG. 16B is a schematic diagram showing the view of the driver in front from inside the vehicle. The upper part of the driver's field of view is the windshield 110, and a display device 111 having a display screen is installed in the lower part of the field of view.

[0217] The upper part of the driver's field of view is the windshield 110, and the windshield 110 is sandwiched between the pillars 112. In FIG. 16A, an example of installing the front image sensor 114a at a position close to the driver's line of sight is shown, but it is not particularly limited and may be installed on the front grille or the front bumper. Also, in the present embodiment, a right-hand drive vehicle is shown as an example, but it is not particularly limited. If it is a left-hand drive vehicle, it may be installed according to the position of the driver.

[0218] Among these image sensors, it is preferable to use the image sensor chip shown in Embodiment 7 for at least one of them.

[0219] The driver accelerates, steers, and brakes mainly while looking at the display device 111, and additionally checks the outside of the vehicle through the windshield. The display device 111 may use any one of a liquid crystal display device, an EL (Electro Luminescence) display device, and a micro LED (Light Emitting Diode) display device. Here, an LED chip with a side dimension exceeding 1 mm is called a macro LED, an LED chip larger than 100 μm and less than or equal to 1 mm is called a mini LED, and an LED chip less than or equal to 100 μm is called a micro LED. It is particularly preferable to use a micro LED as the LED element applied to the pixel. By using a micro LED, a very high-definition display device can be realized. The higher the fineness of the display device 111, the more preferable it is. The pixel density of the display device 111 can be set to a pixel density of 100 ppi or more and 5000 ppi or less, preferably 200 ppi or more and 2000 ppi or less.

[0220] For example, the central part 111a of the display screen of the display device displays an image acquired from an imaging device installed in front of the vehicle outside. Also, on a part 111b, 111c of the display screen, meter displays such as speed, predicted travelable distance, and abnormal warning display are performed. Also, on the lower left part 111L of the display screen, an image of the left side outside the vehicle is displayed, and on the lower right part 111R of the display screen, an image of the right side outside the vehicle is displayed.

[0221] The lower left part 111L and the lower right part 111R of the display screen can also eliminate the side mirror protruding parts (also called door mirrors) that electronically transform the side mirrors and protrude significantly outside the vehicle.

[0222] By making the display screen of the display device 111 touch-input operable, it may be configured to enlarge, reduce, change the display position, expand the area of the display region, etc. of a part of the video.

[0223] Since the image on the display screen of the display device 111 is synthesized from data from a plurality of imaging devices or sensors, it is created using an image signal processing device such as a GPU.

[0224] By using the driving support system shown in Embodiment 1, black-and-white image data with a wide dynamic range can be acquired, only the distant region can be extracted, inference can be performed and colorized, and the emphasized image can be enlarged and output to the display device 111.

[0225] By appropriately using AI, the driver can mainly operate the vehicle by looking at the display image of the display device, that is, the image using the image sensor and AI, and looking at the front of the windshield can be used as an auxiliary. Operating the vehicle by looking at the image using AI rather than driving only with the driver's eyes can be a safe driving. Also, the driver can operate the vehicle while obtaining a sense of security.

[0226] Note that the display device 111 can be applied around the driver's seat (also called the cockpit part) of various types of vehicles including large vehicles, medium-sized vehicles, and small vehicles. It can also be applied around the driver's seat of vehicles such as airplanes and ships.

[0227] Also, in this embodiment, an example is shown in which the front image sensor 114a is installed below the windshield, but it is not particularly limited, and the imaging camera shown in FIG. 17 may be installed on the bonnet or around the rearview mirror inside the vehicle.

[0228] The imaging camera in Fig. 17 can also be called a drive recorder and has a housing 961, a lens 962, a support part 963, etc. By attaching a double-sided tape or the like to the support part 963, it can be installed on the front glass, the bonnet, the rearview mirror support part, or the like.

[0229] An image sensor is arranged in the imaging camera in Fig. 17, and the running video can be recorded and stored in the imaging camera or a memory device mounted on the vehicle.

[0230] This embodiment can be freely combined with other embodiments.

Explanation of Reference Numerals

[0231] 10: Data acquisition device, 11: Solid-state imaging device, 12: Analog arithmetic circuit, 13: A / D circuit, 14: Memory section, 15: Display device, 16a: Neural network section, 16b: Neural network section, 16c: Neural network section, 17: Data extraction section, 18a: Storage section, 18b: Storage section, 18c: Storage section, 19: Display section, 21: Display example, 22: Remote area, 23: Colorized image, 24: Emphasized image, 31: Driving assistance system, 41: Imaging system, 110: Windshield, 111: Display device, 111a: Central part, 111b: Part, 111c: Part, 111L: Lower left part, 111R: Lower right part, 112: Pillar, 114a: Front image sensor, 114L: Left side image sensor, 120: Vehicle, 200: Pixel block, 201: Circuit, 202: Capacitor, 203: Transistor, 204: Transistor, 205: Transistor, 206: Transistor, 207: Resistor, 211: Wiring, 212: Wiring, 213: Wiring, 215: Wiring, 216: Wiring, 217: Wiring, 218: Wiring, 219: Wiring, 300: Pixel array, 301: Circuit, 302: Circuit, 303: Circuit, 304: Circuit, 305: Circuit, 306: Circuit, 311: Wiring, 400: Pixel, 401: Photoelectric conversion device, 402: Transistor, 403: Transistor, 404: Transistor, 405: Transistor, 406: Transistor, 407: Capacitor, 410: Package substrate, 411: Wiring, 412: Wiring, 413: Wiring, 413a: Wiring, 413b: Wiring, 413c: Wiring, 413d: Wiring, 413e: Wiring, 413f: Wiring, 413g: Wiring, 413h: Wiring, 413i: Wiring, 413j: Wiring, 414: Wiring, 415: Wiring, 417: Wiring, 420: Cover glass, 421: Wiring, 422: Wiring, 423: Wiring, 424: Wiring, 430: Adhesive, 431: Package substrate, 432: Lens cover, 435: Lens, 440: Bump, 441: Land, 450: Transistor, 450a: Transistor, 450b: Transistor, 450c: Transistor, 450d: Transistor, 450e: Transistor, 450f: Transistor, 450g: Transistor, 450h: Transistor, 450i: Transistor, 450j: Transistor, 451: Image sensor chip, 452: Image sensor chip, 460: Electrode pad461: Electrode pad, 470: Wire, 471: Wire, 490: IC chip, 545: Semiconductor layer, 546: Insulating layer, 560: Layer, 561: Layer, 563: Layer, 565a: Layer, 565b: Layer, 566a: Layer, 566b: Layer, 566c: Layer, 566d: Layer, 567a: Layer, 567b: Layer, 567c: Layer, 567d: Layer, 567e: Layer, 632: Silicon substrate, 633: Insulating layer, 634: Insulating layer, 635: Insulating layer, 636: Conductive layer, 637: Insulating layer, 638: Insulating layer, 647: Insulating layer, 651: Insulating layer, 652: Insulating layer, 653: Insulating layer, 654: Insulating layer, 655: Conductive layer, 661: Insulating layer, 662: Insulating layer, 664: Insulating layer, 665: Insulating layer, 671: Light-shielding layer, 673: Microlens array, 683: Conductive layer, 684: Conductive layer, 685: Conductive layer, 686: Conductive layer, 961: Housing, 962: Lens, 963: Support part,

Claims

[Claim 1] A step of running a vehicle equipped with an imaging device; A step of capturing a black and white image of an area in front of a traveling vehicle by the imaging device; performing a segmentation process on the black and white image including the far region to infer at least sky, car and road regions; performing a depth estimation process on the monochrome image including the distant region to infer a specific distant region; determining a center of a crop from the monochrome image based on the segmentation process and the depth estimation process; A step of extracting a rectangular area having the center as a central portion, inputting the extracted data, and performing a super-resolution process; A step of inputting an output result of the super-resolution processing and performing a colorization processing for accurately highlighting an object included in the distant region; and enlarging and displaying the colorized distant area.

Images