Image Processing System

Abstract

To provide a travel support system and an electronic apparatus for vehicles suitable for a vehicle which performs semiautomatic driving, and to provide an image processing system to be used for a display unit of the electronic apparatus for vehicles.SOLUTION: While getting on a vehicle, a direction of travel is imaged by an imaging device, and a region of a person who should be paid attention to is extracted from image data which are captured using a neural network. Thereafter, an intensified indication is performed by displaying an indication other than the person in black and white and coloring the person and a periphery thereon (in a color of high visual attractiveness other than black and white), thereby providing a video image for assisting driving of a driver.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present invention relates to a neural network and an image processing system using the same. Another aspect of the present invention relates to an electronic device using the neural network. Another aspect of the present invention relates to a vehicle using the neural network. An imaging system that processes images based on images obtained by a solid-state imaging element. A video surveillance system, a security system, a safety information providing system, or a driving support system using the imaging system.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. One embodiment of the present invention disclosed in the present specification relates to an object, a method, or a manufacturing method. One embodiment of the present invention relates to a process, a machine, a manufacture, or a composition of matter. Therefore, examples of the technical field of one embodiment of the present invention disclosed in the present specification include a semiconductor device, a display device, a light-emitting device, a power storage device, a memory device, an electronic device, a lighting device, an input device, an input / output device, a driving method thereof, or a manufacturing method thereof. One embodiment of the present invention relates to a vehicle or an electronic device for a vehicle provided in a vehicle.

[0003] Note that in this specification and the like, a semiconductor device refers to any device that can function by utilizing semiconductor characteristics. A transistor and a semiconductor circuit are one embodiment of a semiconductor device. Further, a memory device, a display device, an imaging device, and an electronic device may include a semiconductor device.

[0004] Another aspect of the present invention relates to a program using a neural network. One aspect of the present invention can be considered as a technology using AI (Artificial Intelligence).

[0005] The present invention also relates to a driving system that enables a vehicle such as an automobile to freely switch between a safety support driving state and a semi-automatic driving state. [Background technology]

[0006] A technology for forming a transistor using an oxide semiconductor thin film formed over a substrate has attracted attention. For example, Patent Document 1 discloses an imaging device in which a transistor that includes an oxide semiconductor and has an extremely low off-state current is used in a pixel circuit.

[0007] Patent Document 1 discloses a display device that corrects images in accordance with the external light environment. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2011-119711 A [Patent Document 2] JP 2018-36639 A Summary of the Invention [Problem to be solved by the invention]

[0009] With the advancement of technology, imaging devices equipped with solid-state imaging elements such as CMOS image sensors can easily capture high-quality images. In the next generation, imaging devices will be required to be equipped with even more intelligent functions.

[0010] One object of one embodiment of the present invention is to provide an imaging device capable of performing image processing. Another object is to provide an imaging device with low power consumption. Another object is to provide an imaging device with high reliability. Another object is to provide a novel imaging device or the like. Another object is to provide a driving method of the imaging device. Another object is to provide a novel semiconductor device or the like.

[0011] Another object of one embodiment of the present invention is to provide a driving assistance system and an electronic device for a vehicle that is suitable for a semi-autonomous driving vehicle, and to provide an image processing system for use in a display unit of the electronic device for the vehicle.

[0012] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0013] While in the vehicle, the image sensor captures the image in the direction of travel, and a neural network is used to extract areas of people to watch out for from the captured image data. After that, everything other than the person is displayed in black and white, and the person and its surroundings are colored (in a highly conspicuous color other than black and white) to highlight them, providing an image to assist the driver in driving.

[0014] Also, when driving in a dark environment, the headlights of oncoming vehicles illuminate the surroundings, making it difficult to see the surroundings. In particular, the pupils of the human eye become smaller, making it difficult to see the surroundings. Therefore, by using an image display by an imaging device with an exposure setting that can be set as desired, it becomes easier to ensure the safety of the surroundings. It is desirable for this image to have good visibility.

[0015] It is desirable to segment such images and highlight any humans detected in the image, but the detection is not limited to humans, it could also be vehicles, or both.

[0016] The invention disclosed in this specification is an image processing system that includes an image data acquisition means obtained by an imaging element, a calculation means that performs segmentation processing on the image data, and a synthesis means that detects a first region and other second regions through the segmentation processing, and generates second image data by overlaying the first image data with color information that makes the first region a highly conspicuous color.

[0017] In the above configuration, the first region is a person or a car that is labeled by the segmentation process. Also, the color information of the second image data is only three colors including color information that is a highly conspicuous color. Note that the color information here is called two colors, even if there are, for example, white and black and a combination of these, gray.

[0018] In addition, as the vehicle speed increases, the driver's visual field narrows. In addition, when the vehicle speed increases, the driver's ability to recognize the scenery outside the vehicle may decrease if the driver is older. In addition, elderly drivers may have poor visibility outside the vehicle even when driving in a dark environment.

[0019] It is particularly difficult for the human eye to recognize pedestrians wearing black, gray, or camouflage clothing. By having the system learn images of people in clothing that are difficult for the human eye to recognize, it is possible to improve the accuracy of detecting people in images.

[0020] Neural network processing is processing performed using an artificial neural network, and is realized by software calculations by a microcontroller. A microcontroller is a computer system built into a single integrated circuit (IC). When the scale of calculations or the data to be handled is large, a neural network may be configured by combining multiple ICs. In addition, a microcontroller equipped with Linux (registered trademark) is preferable because it can use free software, which can reduce the total cost for configuring a neural network. In addition, it is not limited to Linux (registered trademark), and other OS (operating systems) may be used. In addition, when neural network processing is executed by software, a computer in which a program constituting the software is built into the hardware, or a program may be installed from a network or a recording medium. A program recorded on a recording medium such as a computer-readable CD-ROM (Compact Disk Read Only Memory) is installed, and the program for neural network processing is executed. The processing performed by the program is not limited to processing in a sequence, and does not have to be chronological, and may be performed, for example, in parallel.

[0021] Another configuration of the invention disclosed in this specification is an image processing system that includes an image data acquisition means obtained by an imaging element, a calculation means that performs segmentation processing on the image data, and a synthesis means that detects human areas using the segmentation processing, and overlays black and white image data with color information that makes the detected human areas a highly conspicuous color to generate new image data including color information.

[0022] In the above configuration, the color information of the new image data is only color information regarding highly conspicuous colors, white information, and black information. Note that in this specification, although a combination of a highly conspicuous color and gray may be formed, it is considered that the color information and the number of colors do not increase. In the case of simple superimposition, superimposing a highly conspicuous color and black results in black. In this configuration, instead of simple superimposition, the image of a person wearing black clothing immediately after being captured by the imaging element is synthesized after the above neural network processing to create an image of the color of the clothing that is a combination of a highly conspicuous color and gray.

[0023] This makes it possible to make the background black and white and the person stand out as a color (excluding white and black). A person wearing black clothing becomes an image of a person wearing clothing that combines a highly eye-catching color with gray, making it easier to recognize on the display screen.

[0024] In the above configuration, the image data may be converted into a black and white image before the segmentation process is performed.

[0025] In the above-mentioned configuration, the image processing device may further include a calculation unit for performing a depth estimation process, and the new image data may include distance information obtained by performing depth estimation. Color information of the new image data has a shade based on the distance information.

[0026] The image processing system can clearly capture images in relatively dark places and provide the driver with an image highlighting the area that the driver should pay attention to. The image processing system can prevent accidents in dark places with little lighting, such as in the evening or at night, when the vehicle is traveling.

[0027] In addition, the software program that executes the inference program for performing neural network processing used for inference in segmentation processing can be written in various programming languages ​​such as Python, Go, Perl, Ruby, Prolog, Visual Basic, C, C++, Swift, Java (registered trademark), and .NET. In addition, the application may be created using a framework such as Chainer (available in Python), Caffe (available in Python and C++), and TensorFlow (available in C, C++, and Python). For example, the LSTM algorithm is programmed in Python and uses a CPU (Central Processor Unit) or a GPU (Graphics Processing Unit). In addition, a chip that integrates a CPU and a GPU into one is sometimes called an APU (Accelerated Processing Unit), and this APU chip can also be used. In addition, an IC (also called an inference chip) incorporating an AI (system) may be used. An IC incorporating an AI system may also be called a circuit (microprocessor) that performs neural network calculations.

[0028] In addition, the above-mentioned image processing system can be applied to a vehicle capable of semi-automated driving by combining a camera or radar that captures images around the vehicle with an ECU (Electronic Control Unit) that performs image processing, etc. A vehicle using an electric motor has multiple ECUs, and the ECU performs engine control, etc. The ECU includes a microcomputer. The ECU is connected to a CAN (Controller Area Network) provided in the electric vehicle. CAN is one of the serial communication standards used as an in-vehicle LAN. The ECU uses a CPU or a GPU. For example, the ECU performs inference via the CAN on image data captured using at least one of multiple cameras (such as a camera for a drive recorder or a rear camera) mounted on the electric vehicle, and creates a composite image that emphasizes people, which can be displayed on the display unit of a display device inside the vehicle. Note that the image displayed on the display unit of the display device shows the scenery outside the vehicle in black and white, and the area of ​​the person in the black and white image is a highly eye-catching color.

[0029] Furthermore, when segmentation is performed by software, the program constituting the software may be installed from a computer having a built-in hardware, or from a network or a recording medium. A program recorded on a recording medium such as a computer-readable CD-ROM (Compact Disk Read Only Memory) is installed, and the program for segmentation is executed. The processing performed by the program is not limited to processing in a sequence, and does not have to be chronological, and may be performed in parallel, for example. Effect of the Invention

[0030] Using a neural network, the system can provide the driver with an image display that highlights areas that require attention and displays other areas in black and white. This is mainly useful for providing an image display that assists the driver.

[0031] In addition, this has a particularly significant effect in environments with insufficient light, at dusk, at night, early in the morning, when passing through a long tunnel, etc., since it can provide the driver with a clear color display of the distant area in the direction the vehicle is traveling.

[0032] In addition, by having the learning model learn clothing patterns and colors that are difficult for the human eye to recognize, it is possible to accurately detect human figures that are difficult for the human eye to recognize. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a diagram showing an example of a flowchart illustrating one embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a flowchart illustrating one embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram showing an example of a flowchart illustrating one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a flowchart illustrating one embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing an example of a flowchart illustrating one embodiment of the present invention. [Figure 6] FIG. 6 is a block diagram illustrating an example of the configuration of the imaging unit. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of a pixel block 200 and a circuit 201. As shown in FIG. [Figure 8] FIG. 8 is a diagram illustrating an example of a pixel configuration. [Figure 9] 9A to 9C are diagrams showing filters. [Figure 10] Fig. 10A is a diagram showing a configuration example of a pixel, and Figs. 10B to 10D are diagrams showing configuration examples of a photoelectric conversion device. [Figure 11] 11A1 to 11A3 and 11B1 to 11B3 are perspective views of a package and a module that house an imaging device. [Figure 12]FIG. 12(A) is a diagram showing the exterior of a vehicle, and FIG. 12(B) is a schematic diagram showing the driver's field of vision looking forward from inside the vehicle. [Figure 13] FIG. 13 shows an example of an application product to which the imaging system according to an embodiment of the present invention can be applied. [Figure 14] FIG. 14 is an example of a program according to this embodiment. [Figure 15] FIG. 15 is an example of a program according to this embodiment. [Figure 16] FIG. 16 is an example of a program according to this embodiment. [Figure 17] FIG. 17 shows an example of a program according to this embodiment. [Figure 18] FIG. 18 is an example of a program according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be easily understood by those skilled in the art that the form and details of the present invention can be modified in various ways. Furthermore, the present invention is not to be interpreted as being limited to the description of the embodiments shown below.

[0035] (Embodiment 1) In this embodiment, an example of the flow of a driving support system is shown in FIG. 1. The system uses a neural network to extract people outside the vehicle who should be watched out for while driving from image data obtained by a solid-state image sensor mounted in the vehicle, colors the areas of the people, and provides the driver with an image in which the people are highlighted.

[0036] An imaging system having a hardware configuration including a solid-state imaging element is installed in a part of the vehicle (such as the bonnet, interior, or roof) capable of capturing images in the traveling direction of the vehicle (including distant areas), and is started up to begin continuous imaging.

[0037] In the on-board hardware configuration, one or more processors mainly control the operation at each step. When performing neural network processing, hardware that has a storage unit such as a sufficient memory for storing learning data and is capable of sufficient arithmetic processing is required. The storage unit refers to a small-sized mass storage device (e.g., SSD or hard disk) that can be mounted on the vehicle. The processing in the flowchart shown in Figure 1 can be realized by executing a program stored in the small-sized mass storage device.

[0038] First, preparation for acquiring data begins (S1).

[0039] Image data is acquired using a solid-state imaging element (S2). The image data may be full color or monochrome. Also, on the left side of FIG. 1, a display image example 29 of the captured image immediately after step S2 is shown. However, the display image example 29 is illustrated for ease of explanation and is not actually displayed. The display image example 29 shows an example in which a person 28 is captured.

[0040] Next, segmentation inference is performed (S21B).

[0041] Segmentation inference is also called segmentation. Segmentation refers to a process of identifying which object each pixel of an input image belongs to. It is also called semantic segmentation. Software that generates multiple image segments for use in image analysis is executed by neural network processing. Specifically, segmentation is performed based on the learned content using image processing and U-net and FCRN (Fully Convolutional Residual Networks), which are types of convolutional neural networks (CNNs). Segmentation labels are classified into vehicle, sky, plant, ground, human, building, and the like. In addition, in this embodiment, inference is performed inside the vehicle, so pre-learned features (segmentation weights) are used.

[0042] Example images are also shown that show the results of segmentation inference as images. Labeled areas are color coded. Example image 26 shows areas 27 that are labeled as human. Example image 26 is shown for illustrative purposes only and is not intended to be displayed.

[0043] Moreover, if no person is detected in the results obtained by segmentation inference, the subsequent processing is not performed, and the image as captured is displayed on the display device.

[0044] Next, if a person is detected in the results obtained by segmentation inference, the label (person) of area 27 in image example 26 is extracted (S22). For example, the pixel values ​​R and G of the position (matrix) corresponding to area 27 are set to 255 and 255, respectively. Note that the color when pixel values ​​R, G, and B are 255, 255, and 0, respectively, is yellow.

[0045] Next, composite image data is generated by adding the pixel values ​​of the area 27 to the pixel values ​​of the display image example 29. Coloring processing is performed by a program for coloring that includes this addition (S25).

[0046] An example of a flowchart for this program is shown in Figure 2.

[0047] The pixel values ​​of the pixel matrix data (pixel data) obtained by the image sensor are input one by one into the program and executed. First, if there is pixel data that has not been determined to be a "person" label, the pixel data (pixel values ​​(R,G,B) are obtained. Then, it is determined whether the sum of the pixel values ​​(R+G+B) is less than 100. If it is less than 100, 100 is added to the R of that pixel value and 100 is also added to G.

[0048] Yellow coloring can be achieved by adding 100 to the R and G pixel values. In the embodiment, if the pixel value is greater than 100, the pixel value is not changed. This is because if a pixel value is added to an object that is sufficiently bright and has good visibility, the unnaturalness of the contours of the image of that area becomes noticeable, making it difficult to grasp the features and reducing visibility. By setting conditions for the sum of pixel values ​​(R+G+B) as described above, it is possible to prevent a reduction in visibility.

[0049] This procedure is carried out for each pixel to determine the new pixel value. In this embodiment, an example is given in which 100 is added to R and G, but this is not particularly limited and may be any value between 50 and 255. In this embodiment, the values ​​are adjusted so that they do not exceed 255 even when added together. Also, since the color changes when blue is mixed in, B is adjusted to be 0. Also, in the above, the threshold value for the sum of pixel values ​​(R+G+B) was 100, but it may be changed to a predetermined value between 0 and 255. In this case, the sum of pixel values ​​to be added to R and G should be adjusted so that it does not exceed 255 even when added together.

[0050] Based on the composite image data thus obtained, an emphasis image is displayed on the display device of the vehicle. A display example 25 is actually displayed on the display device, and the area 24 of the person is colored yellow.

[0051] In this embodiment, yellow is used as an example of a highly conspicuous color, but there is no particular limitation, and it is preferable to use a color that is easy to distinguish when overlapping with roads or vegetation, such as pink or light blue.

[0052] Furthermore, although an example of extracting a human label is shown in this embodiment, this is not particularly limited, and both a vehicle label and a human label may be extracted and the extracted area may be colored.

[0053] The above-mentioned acquisition of the emphasized images is repeated.

[0054] If the image data is a black and white image, the area where the person is located will be colored yellow, resulting in a three-color image display. If the background is a black and white image with some parts colored yellow, the image will be easy for the driver to recognize, especially in dimly lit images such as at night, when some parts are displayed in yellow. If the image data is full color, the area where the person is located will be colored yellow, and the unnatural coloring can increase the driver's recognition.

[0055] Since the output result of segmentation is almost the same whether the image data is a black and white image or a full color image, the solid-state imaging device to be used is not particularly limited. The solid-state imaging device may be selected according to the size and resolution of the display device on which the image is to be displayed.

[0056] In this embodiment, the flow is shown in the order of steps S21B, S22, and S25, but depending on the configuration of the program, the parts enclosed by the dotted lines in Fig. 1 may be processed in parallel. Also, depending on the CPU or GPU that processes the program, it may be possible to execute the program in a short time, making it possible to highlight the program in real time.

[0057] Furthermore, the learning model for segmentation inference is not particularly limited, and a general-purpose segmentation inference model using a color image as a teacher data set can be used. In addition, in this embodiment, since a human outside the vehicle is detected in the vehicle, it is preferable to use images from a drive recorder or the like in which a human is captured as teacher data. Furthermore, by using image data of a person wearing clothing in black, gray, camouflage, or the like that is difficult for the human eye to recognize as teacher data, when it is dark outside the vehicle or when the vehicle speed is high, it is possible to identify a human more accurately than the human eye and highlight the human on the display device.

[0058] According to this embodiment, an image display that alerts the driver to a location where a person is outside the vehicle can be displayed on the display device.

[0059] (Embodiment 2) In this embodiment, an imaging element that does not have a color filter is used. In this embodiment, an example that is partially different from the first embodiment is shown in Figures 3 and 4, and the same reference numerals are used for the same parts as in Figure 1.

[0060] The flowchart shown in Figure 3 is an example in which black-and-white image data obtained by an imaging element that does not have a color filter is used to perform segmentation inference, followed by colorization, and the distance information from which depth inference has been performed is then subjected to coloring processing.

[0061] First, preparation for acquiring data begins (S1).

[0062] Black and white image data is acquired using a solid-state imaging device that does not have a color filter (S11).

[0063] When an image sensor without a color filter is used, image data with a wider dynamic range can be obtained compared to an image sensor with a color filter.

[0064] Next, segmentation inference is performed (S21B). The segmentation inference can be performed in the same manner as in embodiment 1. Even if the image data used is black and white image data, the same learning model can be used to obtain an output result.

[0065] Next, if a person is detected in the results obtained by the segmentation inference, a label (person) is extracted from the black-and-white image data (S22).

[0066] The black and white image data is then colorized. The resulting image often does not have natural colors, but instead is highlighted, making it easier for the driver to recognize.

[0067] In addition, a depth estimation process is performed on the monochrome image data (S21A). By using the output result of the depth estimation process, distance information of the label (person) can be obtained.

[0068] Next, composite image data is created. In this embodiment, based on the distance information of the label (person), the area of ​​the person in the colorized image is colored in multiple shades of yellow (light yellow, dark yellow) according to the distance. Partial coloring is performed by a program for gradual coloring (S25).

[0069] The above-mentioned acquisition of the emphasized images is repeated.

[0070] With an imaging element having a color filter, it is possible to obtain distinguishable colorized image data even when the amount of light is low and distinguishing images is impossible (for example, at night).

[0071] In the above configuration, the segmentation process uses a first neural network process, the depth estimation process uses a second neural network process, and the colorization process uses a third neural network process.

[0072] As the teacher dataset for learning segmentation processing, MSCOCO, cityscape, etc. can be used. As the teacher dataset for learning depth estimation processing, KITTI, etc. can be used. As the teacher dataset for learning colorization processing, there are no particular limitations as long as it is color, and Imagenet and color images taken with a dashcam can be processed and used.

[0073] In addition, while Fig. 3 shows an example in which colorization is performed, this is not particularly limited, and an example in which colorization is not performed may be used as shown in Fig. 4. Compared to Fig. 3, the number of neural network processes in Fig. 4 can be reduced.

[0074] This embodiment mode can be freely combined with the first embodiment mode.

[0075] (Embodiment 3) In this embodiment, an imaging element having a color filter is used. In this embodiment, an example that is partially different from the first embodiment is shown in Fig. 5, and the same reference numerals are used for the same parts as in Fig. 4.

[0076] The flowchart shown in FIG. 5 is an example in which image data obtained by an imaging element having a color filter is converted to black and white, segmentation inference is performed, and then the distance information obtained by depth inference is subjected to coloring processing.

[0077] First, preparation for acquiring data begins (S1).

[0078] The image data is converted into black and white image data using a solid-state image sensor having a color filter (S3). The means for converting the image data into black and white image data may be publicly known application software.

[0079] Next, segmentation inference is performed (S21B). The segmentation inference can be performed in the same manner as in embodiment 1. Even if the image data used is black and white image data, the same learning model can be used to obtain an output result.

[0080] Next, if a person is detected in the results obtained by the segmentation inference, a label (person) is extracted from the black-and-white image data (S22).

[0081] In addition, a depth estimation process is performed on the black and white image data. Using the output result of the depth estimation process, distance information of the label (person) can be obtained.

[0082] Next, composite image data is created. In this embodiment, based on the distance information of the label (person), the area of ​​the person in the colorized image is colored in multiple shades of yellow (light yellow, dark yellow) according to the distance. Partial coloring is performed by a program for gradual coloring (S25).

[0083] The above-mentioned acquisition of the emphasized images is repeated.

[0084] This embodiment uses a different type of imaging element from embodiment 2, but it can obtain an enhanced image by converting the image into black and white and coloring only the labeled parts of people in multiple shades of yellow (light yellow, dark yellow) that correspond to the distance.

[0085] When there is a sufficient amount of light outside the vehicle, such as on a clear day, it is an environment in which it is easy to detect people even if an image sensor (with a color filter) is used. When there is little light, such as at night, an image sensor without a color filter has a wider dynamic range and can be used as an imaging system that easily detects people.

[0086] This embodiment can be freely combined with other embodiments. For example, by combining this embodiment with embodiment 2 and mounting two types of imaging elements on a vehicle, it is possible to switch between them depending on the situation outside the vehicle. For example, on a sunny day, an imaging system using the imaging element (without a color filter) shown in FIG. 5 may be used, and at night, an imaging system using the imaging element (with a color filter) shown in FIG. 4 may be used. In this case, since there are many common parts, a highlighting system can also be created by switching only the imaging element.

[0087] (Embodiment 4) In the present embodiment, an image pickup device not having the color filter of the embodiment 2 will be described below.

[0088] The imaging system 41 includes a pixel array 300, a circuit 201, a circuit 302, a circuit 303, a circuit 304, a circuit 305, and a circuit 306. Note that each of the circuit 201 and the circuits 301 to 306 is not limited to a single circuit configuration, and may be configured by a combination of multiple circuits. Alternatively, any of multiple circuits described above may be integrated. Furthermore, a circuit other than the above may be connected.

[0089] The pixel array 300 has an imaging function and a calculation function. The circuits 201 and 302 have a calculation function. The circuit 302 has a calculation function or a data conversion function. The circuits 303, 304, and 306 have a selection function. The circuit 303 is electrically connected to the pixel block 200 through a wiring 424. The circuit 304 is electrically connected to the pixel block 200 through a wiring 423. The circuit 305 has a function of supplying a potential for a product-sum calculation to the pixel. A shift register, a decoder, or the like can be used as the circuit having the selection function. The circuit 306 is electrically connected to the pixel block 200 through a wiring 413. Note that the circuits 301 and 302 may be provided outside the pixel block 200.

[0090] 7, the pixel array 300 includes a plurality of pixel blocks 200. As shown in FIG. 7, the pixel block 200 includes a plurality of pixels 400 arranged in a matrix, and each of the pixels 400 is electrically connected to a circuit 201 via a wiring 412. Note that the circuit 201 can also be provided within the pixel block 200.

[0091] In addition, the pixel 400 is electrically connected to the adjacent pixel 400 through a transistor 450 (transistors 450a to 450f). The function of the transistor 450 will be described later.

[0092] In the pixel 400, image data can be acquired and data can be generated by adding the image data and a weighting factor. In FIG. 7, the number of pixels in the pixel block 200 is 3×3 as an example, but the number is not limited to this. For example, it can be 2×2, 4×4, etc. Alternatively, the number of pixels in the horizontal direction and the vertical direction may be different. Also, some pixels may be shared by adjacent pixel blocks. In FIG. 7, an example is shown in which nine transistors 450 (transistors 450a to 450f) are provided between the pixels 400, but the number of transistors 450 may be increased. Also, some transistors may be omitted from the transistors 450g to 450j to eliminate parallel paths. Wirings 413g to 413j are connected to the transistors 450g to 450j as gates, respectively.

[0093] The pixel block 200 and the circuit 201 can be operated as a multiply-accumulate circuit.

[0094] The pixel 400 may include a photoelectric conversion device 401, a transistor 402, a transistor 403, a transistor 404, a transistor 405, a transistor 406, and a capacitor 407, as shown in FIG.

[0095] One electrode of the photoelectric conversion device 401 is electrically connected to one of the source or drain of a transistor 402. The other of the source or drain of the transistor 402 is electrically connected to one of the source or drain of a transistor 403, the gate of a transistor 404, and one electrode of a capacitor 407. One of the source or drain of the transistor 404 is electrically connected to one of the source or drain of a transistor 405. The other electrode of the capacitor 407 is electrically connected to one of the source or drain of a transistor 406.

[0096] The other electrode of the photoelectric conversion device 401 is electrically connected to a wiring 414. The other of the source and drain of the transistor 403 is electrically connected to a wiring 415. The other of the source and drain of the transistor 405 is electrically connected to a wiring 412. The other of the source and drain of the transistor 404 is electrically connected to a GND wiring or the like. The other of the source and drain of the transistor 406 is electrically connected to a wiring 411. The other electrode of the capacitor 407 is electrically connected to a wiring 417.

[0097] A gate of the transistor 402 is electrically connected to a wiring 421. A gate of the transistor 403 is electrically connected to a wiring 422. A gate of the transistor 405 is electrically connected to a wiring 423. A gate of the transistor 406 is electrically connected to a wiring 424.

[0098] Here, an electrical connection point between the other of the source or drain of the transistor 402, one of the source or drain of the transistor 403, one electrode of the capacitor 407, and the gate of the transistor 404 is defined as a node FD. Also, an electrical connection point between the other electrode of the capacitor 407 and one of the source or drain of the transistor 406 is defined as a node FDW.

[0099] The wiring 414 and the wiring 415 can function as power supply lines. For example, the wiring 414 can function as a high-potential power supply line, and the wiring 415 can function as a low-potential power supply line. The wiring 421, the wiring 422, the wiring 423, and the wiring 424 can function as signal lines for controlling the conduction of each transistor. The wiring 411 can function as a wiring for supplying a potential corresponding to a weighting coefficient to the pixel 400. The wiring 412 can function as a wiring for electrically connecting the pixel 400 and the circuit 201. The wiring 417 can function as a wiring for electrically connecting the other electrode of the capacitor 407 of the pixel to the other electrode of the capacitor 407 of another pixel through a transistor 450 (see FIG. 7).

[0100] The wiring 412 may be electrically connected to an amplifier circuit or a gain adjustment circuit.

[0101] A photodiode can be used as the photoelectric conversion device 401. Any type of photodiode can be used, including a Si photodiode having silicon in the photoelectric conversion layer, an organic photodiode having an organic photoconductive film in the photoelectric conversion layer, etc. Note that, when it is desired to increase the light detection sensitivity at low illuminance, it is preferable to use an avalanche photodiode.

[0102] The transistor 402 can have a function of controlling the potential of the node FD. The transistor 403 can have a function of initializing the potential of the node FD. The transistor 404 can have a function of controlling a current flowing from the circuit 201 in accordance with the potential of the node FD. The transistor 405 can have a function of selecting a pixel. The transistor 406 can have a function of supplying a potential corresponding to a weighting coefficient to the node FDW.

[0103] When an avalanche photodiode is used for the photoelectric conversion device 401, a high voltage may be applied, and therefore a high-voltage transistor is preferably used for the transistor connected to the photoelectric conversion device 401. For example, a transistor using a metal oxide in a channel formation region (hereinafter, referred to as an OS transistor) can be used for the high-voltage transistor. Specifically, an OS transistor is preferably used for the transistor 402.

[0104] In addition, the off-state current of the OS transistor is extremely low. By using OS transistors for the transistors 402, 403, and 406, the period during which charges can be held at the nodes FD and FDW can be extremely long. Therefore, a global shutter system in which charges are accumulated simultaneously in all pixels can be applied without complicating the circuit configuration or operation method. In addition, while image data is held in the node FD, multiple calculations can be performed using the image data.

[0105] On the other hand, it may be desirable for the transistor 404 to have excellent amplification characteristics. In addition, it may be preferable to use a transistor with high mobility that can operate at high speed as the transistor 406. Therefore, a transistor using silicon for a channel formation region (hereinafter, referred to as a Si transistor) may be used as the transistor 404 and the transistor 406.

[0106] Note that the present invention is not limited to the above, and any combination of OS transistors and Si transistors may be used. All the transistors may be OS transistors. Alternatively, all the transistors may be Si transistors. Examples of Si transistors include transistors having amorphous silicon and transistors having crystalline silicon (microcrystalline silicon, low-temperature polysilicon, and single crystal silicon).

[0107] The potential of the node FD in the pixel 400 is determined by a potential obtained by adding a reset potential supplied from the wiring 415 and a potential (image data) generated by photoelectric conversion by the photoelectric conversion device 401. Alternatively, the potential is determined by further capacitive coupling with a potential corresponding to a weighting factor supplied from the wiring 411. Therefore, a current corresponding to data obtained by adding an arbitrary weighting factor to image data can flow through the transistor 405.

[0108] It should be noted that the above is only one example of the circuit configuration of the pixel 400, and the photoelectric conversion operation can be performed using other circuit configurations.

[0109] 7, the pixels 400 are electrically connected to each other through wirings 412. The circuit 201 can perform calculations using the sum of currents flowing through the transistors 404 of the pixels 400.

[0110] The circuit 201 includes a capacitor 202 , a transistor 203 , a transistor 204 , a transistor 205 , a transistor 206 , and a resistor 207 .

[0111] One electrode of the capacitor 202 is electrically connected to one of the source or drain of the transistor 203. One of the source or drain of the transistor 203 is electrically connected to the gate of the transistor 204. One of the source or drain of the transistor 204 is electrically connected to one of the source or drain of the transistor 205. One of the source or drain of the transistor 205 is electrically connected to one of the source or drain of the transistor 206. One electrode of the resistor 207 is electrically connected to the other electrode of the capacitor 202.

[0112] The other electrode of the capacitor 202 is electrically connected to a wiring 412. The other of the source and drain of the transistor 203 is electrically connected to a wiring 218. The other of the source and drain of the transistor 204 is electrically connected to a wiring 219. The other of the source and drain of the transistor 205 is electrically connected to a reference power supply line such as a GND wiring. The other of the source and drain of the transistor 206 is electrically connected to a wiring 212. The other electrode of the resistor 207 is electrically connected to a wiring 217.

[0113] The wiring 217, the wiring 218, and the wiring 219 can function as power supply lines. For example, the wiring 218 can function as a wiring that supplies a dedicated potential for reading. The wiring 217 and the wiring 219 can function as high-potential power supply lines. The wiring 213, the wiring 215, and the wiring 216 can function as signal lines that control the conduction of each transistor. The wiring 212 is an output line and can be electrically connected to, for example, the circuit 301 shown in FIG. 6.

[0114] The transistor 203 can have a function of resetting the potential of the wiring 211 to the potential of the wiring 218. The wiring 211 is a wiring connected to one electrode of the capacitor 202, one of the source and the drain of the transistor 203, and the gate of the transistor 204. The transistors 204 and 205 can function as source follower circuits. The transistor 206 can have a function of controlling reading. Note that the circuit 201 has a function as a correlated double sampling circuit (CDS circuit) and can be replaced with a circuit having another structure having the function.

[0115] In one embodiment of the present invention, offset components other than the product of image data (X) and weighting coefficient (W) are removed to extract the target WX. WX can be calculated using data with and without imaging for the same pixel, and data when weights are applied to each of the data.

[0116] The current (I p ) is the sum of kΣ(XV th ) 2 , the current (I p ) is kΣ(W+XV th ) 2 In addition, the current (I ref ) is the sum of kΣ(0-V th ) 2 , the current (I ref ) is the sum of kΣ(WV th ) 2 Here, k is a constant, V th is the threshold voltage of transistor 405.

[0117] First, calculate the difference (data A) between the captured data and the weighted data. kΣ((XV th ) 2 -(W+XV th ) 2 )=kΣ(-W 2 -2W·X+2W·Vth )

[0118] Next, the difference between the no-image data and the weighted data (data B) is calculated. kΣ((0-V th ) 2 -(WV th ) 2 )=kΣ(-W 2 +2W·V th )

[0119] Then, take the difference between data A and data B. kΣ(-W 2 -2W·X+2W·V th -(-W 2 +2W·V th ))=kΣ(-2W·X). In other words, it is possible to remove offset components other than the product of the image data (X) and the weighting coefficient (W).

[0120] The circuit 201 can read out data A and data B. Note that the difference between data A and data B can be calculated by the circuit 301, for example.

[0121] Here, the weights supplied to the entire pixel block 200 function as a filter. For example, a convolution filter of a convolutional neural network (CNN) can be used as the filter. Alternatively, an image processing filter such as an edge extraction filter can be used. Examples of edge extraction filters include the Laplacian filter shown in FIG. 9(A), the Prewitt filter shown in FIG. 9(B), and the Sobel filter shown in FIG. 9(C).

[0122] When the pixel block 200 has 3×3 pixels 400, the elements of the edge extraction filter can be assigned as weights to each pixel 400 and supplied. As described above, data A and data B can be calculated using data with and without imaging and data when weights are added to each of the data. Here, the data with and without imaging is data with no weight added, and can be rephrased as data with a weight of 0 added to all pixels 400.

[0123] 9(A) to 9(C) are filters in which the sum (ΣΔW / N, N is the number of elements) of the filter elements (weight: ΔW) is 0. Therefore, even if an operation of newly supplying ΔW=0 from another circuit is not performed, by performing an operation of acquiring ΣΔW / N, it is possible to acquire data in which ΔW=0 has been added to all pixels 400.

[0124] This operation corresponds to turning on the transistors 450 (transistors 450a to 450f) provided between the pixels 400 (see FIG. 7). By turning on the transistors 450, the nodes FDW of the pixels 400 are all short-circuited via the wiring 417. At this time, the charge accumulated in the nodes FDW of the pixels 400 is redistributed, and when the edge extraction filters illustrated in FIGS. 9(A) to 9(C) are used, the potential (ΔW) of the nodes FDW becomes 0 or approximately 0. Therefore, data with ΔW=0 added can be acquired.

[0125] When rewriting the weight (ΔW) by supplying charge from a circuit outside the pixel array 300, it takes time to complete the rewriting due to the capacitance of the long-distance wiring 411. On the other hand, the pixel block 200 is a very small area, and the wiring 417 is short and has a small capacity. Therefore, in the operation of redistributing the charge accumulated in the node FDW in the pixel block 200, the weight (ΔW) can be rewritten at high speed.

[0126] 7 illustrates a configuration in which the transistors 450a to 450f are electrically connected to different gate lines (wirings 413a to 413f), respectively. In this configuration, the conduction of the transistors 450a to 450f can be controlled independently, and an operation for obtaining ΣΔW / N can be selectively performed.

[0127] 9B or 9C is used, there is a pixel to which ΔW=0 is initially supplied. On the assumption that ΣΔW / N=0, the pixel to which ΔW=0 is supplied may be excluded from the pixels to be subjected to the summation. By excluding the pixel, it is not necessary to supply a potential for operating some of the transistors 450a to 450f, and therefore power consumption can be reduced.

[0128] The data resulting from the product-sum operation output from the circuit 201 is sequentially input to the circuit 301. The circuit 301 may have various calculation functions in addition to the above-mentioned function of calculating the difference between data A and data B. For example, the circuit 301 may have the same configuration as the circuit 201. Alternatively, the function of the circuit 301 may be replaced by software processing.

[0129] The circuit 301 may also include a circuit that performs an activation function calculation. For example, a comparator circuit can be used for the circuit. The comparator circuit compares input data with a set threshold value and outputs the result as binary data. That is, the pixel block 200 and the circuit 301 can act as a part of a neural network.

[0130] Data output from the circuit 301 is sequentially input to the circuit 302. The circuit 302 can have a configuration including, for example, a latch circuit and a shift register. With this configuration, parallel-serial conversion can be performed, and data input in parallel can be output to the wiring 311 as serial data.

[0131] [Pixel configuration example] 10A is a diagram showing an example of the configuration of a pixel 400. The pixel 400 can have a stacked structure of a layer 561 and a layer 563.

[0132] The layer 561 includes the photoelectric conversion device 401. The photoelectric conversion device 401 can include a layer 565a and a layer 565b as shown in FIG 10B. Note that a layer may be referred to as a region in some cases.

[0133] 10B is a pn junction photodiode, and for example, a p-type semiconductor can be used for the layer 565a and an n-type semiconductor can be used for the layer 565b. Alternatively, an n-type semiconductor can be used for the layer 565a and a p-type semiconductor can be used for the layer 565b.

[0134] The pn junction photodiode can be typically formed using single crystal silicon.

[0135] 10(C), the photoelectric conversion device 401 included in the layer 561 may be a stack of layers 566a, 566b, 566c, and 566d. The photoelectric conversion device 401 shown in FIG. 10(C) is an example of an avalanche photodiode, in which the layers 566a and 566d correspond to electrodes, and the layers 566b and 566c correspond to a photoelectric conversion unit.

[0136] Layer 566a is preferably a low resistance metal layer, etc. For example, aluminum, titanium, tungsten, tantalum, silver, or a laminate of these may be used.

[0137] The layer 566d is preferably a conductive layer having a high light-transmitting property to visible light. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, graphene, or the like can be used. Note that the layer 566d may be omitted.

[0138] The layers 566b and 566c of the photoelectric conversion unit may be configured as a pn junction photodiode with a photoelectric conversion layer made of, for example, a selenium-based material. It is preferable that the layer 566b is made of a selenium-based material that is a p-type semiconductor, and the layer 566c is made of gallium oxide or the like that is an n-type semiconductor.

[0139] Photoelectric conversion devices using selenium-based materials have the characteristic of high external quantum efficiency for visible light. In such photoelectric conversion devices, avalanche multiplication can be used to increase the amplification of electrons relative to the amount of incident light. In addition, selenium-based materials have a high optical absorption coefficient, which offers the advantage in terms of production, such as the ability to fabricate a thin photoelectric conversion layer. Thin films of selenium-based materials can be formed using vacuum deposition, sputtering, or the like.

[0140] As the selenium-based material, crystalline selenium such as single crystal selenium or polycrystalline selenium, amorphous selenium, a compound of copper, indium, and selenium (CIS), or a compound of copper, indium, gallium, and selenium (CIGS), or the like can be used.

[0141] The n-type semiconductor is preferably made of a material that has a wide band gap and is transparent to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or a mixture of these oxides can be used. These materials also function as a hole injection blocking layer, and can reduce dark current.

[0142] 10(D), the photoelectric conversion device 401 included in the layer 561 may be a laminate of layers 567a, 567b, 567c, 567d, and 567e. The photoelectric conversion device 401 shown in FIG. 10(D) is an example of an organic photoconductive film, in which the layer 567a is a lower electrode, the layer 567e is a light-transmitting upper electrode, and the layers 567b, 567c, and 567d correspond to photoelectric conversion units.

[0143] In the photoelectric conversion portion, one of the layer 567b and the layer 567d can be a hole-transporting layer, the other of the layer 567b and the layer 567d can be an electron-transporting layer, and the layer 567c can be a photoelectric conversion layer.

[0144] For the hole transport layer, for example, molybdenum oxide can be used. For the electron transport layer, for example, C 60 , C 70 or derivatives thereof can be used.

[0145] As the photoelectric conversion layer, a mixed layer (bulk heterojunction structure) of an n-type organic semiconductor and a p-type organic semiconductor can be used.

[0146] 10A includes, for example, a silicon substrate. Si transistors and the like are provided over the silicon substrate. The pixel 400 can be formed using the Si transistors. In addition, the circuit 201 and the circuits 301 to 306 shown in FIG. 6 can be formed.

[0147] (Embodiment 5) In this embodiment, a package that houses an imaging unit, that is, an image sensor chip, will be described below.

[0148] Fig. 11(A1) is a perspective view showing the appearance of the upper surface of a package containing an image sensor chip. The package includes a package substrate 410 for fixing an image sensor chip 452 (see Fig. 11(A3)), a cover glass 420, and an adhesive 430 for bonding the two together.

[0149] 11(A2) is an external perspective view of the bottom surface of the package. The bottom surface of the package has a BGA (Ball grid array) with solder balls as bumps 440. Note that the package is not limited to a BGA, and may have an LGA (Land grid array) or a PGA (Pin Grid Array), etc.

[0150] 11(A3) is a perspective view of the package with a portion of the cover glass 420 and the adhesive 430 omitted. An electrode pad 460 is formed on the package substrate 410, and the electrode pad 460 and the bump 440 are electrically connected via a through hole. The electrode pad 460 is electrically connected to the image sensor chip 452 by a wire 470.

[0151] 11(B1) is a perspective view of the appearance of the upper surface side of a camera module in which an image sensor chip is housed in a lens-integrated package. The camera module has a package substrate 431 for fixing an image sensor chip 451 (FIG. 11(B3)), a lens cover 432, a lens 435, and the like. In addition, an IC chip 490 (FIG. 11(B3) having functions such as a drive circuit and a signal conversion circuit of an imaging device is also provided between the package substrate 431 and the image sensor chip 451, and the camera module has a configuration as a SiP (System in Package).

[0152] 11B2 is a perspective view of the appearance of the bottom surface side of the camera module. The bottom surface and side surfaces of the package substrate 431 have a QFN (Quad flat no-lead package) configuration with mounting lands 441 provided thereon. Note that this configuration is one example, and a QFP (Quad flat package) or the above-mentioned BGA may also be provided.

[0153] 11(B3) is a perspective view of the module, with a portion of the lens cover 432 and the lens 435 omitted. The land 441 is electrically connected to the electrode pad 461, and the electrode pad 461 is electrically connected to the image sensor chip 451 or the IC chip 490 by a wire 471.

[0154] By housing the image sensor chip in a package of the above-mentioned type, mounting on a printed circuit board or the like becomes easy, and the image sensor chip can be incorporated into various semiconductor devices and electronic devices.

[0155] This embodiment mode can be appropriately combined with the descriptions of other embodiment modes.

[0156] (Embodiment 6) A driving assistance device suitable for a vehicle performing semi-automatic driving is provided by using the driving assistance system according to the above-described embodiment.

[0157] In Japan, the automation levels of driving systems for automobiles and other vehicles are defined as four levels, from level 1 to level 4. Level 1 refers to the automation of either acceleration, steering, or braking, and is called a safe driving support system. Level 2 automates multiple operations out of acceleration, steering, and braking at the same time, and is called a quasi-automated driving system (also called semi-automated driving). Level 3 is when acceleration, steering, and braking are all automated, with the driver only taking action in emergencies, and is also called a quasi-automated driving system (also called semi-automated driving). Level 4 is when acceleration, steering, and braking are all automated, and is called fully automated driving, with little to no driver involvement.

[0158] In this specification, a new configuration or a new driving assistance system is proposed, mainly assuming semi-autonomous driving at level 2 or level 3.

[0159] In order to display a warning to the driver of danger based on the situation obtained from various cameras and sensors, a display area commensurate with the number of cameras and sensors is required.

[0160] Fig. 12(A) shows an external view of a vehicle 120. Fig. 12(A) shows an example of the locations of a forward image sensor 114a and a left-side image sensor 114L. Fig. 12(B) is a schematic diagram showing the driver's field of vision from inside the vehicle. The upper part of the driver's field of vision is the windshield 110, and the lower part of the field of vision is where a display device 111 having a display screen is installed.

[0161] The upper part of the driver's field of vision is a windshield 110, which is sandwiched between pillars 112. In Fig. 12(A), an example is shown in which the forward image sensor 114a is installed in a position close to the driver's line of sight, but this is not particularly limited, and the forward image sensor 114a may be installed in the front grille or the front bumper. In addition, although a right-hand drive vehicle is shown as an example in this embodiment, this is not particularly limited, and if the vehicle is a left-hand drive vehicle, the forward image sensor 114a may be installed according to the driver's position.

[0162] It is preferable to use the image sensor chip shown in the fifth embodiment as at least one of these image sensors.

[0163] The driver accelerates, steers, and brakes while mainly looking at the display device 111, and checks the outside of the vehicle from the windshield as an auxiliary. The display device 111 may be any one of a liquid crystal display device, an EL (Electro Luminescence) display device, and a micro LED (Light Emitting Diode) display device. Here, an LED chip with a side dimension of more than 1 mm is called a macro LED, an LED chip with a side dimension of more than 100 μm and 1 mm or less is called a mini LED, and an LED chip with a side dimension of 100 μm or less is called a micro LED. It is particularly preferable to use a micro LED as an LED element applied to a pixel. By using a micro LED, a very high-definition display device can be realized. The higher the resolution of the display device 111, the more preferable it is. The pixel density of the display device 111 can be 100 ppi or more and 5000 ppi or less, preferably 200 ppi or more and 2000 ppi or less.

[0164] For example, the central portion 111a of the display screen of the display device displays an image acquired from an imaging device installed in front of the vehicle. Also, parts 111b and 111c of the display screen display meters such as speed, predicted driving distance, and abnormality warning display. Also, the lower left portion 111L of the display screen displays an image of the left side outside the vehicle, and the lower right portion 111R of the display screen displays an image of the right side outside the vehicle.

[0165] The left lower portion 111L and the right lower portion 111R of the display screen can be electronic side mirrors (also called door mirrors), eliminating the side mirror protrusions that protrude significantly outside the vehicle.

[0166] The display screen of the display device 111 may be configured to be touch-input operable to enlarge or reduce a part of the image, change the display position, or enlarge the area of ​​the display region.

[0167] The image on the display screen of the display device 111 is created by combining data from a plurality of image capture devices and sensors, and is therefore created using an image signal processing device such as a GPU.

[0168] By using the driving assistance system described in embodiment 1, it is possible to acquire black-and-white image data with a wide dynamic range, perform segmentation inference to extract human labels, and highlight the human labeled parts by coloring them yellow in a composite image, which can be enlarged and output to the display device 111.

[0169] The display device 111 can be applied to the driver's seat (also called the cockpit) of various types of vehicles, including large, medium, and small vehicles, and can also be applied to the driver's seat of vehicles such as airplanes and ships.

[0170] In addition, in this embodiment, an example is shown in which the forward image sensor 114a is installed below the windshield, but this is not particularly limited, and the imaging camera shown in FIG. 13 may be installed on the hood or around the rearview mirror inside the vehicle.

[0171] 13 can also be called a drive recorder, and includes a housing 961, a lens 962, a support portion 963, etc. By attaching double-sided tape or the like to the support portion 963, the imaging camera can be installed on the windshield, the hood, a rearview mirror support portion, etc.

[0172] An image sensor is disposed in the imaging camera of FIG. 13, and driving images can be recorded and stored inside the imaging camera or in a storage device mounted on the vehicle.

[0173] This embodiment mode can be freely combined with other embodiment modes. EXAMPLES

[0174] In this embodiment, an image pickup device capable of picking up an image of the outside of the vehicle in the traveling direction and a display device visible to the driver are installed in the vehicle.

[0175] A display device is used that forms a display surface by treating the display area obtained by the imaging element as one image data group of 162 pixel rows and 240 pixel columns.

[0176] The gradation of the display device ranges from 0 to 255. The display device is capable of full-color display, and the display color is determined by setting each pixel value (R, G, B) to a gradation of 0 to 255.

[0177] An example of a program corresponding to this embodiment is shown in Figures 14 and 15. A CPU or GPU capable of constructing a learning model uses data in memory to access a program (Python in this embodiment) stored in an SSD (or hard disk), reads the program, and the program stored in the SSD (or hard disk) is loaded into memory and deployed as a process on the memory. The flow diagram corresponds to Figures 1 and 2.

[0178] In the program shown in Figure 14, the image data is (row, column, color matrix data). Processing is performed on the data for each coordinate. Note that the data is referenced within the program, but since it is a huge amount of data, only the file name is shown here and the contents are omitted. If the sum of B, G, and R (colsum) is 100 or less, it is set to 0. Note that if the sum of B, G, and R is 100 or more, no processing is performed. Note that B is set to 0 to prevent blue from being mixed in.

[0179] In this embodiment, G is added by 100, and R is added by 100. In this way, the area of ​​people can be colored yellow. At the end of the program in Figure 15, the image is saved.

[0180] By executing the programs in Figures 14 and 15, the output results of segmentation processing of a black-and-white image or a color image can be used to create composite image data in which people in the image are colored yellow.

[0181] Furthermore, when performing the depth inference shown in Fig. 4, the depth inference shown in Fig. 4 can be performed by modifying parts of Fig. 14 and Fig. 15 and executing the program. Fig. 16, Fig. 17, and Fig. 18 show parts of the program to be modified. Depth data 3d is added as shown in Fig. 16, Fig. 17, and Fig. 18.

[0182] Using the programs of Figures 16, 17, and 18, people in image data can be colored yellow and the yellow can be shaded based on depth information, for example, closer areas can be colored darker yellow and farther areas can be colored lighter yellow. [Explanation of symbols]

[0183] 24 Area 25 Display example 26 Image example 27 areas 28 humans 29 Example of display image 41 Imaging System 110 Windshield 111 Display device 111a central part 111b part 111c part 111L lower left 111R Lower right 112 Pillar 114a Front image sensor 114L Left side image sensor 120 vehicles 162 pixel rows 200 pixel block 201 Circuit 202 Capacitor 203 Transistor 204 Transistor 205 Transistor 206 Transistor 207 Resistance 211 Wiring 212 Wiring 213 Wiring 215 Wiring 216 Wiring 217 Wiring 218 Wiring 219 Wiring 240 pixel rows 300 pixel array 301 Circuit 302 Circuit 303 Circuit 304 Circuit 305 Circuit 306 Circuit 311 Wiring 400 pixels 401 Photoelectric conversion devices 402 Transistor 403 Transistor 404 Transistor 405 Transistor 406 Transistor 407 Capacitor 410 Package Substrate 411 Wiring 412 Wiring 413 Wiring 413a Wiring 413f Wiring 413g Wiring 413j Wiring 414 Wiring 415 Wiring 417 Wiring 420 Coverslip 421 Wiring 422 Wiring 423 Wiring 424 Wiring 430 Adhesive 431 Package Substrate 432 Lens cover 435 Lens 440 Bump 441 rand 450 Transistors 450a transistor 450f transistor 450g transistor 450j transistor 451 Image Sensor Chip 452 Image sensor chip 460 Electrode Pads 461 Electrode Pads 470 Wire 471 Wire 490 IC chips 561 layers 563 layers 565a layer 565b layer 566a layer 566b layer 566c layer 566d layer 567a layer 567b layer 567c layer 567d layer 567e layer 961 Case 962 Lens 963 Support part

Claims

1. A means for acquiring first image data consisting of full color obtained by an imaging element; a computing means for converting the first image data into black and white image data, and then performing a segmentation process, and when a person is detected in the segmentation process, detecting an area of ​​the person as a first area and other areas as second areas; a synthesizing means for generating second image data by superimposing the black and white image data and color information that makes the first region a highly conspicuous color; and a display means for displaying the second image data when the person is detected, and displaying the first image data when the person is not detected.

2. In claim 1, An image processing system in which the color information of the second image data is only color information regarding the highly conspicuous color, white information, and black information.

3. In claim 1 or claim 2, The imaging device is mounted so as to capture an image in the traveling direction of the vehicle.

Images