Solid-state imaging device and electronic apparatus

Abstract

There is provided a solid-state imaging device including a pixel array unit including a plurality of pixels each configured to generate an electric charge by photoelectric conversion, in which the plurality of pixels include a plurality of event pixels and a plurality of gradation pixels, each of the event pixels generates an event signal based on a brightness change of incident light, and each of the gradation pixels generates a brightness signal based on a light amount of the incident light. Each of the event pixels is associated with a white or cyan color filter, and each of the grayscale pixels is associated with a red, green, or blue color filter.

Description

Cross - reference to related applications

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2022 - 199558, filed on December 14, 2022, the entire content of which is incorporated herein by reference. Technical field

[0002] The present invention relates to a solid - state imaging device and an electronic device. Background art

[0003] Solid - state imaging devices proposed in the past include event pixels and grayscale pixels. The event pixels detect the occurrence of an event based on the amount of change in charge generated from incident light entering a photodiode, while each grayscale pixel outputs a pixel signal corresponding to the amount of charge generated from incident light entering the photodiode. Citation list Patent literature

[0004] Patent Document 1: PCT Patent Publication No. WO2021 / 117350 Summary of the invention Technical problem

[0005] According to the above - mentioned technology, for example, color mixing may occur from a predetermined grayscale pixel to an adjacent grayscale pixel. In this case, the image quality of the grayscale pixel may deteriorate.

[0006] Therefore, the present disclosure proposed in view of the above problems provides a solid - state imaging device capable of reducing the deterioration of the image quality of grayscale pixels. Problem - solving means

[0007] A solid-state imaging device according to a first aspect of the present invention includes a pixel array unit including a plurality of pixels each configured to generate charges through photoelectric conversion. The plurality of pixels include a plurality of event pixels and a plurality of gradation pixels. The event pixels are each configured to generate an event signal based on a change in luminance of incident light, and the gradation pixels are each configured to generate a luminance signal based on the amount of incident light. The color arrangements of the event pixels and the gradation pixels do not have 180-degree rotational symmetry. In this configuration, by providing a white color filter having a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the gradation pixels from the event pixels as compared with a state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce a decrease in sensitivity for acquiring luminance information. Further, by providing a cyan color filter on each event pixel, color mixing caused by incident light invading the gradation pixels from the event pixels can be reduced as compared with a state without a color filter. In addition, the event pixels each including a cyan color filter can more effectively reduce color mixing of long-wavelength light as compared with a configuration including a white color filter.

[0008] Further, in this first aspect, the event pixels and the gradation pixels each have a color filter for transmitting light having a predetermined wavelength band. In this configuration, by providing a white color filter having a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the gradation pixels from the event pixels as compared with a state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce a decrease in sensitivity for acquiring luminance information. Further, by arranging a cyan color filter on each event pixel, color mixing caused by incident light invading the gradation pixels from the event pixels can be reduced as compared with a state without a color filter. In addition, the event pixels each including a cyan color filter can better reduce color mixing of long-wavelength light as compared with a configuration including a white color filter.

[0009] In addition, in this first aspect, the color filter of the event pixel includes a white color filter or a cyan color filter, and the color filter of the grayscale pixel includes a red color filter, a green color filter, and a blue color filter. In this configuration, by providing a white color filter with a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the grayscale pixel from the event pixel as compared to the state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce the decrease in sensitivity for obtaining luminance information. In addition, by providing a cyan color filter on each event pixel, color mixing caused by incident light invading the grayscale pixel from the event pixel can be reduced as compared to the state without a color filter. In addition, as compared to the configuration including a white color filter, the event pixel including a cyan color filter can better reduce color mixing of long-wavelength light. In addition, by providing a lens element on the event pixel adjacent to the grayscale pixel other than the blue grayscale pixel, these event pixels can ensure a larger amount of incident light and reduce the decrease in sensitivity.

[0010] In addition, in this first aspect, the color filter of the event pixel includes a white color filter and a cyan color filter, and the color filter of the grayscale pixel includes a red color filter, a green color filter, and a blue color filter. In this configuration, the pixel array unit can reduce color mixing of long-wavelength light from the event pixel having a cyan color filter to the adjacent blue grayscale pixel. In addition, by arranging a white color filter on each event pixel adjacent to the grayscale pixel other than the blue grayscale pixel, the pixel array unit can reduce color mixing from these event pixels to the adjacent grayscale pixel and simultaneously reduce the decrease in sensitivity.

[0011] In addition, in this first aspect, the event pixel adjacent to the grayscale pixel having a blue color filter has a cyan color filter. In this configuration, the pixel array unit can reduce color mixing of long-wavelength light from the event pixel having a cyan color filter to the adjacent blue grayscale pixel.

[0012] In addition, in this first aspect, the event pixel adjacent to the grayscale pixel having a color filter of a color other than blue has a white color filter. In this configuration, the event pixel adjacent to the grayscale pixel other than the blue grayscale pixel has a white color filter. Therefore, the pixel array unit can reduce color mixing from this event pixel to the adjacent grayscale pixel and simultaneously reduce the decrease in sensitivity.

[0013] In addition, in this first aspect, each event pixel located in the central portion of the pixel array unit has a white color filter, and each event pixel located in the peripheral portion of the pixel array unit has a cyan color filter. With this configuration, compared to the state without a color filter, color mixing caused by incident light invading the gray-scale pixels from the event pixels can be reduced. In particular, color mixing of long-wavelength light with the blue gray-scale pixels in the peripheral portion of the pixel array unit can be reduced. In addition, the event pixels located in the central portion of the pixel array unit can ensure a larger amount of incident light, and thereby reduce the decrease in sensitivity.

[0014] In addition, in this first aspect, the ratio of the number of event pixels included in the pixel array unit to the total number of event pixels and gray-scale pixels is 25% or less. In this configuration, by providing a white color filter having a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the gray-scale pixels from the event pixels compared to the state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce the decrease in sensitivity for acquiring luminance information. In addition, by providing a cyan color filter on each event pixel, color mixing caused by incident light invading the gray-scale pixels from the event pixels can be reduced compared to the state without a color filter. In addition, compared to the configuration including a white color filter, the event pixels each including a cyan color filter can better reduce color mixing of long-wavelength light.

[0015] In addition, in this first aspect, each event pixel includes an EVS pixel. In this configuration, by providing a white color filter having a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the gray-scale pixels from the event pixels compared to the state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce the decrease in sensitivity for acquiring luminance information. In addition, by having a cyan color filter on each event pixel, color mixing caused by incident light invading the gray-scale pixels from the event pixels can be reduced compared to the state without a color filter. In addition, compared to the configuration including a white color filter, the event pixels each including a cyan color filter can better reduce color mixing of long-wavelength light.

[0016] In addition, in this first aspect, a first light-shielding wall is arranged between the color filters of the plurality of pixels. With this configuration, the pixel array unit having the first light-shielding wall can reduce color mixing between the color filters. In addition, the pixel array unit can improve the quantum efficiency.

[0017] In addition, in this first aspect, each first light-shielding wall includes a low-refractive-index material structure or an air structure. In this configuration, the pixel array unit having the first light-shielding wall can reduce color mixing between color filters. In addition, the refractive index of each first light-shielding wall having an air structure can be lower than that of the low-refractive-index material structure. In addition, the pixel array unit can improve the quantum efficiency.

[0018] In addition, in this first aspect, the thickness of the first light-shielding wall disposed between the grayscale pixel and the adjacent event pixel is different from the thickness of the first light-shielding wall disposed between the grayscale pixel and the adjacent grayscale pixel. In this configuration, color mixing caused by light from the event pixel entering the grayscale pixel having a blue color filter can be further reduced. In addition, the sensitivity of the grayscale pixel and the event pixel can also be improved.

[0019] In addition, in this first aspect, the thickness of the first light-shielding wall disposed between the grayscale pixel having a blue color filter and the adjacent event pixel is greater than the thickness of the first light-shielding wall disposed between the grayscale pixel and the adjacent grayscale pixel. In this configuration, color mixing caused by light from the event pixel entering the grayscale pixel having a blue color filter can be further reduced.

[0020] In addition, in this first aspect, the thickness of the first light-shielding wall disposed between the grayscale pixel having a color filter of any color other than blue and the adjacent event pixel is less than the thickness of the first light-shielding wall disposed between the grayscale pixel and the adjacent grayscale pixel. With this configuration, the sensitivity of the grayscale pixel and the event pixel can be improved.

[0021] In addition, in this first aspect, the event pixel and the grayscale pixel each further have an on-chip lens disposed on the color filter and used for collecting incident light, and a second light-shielding wall is disposed between the on-chip lenses of multiple pixels. In this configuration, the pixel array unit can reduce color mixing caused by light from the on-chip lens entering adjacent pixels.

[0022] In addition, in this first aspect, waveguides that respectively form the optical path of incident light are disposed on the color filter. In this configuration, the pixel array unit can reduce color mixing in adjacent pixels by the function of the waveguides as channels for incident light.

[0023] In addition, in this first aspect, waveguides are disposed on the white color filter or the cyan color filter. In this configuration, the pixel array unit can reduce color mixing in adjacent pixels by the function of the waveguides as channels for incident light.

[0024] In addition, in this first aspect, the event pixels and the grayscale pixels each further include a photodiode, which is located in a semiconductor substrate below the color filter and is used to perform photoelectric conversion; and a third light-shielding wall penetrating the interior of the semiconductor substrate is provided between the photodiodes of the plurality of pixels. In this configuration, the semiconductor substrate can reduce color mixing caused by light entering the photodiode.

[0025] In addition, in this first aspect, in the insulating film below the semiconductor substrate, a fourth light-shielding wall is provided between the plurality of pixels. In this configuration, the insulating film can reduce color mixing caused by light entering the insulating film.

[0026] The electronic device according to the second aspect of the present invention is an electronic device including an imaging device. The imaging device includes a pixel array unit, and the pixel array unit includes a plurality of pixels, which are each configured to generate charges through photoelectric conversion. The plurality of pixels include a plurality of event pixels and a plurality of grayscale pixels. The event pixels are each configured to generate an event signal based on the brightness change of incident light, and the grayscale pixels are each configured to generate a brightness signal based on the amount of incident light. The color arrangements of the event pixels and the grayscale pixels do not have 180-degree rotational symmetry. In this configuration, by providing a white color filter with a relatively high refractive index on each event pixel, the pixel array unit can reduce color mixing caused by incident light invading the grayscale pixels from the event pixels compared to the state without a color filter. In addition, the pixel array unit having a white color filter on each event pixel can reduce the decrease in the sensitivity of obtaining brightness information. In addition, by providing a cyan color filter on each event pixel, color mixing caused by incident light invading the grayscale pixels from the event pixels can be reduced compared to the state without a color filter. In addition, compared with the configuration including a white color filter, the event pixels each including a cyan color filter can more effectively reduce the color mixing of long-wavelength light. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram showing a configuration example of an imaging device according to the first embodiment. Figure 2 is a block diagram showing a configuration example of a solid-state imaging device according to the first embodiment. Figure 3 is a plan view showing the layout and the like of each pixel in a pixel array unit according to the first embodiment. Figure 4 is a cross-sectional view along line a-a' showing a pixel array unit including grayscale pixels and event pixels according to the first embodiment. Figure 5 is a plan view showing the layout and the like of each pixel in a pixel array unit according to the second embodiment. Figure 6It is a diagram showing examples of the transmittance of a white color filter and a cyan color filter respectively. Figure 7 It is a plan view showing the layout of each pixel in the pixel array unit according to the third embodiment. Figure 8 It is a plan view showing the layout of each pixel in the pixel array unit according to the fourth embodiment. Figure 9 It is a plan view showing the layout of each pixel in the pixel array unit according to the fifth embodiment. Figure 10 It is a plan view showing the layout of each pixel in the pixel array unit according to the sixth to eighth embodiments. Figure 11 It is a plan view showing the layout of each pixel in the pixel array unit according to the ninth to tenth embodiments. Figure 12 It is a plan view showing the layout etc. of each pixel in the pixel array unit according to the eleventh embodiment. Figure 13 It shows an example of a cross-sectional view of a pixel array unit etc. according to the twelfth embodiment. Figure 14 It is an example of a cross-sectional view of a pixel array unit according to the thirteenth embodiment. Figure 15 It is an example of a cross-sectional view of a pixel array unit according to the fourteenth embodiment. Figure 16 It is an example of a cross-sectional view of a pixel array unit according to the fifteenth embodiment. Figure 17 It shows an example of a cross-sectional view of a pixel array unit etc. according to the sixteenth embodiment. Figure 18 It is a block diagram showing a configuration example of a vehicle control system. Figure 19 It is a diagram showing an example of a sensing area. Detailed Description of the Invention

[0028] Embodiments according to the present disclosure will be described below with reference to the accompanying drawings.

[0029] (First Embodiment) Figure 1 It is a block diagram showing a configuration example of an imaging device according to the first embodiment.

[0030] Figure 1 The imaging device 100 in it includes an imaging lens 110, a solid-state imaging device 200, a control unit 130, and a data processing unit 120. Examples of the imaging device 100 include a camera mounted on an industrial robot and an in-vehicle camera.

[0031] The imaging lens 110 collects incident light and guides the collected light into the solid-state imaging device 200. The solid-state imaging device 200 generates a luminance signal at a gray level corresponding to the amount of incident light that has entered through the imaging lens 110, and outputs the generated luminance signal to the data processing unit 120. In addition, the solid-state imaging device 200 detects the fact that the luminance has changed by more than a predetermined threshold due to the entry of incident light as an event, generates an event signal, and outputs the generated event signal to the data processing unit 120. More specifically, the solid-state imaging device 200 detects as an event whether the change in the photocurrent corresponding to the luminance of the incident light exceeds a predetermined threshold.

[0032] The control unit 130 performs overall control of the imaging device 100. For example, the control unit 130 causes the solid-state imaging device 200 to capture image data.

[0033] The data processing unit 120 includes a data generation unit 150 and a recording unit 140. The data processing unit 120 performs data processing based on the signals received from the solid-state imaging device 200. Details of the data generation unit 150 and the recording unit 140 will be described below.

[0034] The data generation unit 150 performs predetermined data processing on the luminance signal received from the solid-state imaging device 200. In addition, the data generation unit 150 performs predetermined data processing and the like using the event signal received from the solid-state imaging device 200. The data generation unit 150 outputs the processed data as a data processing result to an external device (not shown in the figure). Alternatively, the data generation unit 150 may output the luminance signal and the event signal provided from the solid-state imaging device 200 to the external device as they are.

[0035] The recording unit 140 records the data received from the solid-state imaging device 200.

[0036] Figure 2 is a block diagram showing a schematic configuration example of a solid-state imaging device according to the first embodiment.

[0037] The solid-state imaging device 200 includes a pixel array unit 10, a driving unit 2, an arbiter (arbitration unit) 3, an event signal processing unit 4, and a luminance signal processing unit 5.

[0038] The pixel array unit 10 has a plurality of pixels 9, which are arranged in a grid pattern and generate charges through photoelectric conversion respectively. In addition, the pixels 9 include event pixels and grayscale pixels. The event pixels respectively detect the brightness change of incident light as events, while the grayscale pixels respectively generate brightness signals for indicating the grayscale level corresponding to the amount of incident light. These event pixels and grayscale pixels are arranged in various types of layouts. For example, the event pixels are respectively event-based Vision (EVS) pixels. In addition, various layout types will be described separately below.

[0039] The driving unit 2 controls and drives each grayscale pixel included in the pixel array unit 10.

[0040] The arbiter 3 arbitrates requests sent from the event pixels within the pixel array unit 10 and returns a reply indicating whether to allow or not allow the output of event signals to each event pixel that sends a request. Each event pixel that receives an allow reply from the arbiter 3 is allowed to output an event signal to the event signal processing unit 4. The event signals are transmitted to the event signal processing unit 4 for each row. In addition, the event signals of the event pixels that are included in multiple event pixels in the same row and do not cause events will be discarded by the event signal processing unit 4. The arbiter 3 provides a reset signal for resetting event detection to each event pixel.

[0041] The event signal processing unit 4 performs necessary processing on the event signals received from the respective event pixels of the pixel array unit 10 and transmits the processed event signals to the data processing unit 120.

[0042] The brightness signal processing unit 5 performs necessary processing on the brightness signals received from the respective grayscale pixels of the pixel array unit 10 and transmits the processed brightness signals to the data processing unit 120.

[0043] Figure 3 is a plan view showing the layout etc. of each pixel in the pixel array unit according to the first embodiment.

[0044] Figure 3 A of shows the layout of the pixel array unit 10 including the grayscale pixel 9a and the event pixel 9b according to the present embodiment, while Figure 3 B of shows the layout of the pixel array unit 10 including the grayscale pixel 9a and the event pixel 9b in another mode of the present embodiment.

[0045] Figure 3The X-axis, Y-axis, and Z-axis perpendicular to each other are shown. The X-direction and Y-direction respectively correspond to the lateral direction (horizontal direction), while the Z-direction corresponds to the longitudinal direction (vertical direction). In addition, the +Z direction corresponds to the upward direction, while the -Z direction corresponds to the downward direction. Note that the Z-direction may or may not be exactly the same as the direction of gravity.

[0046] According to Figure 3 A of Figure 3 and B of Figure 3 A of Figure 3 and B of

[0047] According to Figure 3 A of Figure 3 and B of

[0048] In the pixel array unit 10, color filters for transmitting light within a predetermined wavelength range are arranged on respective pixels 9 of the corresponding colors in order to achieve the color arrangements of red R, blue B, and green G. Hereinafter, the gray-scale pixels 9a each having a red R color filter, the gray-scale pixels 9a each having a green G color filter, and the gray-scale pixels 9a each having a blue B color filter are also respectively referred to as red R gray-scale pixels 9a, green G gray-scale pixels 9a, and blue B gray-scale pixels 9a.

[0049] In addition, in Figure 3 A of Figure 3 and B of Figure 3 A of Figure 3The event pixels 9b in A each have a white arrangement. Figure 3 The event pixels 9b in B each have a white color filter instead of a lens element and have a white arrangement different from the arrangement of the lens elements. Specifically, both the lens element and the white color filter can transmit light of various colors included in visible light. However, the composition and characteristics of the light transmitted through the white color filter are different from those of the light transmitted through the lens element. To distinguish the color arrangements of the lens element and the white color filter, the white arrangement of the white color filter is hereinafter represented as white W, while the white arrangement of the lens element is represented as white W'.

[0050] According to the present embodiment, each of the grayscale pixels 9a and event pixels 9b included in the pixel array unit 10 has a color filter and has a predetermined color arrangement. In addition, after the 4×4 pixels 9 in the pixel array unit 10 are rotated 180 degrees in the XY plane around the rotation axis located at the center of the 4×4 pixels 9, any part of their color arrangement does not match the corresponding part of the color arrangement in the pixel array unit 10 before rotation. According to this example, when the 4×4 pixels 9 are rotated 180 degrees in the XY plane around the rotation axis located at the center of the 4×4 pixels 9, the positions of the red R grayscale pixels 9a and blue B grayscale pixels 9a will be opposite to their initial positions before rotation.

[0051] The inconsistency between any part of the color arrangement of the color arrangement unit of the pixel 9 after being rotated 180 degrees in the XY plane around the rotation axis located at the center of the color arrangement unit of the pixel 9 and the corresponding color arrangement in the pixel array unit 10 before rotation is referred to as "not having 180-degree rotational symmetry". In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0052] Figure 4 is a cross-sectional view taken along the a-a' line showing a pixel array unit including grayscale pixels 9a and event pixels 9b according to the first embodiment.

[0053] Figure 4 A of Figure 3 is a cross-sectional view taken along the a-a' line in A of Figure 4 and B of Figure 3 is a cross-sectional view taken along the a-a' line in B of

[0054] In addition, according to Figure 4 A of Figure 4 and the example shown in B of Figure 4 A of Figure 4Other components are not shown in the example shown in B.

[0055] The on-chip lenses 90 collect incident light respectively. The inter-pixel light shielding films 91 each mainly prevent color mixing caused by light entering a predetermined pixel 9 at an oblique angle and invading adjacent different pixels 9. In addition, the photodiodes 92 are each used to realize photoelectric conversion of incident light.

[0056] According to Snell's law, light bends from a medium with a low refractive index to a medium with a high refractive index. Assume that the refractive indices of air, lens element, red R filter, green G filter, blue B filter, and white W filter are n1, n2, and n3, respectively. w1 、n R 、n G 、n B and n w2 , then the respective refractive indices generally have the following relationship in Expression (1). n1 <n w1 <n R 、n G 、n B <n w2 ...(1)

[0057] Figure 4 A shows the lens element (refractive index n) as light enters the event pixel 9b. w1 ) and intrudes into the green G color filter (refractive index n G ) and blue B filter (refractive index n B ) is an example of color mixing caused by the green G filter. G and the refractive index n of the blue B filter B are higher than the refractive index n of the lens element L . Therefore, the incident light entering the lens element is refracted according to this relationship and invades the green G and blue B filters. Therefore, color mixing from the event pixel 9b to the grayscale pixel 9a occurs. In particular, light with a long wavelength such as red is mixed with the blue B filter and may cause image quality deterioration.

[0058] Figure 4 B shows an example of color mixing reduction according to this embodiment. The refractive index n of the white W filter is w2 Higher than the refractive index n of the green G filter G and the refractive index n of the blue B filter B In this relationship, incident light does not invade the green G and blue B filters respectively adjacent to the white W filter. Therefore, color mixing can be reduced.

[0059] According to this embodiment, a white W color filter with a relatively high refractive index is provided on each of the event pixels 9b. Therefore, compared with the state without a color filter, the pixel array unit 10 can reduce color mixing caused by incident light invading the gray-scale pixels 9a from the event pixels 9b. In addition, the pixel array unit 10 can also prevent image quality degradation by reducing uneven color mixing caused at the gray-scale pixels 9a (e.g., green G and blue B gray-scale pixels 9a) adjacent to the event pixels 9b.

[0060] In addition, according to this embodiment, the pixel array unit 10 having a white W color filter on each of the event pixels 9b can reduce the decrease in sensitivity for obtaining luminance information.

[0061] (Second Embodiment) Figure 5 FIG. is a plan view showing the layout and the like of each pixel 9 in the pixel array unit according to the second embodiment.

[0062] Different from the first embodiment, according to this embodiment, the event pixels 9b each have a cyan C color filter instead of a white W color filter. Different from the white W color filter, the cyan C color filter hardly transmits light with a long wavelength (about 600 nm or more). Hereinafter, the event pixels 9b having a cyan C color filter will also be referred to as cyan C event pixels 9b.

[0063] Figure 6 FIG. is a diagram showing examples of the transmittance of the white color filter and the cyan color filter, respectively.

[0064] Figure 6 A of FIG. shows an example of the transmittance of the white W color filter at each wavelength, while Figure 6 B of FIG. shows an example of the transmittance of the cyan C color filter at each wavelength.

[0065] As Figure 6 shown in A of FIG., the white W color filter can transmit light with a wavelength of about 600 nm or more. On the other hand, according to Figure 6 the characteristics of the cyan C color filter shown in B of FIG., the cyan C color filter hardly transmits light with a wavelength of about 600 nm or more. Therefore, by arranging the cyan C color filters on the event pixels 9b respectively, color mixing caused by long-wavelength light invading adjacent gray-scale pixels 9a can be reduced.

[0066] According to this embodiment, since the cyan C color filters are arranged on the respective event pixels 9b, the pixel array unit 10 can reduce color mixing caused by incident light invading the gray-scale pixels 9a from the event pixels 9b compared with the state without a color filter.

[0067] In addition, according to this embodiment in which a cyan C color filter is disposed on the event pixel 9b respectively, the pixel array unit 10 can reduce the color mixing of long-wavelength light more than the configuration including a white W color filter. In particular, the pixel array unit 10 can prevent image quality degradation by reducing the color mixing caused at the blue B grayscale pixel 9a adjacent to the event pixel 9b.

[0068] (Third Embodiment) Figure 7 is a plan view showing the layout of each pixel in the pixel array unit according to the third embodiment.

[0069] The arrangement of each event pixel 9b and each grayscale pixel 9a in the pixel array unit 10 of this embodiment is similar to the arrangement in the first embodiment. In addition, different from the first embodiment, the event pixels 9b adjacent to the blue B grayscale pixel 9a in the pixel array unit 10 of this embodiment respectively have cyan color filters. In addition, the event pixels 9b adjacent to pixels other than the blue B grayscale pixel 9a respectively have lens elements.

[0070] Similar to the above embodiments, the color arrangement of the grayscale pixel 9a and the event pixel 9b included in the pixel array unit 10 according to this embodiment does not have 180-degree rotational symmetry. In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0071] According to this embodiment, the color mixing between the cyan C event pixel 9b and the adjacent blue B grayscale pixel 9a can be reduced. In addition, the event pixels 9b adjacent to grayscale pixels 9a other than the blue B grayscale pixel 9a are respectively provided with lens elements. Therefore, the event pixels 9b that can ensure a larger incident light amount can effectively reduce the decrease in sensitivity.

[0072] (Fourth Embodiment) Figure 8 is a plan view showing the layout of each pixel in the pixel array unit according to the fourth embodiment.

[0073] The arrangement of each event pixel 9b and each grayscale pixel 9a in the pixel array unit 10 of this embodiment is similar to the arrangement in the first embodiment. In addition, different from the first embodiment, in the pixel array unit 10 of this embodiment, the event pixels 9b adjacent to the blue B grayscale pixel 9a respectively have cyan color filters. In addition, the event pixels 9b adjacent to pixels other than the blue B grayscale pixel 9a respectively have white W color filters.

[0074] Similar to the above embodiments, the color arrangement of the grayscale pixels 9a and event pixels 9b included in the pixel array unit 10 according to this embodiment does not have 180-degree rotational symmetry. In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0075] According to this embodiment, the pixel array unit 10 can reduce color mixing of long-wavelength light from the cyan C event pixels 9b to adjacent blue B grayscale pixels 9a. In addition, the event pixels 9b adjacent to grayscale pixels 9a other than the blue B grayscale pixels 9a each have a white W color filter. Therefore, the pixel array unit 10 can reduce color mixing from the event pixels 9b with white W color filters to adjacent grayscale pixels 9a and simultaneously reduce a decrease in sensitivity.

[0076] (Fifth Embodiment) Figure 9 is a plan view showing the layout of each pixel in the pixel array unit according to the fifth embodiment.

[0077] The arrangement of each event pixel 9b and each grayscale pixel 9a in the pixel array unit 10 in this embodiment is similar to the arrangement in the first embodiment.

[0078] Meanwhile, in the pixel array unit 10 according to this embodiment, the color arrangement in the central portion 6 (the central portion within the viewing angle) of the pixel array unit 10 is different from the color arrangement in the peripheral portion 7 (the outer portion within the viewing angle) of the pixel array unit 10. The event pixels 9b in the central portion 6 each have a white W color filter, while the event pixels 9b in the peripheral portion 7 each have a cyan C color filter.

[0079] According to this embodiment, the color arrangement of the grayscale pixels 9a and event pixels 9b respectively included in the central portion 6 and the peripheral portion 7 of the pixel array unit 10 does not have 180-degree rotational symmetry. In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0080] The ranges of the central portion 6 and the peripheral portion 7 are sufficient to determine the relative positional relationship. Specifically, the following settings are sufficient: the event pixels 9b located in the central portion of the viewing angle each have a white W color filter, while the event pixels 9b located in the outer portion of the viewing angle in the area surrounding the central portion each have a cyan C color filter.

[0081] Generally, in the peripheral region 7 where the incident light has a relatively large incident angle, color mixing is more likely to occur than in the central region 6. According to this embodiment, compared with the state where no color filter is used, color mixing caused by the incident light invading the gray-scale pixel 9a from the event pixel 9b can be reduced. In particular, color mixing between high-wavelength light and the blue B gray-scale pixel 9a in the peripheral region 7 can be reduced.

[0082] In addition, a white W color filter is disposed on the event pixel 9b in the central portion 6, respectively. Therefore, a larger amount of incident light can be ensured, and thus a reduction in sensitivity can be reduced.

[0083] (Sixth to Eighth Embodiments) Figure 10 is a plan view showing the layout of each pixel in the pixel array unit according to the sixth to eighth embodiments.

[0084] Figure 10 A of is the layout of each pixel 9 in the pixel array unit 10 according to the sixth embodiment, Figure 10 B of is the layout of each pixel 9 in the pixel array unit 10 according to the seventh embodiment, and Figure 10 C of is the layout of each pixel 9 in the pixel array unit 10 according to the eighth embodiment.

[0085] According to Figure 10 A to Figure 10 C of, the pixel array unit 10 has a color arrangement unit composed of four gray-scale pixels 9a or event pixels 9b arranged in the row direction (X direction) and four gray-scale pixels 9a or event pixels 9b arranged in the column direction (Y direction). In addition, this unit is arranged in a cycle in the X direction and the Y direction. Figure 10 A to Figure 10 The portions respectively surrounded by the dashed lines in C of represent one color arrangement unit. The color arrangement will be described below based on this unit.

[0086] In Figure 10 the example of A (seventh embodiment) of, in the first row and the second row of the pixel array unit 10, red R gray-scale pixels 9a, red R gray-scale pixels 9a, green G gray-scale pixels 9a, and green G gray-scale pixels 9a are arranged in sequence from the left. In addition, in the third row and the fourth row, green G gray-scale pixels 9a, green G gray-scale pixels 9a, event pixels 9b, and blue B gray-scale pixels 9a are arranged in sequence from the left.

[0087] In Figure 10In the example of B (the eighth embodiment), in the first row and the second row of the pixel array unit 10, event pixels 9b, red R grayscale pixels 9a, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged in sequence from the left. In addition, in the third row and the fourth row, green G grayscale pixels 9a, green G grayscale pixels 9a, event pixels 9b, and blue B grayscale pixels 9a are arranged in sequence from the left.

[0088] In Figure 10 the example of C (the ninth embodiment), in the first row of the pixel array unit 10, red R grayscale pixels 9a, red R grayscale pixels 9a, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged in sequence from the left. In addition, in the second row of the pixel array unit 10, red R grayscale pixels 9a, event pixels 9b, event pixels 9b, and green G grayscale pixels 9a are arranged. In addition, in the third row of the pixel array unit 10, green G grayscale pixels 9a, event pixels 9b, event pixels 9b, and blue B grayscale pixels 9a are arranged. In addition, in the fourth row, green G grayscale pixels 9a, green G grayscale pixels 9a, blue B grayscale pixels 9a, and blue B grayscale pixels 9a are arranged.

[0089] In Figure 10 from A to Figure 10 C, at least a part of the event pixels 9b in the pixel array unit 10 may respectively have a white W color filter or a cyan C color filter. In addition, at least a part of the event pixels 9b may respectively have a lens element. For example, the event pixels 9b may respectively have a lens element, a white W color filter, or a cyan C color filter. In addition, the event pixels 9b adjacent to the blue B grayscale pixels 9a may respectively have a cyan C color filter, and the event pixels 9b adjacent to pixels other than the blue B grayscale pixels 9a may respectively have a lens element or a white W color filter.

[0090] Similar to the above embodiments, the color arrangement of 4×4 pixels in the pixel array unit 10 according to the sixth to eighth embodiments does not have 180-degree rotational symmetry. In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0091] According to the present embodiment, the pixel array unit 10 is allowed to have various arrangements of event pixels 9b and grayscale pixels 9a in a form suitable for pixel rearrangement.

[0092] (The ninth to tenth embodiments) Figure 11 is a plan view showing the layout of each pixel in the pixel array unit according to the ninth to tenth embodiments.

[0093] Figure 11 A is the layout of each pixel 9 in the pixel array unit 10 according to the ninth embodiment, and Figure 11 B is the layout of each pixel 9 in the pixel array unit 10 according to the tenth embodiment.

[0094] According to Figure 11 A of Figure 11 and B of Figure 11 A of Figure 11 and B of

[0095] In Figure 10 the example of A of

[0096] (ninth embodiment), in the first row of the pixel array unit 10, red R grayscale pixels 9a, red R grayscale pixels 9a, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged in sequence from the left. In addition, in the second row of the pixel array unit 10, red R grayscale pixels 9a, event pixels 9b, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged. In addition, in the third row of the pixel array unit 10, green G grayscale pixels 9a, green G grayscale pixels 9a, event pixels 9b, and blue B grayscale pixels 9a are arranged. In addition, in the fourth row of the pixel array unit 10, green G grayscale pixels 9a, green G grayscale pixels 9a, blue B grayscale pixels 9a, and blue B grayscale pixels 9a are arranged.

[0096] In Figure 10 the example of B of

[0097] In Figure 11 A of Figure 11In B of, at least some of the event pixels 9b in the pixel array unit 10 may respectively have a white W color filter or a cyan C color filter. In addition, at least some of the event pixels 9b may respectively have a lens element. For example, the event pixels 9b may respectively have a lens element, a white W color filter, or a cyan C color filter. In addition, the event pixels 9b adjacent to the blue B grayscale pixels 9a may have a cyan C color filter, and the event pixels 9b adjacent to pixels other than the blue B grayscale pixels 9a may have a lens element or a white W color filter.

[0098] Similar to the above-described embodiments, the color arrangement of the grayscale pixels 9a and the event pixels 9b included in the pixel array unit 10 according to the ninth to tenth embodiments does not have 180-degree rotational symmetry. In addition, the proportion of the number of event pixels 9b included in the pixel array unit 10 to the total number of event pixels 9b and grayscale pixels 9a is 25% or less.

[0099] According to the present embodiment, the pixel array unit 10 is allowed to have various arrangements of event pixels 9b and grayscale pixels 9a in a form suitable for pixel rearrangement.

[0100] (Eleventh Embodiment) Figure 12 is a plan view showing the layout and the like of each pixel in the pixel array unit according to the eleventh embodiment.

[0101] According to Figure 12 In the example of, one color arrangement unit is composed of four grayscale pixels 9a or event pixels 9b arranged in the row direction (X direction) and eight grayscale pixels 9a or event pixels 9b arranged in the column direction (Y direction). This unit will also be simply referred to as 4×8 pixels 9. In addition, this unit is arranged in a cycle in the X direction and the Y direction. Figure 12 The portion surrounded by the dashed line in is a color arrangement unit. The color arrangement will be described below based on this unit.

[0102] In Figure 12 In the example of, in the first row and the second row of the pixel array unit 10, red R grayscale pixels 9a, red R grayscale pixels 9a, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged in order from the left. In addition, in the third row and the fourth row of the pixel array unit 10, green G grayscale pixels 9a, green G grayscale pixels 9a, event pixels 9b, and blue B grayscale pixels 9a are arranged. In addition, in the fifth row and the sixth row of the pixel array unit 10, red R grayscale pixels 9a, event pixels 9b, green G grayscale pixels 9a, and green G grayscale pixels 9a are arranged. In addition, in the seventh row and the eighth row of the pixel array unit 10, green G grayscale pixels 9a, green G grayscale pixels 9a, blue B grayscale pixels 9a, and blue B grayscale pixels 9a are arranged.

[0103] In Figure 12 it, at least a part of the event pixels 9b in the pixel array unit 10 may respectively have a white W color filter or a cyan C color filter. In addition, at least a part of the event pixels 9b may respectively have a lens element. For example, the event pixels 9b may respectively have a lens element, a white W color filter, or a cyan C color filter. In addition, the event pixels 9b adjacent to the blue B grayscale pixels 9a may have a cyan C color filter, and the event pixels 9b adjacent to pixels other than the blue B grayscale pixels 9a may have a lens element or a white W color filter.

[0104] According to the eleventh embodiment, the color arrangement unit of the grayscale pixels 9a and the event pixels 9b included in the 4×8 pixels 9 in the pixel array unit 10 does not have rotational symmetry compared with the color arrangement before rotation when rotated 180 degrees around the rotation axis located at the center of the 4×8 pixels 9 in the XY plane. In addition, the proportion of the number of the event pixels 9b included in the pixel array unit 10 to the total number of the event pixels 9b and the grayscale pixels 9a is 25% or less.

[0105] According to the present embodiment, the pixel array unit 10 is allowed to have various arrangements of the event pixels 9b and the grayscale pixels 9a in a form suitable for pixel rearrangement.

[0106] (Twelfth Embodiment) Figure 13 An example of a cross-sectional view of a pixel array unit and the like according to the twelfth embodiment is shown.

[0107] Figure 13 A in Figure 13 is an example of a cross-sectional view of the grayscale pixels 9a and the event pixels 9b included in the pixel array unit 10 in the comparative example, while Figure 13 B in

[0108] In Figure 13 is an example of a cross-sectional view of the grayscale pixels 9a and the event pixels 9b included in the pixel array unit 10 according to the present embodiment. The pixel arrangement and the color arrangement in the present embodiment are not limited to the arrangement shown in

[0109] Meanwhile, in Figure 13In the cross-sectional view shown in FIG. B, first light-shielding walls 93 are provided between the respective color filters in the pixel array unit 10. These first light-shielding walls 93 are walls that include a low-refractive-index material and are used to separate the respective color filters. For example, the first light-shielding walls 93 are each constituted by a silicon oxide film. The material of the first light-shielding walls 93 is not limited to this example and can be any material as long as the following relationship is satisfied: the refractive indices of the low-refractive-index materials of the first light-shielding walls 93 are all lower than the refractive index of the color filter. In this text, the structure of the first light-shielding walls 93 is referred to as a low-refractive-index material structure or an NKB (low-N wall) structure.

[0110] According to this embodiment, the pixel array unit 10 having the first light-shielding walls 93 can reduce color mixing between the color filters and can improve the quantum efficiency (Qe).

[0111] (Thirteenth Embodiment) Figure 14 FIG. is an example of a cross-sectional view of a pixel array unit according to the thirteenth embodiment.

[0112] Figure 14 The cross-sectional view shown is an example of the grayscale pixels 9a and event pixels 9b included in the pixel array unit 10 according to this embodiment. The pixel arrangement and color arrangement in this embodiment are not limited to Figure 14 the arrangement shown. For example, the pixel arrangement and color arrangement in any one of the first to eleventh embodiments can be adopted.

[0113] Similar to Figure 13 the example in FIG. B, Figure 14 the pixel array unit 10 in the cross-sectional view shown has first light-shielding walls 93' between the respective grayscale pixels 9a and the respective event pixels 9b. Different from Figure 13 the example in FIG., the first light-shielding walls 93' according to this embodiment each have a structure including air in the wall. The refractive index of air is approximately 1. Therefore, a low-refractive-index structure is generated due to the presence of air. In this text, the structure of the first light-shielding walls 93' is referred to as an air structure or an AKB (air wall) structure.

[0114] According to this embodiment, the first light-shielding walls 93' each have an air structure. Therefore, the refractive index can become lower than the refractive index of the low-refractive-index material structure. In addition, the first light-shielding walls 93' can each reduce color mixing between the color filters. In addition, the pixel array unit 10 can improve the quantum efficiency.

[0115] (Fourteenth Embodiment) Figure 15 FIG. is an example of a cross-sectional view of a pixel array unit according to the fourteenth embodiment.

[0116] Figure 15The cross-sectional view shown is an example of the grayscale pixels 9a and event pixels 9b included in the pixel array unit 10 according to the present embodiment. The pixel arrangement and color arrangement in the present embodiment are not limited to Figure 15 the arrangement shown. For example, the pixel arrangement and color arrangement in any one of the first to eleventh embodiments may be adopted.

[0117] Similar to Figure 13 the example in B of Figure 15 In the pixel array unit 10 in the cross-sectional view shown, there are first light-shielding walls 93 and first light-shielding walls 93'' and 93''' between the respective grayscale pixels 9a and event pixels 9b. Similar to Figure 13 B of Figure 14 and the example in

[0118] According to the present embodiment, the first light-shielding walls 93 to 93''' are formed such that the thickness of each of the first light-shielding walls 93'' and 93''' provided between the grayscale pixel 9a and the adjacent event pixel 9b is different from the thickness of the first light-shielding wall 93 provided between the grayscale pixel 9a and the grayscale pixel 9a. Figure 15 According to the structure in

[0119] In addition, according to the present embodiment, the thickness of each of the first light-shielding walls 93'' provided between the grayscale pixels 9a having color filters other than the blue B color filter and the adjacent event pixels 9b is smaller than the thickness of the first light-shielding wall 93 provided between the grayscale pixel 9a and the adjacent grayscale pixel 9a. According to Figure 15 the structure in

[0120] In addition, although Figure 15 an example in which the first light-shielding walls 93 to 93''' each have a low refractive index material structure is shown, an air structure may also be adopted.

[0121] According to the present embodiment, the pixel array unit 10 can further reduce color mixing caused by light intrusion from the event pixel 9b into the blue B grayscale pixel 9a by increasing the thickness of the first light-shielding wall 93''' provided between the blue B grayscale pixel 9a and the adjacent event pixel 9b having a cyan C color filter.

[0122] In addition, according to the present embodiment, the pixel array unit 10 can improve the sensitivity of the grayscale pixel 9a and the event pixel 9b by reducing the thickness of the first light-shielding wall 93 provided between the grayscale pixel 9a having a color filter (e.g., red R and green G) less affected by color mixing and the adjacent event pixel 9b.

[0123] (The fifteenth embodiment) Figure 16 FIG. 15 is an example of a cross-sectional view of a pixel array unit according to the fifteenth embodiment.

[0124] Figure 16 The cross-sectional view shown is an example of a cross-sectional view of the grayscale pixel 9a and the event pixel 9b included in the pixel array unit 10. The pixel arrangement and color arrangement in this embodiment are not limited to Figure 16 the arrangement shown. For example, the pixel arrangement and color arrangement in any one of the first to eleventh embodiments can be adopted.

[0125] Similar to Figure 13 the example in FIG. 15B, Figure 16 the pixel array unit 10 in the cross-sectional view shown has a first light-shielding wall 133 provided between each grayscale pixel 9a and each event pixel 9b.

[0126] According to the present embodiment, second light-shielding walls 94 as low-refractive-index material walls are provided between the color filters of the event pixel 9b and the grayscale pixel 9a and between the corresponding color filters and the on-chip lens 90, respectively. For example, the second light-shielding walls 94 are each formed of a silicon oxide film.

[0127] In addition, according to this example, waveguides 95 serving as optical paths of incident light are respectively arranged on each white W color filter included in the event pixel 9b. For example, the waveguides 95 each include a high-refractive-index material such as silicon nitride. In this case, the waveguides 95 each serve as a channel for light entering the upper side of the event pixel 9b. Therefore, color mixing in adjacent grayscale pixels 9a can be reduced. Although an example in which the waveguides 95 are provided on the white W color filters is described herein, the waveguides 95 can also be provided on the cyan C color filters.

[0128] Alternatively, the waveguides 95 can also have a column shape. In this case, the waveguides 95 can each collect a larger amount of incident light.

[0129] According to the present embodiment, the second light-shielding walls 94 are provided on the color filters. Therefore, the pixel array unit 10 can reduce color mixing caused by incident light entering the adjacent pixels 9 from the on-chip lens 90.

[0130] Alternatively, the waveguides 95 are provided on the white W color filters or the cyan C color filters. Therefore, the pixel array unit 10 can reduce color mixing in adjacent pixels by utilizing the function of the waveguides 95 as optical channels for incident light.

[0131] (Sixteenth Embodiment) Figure 17 FIG. shows an example of a cross-sectional view of a pixel array unit and the like according to the sixteenth embodiment.

[0132] Figure 17 A in FIG. is an example of a cross-sectional view of a grayscale pixel 9a and an event pixel 9b included in the pixel array unit 10 in the comparative example, while Figure 17 B in FIG. is an example of a cross-sectional view of a grayscale pixel 9a and an event pixel 9b included in the pixel array unit 10 according to the present embodiment. The pixel arrangement and color arrangement in the present embodiment are not limited to Figure 17 the arrangement shown in B of FIG. For example, the pixel arrangement and color arrangement in any one of the first to eleventh embodiments can be adopted.

[0133] Reference will be made to Figure 17 A in FIG. to illustrate an example of color mixing caused in the semiconductor substrate 88 and the insulating film 89.

[0134] The light of the lens material that enters the event pixel 9b through the on-chip lens 90 is photoelectrically converted by the photodiode 92 in the semiconductor substrate 88. A part of the light that enters the photodiode 92 undergoes total internal reflection on the element isolation insulating film 96 and enters the region of the adjacent photodiode 92. In addition, a part of the light of the lens material that enters the event pixel 9b through the on-chip lens 90 undergoes total internal reflection in the insulating film 89 including the transfer transistor 99 and the like and enters the region of the adjacent photodiode 92. In this way, color mixing is caused due to different incident heights of the light entering the adjacent photodiodes 92.

[0135] In Figure 17 B of FIG., the photodiodes 92 are provided under each color filter in the semiconductor substrate 88. In addition, a third light-shielding wall 97 is provided between the respective photodiodes 92. The third light-shielding walls 97 are low-refractive-index material walls that penetrate the inside of the semiconductor substrate 88. For example, the third light-shielding walls 97 respectively include materials with a refractive index lower than that of the photodiode 92.

[0136] In addition, as Figure 17 shown in B of FIG., in the insulating film 89 below the semiconductor substrate 88, a fourth light-shielding wall 98 as a low-refractive-index material wall is provided between the respective pixels 9. For example, the fourth light-shielding walls 98 respectively include materials with a refractive index lower than that of the insulating film 89.

[0137] Although in Figure 17In an example of B, the third light shielding wall 97 and the fourth light shielding wall 98 are integrally formed with each other, but a structure including only the third light shielding wall 97 or the fourth light shielding wall 98 may also be adopted. In addition, the third light shielding wall 97 and the fourth light shielding wall 98 may be combined with the first light shielding walls 93 to 93''', the second light shielding wall 94, and the waveguide 95 described above and mounted on the pixel array unit 10.

[0138] According to the present embodiment, the semiconductor substrate 88 including the third light shielding wall 97 reduces color mixing caused by light entering the photodiode 92.

[0139] In addition, according to the present embodiment, the insulating film 89 including the fourth light shielding wall 98 can reduce color mixing caused by light entering the insulating film 89.

[0140] "Example of the Structure of a Vehicle Control System" Figure 18 FIG. is a block diagram showing an example of the structure of a vehicle control system 11, which is shown as an example of a mobile device control system to which the present technology is applied.

[0141] The vehicle control system 11 is provided in the vehicle 1 and performs processes related to driving assistance and autonomous driving of the vehicle 1.

[0142] The vehicle control system 11 includes a vehicle control electronic control unit (ECU) 21, a communication unit 22, a map information accumulation unit 23, a position information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving assistance and autonomous driving control unit 29, a driver monitoring system (DMS) 30, a human machine interface (HMI) 31, and a vehicle control unit 32.

[0143] The vehicle control ECU 21, communication unit 22, map information accumulation unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, storage unit 28, driving assistance and autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are communicatively connected to each other via a communication network 41. For example, the communication network 41 includes in-vehicle communication networks that conform to digital two-way communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark), and also includes buses and the like. Different types of communication networks 41 can be selected according to the type of data to be transmitted. For example, CAN can be applied to data related to vehicle control, while Ethernet can be applied to mass data. Note that each unit of the vehicle control system 11 is not connected via the communication network 41, but in some cases is directly connected via wireless communication when assuming that the communication to be established is relatively short-distance communication such as near-field communication (NFC) and Bluetooth (registered trademark).

[0144] Note that even if each unit of the vehicle control system 11 communicates with each other via the communication network 41, the description of the communication network 41 will be omitted hereinafter. For example, in the case where the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, this case will be simply described as "the vehicle control ECU 21 and the communication unit 22 communicate with each other".

[0145] For example, the vehicle control ECU 21 includes a processor selected from various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls all or part of the functions of the vehicle control system 11.

[0146] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc. to send and receive various data. In this case, the communication unit 22 can use a variety of communication systems to achieve communication.

[0147] The communication with the outside of the vehicle that can be achieved by the communication unit 22 will be outlined. For example, the communication unit 22 uses wireless communication systems such as 5G (fifth-generation mobile communication system), LTE (Long-Term Evolution), and DSRC (Dedicated Short Range Communication) to communicate with a server (hereinafter referred to as an external server) existing on an external network through a base station or an access point. For example, the communication unit 22 communicates with an external network such as the Internet, a cloud network, or a vendor-specific network. The communication system adopted by the communication unit 22 when communicating with the external network is not particularly limited as long as it is a wireless communication system capable of achieving digital two-way communication at a predetermined or higher communication speed and over a predetermined or longer distance.

[0148] In addition, for example, the communication unit 22 can use peer-to-peer (P2P) technology to communicate with a terminal located near the vehicle. For example, a terminal located near the vehicle can be a terminal attached to a moving body moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed at a fixed position such as a store, or a machine type communication (MTC) terminal. In addition, the communication unit 22 can implement V2X communication. V2X communication refers to the communication between the vehicle and other vehicles, such as vehicle-to-vehicle communication, vehicle-infrastructure communication for communicating with roadside equipment, etc., vehicle-home communication, and vehicle-pedestrian communication for communicating with a terminal carried by a pedestrian.

[0149] For example, the communication unit 22 can receive a program (wireless method) for updating the software that controls the operation of the vehicle control system 11 from the outside. The communication unit 22 can also receive map information, traffic information, information related to the surrounding environment of the vehicle 1, etc. from the outside. In addition, for example, the communication unit 22 can transmit information related to the vehicle 1, information related to the surrounding environment of the vehicle 1, etc. to the outside. Examples of the information related to the vehicle 1 transmitted from the communication unit 22 to the outside include data indicating the state of the vehicle 1 and the recognition result obtained by the recognition unit 73. In addition, for example, the communication unit 22 implements communication in accordance with a vehicle emergency reporting system such as e-call.

[0150] For example, the communication unit 22 receives electromagnetic waves transmitted from a radio wave beacon, an optical beacon, and a vehicle information and communication system (VICS) (registered trademark) provided by FM multiplex broadcasting, etc.

[0151] The communication with the interior of the vehicle that can be achieved by the communication unit 22 will be outlined. For example, the communication unit 22 can communicate with each in-vehicle device via wireless communication. The communication unit 22 can wirelessly communicate with in-vehicle devices via a communication system such as wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB), which allows digital two-way communication at a predetermined or higher communication speed via wireless communication. In addition, the communication unit 22 can communicate with each in-vehicle device via wired communication. For example, the communication unit 22 can communicate with each in-vehicle device via wired communication using a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each in-vehicle device via a communication system such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-Definition Link), which allows digital two-way communication at a predetermined or higher communication speed via wired communication.

[0152] For example, the in-vehicle devices in this document refer to devices installed inside the vehicle and not connected to the communication network 41. Examples of in-vehicle devices are assumed to include mobile devices or wearable devices carried by occupants such as the driver, as well as information devices brought into the vehicle and temporarily installed.

[0153] The map information accumulation unit 23 accumulates maps acquired from the outside and / or maps created by the vehicle 1. For example, the map information accumulation unit 23 accumulates a three-dimensional high-precision map and a global map with a lower precision than the high-precision map and covering a wide area.

[0154] For example, the high-precision map can be a dynamic map, a point cloud map, or a vector map. For example, a dynamic map is a map having four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map composed of point clouds (point cloud data). For example, a vector map is a map that associates traffic information such as the positions of lanes and traffic lights with the point cloud map for applying traffic information and the like to an Advanced Driver Assistance System (ADAS) and Autonomous Driving (AD).

[0155] For example, both the point cloud map and the vector map can be provided from an external server or the like, or can be created by the vehicle 1 based on the sensing results obtained from the camera 51, the radar 52, the lidar 53, etc. to match the local map described below, and accumulated in the map information accumulation unit 23. Further, in the case of providing a high-precision map from an external server or the like, for example, map data for indicating a map with a side length of several hundred meters and associated with the planned route on which the vehicle 1 plans to travel is acquired from an external server or the like to reduce the communication volume.

[0156] The position information acquisition unit 24 receives GNSS signals from Global Navigation Satellite System (GNSS) satellites to acquire the position information related to the vehicle 1. The acquired position information is provided to the driving assistance and autonomous driving control unit 29. Note that the position information acquisition unit 24 is not limited to a system based on GNSS signals, and for example, beacons can also be used to acquire position information.

[0157] The external recognition sensor 25 includes various sensors for recognizing the situation outside the vehicle 1, and provides the sensor data received from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 can be of any type and number.

[0158] For example, the external recognition sensor 25 includes the camera 51, the radar 52, the Light Detection and Ranging (LiDAR) 53, and the ultrasonic sensor 54. Alternatively, the external recognition sensor 25 can include at least one sensor selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 is not particularly limited as long as the number is the number that can be actually installed on the vehicle 1. Further, the types of sensors included in the external recognition sensor 25 are not limited to these examples. The external recognition sensor 25 can have other types of sensors. Examples of the sensing areas of the respective sensors included in the external recognition sensor 25 will be described below.

[0159] Note that the imaging method adopted by the camera 51 is not limited to a specific method. For example, various imaging methods capable of performing distance measurement can be applied to the camera 51 as needed, such as a ToF (Time of Flight) camera, a stereo camera, a monocular camera, and an infrared camera. Alternatively, the camera 51 can be only a camera for acquiring captured images, rather than a camera with a distance measurement function.

[0160] In addition, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment of the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, meteorology, and brightness, and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and a brightness sensor.

[0161] In addition, for example, the external recognition sensor 25 includes a microphone for detecting the sound, sound source position around the vehicle 1, and for other purposes.

[0162] The in-vehicle sensors 26 include various sensors for detecting in-vehicle information, and provide the sensor data received from each sensor to each unit of the vehicle control system 11. The type and number of each of the various sensors included in the in-vehicle sensors 26 are not particularly limited as long as the type and number are the actual types and numbers that can be installed on the vehicle 1.

[0163] For example, the in-vehicle sensors 26 may include at least one sensor selected from a camera, a radar, a seat sensor, a steering wheel sensor, a microphone, and a biological sensor. For example, cameras using various imaging methods capable of distance measurement such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera are applicable to the cameras included in the in-vehicle sensors 26. Alternatively, the cameras included in the in-vehicle sensors 26 may be cameras only for acquiring images, rather than cameras having a distance measurement function. The biological sensors included in the in-vehicle sensors 26 are arranged, for example, on the seat or the steering wheel, and detect various biological information related to an occupant such as a driver.

[0164] The vehicle sensors 27 include various sensors for detecting the state of the vehicle 1, and provide the sensor data received from each sensor to each unit of the vehicle control system 11. The type and number of the various sensors included in the vehicle sensors 27 are not particularly limited as long as the type and number are the actual types and numbers that can be installed on the vehicle 1.

[0165] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, a throttle sensor for detecting the operation amount of the accelerator pedal, and a brake sensor for detecting the operation amount of the brake pedal. For example, the vehicle sensor 27 includes a rotational speed sensor for detecting the rotational speed of the engine or motor, a tire pressure sensor for detecting the tire pressure, a slip ratio sensor for detecting the tire slip ratio, and a wheel speed sensor for detecting the wheel rotational speed. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining battery charge and temperature, and a shock sensor for detecting external shocks.

[0166] The storage unit 28 includes at least a non-volatile storage medium or a volatile storage medium, and stores data and programs. For example, the storage unit 28 functions as an EEPROM (electrically erasable programmable read-only memory) and a RAM (random access memory). As the storage medium, a magnetic storage device (e.g., HDD (hard disk drive)), a semiconductor storage device, an optical storage device, and a magneto-optical storage device can be used. The storage unit 28 stores various programs and data used by each unit of the vehicle control system 11. For example, the storage unit 28 includes an Event Data Recorder (EDR) and a Data Storage System for Automated Driving (DSSAD), and stores information related to the vehicle 1 before and after an event such as an accident and information acquired by the in-vehicle sensors 26.

[0167] The driving assistance and automated driving control unit 29 performs driving assistance and automated driving control of the vehicle 1. For example, the driving assistance and automated driving control unit 29 includes an analysis unit 61, a behavior planning unit 62, and an action control unit 63.

[0168] The analysis unit 61 performs an analysis process for analyzing the internal and surrounding situations of the vehicle 1. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and an identification unit 73.

[0169] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data received from the external recognition sensor 25 and the high-precision map accumulated in the map information accumulation unit 23. For example, the self-position estimation unit 71 creates a local map based on the sensor data received from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. For example, the position of the vehicle 1 is defined based on a reference point located at the axis center of a pair of rear wheels.

[0170] For example, the local map is a three-dimensional high-precision map created by technologies such as Simultaneous Localization and Mapping (SLAM), or an occupancy grid map. For example, the three-dimensional high-precision map is the above-mentioned point cloud map. The occupancy grid map is a map generated by dividing the three-dimensional or two-dimensional space around the vehicle 1 into grid cells each having a predetermined size to indicate the occupancy state of an object based on the grid cells. For example, the occupancy state of an object is represented by the presence or absence of the object or the probability of its presence. For example, the local map is also used to identify the detection process and the recognition process performed by the identification unit 73 for detecting and recognizing the external situation of the vehicle 1.

[0171] Note that the self-position estimation unit 71 can estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data received from the vehicle sensor 27.

[0172] The sensor fusion unit 72 performs a sensor fusion process for generating new information by combining multiple different types of sensor data (for example, the image data provided by the camera 51 and the sensor data provided by the radar 52). The different types of sensor data are combined by methods such as integration, fusion, and association.

[0173] The identification unit 73 performs a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1.

[0174] For example, the identification unit 73 performs the detection process and the recognition process related to the situation outside the vehicle 1 based on the information received from the external identification sensor 25, the information received from the self-position estimation unit 71, the information received from the sensor fusion unit 72, and the like.

[0175] Specifically, for example, the identification unit 73 performs a detection process, a recognition process, etc. related to the objects around the vehicle 1. For example, the detection process for detecting an object is a process of detecting whether the object exists and the size, shape, position, movement, etc. of the object. For example, the recognition process for recognizing an object is a process of recognizing the attributes of the object (for example, the type of the object) and determining a specific object. Note that the detection process and the recognition process are not necessarily clearly distinguishable processes, but may overlap with each other.

[0176] For example, the identification unit 73 detects the objects around the vehicle 1 based on clustering, which classifies the point cloud data corresponding to the sensor data acquired by the radar 52, the lidar 53, etc. into point cloud groups. In this way, the presence or absence, size, shape, and position of the objects around the vehicle 1 are detected.

[0177] For example, the recognition unit 73 detects the movement of an object around the vehicle 1 by tracking the movement of the point cloud group classified by clustering. In this way, the speed and traveling direction (motion vector) of the object around the vehicle 1 are detected.

[0178] For example, the recognition unit 73 detects or recognizes a vehicle, a person, a bicycle, an obstacle, a building, a road, a traffic signal, a traffic sign, a road marking, etc. based on the image data provided by the camera 51. In addition, the recognition unit 73 can identify the type of an object around the vehicle 1 by performing a recognition process such as semantic segmentation.

[0179] For example, the recognition unit 73 can perform a recognition process for recognizing traffic rules around the vehicle 1 based on the map accumulated in the map information accumulation unit 23, the own position estimation result obtained by the own position estimation unit 71, and the recognition result related to the object around the vehicle 1 obtained by the recognition unit 73. The recognition unit 73 that executes this process can identify the position and state of the traffic signal, the details of the traffic sign and the road marking, the details of the traffic rules, and the lane in which the vehicle 1 is allowed to travel, etc.

[0180] For example, the recognition unit 73 can perform a recognition process for recognizing the environment around the vehicle 1. Examples of the surrounding environment specified as the recognition target by the recognition unit 73 include weather, temperature, humidity, brightness, and road surface conditions.

[0181] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates a behavior plan by executing processes for route planning and route tracking.

[0182] Note that route planning (global path planning) is a process for planning a rough route from a starting point to an end point. This route planning also includes a process called trajectory planning, which is used to perform trajectory formation (local path planning), and this trajectory formation forms a route near the vehicle 1 while considering the motion characteristics of the vehicle 1 and ensures that the vehicle 1 can drive safely and smoothly along the planned route.

[0183] Route tracking is a process for planning an action that enables safe and accurate driving along the route planned by the route planning within a predetermined time. For example, the behavior planning unit 62 can calculate the target speed and target angular velocity of the vehicle 1 based on the result of this route tracking process.

[0184] The motion control unit 63 controls the motion of the vehicle 1 to achieve the behavior plan created by the behavior planning unit 62.

[0185] For example, the motion control unit 63 controls the steering control unit 81, the braking control unit 82, and the drive control unit 83 included in the following vehicle control unit 32 to achieve acceleration / deceleration control and direction control, so that the vehicle 1 can travel on the trajectory calculated by the trajectory planning. For example, the motion control unit 63 performs cooperative control for implementing ADAS functions such as collision avoidance or shock absorption, following driving, maintaining vehicle speed driving, early warning of vehicle impact, and early warning of vehicle lane departure. For example, the motion control unit 63 performs cooperative control for autonomous driving purposes or other purposes, and the autonomous driving realizes autonomous driving without driver operation.

[0186] The DMS 30 performs a verification process for verifying the driver, an identification process for identifying the driver's state, and other processes based on the sensor data received from the in-vehicle sensor 26, the input data input to the following HMI 31, and the like. Examples of the driver state designated as corresponding to the identification target include physical condition, wakefulness, degree of concentration, fatigue degree, line of sight direction, degree of intoxication, driving operation, and posture.

[0187] Note that the DMS 30 can also perform an authentication process for verifying occupants other than the driver and an identification process for identifying the state of the occupant. In addition, for example, the DMS 30 can perform an identification process for identifying the interior situation of the vehicle based on the sensor data received from the in-vehicle sensor 26. Examples of the interior situation of the vehicle designated as the identification target include temperature, humidity, brightness, and odor.

[0188] The HMI 31 receives inputs of various data, instructions, etc., and presents various data to the driver and the like.

[0189] The data input implemented by the HMI 31 will be briefly described. The HMI 31 includes an input device for inputting data through manual operations. The HMI 31 generates an input signal based on the data, instructions, etc. input through the input device, and provides the generated input signal to each unit of the vehicle control system 11. The HMI 31 includes operation elements such as a touch panel, buttons, switches, and control levers as input devices. In addition, the HMI 31 can also include an input device that allows information to be input by methods other than manual operations such as voice and gestures. In addition, for example, the HMI 31 can use an external connection device such as a remote controller using infrared or radio waves, a mobile device that processes the operations of the vehicle control system 11, and a wearable device as an input device.

[0190] A brief description will be given of the data presentation implemented by the HMI 31. The HMI 31 generates visual information, auditory information, and tactile information provided to the occupants or outside the vehicle. In addition, the HMI 31 performs output control for controlling the output, output content, output timing, output method, etc. of each piece of generated information. For example, as visual information, the HMI 31 generates and outputs information indicated by images such as an operation screen, a display of the state of the vehicle 1, a warning display, and a monitoring image indicating the situation around the vehicle 1, or by a headlight indication. In addition, as auditory information, the HMI 31 generates and outputs information indicated by sounds such as voice guidance, warning sounds, and warning messages. In addition, as tactile information, the HMI 31 generates and outputs information that gives the occupants a sense of touch through force, vibration, movement, etc.

[0191] As an output device for outputting visual information from the HMI 31, examples that can be adopted include: a display device that displays an image itself to present visual information, or a projector device that projects an image to present visual information. Note that the display device can be a device for displaying visual information within the field of view of the occupants, such as a head-up display, a transmissive display, and a wearable device with an AR (augmented reality) function, rather than a display device with an ordinary display. In addition, the HMI 31 can use a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic rearview mirror, a headlight, or other devices provided on the vehicle 1 as an output device for outputting visual information.

[0192] As an output device for outputting auditory information from the HMI 31, for example, an audio speaker, a headset, earplugs, etc. can be adopted.

[0193] As an output device for outputting tactile information from the HMI 31, examples that can be adopted include tactile elements that apply tactile technology. For example, the tactile elements are provided at parts in contact with the occupants of the vehicle 1, such as a steering wheel and a seat.

[0194] The vehicle control unit 32 controls each unit of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a headlight control unit 85, and a horn control unit 86.

[0195] The steering control unit 81 realizes the detection, control, etc. of the state of the steering system of the vehicle 1. For example, the steering system includes a steering mechanism equipped with a steering wheel, etc., and an electric power steering mechanism. For example, the steering control unit 81 includes a steering ECU for controlling the steering system and an actuator for driving the steering system.

[0196] The brake control unit 82 realizes the detection, control, etc. of the state of the braking system of the vehicle 1. For example, the braking system includes a braking mechanism equipped with a brake pedal, etc., an ABS (antilock braking system), and a regenerative braking mechanism. For example, the brake control unit 82 includes a brake ECU for controlling the braking system and an actuator for driving the braking system.

[0197] The drive control unit 83 realizes the detection and control, etc. of the state of the drive system of the vehicle 1. For example, the drive system includes an accelerator pedal, a driving force generation device for generating a driving force for an internal combustion engine, a drive motor, etc., and a driving force transmission mechanism for transmitting the driving force to the wheels. For example, the drive control unit 83 includes a drive ECU for controlling the drive system and an actuator for driving the drive system.

[0198] The body system control unit 84 realizes the detection and control, etc. of the state of the body system of the vehicle 1. For example, the body system includes a keyless entry system, a smart key system, an automatic window device, an electric seat, an air conditioner, a safety airbag, a seat belt, and a gear shift lever. The body system control unit 84 includes a body system ECU for controlling the body system and an actuator for driving the body system.

[0199] The headlight control unit 85 realizes the detection, control, etc. of the state of various headlights of the vehicle 1. Examples of headlights assumed to be designated corresponding to the control target include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The headlight control unit 85 includes a headlight ECU for controlling the headlights and an actuator for driving the headlights.

[0200] The horn control unit 86 realizes the detection, control, etc. of the state of the vehicle horn of the vehicle 1. For example, the horn control unit 86 includes a horn ECU for controlling the vehicle horn and an actuator for driving the vehicle horn.

[0201] Figure 19 is a diagram showing Figure 18 an example diagram of the sensing areas of the camera 51, radar 52, lidar 53, ultrasonic sensor 54, etc. included in the external recognition sensor 25 shown. Note that Figure 19 schematically shows the state of the vehicle 1 as viewed from above. The left end side corresponds to the front end side (front side) of the vehicle 1, and the right end side corresponds to the rear end side (rear side) of the vehicle 1.

[0202] The sensing area 101F and the sensing area 101B are examples of the sensing areas of the ultrasonic sensor 54. The sensing area 101F covers the periphery of the front end of the vehicle 1 using a plurality of ultrasonic sensors 54. The sensing area 101B covers the periphery of the rear end of the vehicle 1 using a plurality of ultrasonic sensors 54.

[0203] For example, the sensing results obtained within the sensing regions 101F and 101B can be used for parking assistance of the vehicle 1 and the like.

[0204] The sensing regions 102F to 102B are examples of the short - distance and medium - distance sensing regions of the radar 52. The sensing region 102F covers the region in front of the vehicle 1 up to a position farther than the sensing region 101F. The sensing region 102B covers the region behind the vehicle 1 up to a position farther than the sensing region 101B. The sensing region 102L covers the periphery of the left rear side of the vehicle 1. The sensing region 102R covers the periphery of the right rear side of the vehicle 1.

[0205] For example, the sensing results obtained within the sensing region 102F can be used to detect vehicles, pedestrians, etc. in front of the vehicle 1. For example, the sensing results obtained within the sensing region 102B can be used for the rear - collision prevention function of the vehicle 1. For example, the sensing results obtained within the sensing regions 102L and 102R can be used to detect objects located in the side blind spots of the vehicle 1.

[0206] The sensing regions 103F to 103B are examples of the sensing regions of the camera 51. The sensing region 103F covers the region in front of the vehicle 1 up to a position farther than the sensing region 102F. The sensing region 103B covers the region behind the vehicle 1 up to a position farther than the sensing region 102B. The sensing region 103L covers the periphery of the left side of the vehicle 1. The sensing region 103R covers the periphery of the right side of the vehicle 1.

[0207] For example, the sensing results obtained within the sensing region 103F can be used for the recognition of traffic lights or traffic signs, the lane - departure prevention support system, and the automatic headlight control system. For example, the sensing results obtained within the sensing region 103B can be used for parking assistance and the surround - view system. For example, the sensing results obtained within the sensing regions 103L and 103R can be used for the surround - view system.

[0208] The sensing region 104 is an example of the sensing region of the lidar 53. The sensing region 104 covers the region in front of the vehicle 1 up to a position farther than the sensing region 103F. At the same time, the range of the sensing region 104 in the left - right direction is narrower than that of the sensing region 103F.

[0209] For example, the sensing results obtained within the sensing region 104 can be used to detect objects such as surrounding vehicles.

[0210] The sensing region 105 is an example of the long - distance sensing region of the radar 52. The sensing region 105 covers the region in front of the vehicle 1 up to a position farther than the sensing region 104. At the same time, the range of the sensing region 105 in the left - right direction is narrower than that of the sensing region 104.

[0211] For example, the sensing results obtained within the sensing area 105 can be used for ACC (Adaptive Cruise Control), emergency braking, and collision avoidance.

[0212] Note that the sensing areas of the respective sensors included in the external recognition sensor 25 (i.e., the camera 51, the radar 52, the lidar 53, and the ultrasonic sensor 54) can have various configurations other than Figure 19 the configuration shown. Specifically, the ultrasonic sensor 54 can also sense the side of the vehicle 1, and the lidar 53 can sense the rear of the vehicle 1. In addition, the installation positions of the respective sensors are not limited to the above respective examples. Further, each sensor can be composed of a single sensor or a plurality of sensors.

[0213] In addition, for example, the present disclosure can also have the following configuration.

[0214] (1) A solid-state imaging device, comprising: a pixel array unit including a plurality of pixels, wherein the plurality of pixels are each configured to generate charges through photoelectric conversion, and wherein the plurality of pixels include: a plurality of event pixels, wherein the plurality of event pixels are each configured to generate an event signal based on a change in the luminance of incident light; and a plurality of grayscale pixels, wherein the plurality of grayscale pixels are each configured to generate a luminance signal based on the amount of light of incident light; and a plurality of color filters, wherein at least one color filter of the plurality of color filters is disposed above each of the plurality of pixels, wherein the color filter disposed above the event pixels is at least one of a white color filter or a cyan color filter, and wherein the color filter disposed above the grayscale pixels is at least one of a red color filter, a green color filter, or a blue color filter.

[0215] (2) The solid-state imaging device according to (1), wherein the arrangement of the event pixels and the grayscale pixels does not have 180-degree rotational symmetry.

[0216] (3) The solid-state imaging device according to (1) or (2), wherein the color filter disposed above the event pixels includes a white color filter or a cyan color filter, and wherein the color filter disposed above the grayscale pixels includes a red color filter, a green color filter, and a blue color filter.

[0217] (4) The solid-state imaging device according to any one of (1) to (3), wherein the color filter disposed above the event pixels includes a white color filter and a cyan color filter, and Among them, the color filters disposed above the gray-scale pixels include a red color filter, a green color filter, and a blue color filter.

[0218] (5) The solid-state imaging device according to any one of (1) to (4), wherein event pixels adjacent to gray-scale pixels having a blue color filter have a cyan color filter.

[0219] (6) The solid-state imaging device according to any one of (1) to (4), wherein event pixels adjacent to gray-scale pixels having a color filter of any color other than blue have a white color filter.

[0220] (7) The solid-state imaging device according to (4), wherein the event pixels located in the central portion of the pixel array unit respectively have white color filters, and wherein the event pixels located in the peripheral portion of the pixel array unit respectively have cyan color filters.

[0221] (8) The solid-state imaging device according to any one of (1) to (7), wherein the proportion of the number of event pixels included in the pixel array unit to the total number of event pixels and gray-scale pixels is 25% or less.

[0222] (9) The solid-state imaging device according to claim 8, wherein the proportion of the number of event pixels included in the pixel array unit to the total number of event pixels and gray-scale pixels is 12.5% or less.

[0223] (10) The solid-state imaging device according to any one of (1) to (9), wherein each of the event pixels includes an event-based vision sensor (EVS) pixel.

[0224] (11) The solid-state imaging device according to any one of (1) to (10), wherein a first light-shielding wall is provided between the color filters of the plurality of pixels.

[0225] (12) The solid-state imaging device according to (11), wherein the first light-shielding walls respectively include a low-refractive-index material structure or an air structure.

[0226] (13) The solid-state imaging device according to (11) or (12), wherein the thickness of each first light-shielding wall provided between the color filter of the gray-scale pixel and the color filter of the adjacent event pixel is different from the thickness of each first light-shielding wall provided between the color filter of the gray-scale pixel and the color filter of the adjacent gray-scale pixel.

[0227] (14) The solid-state imaging device according to (13), wherein the thickness of each of the first light-shielding walls provided between the color filters of the grayscale pixels each having a blue color filter and the color filters of the adjacent event pixels is greater than the thickness of each of the first light-shielding walls provided between the color filters of the grayscale pixels and the color filters of the adjacent grayscale pixels.

[0228] (15) The solid-state imaging device according to (13), wherein the thickness of each of the first light-shielding walls provided between the color filters of the grayscale pixels each having a color filter other than blue and the color filters of the adjacent event pixels is less than the thickness of each of the first light-shielding walls provided between the color filters of the grayscale pixels and the color filters of the adjacent grayscale pixels.

[0229] (16) The solid-state imaging device according to any one of (1) to (15), further comprising: a plurality of on-chip lenses, wherein one of the on-chip lenses of the plurality of on-chip lenses is disposed above the color filter of each of the plurality of pixels, and wherein each of the on-chip lenses collects incident light; a plurality of second light-shielding walls, wherein one of the second light-shielding walls of the plurality of second light-shielding walls is disposed between the on-chip lenses of each adjacent pair of the plurality of pixels.

[0230] (17) The solid-state imaging device according to any one of (1) to (16), further comprising: a plurality of waveguides, wherein one of the waveguides of the plurality of waveguides is provided on each of at least some of the plurality of color filters, and wherein each of the waveguides forms an optical path of incident light.

[0231] (18) The solid-state imaging device according to (17), wherein the waveguide is provided on the white color filter or the cyan color filter.

[0232] (19) The solid-state imaging device according to any one of (1) to (18), wherein the plurality of pixels further each include a photodiode located in a semiconductor substrate below the color filter and performing photoelectric conversion, and the solid-state imaging device further comprises: a third light-shielding wall, wherein the third light-shielding wall penetrates the inside of the semiconductor substrate and is provided between the photodiodes of the plurality of pixels.

[0233] (20) The solid-state imaging device according to (19), further comprising: A fourth light-shielding wall, wherein the fourth light-shielding wall is disposed between the plurality of pixels in an insulating film below the semiconductor substrate.

[0234] (21) The solid-state imaging device according to any one of (1) to (20), wherein the pixels are arranged in subsets of 4×4 pixels, and each subset of 4×4 pixels includes grayscale pixels surrounding the periphery of the subset and a 2×2 set of event pixels centrally located in the subset.

[0235] (22) The solid-state imaging device according to any one of (1) to (4), (6), (9) to (17), (19) or (20), wherein the pixels are arranged in subsets of 4×4 pixels, and for each subset of 4×4 pixels: The first row includes: a first grayscale pixel and a second grayscale pixel with a red color filter disposed above, and a third grayscale pixel and a fourth grayscale pixel with a green color filter disposed above; The second row includes: a first grayscale pixel and a second grayscale pixel with a red color filter disposed above, a first event pixel, and a third grayscale pixel with a green color filter disposed above; The third row includes: a first grayscale pixel and a second grayscale pixel with a green color filter disposed above, a first event pixel, and a third grayscale pixel with a blue color filter disposed above; and The fourth row includes: a first grayscale pixel and a second grayscale pixel with a green color filter disposed above, and a third grayscale pixel and a fourth grayscale pixel with a blue color filter disposed above.

[0236] (23) The solid-state imaging device according to any one of (1) to (19), wherein for each pixel among the plurality of pixels, each event pixel is adjacent to another event pixel, and each pair of adjacent event pixels has the same type of color filter.

[0237] (24) An electronic device, comprising: An imaging device, wherein the imaging device includes: A pixel array unit including a plurality of pixels, wherein each pixel among the plurality of pixels is configured to generate charges through photoelectric conversion, and the plurality of pixels include: a plurality of event pixels, wherein each event pixel among the plurality of event pixels is configured to generate an event signal based on a change in luminance of incident light; and a plurality of grayscale pixels, wherein each grayscale pixel among the plurality of grayscale pixels is configured to generate a luminance signal based on the amount of incident light; and Multiple color filters, wherein at least one color filter among the multiple color filters is disposed above each of the multiple pixels, wherein the color filter disposed above the event pixel is at least one of a white color filter or a cyan color filter, and wherein the color filter disposed above the gray-scale pixel is at least one of a red color filter, a green color filter, or a blue color filter.

[0238] Those skilled in the art should understand that various modifications, combinations, sub-combinations, and changes can be made according to design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents. List of reference numerals

[0239] 1: Vehicle 2: Driving unit 3: Arbiter 4: Event signal processing unit 5: Luminance signal processing unit 6: Central part of the pixel array unit 7: Peripheral part of the pixel array unit 9: Pixel 9a: Gray-scale pixel 9b: Event pixel 10: Pixel array unit 11: Vehicle control system 21: Vehicle control ECU 22: Communication unit 23: Map information accumulation unit 24: Position information acquisition unit 25: External recognition sensor 26: On-vehicle sensor 27: Vehicle sensor 28: Storage unit 29: Driving assistance and autonomous driving control unit 30: DMS 31: HMI 32: Vehicle control unit 41: Communication network 51: Camera 52: Radar 53: Lidar 54: Ultrasonic sensor 61: Analysis unit 62: Behavior planning unit 63: Action control unit 71: Self-position estimation unit 72: Sensor fusion unit 73: Recognition unit 81: Steering control unit 82: Brake control unit 83: Drive control unit 84: Body system control unit 85: Headlamp control unit 86: Horn control unit 88: Semiconductor substrate 89: Insulating film 90: On-chip lens 91: Inter-pixel light-shielding film 92: Photodiode 93: First light-shielding wall 93': First light-shielding wall 93'': First light-shielding wall 93''': First light-shielding wall 94: Second light-shielding wall 95: Waveguide 96: Element isolation insulating film 97: Third light-shielding wall 98: Fourth light-shielding wall 99: Transfer transistor 100: Imaging device 110: Imaging lens 120: Data processing unit 130: Control unit 140: Recording unit 150: Data generation unit 200: Solid-state imaging device R: Red G: Green B: Blue W: White W': White C: Cyan

Claims

1. A solid-state imaging device, comprising: A pixel array unit including a plurality of pixels, wherein each of the plurality of pixels is configured to generate charges through photoelectric conversion, and wherein the plurality of pixels include: a plurality of event pixels, wherein each of the plurality of event pixels is configured to generate an event signal based on a change in luminance of incident light; and a plurality of grayscale pixels, wherein each of the plurality of grayscale pixels is configured to generate a luminance signal based on the amount of incident light; and A plurality of color filters, wherein at least one color filter of the plurality of color filters is disposed above each of the plurality of pixels, wherein the color filter disposed above the event pixel is at least one of a white color filter or a cyan color filter, and wherein the color filter disposed above the grayscale pixel is at least one of a red color filter, a green color filter, or a blue color filter.

2. The solid-state imaging device according to claim 1, wherein, The arrangement of the event pixels and the grayscale pixels does not have 180-degree rotational symmetry.

3. The solid-state imaging device according to claim 1, wherein, The color filter disposed above the event pixel includes a white color filter or a cyan color filter, and wherein the color filter disposed above the grayscale pixel includes a red color filter, a green color filter, and a blue color filter.

4. The solid-state imaging device according to claim 1, Among them, The color filter disposed above the event pixel includes a white color filter and a cyan color filter, and wherein the color filter disposed above the grayscale pixel includes a red color filter, a green color filter, and a blue color filter.

5. The solid-state imaging device according to claim 4, wherein, An event pixel adjacent to a grayscale pixel having a blue color filter has a cyan color filter.

6. The solid-state imaging device according to claim 4, wherein, An event pixel adjacent to a grayscale pixel having a color filter other than blue has a white color filter.

7. The solid-state imaging device according to claim 4, Among them, Each of the event pixels located in the central portion of the pixel array unit has a white color filter, and wherein each of the event pixels located in the peripheral portion of the pixel array unit has a cyan color filter.

8. The solid-state imaging device according to claim 1, wherein, The proportion of the number of event pixels included in the pixel array unit to the total number of event pixels and grayscale pixels is 25% or less.

9. The solid-state imaging device according to claim 8, wherein, The proportion of the number of event pixels included in the pixel array unit to the total number of event pixels and grayscale pixels is 12.5% or less.

10. The solid-state imaging device according to claim 1, wherein, Each of the event pixels includes an event-based vision sensor (EVS) pixel.

11. The solid-state imaging device according to claim 1, wherein, A first light-shielding wall is provided between the color filters of the plurality of pixels.

12. The solid-state imaging device according to claim 11, wherein, Each of the first light-shielding walls includes a low refractive index material structure or an air structure.

13. The solid-state imaging device according to claim 11, wherein, The thickness of each of the first light-shielding walls provided between the color filter of the grayscale pixel and the color filter of the adjacent event pixel is different from the thickness of each of the first light-shielding walls provided between the color filter of the grayscale pixel and the color filter of the adjacent grayscale pixel.

14. The solid-state imaging device according to claim 13, wherein, The thickness of each of the first light-shielding walls disposed between the color filters of the gray-scale pixels each having a blue color filter and the color filters of the adjacent event pixels is greater than the thickness of each of the first light-shielding walls disposed between the color filters of the gray-scale pixels and the color filters of the adjacent gray-scale pixels.

15. The solid-state imaging device according to claim 13, wherein, The thickness of each of the first light-shielding walls disposed between the color filters of the gray-scale pixels each having a color filter other than blue and the color filters of the adjacent event pixels is less than the thickness of each of the first light-shielding walls disposed between the color filters of the gray-scale pixels and the color filters of the adjacent gray-scale pixels.

16. The solid-state imaging device according to claim 1, further comprising: A plurality of on-chip lenses, wherein one of the plurality of on-chip lenses is disposed above the color filter of each of the plurality of pixels, and wherein each of the on-chip lenses collects incident light; A plurality of second light-shielding walls, wherein one of the plurality of second light-shielding walls is disposed between the on-chip lenses of each adjacent pair of the plurality of pixels.

17. The solid-state imaging device according to claim 16, further comprising: A plurality of waveguides, wherein one of the plurality of waveguides is provided on each of at least some of the plurality of color filters, and wherein each of the waveguides constitutes an optical path of incident light.

18. The solid-state imaging device according to claim 17, wherein, The waveguide is provided on the white color filter or the cyan color filter.

19. The solid-state imaging device according to claim 1, wherein Each of the plurality of pixels further includes a photodiode, the photodiode being located within a semiconductor substrate below the color filter and performing photoelectric conversion, and the solid-state imaging device further comprising: A third light-shielding wall, wherein the third light-shielding wall penetrates the interior of the semiconductor substrate and is disposed between the photodiodes of the plurality of pixels.

20. The solid-state imaging device according to claim 19, further comprising: A fourth light-shielding wall, wherein the fourth light-shielding wall is disposed within an insulating film below the semiconductor substrate between the plurality of pixels.

21. The solid-state imaging device according to claim 1, wherein, The pixels are arranged in subsets of 4×4 pixels, wherein for each of the subsets of 4×4 pixels: The first row includes: a first gray-scale pixel and a second gray-scale pixel having a red color filter disposed above, and a third gray-scale pixel and a fourth gray-scale pixel having a green color filter disposed above; The second row includes: a first gray-scale pixel having a red color filter disposed above, a first event pixel, a second event pixel, and a second gray-scale pixel having a green color filter disposed above; The third row includes: a first gray-scale pixel having a green color filter disposed above, a first event pixel, a second event pixel, and a second gray-scale pixel having a blue color filter disposed above; and The fourth row includes: a first gray-scale pixel and a second gray-scale pixel having a green color filter disposed above, and a third gray-scale pixel and a fourth gray-scale pixel having a blue color filter disposed above.

22. The solid-state imaging device according to claim 1, wherein, The pixels are arranged in subsets of 4×4 pixels, wherein for each of the subsets of 4×4 pixels: The first row includes: a first grayscale pixel and a second grayscale pixel with a red color filter disposed above, and a third grayscale pixel and a fourth grayscale pixel with a green color filter disposed above; The second row includes: a first grayscale pixel and a second grayscale pixel with a red color filter disposed above, a first event pixel, and a third grayscale pixel with a green color filter disposed above; The third row includes: a first grayscale pixel and a second grayscale pixel with a green color filter disposed above, a first event pixel, and a third grayscale pixel with a blue color filter disposed above; and The fourth row includes: a first grayscale pixel and a second grayscale pixel with a green color filter disposed above, and a third grayscale pixel and a fourth grayscale pixel with a blue color filter disposed above.

23. The solid-state imaging device according to claim 1, wherein, For each pixel of the plurality of pixels, each of the event pixels is adjacent to another of the event pixels, and wherein each pair of adjacent event pixels has the same type of color filter.

24. An electronic device, comprising: An imaging device, Wherein, the imaging device includes: A pixel array unit, the pixel array unit including a plurality of pixels, wherein each pixel of the plurality of pixels is configured to generate charge through photoelectric conversion, wherein the plurality of pixels includes: a plurality of event pixels, wherein each pixel of the plurality of event pixels is configured to generate an event signal based on a change in the brightness of incident light; and a plurality of grayscale pixels, wherein each pixel of the plurality of grayscale pixels is configured to generate a brightness signal based on the amount of incident light; and A plurality of color filters, wherein at least one of the plurality of color filters is disposed above each pixel of the plurality of pixels, wherein the color filter disposed above the event pixel is at least one of a white color filter or a cyan color filter, and wherein the color filter disposed above the grayscale pixel is at least one of a red color filter, a green color filter, or a blue color filter.

Images