Light receiving device, solid-state imaging device, electronic device, and information processing system

Abstract

The object of the present invention is to enable high-speed image recognition processing. According to the embodiment, a light receiving device is provided with: a plurality of first filters (130), each of which transmits edge components in a predetermined direction in an incident image; a plurality of second filters (150), each of which transmits light of a predetermined wavelength band in the incident light; and a plurality of photoelectric conversion elements (PD), each of which performs photoelectric conversion on light transmitted through one of a plurality of convolution filters and one of a plurality of color filters.

Description

Technical Field

[0001] The present disclosure relates to a light receiving device, a solid-state imaging device, an electronic device, and an information processing system. Background Art

[0002] In recent years, technology for recognizing an object included in an image by performing image processing of a convolution operation on image data acquired by an imaging device has been developed.

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent literature 1: Huaijin G.Chen, Suren Jayasuriya, Jiyue Yang, Judy Stephen, Sriram Sivaramakrishnan, Ashok Veeraraghavan, Alyosha C.Molnar; ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels (CVPR) 2016, pp.903-912. Summary of the invention

[0006] Technical issues

[0007] However, since image recognition processing using convolution operations has a large amount of data to be processed and the processing itself is complicated, there is a problem that it is difficult to achieve higher real-time performance.

[0008] Therefore, the present disclosure proposes a light receiving device, a solid-state imaging apparatus, an electronic device, and an information processing system that can realize high-speed image recognition processing.

[0009] Solution to the problem

[0010] In order to solve the above-mentioned problems, according to one aspect of the present disclosure, a light receiving device includes: a plurality of first filters, each of which transmits edge components in a predetermined direction in an incident image; a plurality of second filters, each of which transmits light in a predetermined wavelength band in the incident light; and a plurality of photoelectric conversion elements, each of which performs photoelectric conversion on light transmitted through a first filter among the plurality of first filters and a second filter among the plurality of second filters. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram showing a schematic configuration example of an information processing system including an electronic device according to an embodiment.

[0012] Figure 2 is a block diagram showing a schematic configuration example of a solid-state imaging device according to the embodiment.

[0013] Figure 3 is a circuit diagram showing a circuit configuration example of a unit pixel according to an embodiment.

[0014] Figure 4 is a perspective view showing a stacked configuration example of a solid-state imaging device according to the embodiment.

[0015] Figure 5 is a perspective view showing a stacked configuration example of a solid-state imaging device according to a modification of the embodiment.

[0016] Figure 6 This is a diagram used to illustrate a general CNN.

[0017] Figure 7 is a diagram for describing an overview of a convolutional layer which is the first layer of a CNN.

[0018] Figure 8 is a diagram for explaining a case where the embodiment is applied to a convolutional layer which is the first layer of a CNN.

[0019] Fig. 9 is a diagram illustrating an example of a convolution filter according to an embodiment.

[0020] Fig.10 is a diagram illustrating an example of a convolution filter array according to an embodiment.

[0021] Fig.11 is a diagram illustrating an example of a frequency spectrum of an edge component acquired by a convolution filter array according to an embodiment.

[0022] Fig.12 It is to show that the composition can obtain Fig.11 A schematic diagram of an example of a convolution filter unit of a convolution filter array showing edge components of a frequency spectrum.

[0023] Fig.13 : is a plan view showing a schematic configuration example of a combined filter combining a convolution filter array and a color filter array according to the embodiment.

[0024] Fig.14 is a diagram illustrating an example of frame data generated by an image sensor according to an embodiment.

[0025] Fig.15 is a plan view showing a schematic configuration example of a combined filter according to a modification of the embodiment.

[0026] Fig.16is a diagram for explaining an overview of an optical convolution operation according to an embodiment.

[0027] Fig.17 is a diagram for explaining an overview of an optical convolution operation (with a color filter) according to an embodiment.

[0028] Fig.18 is a diagram for explaining an outline of a convolution operation according to a modification of the embodiment (Part 1).

[0029] Fig.19 is a diagram for explaining an outline of a convolution operation according to a modification of the embodiment (Part 2).

[0030] Fig. 20 is a diagram for explaining an outline of a convolution operation according to a modification of the embodiment (Part 3).

[0031] Fig.21 is a diagram for explaining an outline of a convolution operation according to a modification of the embodiment (Part 4).

[0032] Fig. 22 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0033] Fig.23 It is a diagram showing an example of the installation positions of the vehicle exterior information detection section and the imaging section. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0035] In addition, the present disclosure will be described according to the following item order.

[0036] 1. Example

[0037] 1.1 Schematic Configuration Example of Electronic Devices

[0038] 1.2 Schematic Configuration Example of Solid-State Imaging Device

[0039] 1.3 Circuit Configuration Example of Unit Pixel

[0040] 1.4 Basic Function Examples of Unit Pixels

[0041] 1.5 Image sensor stack configuration example

[0042] 1.5.1 Modifications

[0043] 1.6 Application Examples of Optical Convolution Operation

[0044] 1.7 Overview of CNN

[0045] 1.8 Application of this embodiment

[0046] 1.9 Convolutional Filters

[0047] 1.10 Example of the function of convolution filter array

[0048] 1.11 Relationship between the pattern of the convolution filter array and the spectrum

[0049] 1.12 Configuration example of combined filter

[0050] 1.12.1 Modification of combined filter

[0051] 1.13 Overview of Convolution Operation (Without Color Filter)

[0052] 1.14 Overview of Convolution Operation (with Color Filter)

[0053] 1.14.1 Modification of convolution operation

[0054] 1.15 Operation and Effect

[0055] 2. Application to mobile objects

[0056] 1. Example

[0057] 1.1 Schematic Configuration Example of Electronic Devices

[0058] Figure 1 1 is a block diagram showing a schematic configuration example of an information processing system including an electronic device according to an embodiment. Figure 1 As shown, the electronic device 1 includes an imaging device 10 and an application processor 20. The imaging device 10 includes an imaging unit 11, a control unit 12, a signal processing unit 13, a digital signal processor (DSP) 14, a memory 15, and an output unit 16.

[0059] The control unit 12 controls each unit in the imaging device 10 according to, for example, a user's operation or a set operation mode.

[0060] The imaging unit 11 includes, for example, an optical system 11a including a zoom lens, a focusing lens, an aperture, etc., and a solid-state imaging device 100 having a configuration in which unit pixels including light receiving elements such as photodiodes are arranged in a two-dimensional matrix. Light incident from the outside is imaged on a light receiving surface on which the light receiving elements in the solid-state imaging device 100 are arranged through the optical system 11a. Each unit pixel of the solid-state imaging device 100 electrically converts the light incident on the light receiving element, thereby readable and storing a charge corresponding to the amount of incident light. Then, the solid-state imaging device 100 outputs a pixel signal as data in units of frames based on the charge accumulated in each unit pixel. Note that the details of the solid-state imaging device 100 will be described later.

[0061] In addition, in the present embodiment, the data read from the solid-state imaging device 100 in units of frames is the result of a convolution operation (optical convolution operation described later) performed using a physical convolution filter described later. Therefore, the data read from the solid-state imaging device 100 is, for example, binary data such as a feature map.

[0062] The signal processing unit 13 performs various types of signal processing on the binary data read from the solid-state imaging device 100. For example, the signal processing unit 13 compresses the transmission amount by compressing the binary data by run-length compression or the like. In addition, in the case where the binary data includes color information, the signal processing unit 13 may convert the binary data into a YUV format, an RGB format, or the like. Furthermore, the signal processing unit 13 may perform processing such as noise removal and white balance adjustment on the binary data as required.

[0063] Note that in the present embodiment, the signal processing unit 13 is not an essential component and can be omitted. In this case, the binary data output from the solid-state imaging device 100 may be directly input to the DSP 14 or the memory 15, or may be output to the external application processor 20 or the like via the output unit 16 without passing through the DSP 14. In addition, the binary data output from the imaging unit 11 may be data compressed by run-length compression or the like.

[0064] For example, the DSP 14 may perform various types of signal processing on the input binary data. The DSP 14 may perform image recognition processing using a deep neural network (DNN) on the input binary data. In this case, the DSP 14 functions as a machine learning unit using the DNN by reading and executing a learning model stored in the memory 15. Then, the DSP 14 functioning as a machine learning unit performs image recognition processing using the DNN by multiplying the dictionary coefficients stored in the memory 15 by the binary data.

[0065] Furthermore, the DSP 14 outputs a result obtained by signal processing the binary data (hereinafter, referred to as a signal processing result) to the memory 15 and / or the output unit 16. Note that a memory controller that controls access to the memory 15 may be incorporated in the DSP 14.

[0066] Note that in the present embodiment, the DSP 14 is not an essential component and can be omitted. Alternatively, the DSP 14 can output the input binary data as it is without performing any signal processing on the input binary data. In these cases, the binary data output from the solid-state imaging device 100 or the signal processing unit 13 can be input to the memory 15, or can be output to the external application processor 20 or the like via the output unit 16.

[0067] The memory 15 stores the signal processing results obtained by the DSP 14 as needed. In addition, the memory 15 can store the algorithm of the learning model executed by the DSP 14 as a program and dictionary coefficients. For example, the program and dictionary coefficients of the learning model created by the external cloud server 30 or the like can be downloaded to the electronic device 1 via the network 40 and stored in the memory 15, or can be stored in the memory 15 before the shipment of the electronic device 1.

[0068] The output unit 16 selectively outputs binary data output from the solid-state imaging device 100, the signal processing unit 13 or the DSP 14, signal processing results output from the DSP 14, or binary data or signal processing results stored in the memory 15, for example, according to a selection control signal from the control unit 12.

[0069] The binary data or signal processing result output from the output unit 16 as described above is input to the application processor 20 that processes display, user interface, etc. The application processor 20 is configured using, for example, a central processing unit (CPU) or the like, and executes an operating system, various application software, etc. The application processor 20 may be equipped with functions such as a graphics processing unit (GPU) and a baseband processor. The application processor 20 performs various types of processing on the input binary data or signal processing result as needed, performs display to the user, or sends the input binary data or signal processing result to the external cloud server 30 via a predetermined network 40.

[0070] Note that various networks such as the Internet, a wired local area network (LAN), a wireless LAN, a mobile communication network, and Bluetooth (registered trademark), for example, can be used as the predetermined network 40. In addition, the transmission destination of the binary data or the signal processing result is not limited to the cloud server 30, and may be various information processing devices (systems) having a communication function, such as a server operating alone or in cooperation with another server, a file server storing various data, and a communication terminal such as a mobile phone.

[0071] 1.2 Schematic Configuration Example of Solid-State Imaging Device

[0072] Figure 2 1 is a block diagram showing a schematic configuration example of a complementary metal oxide semiconductor (CMOS) solid-state imaging device (hereinafter, referred to as an image sensor) according to an embodiment. Here, the CMOS image sensor is an image sensor created by applying or partially using a CMOS process. The solid-state imaging device 100 according to the present embodiment may be a so-called back-illuminated type in which the incident surface is on the side of the surface (hereinafter, referred to as the back side) opposite to the element formation surface in the semiconductor substrate, or may be a so-called front-illuminated type in which the incident surface is on the front surface side.

[0073] like Figure 2 As shown, the image sensor 100 includes, for example, a pixel array unit 101, a vertical driving circuit 102, a column processing circuit 103, a horizontal driving circuit 104, a system control unit 105, a signal processing circuit 108, and a data storage unit 109. In the following description, the vertical driving circuit 102, the column processing circuit 103, the horizontal driving circuit 104, the system control unit 105, the signal processing circuit 108, and the data storage unit 109 are also referred to as peripheral circuits.

[0074] The pixel array unit 101 has unit pixels (hereinafter, it may be simply described as "pixels") 110 each having a photoelectric conversion element that generates and accumulates electric charge according to the amount of light received, arranged in the row direction and the column direction, that is, in a two-dimensional grid pattern in a matrix (hereinafter, referred to as a matrix pattern). Here, the row direction refers to the arrangement direction of pixels in a pixel row (horizontal direction in the drawings), and the column direction refers to the arrangement direction of pixels in a pixel column (vertical direction in the drawings). The specific circuit configuration and pixel structure of the unit pixel will be described in detail later.

[0075] In the pixel array unit 101, with respect to the matrix-shaped pixel array, the pixel drive line LD is wired in the row direction for each pixel row, and the vertical signal line VSL is wired in the column direction for each pixel column. The pixel drive line LD transmits a drive signal for driving when reading a signal from a pixel. Figure 2 , the pixel drive line LD is shown as being wired one by one, but is not limited to being wired one by one. One end of the pixel drive line LD is connected to an output terminal corresponding to each row of the vertical drive circuit 102.

[0076] The vertical driving circuit 102 includes a shift register, an address decoder, etc., and drives all pixels of the pixel array unit 101 simultaneously or in units of rows. That is, the vertical driving circuit 102, together with a system control unit 105 that controls the vertical driving circuit 102, constitutes a driving unit that controls the operation of each pixel of the pixel array unit 101. Although the specific configuration of the vertical driving circuit 102 is not shown, the vertical driving circuit generally includes two scanning systems of a reading scanning system and a sweeping scanning system.

[0077] The reading scanning system selectively scans the unit pixels of the pixel array unit 101 row by row in order to read signals from the unit pixels. The signals read from the unit pixels are analog signals. The sweeping scanning system performs sweeping scanning on the reading row for which the reading scanning is performed by the reading scanning system at an exposure time before the reading scanning.

[0078] By sweeping scanning of the sweeping scanning system, the excess charge is swept out from the photoelectric conversion element of the unit pixel of the read row, and the photoelectric conversion element is reset. Then, by sweeping out (resetting) the unnecessary charge by the sweeping scanning system, the so-called electronic shutter operation is performed. Here, the electronic shutter operation refers to the operation of discarding the charge of the photoelectric conversion element and restarting exposure (starting the accumulation of charge).

[0079] The signal read by the reading operation of the reading scanning system corresponds to the amount of light received after the immediately preceding reading operation or the electronic shutter operation. Then, the time period from the reading timing of the immediately preceding reading operation or the scanning timing of the electronic shutter operation to the reading timing of the current reading operation is the charge accumulation time period in the unit pixel (also referred to as the exposure time period).

[0080] A signal output from each unit pixel of a pixel row selectively scanned by the vertical drive circuit 102 is input to the column processing circuit 103 through each vertical signal line VSL of each pixel column. The column processing circuit 103 performs predetermined signal processing on the signal output from each pixel of the selected row through the vertical signal line VSL of each pixel column of the pixel array unit 101, and temporarily holds the pixel signal after the signal processing.

[0081] Specifically, the column processing circuit 103 performs at least noise removal processing, such as correlated double sampling (CDS) processing or double data sampling (DDS) processing as signal processing. For example, fixed pattern noise unique to pixels, such as reset noise and threshold value variation of the amplifier transistor in the pixel, is removed by CDS processing. The column processing circuit 103 also includes, for example, an analog-to-digital (AD) conversion function, converts an analog pixel signal read and obtained from a photoelectric conversion element into a digital signal, and outputs the digital signal.

[0082] The horizontal drive circuit 104 includes a shift register, an address decoder, etc., and sequentially selects a reading circuit (hereinafter, referred to as a pixel circuit) corresponding to a pixel column of the column processing circuit 103. Through the selective scanning of the horizontal drive circuit 104, a pixel signal subjected to signal processing is sequentially output for each pixel circuit in the column processing circuit 103.

[0083] The system control unit 105 includes a timing generator that generates various timing signals and the like, and performs drive control of the vertical drive circuit 102 , the column processing circuit 103 , the horizontal drive circuit 104 , and the like based on the various timings generated by the timing generator.

[0084] The signal processing circuit 108 has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing on the pixel signal output from the column processing circuit 103. The data storage unit 109 temporarily stores data required for signal processing in the signal processing circuit 108. Note that the signal processing circuit 108 may have the same configuration as or a different configuration from the above-described signal processing unit 13. In addition, the signal processing circuit 108 may be omitted.

[0085] Note that, as described above, the binary data output from the signal processing circuit 108 (or the column processing circuit 103 ) is input to the signal processing unit 13 , the DSP 14 , the memory 15 , or the output unit 16 .

[0086] 1.3 Circuit Configuration Example of Unit Pixel

[0087] Figure 3 1 is a circuit diagram showing an example of a circuit configuration of a unit pixel according to an embodiment. Figure 3 As shown, the unit pixel 110 includes a photodiode PD, a transfer transistor 111 , a reset transistor 112 , an amplifying transistor 113 , a selection transistor 114 , and a floating diffusion layer FD.

[0088] The selection transistor driving line LD114 included in the pixel driving line LD is connected to the gate of the selection transistor 114, the reset transistor driving line LD112 included in the pixel driving line LD is connected to the gate of the reset transistor 112, and the transfer transistor driving line LD111 included in the pixel driving line LD is connected to the gate of the transfer transistor 111. In addition, the vertical signal line VSL having one end connected to the column processing circuit 103 is connected to the drain of the amplification transistor 113 via the selection transistor 114.

[0089] In the following description, the reset transistor 112 , the amplifying transistor 113 , and the selecting transistor 114 are also collectively referred to as a pixel circuit. The pixel circuit may include a floating diffusion layer FD and / or a transfer transistor 111 .

[0090] The photodiode PD performs photoelectric conversion on incident light. The transfer transistor 111 transfers the charge generated in the photodiode PD. The floating diffusion layer FD accumulates the charge transferred by the transfer transistor 111. The amplifier transistor 113 causes a pixel signal having a voltage value corresponding to the charge accumulated in the floating diffusion layer FD to appear in the vertical signal line VSL. The reset transistor 112 releases the charge accumulated in the floating diffusion layer FD. The selection transistor 114 selects the unit pixel 110 to be read.

[0091] The anode of the photodiode PD is grounded, and the cathode is connected to the source of the transfer transistor 111. The drain of the transfer transistor 111 is connected to the source of the reset transistor 112 and the gate of the amplification transistor 113, and the node as the connection point of these transistors constitutes the floating diffusion layer FD. Note that the drain of the reset transistor 112 is connected to a vertical reset input line (not shown).

[0092] The source of the amplifier transistor 113 is connected to a vertical current supply line (not shown). The drain of the amplifier transistor 113 is connected to the source of the selection transistor 114, and the drain of the selection transistor 114 is connected to the vertical signal line VSL.

[0093] The floating diffusion layer FD converts the accumulated charge into a voltage having a voltage value corresponding to the amount of charge. Note that the floating diffusion layer FD may be, for example, a grounding capacitor. However, without limitation thereto, the floating diffusion layer FD may be a capacitor that is increased by intentionally connecting a capacitor or the like to a node where the drain of the transfer transistor 111, the source of the reset transistor 112, and the gate of the amplification transistor 113 are connected.

[0094] 1.4 Basic Function Examples of Unit Pixels

[0095] Next, we will refer to Figure 3 The basic function of the unit pixel 110 is described. The reset transistor 112 controls the discharge (reset) of the charge accumulated in the floating diffusion layer FD according to the reset signal RST supplied from the vertical drive circuit 102 via the reset transistor drive line LD112. Note that when the reset transistor 112 is in the on state, by turning on the transfer transistor 111, in addition to the charge accumulated in the floating diffusion layer FD, the charge accumulated in the photodiode PD can also be discharged (reset).

[0096] When a high-level reset signal RST is input to the gate of the reset transistor 112, the floating diffusion layer FD is clamped to the voltage applied through the vertical reset input line. Therefore, the charge accumulated in the floating diffusion layer FD is discharged (reset).

[0097] Furthermore, when a low-level reset signal RST is input to the gate of the reset transistor 112 , the floating diffusion layer FD is electrically disconnected from the vertical reset input line, and enters a floating state.

[0098] The photodiode PD performs photoelectric conversion on the incident light and generates charges corresponding to the amount of light. The generated charges are accumulated on the cathode side of the photodiode PD. The transfer transistor 111 controls the transfer of charges from the photodiode PD to the floating diffusion layer FD according to the transfer control signal TRG provided from the vertical drive circuit 102 via the transfer transistor drive line LD111.

[0099] For example, when a high-level transfer control signal TRG is input to the gate of the transfer transistor 111, the charge accumulated in the photodiode PD is transferred to the floating diffusion layer FD. On the other hand, when a low-level transfer control signal TRG is supplied to the gate of the transfer transistor 111, the transfer of charge from the photodiode PD is stopped.

[0100] As described above, the floating diffusion layer FD has a function of converting the charge transferred from the photodiode PD via the transfer transistor 111 into a voltage of a voltage value corresponding to the charge amount. Therefore, in a floating state where the reset transistor 112 is off, the potential of the floating diffusion layer FD is modulated according to the charge amount accumulated therein.

[0101] The amplification transistor 113 functions as an amplifier using a potential change of the floating diffusion layer FD connected to its gate as an input signal, and its output voltage signal appears in the vertical signal line VSL as a pixel signal via the selection transistor 114 .

[0102] The selection transistor 114 controls the appearance of the pixel signal in the vertical signal line VSL through the amplifier transistor 113 according to the selection control signal SEL provided from the vertical drive circuit 102 via the selection transistor drive line LD114. For example, when the high-level selection control signal SEL is input to the gate of the selection transistor 114, the pixel signal appears in the vertical signal line VSL through the amplifier transistor 113. On the other hand, when the low-level selection control signal SEL is input to the gate of the selection transistor 114, the appearance of the pixel signal in the vertical signal line VSL is stopped. Therefore, only the output of the selected unit pixel 110 can be extracted in the vertical signal line VSL connected to the plurality of unit pixels 110.

[0103] 1.5 Image sensor stack configuration example

[0104] Figure 4 is a perspective view showing an example of a stacked configuration of an image sensor according to an embodiment. Figure 4In the following description, for simplicity, the case where the image sensor 100 is 4×4 pixels will be exemplified.

[0105] like Figure 4 As shown, the image sensor 100 includes a semiconductor chip 121, a convolution filter array 122, a color filter array 123, and a microlens array 124. Figure 4 In the figure, the semiconductor chip 121, the convolution filter array 122, the color filter array 123 and the microlens array 124 are shown as being separated in the stacking direction, but in fact, the semiconductor chip 121, the convolution filter array 122, the color filter array 123 and the microlens array 124 are constructed as one chip.

[0106] The semiconductor chip 121 includes, for example, Figure 2 The components in the solid-state imaging device 100 illustrated in Figure 3 The semiconductor chip 121 may include one die, or may be a laminated chip in which a plurality of dies are bonded. On the light receiving surface of the semiconductor chip 121, a plurality of photodiodes PD constituting the pixel array unit 101 are arranged in a matrix.

[0107] For example, the convolution filter array 122 is provided on the light receiving surface of the semiconductor chip 121. The convolution filter array 122 has, for example, a configuration in which convolution filters (first filters) 130 corresponding one-to-one to the respective photodiodes PD are arranged in a matrix.

[0108] For example, the color filter array 123 is provided on the convolution filter array 122. The color filter array 123 has, for example, a configuration in which color filters (second filters) 150 corresponding one-to-one to the respective photodiodes PD are arranged in a matrix.

[0109] Note that the repetitive unit pattern of the color filter array 123 according to the present embodiment (hereinafter, referred to as a color filter unit) may be a 2×2 pixel Bayer array including one red (R) pixel, one blue (B) pixel, and two green (G) pixels. However, not limited thereto, various color filter arrays may be used, such as a 3×3 pixel color filter array (hereinafter, referred to as an X-trans (registered trademark) type array) adopted in an X-trans (registered trademark) CMOS sensor, a 4×4 pixel quad Bayer array (also referred to as an orthogonal array), and a 4×4 pixel color filter obtained by combining a white RGB color filter with a Bayer array (hereinafter, referred to as a white RGB array).

[0110] For example, the microlens array 124 is provided on the color filter array 123. The microlens array 124 has, for example, a configuration in which on-chip lenses 160 corresponding to the corresponding photodiodes PD are arranged in a matrix. However, it is not limited to such a configuration, and one on-chip lens 160 may be associated with two or more photodiodes PD. That is, one on-chip lens 160 may be shared by two or more unit pixels 110.

[0111] According to the above configuration, each unit pixel 110 includes a pixel circuit formed in the semiconductor chip 121 , a convolution filter 130 on the photodiode PD in the pixel circuit, a color filter 150 on the convolution filter 130 , and an on-chip lens 160 on the color filter 150 .

[0112] 1.5.1 Modifications

[0113] Note that the location of the convolution filter array 122 is not limited to Figure 4 The position between the semiconductor chip 121 and the color filter array 123 is shown. Figure 5 In the image sensor 100A illustrated in FIG. 1 , the convolution filter array 122 may be disposed between the color filter array 123 and the microlens array 124 .

[0114] 1.6 Application Examples of Optical Convolution Operation

[0115] The convolution filter array 122 according to the present embodiment has, for example, a physical configuration that performs an optical convolution operation on an image of light incident on an array (pixel array unit 101) of unit pixels 110 (specifically, photodiodes PD) (hereinafter, referred to as an incident image). In this specification, the convolution operation performed using the convolution filter array 122 is referred to as an optical convolution operation.

[0116] Here, an application example of the convolution operation will be described using a convolutional neural network (CNN) which is one of the DNNs.

[0117] Figure 6 This is a diagram used to illustrate a general CNN. Figure 6 As shown in the figure, CNN includes an input layer, multiple convolutional layers and pooling layers that are repeated alternately, a fully connected layer, and an output layer.

[0118] The optical convolution operation performed using the convolution filter array 122 according to the present embodiment can be applied to, for example, Figure 6 In this case, the data input to the input layer of the CNN may be an incident image on the image sensor 100.

[0119] However, the optical convolution operation performed using the convolution filter array 122 according to the present embodiment is not limited to a CNN whose first layer is a convolution layer, and can be applied to various types of processing that perform a convolution operation on an input (i.e., an incident image on the image sensor 100).

[0120] 1.7 Overview of CNN

[0121] Here, an outline of a first layer of a CNN to which an optical convolution operation according to an embodiment can be applied will be described.

[0122] Figure 7 is a diagram for describing the outline of the convolutional layer as the first layer of CNN. Note that Figure 7 The case where the number of channels is K and the frame data of each channel is W×W pixels of frame data 1050 is provided to the input layer is illustrated. Note that the number of channels corresponds to the number (3) of color components of, for example, the RGB three primary colors to be described later, and in this description, K=3. M corresponds to the number of types of, for example, the convolution filter 130 to be described later.

[0123] like Figure 7 As shown, the convolutional layer at the first layer receives K channel data Z from the immediately preceding 1-1 layer (l-1) ijk (k=0, ..., K-1). In the first convolutional layer, the received Z (l-1) ijk Execute using M filters hpqkm Convolution operation of (m=0,…,M-1).

[0124] Each filter hpqkm (m=0, ..., M-1) has the same number of channels K as the input, and its size is, for example, H×H×K. Figure 7 In the M filters from m = 0 to M hpqkm The calculation is performed in parallel on (m=0, ..., M-1), and the output u is obtained for each channel ijk .

[0125] After completing this convolution operation, the results are added across all channels for each variable. This addition can be represented by the following equation (1). Note that in equation (2), b ijm is the bias and may be common to all units for each filter.

[0126]

[0127] Then, the activation function is applied to the output u obtained as described above ijm. Therefore, the value expressed by the following equation (2) is the final output and is propagated to the next layer. Note that in the next layer, the size of the input changes from W×W×K to W×W×M.

[0128] z ijm =f(u ijm ) (2)

[0129] 1.8 Application of this embodiment

[0130] Next, a case where the embodiment is applied to the above-mentioned CNN will be described. Figure 8 This is a diagram for explaining a case where the present embodiment is applied to a convolutional layer which is the first layer of a CNN.

[0131] like Figure 8 As shown, in the case where the present embodiment is applied to the first layer of CNN, frame data 50 including frame data 50R, 50G and 50B of color components for the three primary colors of RGB is provided to the input layer. In this case, Figure 7 The number K of channels in is three of R, G, and B. Furthermore, the size of each of the frame data 50R, 50G, and 50B corresponds to W×W.

[0132] In the present embodiment, the frame data 50 provided to the input layer may be, for example, an image of incident light incident on the photodiode PD arranged in the pixel array unit 101 of the image sensor 100. hpqkm (m=0,…,M=1) The corresponding filter 51 can be the convolution filter array 122.

[0133] According to this convolution operation, M number of feature maps 54 of the convolution filter 130 are obtained as output u ijm The feature map 54 is input to an external data processing unit or a data processing device, such as the signal processing circuit 108, the signal processing unit 13, the DSP 14, the application processor 20, or the cloud server 30, for example, and CNN is performed from the pooling layer of the second layer.

[0134] Note that the data input to the input layer is not limited to one page of frame data 50, and may be data of one or more pixels, a row, or a specific region (region of interest ROI). In this case, the optical convolution operation according to the present embodiment may be applied to another DNN, such as a recurrent neural network (RNNN) instead of a CNN.

[0135] 1.9 Convolutional Filters

[0136] For example, a diffraction grating using Talbot diffraction (also referred to as a Talbot diffraction grating) may be used to construct each convolution filter 130 of the convolution filter array 122 that performs such an optical convolution operation.

[0137] Fig. 9 is a diagram showing an example of a convolution filter according to an embodiment. Note that Fig. 9 Two unit pixels 110A and 110B used in pair are shown.

[0138] like Fig. 9 As shown, the convolution filter 130A provided in one unit pixel 110A includes a diffraction grating 131A arranged in an upper stage (upstream side of the path of incident light) and a diffraction grating 132A arranged in a lower stage (downstream side of the path of incident light).

[0139] The diffraction gratings 131A and 132A may have, for example, the same phase, the same pitch, and the same direction. Note that the direction may be, for example, a direction tilted relative to the row direction of the unit pixels 110 arranged in a matrix on the arrangement surface of the unit pixels 110 in the pixel array unit 101 (the light receiving surface of the photodiode PD).

[0140] Furthermore, the convolution filter 130B provided in another unit pixel 110B similarly includes a diffraction grating 131B arranged in an upper stage (upstream side of the path of incident light) and a diffraction grating 132B arranged in a lower stage (downstream side of the path of incident light).

[0141] Diffraction gratings 131B and 132B may have, for example, the same pitch and the same direction. In addition, diffraction gratings 131A and 132A and diffraction gratings 131B and 132B may have the same pitch and the same direction. However, the phases of diffraction gratings 131B and 132B are shifted by 180°.

[0142] In addition, a light shielding material such as tungsten (W) may be used as the material of the diffraction gratings 131A, 132A, 131B, and 132B. However, it is not limited thereto, and various reflective materials and light shielding materials may be used.

[0143] 1.10 Example of the function of convolution filter array

[0144] In this manner, by arranging the convolution filter 130A or 130B in which the diffraction gratings 131A and 132A or the diffraction gratings 131B and 132B having the same pitch and the same direction are arranged one above the other at a predetermined interval on the light receiving surface of the photodiode PD-A or PD-B, the image of the diffraction gratings 131A and 132A or the diffraction gratings 131B and 132B can be transferred to the light receiving surface of the photodiode PD-A or PD-B. That is, by using the Talbot diffraction grating, the convolution filters 130A and 130B that selectively transmit the edge component in a predetermined direction in each incident image can be configured.

[0145] At this time, the image formed on the light receiving surface of the photodiode PD-A or PD-B is affected by the light density of the incident image. Therefore, a component of the incident image having the same direction as the diffraction gratings 131A and 132A or the diffraction gratings 131B and 132B and having the same period (hereinafter, also referred to as frequency) as the diffraction gratings 131A and 132A or the diffraction gratings 131B and 132B (hereinafter, referred to as an edge component) is imaged on the light receiving surface of the photodiode PD-A or PD-B.

[0146] So, for example, Fig.10 As shown, in the case of a convolution filter array 122 in which a unit pattern (hereinafter referred to as a convolution filter unit) 133 is repeated using four convolution filters 130-0, 130-45, 130-90 and 130-135 that differ in direction by 45° and are arranged in a 2×2 matrix, edge components inclined at 0° (i.e., parallel to the row direction), edge components inclined at 45°, edge components inclined at 90°, and edge components inclined at 135° can be acquired as binary data.

[0147] The convolution filter array 122 including the convolution filter 130 having such characteristics can perform a function similar to that of a Gabor filter. That is, in the present embodiment, the Gabor filter is physically realized using the convolution filters 130A and 130B which employ Talbot diffraction.

[0148] Then, by arranging the convolution filter array 122 used as a Gabor filter relative to the incident image, for example, the result (binary data) of the optical convolution operation using the Gabor filter can be directly acquired. Therefore, for example, since the convolution layer of the first layer in the CNN can be omitted and processing can be performed from the pooling layer of the second layer, a higher-speed image recognition process can be performed.

[0149] Note that by making the phase of one convolution filter 130A in phase and the phase of the other convolution filter 130B inverted, and performing subtraction between the pixel values ​​obtained from the corresponding unit pixels 110A and 110B, a direct current (DC) component (also referred to as a constant component) can be removed from the pixel value (binary data) obtained as a result of the optical convolution operation. However, it is not necessary to remove the DC component from the edge component.

[0150] Furthermore, even in the case of removing the DC component, the unit pixel 110A provided with the convolution filter 130A and the unit pixel 110B provided with the convolution filter 130B are not necessarily adjacent to each other.

[0151] 1.11 Relationship between the pattern of the convolution filter array and the spectrum

[0152] Here, the relationship between the pattern of the convolution filter array 122 and the frequency spectrum according to the present embodiment will be described with reference to the drawings.

[0153] Fig.11 is a diagram illustrating an example of a frequency spectrum of an edge component acquired by a convolution filter array according to an embodiment. Fig.12 It is to show that the composition can obtain Fig.11 A schematic diagram of an example of a convolution filter unit of a convolution filter array showing edge components of a frequency spectrum.

[0154] exist Fig.11 In , the horizontal axis represents the frequency fx in the row direction, and the vertical axis represents the frequency fy in the column direction. Fig.11 In the example shown, a total of 25 types of different edge components #1 to #25 are acquired.

[0155] exist Fig.11 and Fig.12 , reference numbers #1 to #25 correspond to each other. Therefore, in order to obtain Fig.11 The spectrum shown, Fig.12 As shown, the convolution filter array 122 includes 25 different convolution filters 130 (#1 to #25).

[0156] exist Fig.11 In the example, the edge component #13 may be a DC component. In this case, the edge component #13 does not include direction information and frequency information. Fig.12 As shown, the convolution filter 130 (#13) for acquiring such edge component #13 includes a null area that does not include a diffraction grating.

[0157] exist Fig.11, edge components #7, #8, #9, #12, #14, #17, #18, and #19 may be, for example, low-frequency edge components. In this case, the pitch of the diffraction grating constituting the convolution filter 130 (#7, #8, #9, #12, #14, #17, #18, and #19) for acquiring edge components #7, #8, #9, #12, #14, #17, #18, and #19 is widely set, such as Fig.12 In this specification, this distance is referred to as the first distance.

[0158] exist Fig.11 , the edge components #1, #3, #5, #11, #15, #21, #23, and #25 may be, for example, high-frequency edge components narrower than the first pitch. In this case, the pitch of the diffraction grating constituting the convolution filter 130 (#1, #3, #5, #11, #15, #21, #23, and #25) for acquiring the edge components #1, #3, #5, #11, #15, #21, #23, and #25) is set to a second pitch narrower than the first pitch, such as Fig.12 shown.

[0159] exist Fig.11 , the edge components #2, #4, #6, #10, #16, #20, #22, and #24 may be, for example, edge components of intermediate frequencies between the first pitch and the second pitch. In this case, the pitch of the diffraction grating constituting the convolution filter 130 (#2, #4, #6, #10, #16, #20, #22, and #24) for acquiring the edge components #2, #4, #6, #10, #16, #20, #22, and #24 is set to a third pitch that is an intermediate pitch between the first pitch and the second pitch, as shown in FIG. Fig.12 shown.

[0160] In addition, Fig.11 In the example, edge components #3, #8, #18, and #23 may be edge components parallel to the row direction (inclination angle θ = 0°). Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#3, #8, #18, and #23) for acquiring edge components #3, #8, #18, and #23 with respect to the row direction may be 0°.

[0161] exist Fig.11 In the example, edge components #11, #12, #14, and #15 may be edge components perpendicular to the row direction (inclination angle θ = 90°). Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#11, #12, #14, and #15) for acquiring edge components #11, #12, #14, and #15 relative to the row direction may be 90°.

[0162] exist Fig.11 In FIG. 5 , edge components #5, #9, #17, and #21 may be edge components inclined at 45° relative to the row direction. In this case, Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#5, #9, #17 and #21) for acquiring edge components #5, #9, #17 and #21 may be 45° with respect to the row direction.

[0163] exist Fig.11 In FIG. 1 , edge components #1, #7, #19, and #25 may be edge components inclined 135° relative to the row direction. In this case, Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#1, #7, #19 and #25) for acquiring edge components #1, #7, #19 and #25 relative to the row direction may be 135°.

[0164] exist Fig.11 In FIG. 1 , edge components #10 and #16 may be edge components inclined 22.5° relative to the row direction. In this case, Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#10 and #16) for acquiring the edge components #10 and #16 with respect to the row direction may be 22.5°.

[0165] exist Fig.11 In FIG. 4 , edge components #4 and #22 may be edge components inclined 67.5° relative to the row direction. In this case, Fig.12 As shown, the tilt angle θ of the diffraction grating constituting the convolution filter 130 (#4 and #22) for acquiring the edge components #4 and #22 with respect to the row direction may be 67.5°.

[0166] exist Fig.11 In FIG. 1 , edge components #2 and #24 may be edge components inclined 112.5° relative to the row direction. In this case, Fig.12 As shown, the inclination angle θ of the diffraction grating constituting the convolution filter 130 (#2 and #24) for acquiring the edge components #2 and #24 with respect to the row direction may be 112.5°.

[0167] exist Fig.11 In FIG. 1 , edge components #6 and #20 may be edge components inclined 157.5° relative to the row direction. In this case, Fig.12 As shown, the tilt angle θ of the diffraction grating constituting the convolution filter 130 (#6 and #20) for acquiring the edge components #6 and #20 with respect to the row direction may be 157.5°.

[0168] As described above, in this embodiment, the convolution filter array 122 is configured using a plurality of types of convolution filters 130 having different pitches and directions of diffraction gratings. This makes it possible to acquire binary data of a plurality of types of edge components having different directions and frequencies in one imaging.

[0169] Note that Fig.11 In the spectrum shown, edge components #14 to #25 substantially overlap with edge components #1 to #12. Therefore, it is also possible to configure not to obtain edge components #14 to #25 or edge components #1 to #12. In this case, the Fig.12 Convolution filters 130 #14 to #25 or #1 to #12 in the convolution filter array 122 are shown.

[0170] Alternatively, in order to obtain more types of edge components, the spacing and / or direction of the diffraction grating of each convolution filter 130 constituting #14 to #25 may be different from the spacing and / or direction of the diffraction grating of each convolution filter 130 constituting #1 to #12. For example, such a convolution filter array 122 may be implemented, for example, by being configured to be not point-symmetrical with respect to the center of the empty convolution filter 130 (#13) for obtaining the DC component located at the center.

[0171] Alternatively, by configuring the diffraction grating constituting the convolution filter 130 with a controllable optical element such as liquid crystal, a convolution filter array 122 including convolution filters 130 having dynamically changeable pitches and orientations may be configured.

[0172] 1.12 Configuration example of combined filter

[0173] In this embodiment, by combining the above-mentioned convolution filter array 122 and color filter array 123, edge components corresponding to the number of types of convolution filters 130 are obtained for each color component of the three primary colors of RGB. Note that in the following description, it is assumed that the convolution filters 130 and the photodiodes PD are associated one to one.

[0174] Fig.13 : is a plan view showing a schematic configuration example of a combined filter combining a convolution filter array and a color filter array according to the embodiment. Fig.14 is a diagram illustrating an example of frame data generated by an image sensor according to an embodiment.

[0175] like Fig.13 As shown, the color filter array 123 has, for example, a configuration in which color filter units 152 in a Bayer array including four color filters 151R, 151G, and 151B are arranged in a matrix.

[0176] Each convolution filter 130 constituting the convolution filter array 122 is arranged one-to-one with respect to each color filter unit 152 of the color filter array 123. Therefore, in a unit pattern of a combined filter (hereinafter, referred to as a combined filter unit) 154, a total of 25 color filter units 152 are combined with a convolution filter unit 133 including a total of 25 convolution filters 130 #1 to #25.

[0177] According to this configuration, Fig.14 As shown, in the image sensor 100 , frame data 50R, 50G, and 50B including 25 types of edge components are generated for each color component of the three primary colors of RGB.

[0178] Note that Fig.14 , K is the number of channels, and in this description, K is the number of color components of the three primary colors of RGB, that is, "3". In addition, W is the number of pixels of the frame data 50R, 50G, and 50B generated by the image sensor 100 for each color component. For example, in the case where the color filter array 123 of the image sensor 100 includes a total of 2500 50×50 color filter units 152, W is "50". Note that in Fig.14 , each of the frame data 50R, 50G, and 50B is a W×W rectangle, but the rectangle is not essential.

[0179] 1.12.1 Modification of combined filter

[0180] also, Fig.13 The case where one color filter unit 152 is combined with one convolution filter 130 is exemplified, but the configuration of the combined filter is not limited to such a configuration. Fig.15 is a plan view showing a schematic configuration example of a combined filter according to a modification of the embodiment.

[0181] like Fig.15 As shown, the combined filter according to the present modification has a configuration in which a convolution filter unit 133 is combined with a color filter 150. Note that Fig.15 An enlarged view of a G component combined filter unit 155G formed by combining the convolution filter unit 133 with the G component color filter 150G is shown, however, the convolution filter unit 133 is similarly combined with other R component color filters 150 and B component color filters 150 to form an R component combined filter unit 155R and a B component combined filter unit 155B.

[0182] Even with this configuration, Fig.13Similar to the combined filter shown in FIG. 1 , frame data 50R, 50G, 50B including 25 types of edge components can also be obtained for each color component of the three primary colors of RGB (see FIG. 1 ). Fig.14 ).

[0183] 1.13 Overview of Convolution Operation (Without Color Filter)

[0184] Next, an overview of the convolution operation will be described. Fig.16 1 is a diagram for explaining an outline of an optical convolution operation according to an embodiment. Note that a convolution operation in a case where the image sensor 100 does not include the color filter array 123 will be described below.

[0185] like Fig.16 As shown, in the case where the image sensor 100 does not include the color filter array 123, according to the arrangement of the convolution filter 130 in the convolution filter array 122, the charge corresponding to the edge component is accumulated in each photodiode PD in the pixel array unit 101 of the image sensor 100. Therefore, in the case where reading is performed on all the unit pixels 110 of the pixel array unit 101, the frame data 950 is read, in which the edge component is arranged according to the arrangement of the convolution filter 130.

[0186] Therefore, in the present embodiment, for example, reading is performed for each type (direction and frequency) of the convolution filter 130 with respect to the pixel array unit 101. For example, in the case of 25 types of convolution filters 130 from #1 to #25 in total, reading is performed 25 times in total in order from #1. Through such a reading operation, the feature maps 954-1 to 954-25 (binary data) for each type of convolution filter 130 can be read as a result of the optical convolution operation.

[0187] Note that the convolution from the frame data 950 to the feature maps 954-1 to 954-25 is not limited to the above-mentioned read control and can be performed, for example, by an external data processing unit or data processing device such as the signal processing circuit 108, the signal processing unit 13 or the DSP 14.

[0188] 1.14 Overview of Convolution Operation (with Color Filter)

[0189] Next, an outline of a convolution operation in the case where the color filter array 123 is provided will be described. Fig.17 is a diagram for explaining an overview of an optical convolution operation (with a color filter) according to an embodiment.

[0190] like Fig.17As shown, in the case of combining the convolution filter array 122 and the color filter array 123, for each color component of the RGB three primary colors in each photodiode PD of the pixel array unit 101, charges corresponding to the number of types of edge components of the convolution filter 130 are accumulated.

[0191] Therefore, in the present embodiment, for example, reading of each type of convolution filter 130 is performed for each of the three primary colors of RGB with respect to the pixel array unit 101. For example, in the case where there are a total of 25 types of convolution filters 130 #1 to #25 and there are three types of color filters 150 of the three primary colors of RGB, first, for the unit pixel 110 including the color filter 150 that selectively transmits the R component, a total of 25 readings are performed in order starting from the unit pixel 110 including the #1 convolution filter 130; then, for the unit pixel 110 including the color filter 150 that selectively transmits the G component, a total of 25 readings are performed in order starting from the unit pixel 110 including the #1 convolution filter 130; and finally, for the unit pixel 110 including the color filter 150 that selectively transmits the B component, a total of 25 readings are performed in order starting from the unit pixel 110 including the #1 convolution filter 130. Note that the reading order for each RGB component and the reading order for the unit pixels 110 including the convolution filters 130 of #1 to #25 are merely examples.

[0192] Through such a reading operation, the characteristic maps 54 - 1 to 54 - 25 (binary data) for each type of convolution filter 130 can be read as the result of the optical convolution operation for each color component of the RGB three primary colors.

[0193] 1.14.1 Modification of convolution operation

[0194] Note that for each color component, the convolution from the frame data 50 to the feature maps 54-1 to 54-25 is not limited to the method of directly reading the feature maps 54-1 to 54-25 from the pixel array unit 101 as described above, and can also be performed by, for example, the signal processing circuit 108, the signal processing unit 13, the DSP 14, etc. At this time, an external data processing unit or a data processing device (such as the signal processing circuit 108, the signal processing unit 13, or the DSP 14) can perform a demosaicing process on the frame data 50 read from the pixel array unit 101 to generate frame data for each color component.

[0195] Figures 18 to 21 1 is a diagram for explaining an outline of a convolution operation according to a modification of the embodiment. Note that in the following description, a case where the signal processing circuit 108 performs a demosaic process will be described as an example.

[0196] First, if Fig.18 As shown, the signal processing circuit 108 extracts the pixel signal read from the unit pixel 110 including the color filter 150 that selectively transmits the wavelength of the R component from the frame data 50 read from the pixel array unit 101, thereby generating frame data 50R including the pixel signal of the R component. Note that, through this demosaicing process, the resolution of the frame data 50R is reduced to the resolution in each of the color filter units 152.

[0197] Then, if Fig.19 As shown, the signal processing circuit 108 extracts the pixel signal read from the unit pixel 110 including the color filter 150 that selectively transmits the wavelength of the G component from the frame data 50 read from the pixel array unit 101, thereby generating frame data 50G including the pixel signal of the G component. Note that, by this demosaicing process, the resolution of the frame data 50G is also reduced to the resolution in each of the color filter units 152.

[0198] Then, if Fig. 20 As shown, the signal processing circuit 108 extracts the pixel signal read from the unit pixel 110 including the color filter 150 that selectively transmits the wavelength of the B component from the frame data 50 read from the pixel array unit 101, thereby generating frame data 50B including the pixel signal of the B component. Note that, by this demosaicing process, the resolution of the frame data 50B is also reduced to the resolution in each of the color filter units 152.

[0199] In this way, when the frame data 50R, 50G, and 50B of each color component of the RGB three primary colors are generated, the signal processing circuit 108 then generates the feature maps 54-1 to 54-25 (binary data) of each type of convolution filter 130 by performing summation on the pixel signals read from the unit pixels 110 included in the same color filter unit 152 in the corresponding frame data 50R, 50G, and 50B of each type of convolution filter 130 as a result of the optical convolution operation, as shown in FIG. Fig.21 shown.

[0200] 1.15 Operation and Effect

[0201] As described above, according to the present embodiment, the convolution operation can be performed using the convolution filter array 122 as a physical configuration. Therefore, for example, since the convolution layer of the first layer in the CNN can be omitted and processing can be performed from the pooling layer of the second layer, higher-speed image recognition processing can be performed.

[0202] In addition, in this embodiment, for example, a convolution operation can be performed on a plurality of channels corresponding to the color components of the three primary colors of RGB. In this way, by using a plurality of channels as input, image recognition processing can be performed with higher accuracy.

[0203] 2. Application to mobile objects

[0204] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile body, such as a car, an electric car, a hybrid car, a motorcycle, a bicycle, a personal mobile body, an airplane, a drone, a ship, and a robot.

[0205] Fig. 22 : is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present disclosure can be applied.

[0206] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig. 22 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as the functional configuration of the integrated control unit 12050.

[0207] The drive system control unit 12010 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 is used as a control device to control: a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine, a drive motor, etc., a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating the braking force of the vehicle, etc.

[0208] The body system control unit 12020 controls the operation of various types of equipment configured for the body according to various programs. For example, the body system control unit 12020 is used as a control device to control the following items: a keyless entry system, a smart key system, a power window device, or various lights such as headlights, reverse lights, brake lights, turn signals, fog lights, etc. In this case, the body system control unit 12020 can receive as input radio waves transmitted from a mobile device that replaces the key or signals from various switches. The body system control unit 12020 receives these input radio waves or signals to control the door lock device, power window device, lights, etc. of the vehicle.

[0209] The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to image an image of the vehicle exterior, and receives the imaged image. Based on the received image, the vehicle exterior information detection unit 12030 may perform a process of detecting an object (such as a person, vehicle, obstacle, sign, symbol, etc. on the road), or perform a process of detecting the distance to the object.

[0210] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0211] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. The in-vehicle information detection unit 12040 may be connected to a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 may calculate the driver's fatigue level or the driver's concentration level, or may distinguish whether the driver is dozing off.

[0212] The microcomputer 12051 can calculate a control target value for a driving force generation device, a steering mechanism, or a braking device based on information about the interior or exterior of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or impact buffering for the vehicle, following driving based on vehicle-to-vehicle distance, vehicle speed maintenance driving, an alarm for vehicle collision, an alarm for vehicle lane deviation, and the like.

[0213] In addition, the microcomputer 12051 can control the driving force generating device, the steering mechanism, and the braking device based on the information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, thereby performing collaborative control intended for automatic driving, etc. that is independent of the driver's operation.

[0214] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information about the exterior of the vehicle obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can control the headlights to change from high beam to low beam based on the position of the preceding vehicle or oncoming vehicle detected by the exterior information detection unit 12030, thereby performing cooperative control aimed at preventing glare by controlling the headlights.

[0215] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device, which can notify information to the passengers of the vehicle or the outside of the vehicle in a visual or auditory manner. Fig. 22 In the example of FIG. 1206 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0216] Fig.23 12031 is a diagram showing an example of the installation position of the imaging unit 12031.

[0217] exist Fig.23 In the figure, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104 and 12105.

[0218] The imaging units 12101, 12102, 12103, 12104 and 12105 may be arranged at the positions of the front nose, side mirrors, rear bumper, rear door and the upper portion of the windshield inside the vehicle 12100. The imaging unit 12101 arranged at the front nose and the imaging unit 12105 arranged at the upper portion of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 arranged at the side mirrors mainly obtain images of the sides of the vehicle 12100. The imaging unit 12104 arranged at the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The imaging unit 12105 arranged at the upper portion of the windshield inside the vehicle is mainly used to detect the front vehicle, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0219] By the way, Fig.23An example of the shooting range of the imaging units 12101 to 12104 is shown. Imaging range 12111 represents the imaging range of the imaging unit 12101 arranged at the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging units 12102 and 12103 arranged at the side mirrors. Imaging range 12114 represents the imaging range of the imaging unit 12104 arranged at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 observed from above can be obtained by superimposing the image data imaged by the imaging units 12101 to 12104.

[0220] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0221] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby extract the nearest three-dimensional object as the leading vehicle, which specifically exists on the driving path of the vehicle 12100 and travels at a predetermined speed (e.g., equal to or greater than 0 km / h) in the substantially same direction as the vehicle 12100. In addition, the microcomputer 12051 can pre-set the following distance to be maintained from the leading vehicle, and perform automatic braking control (including following parking control), automatic acceleration control (including following starting control), etc. Therefore, cooperative control intended for automatic driving, etc. that does not depend on the operation of the driver, can be performed.

[0222] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, and extract the classified three-dimensional object data for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes whether the obstacles around the vehicle 12100 are obstacles that can be visually identified by the driver of the vehicle 12100, or obstacles that are difficult for the driver of the vehicle 12100 to visually identify. Then, the microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 outputs an alarm to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or avoidance steering via the drive system control unit 12010. Thus, the microcomputer 12051 can assist driving to avoid collisions.

[0223] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 is capable of identifying pedestrians by determining whether there are pedestrians in the imaging images of the imaging units 12101 to 12104. This pedestrian recognition is performed, for example, by the following programs: a program for extracting characteristic points in the imaging images of the imaging units 12101 to 12104 as infrared cameras, and a program for determining whether it is a pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there are pedestrians in the imaging images of the imaging units 12101 to 12104 and thus identifies the pedestrians, the sound / image output unit 12052 controls the display unit 12062 to display a square outline superimposed on the identified pedestrian for emphasizing the identified pedestrian. The sound / image output unit 12052 can also control the display unit 12062 to display an icon representing a pedestrian at a desired position.

[0224] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications can be made without departing from the gist of the present disclosure. In addition, constituent elements of different embodiments and modifications can be appropriately combined.

[0225] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limited, and other effects may be provided.

[0226] Furthermore, each of the above-described embodiments may be used alone or may be used in combination with another embodiment.

[0227] Note that the present technology may also have the following configurations. (1)

[0229] A light receiving device, comprising:

[0230] A plurality of first filters, each of which transmits an edge component in a predetermined direction in an incident image;

[0231] a plurality of second filters, each second filter transmitting light of a predetermined wavelength band in the incident light; and

[0232] A plurality of photoelectric conversion elements each photoelectrically converts light transmitted through one first filter among the plurality of first filters and one second filter among the plurality of second filters. (2)

[0234] The light receiving device according to (1), wherein each of the first filters includes a diffraction grating. (3)

[0236] The light receiving device according to (1) or (2), wherein each first filter is a Talbot diffraction grating. (4)

[0238] The light receiving device according to any one of (1) to (3), wherein the plurality of first filters are Gabor filters. (5)

[0240] The light receiving device according to any one of (1) to (4), wherein:

[0241] The first filter includes a third filter that transmits edge components in a first direction and a fourth filter that transmits edge components in a second direction different from the first direction,

[0242] The second filter includes at least two fifth filters that transmit light of a first wavelength band and at least two sixth filters that transmit light of a second wavelength band different from the first wavelength band,

[0243] A fifth filter of the fifth filter and a sixth filter of the sixth filter are associated with the third filter, and

[0244] Another fifth of the fifth filters and another sixth of the sixth filters are associated with the fourth filter. (6)

[0246] The light receiving device according to any one of (1) to (4), wherein:

[0247] The first filter includes at least two third filters transmitting edge components in a first direction and at least two fourth filters transmitting edge components in a second direction different from the first direction,

[0248] The second filter includes a fifth filter that transmits light of a first wavelength band and a sixth filter that transmits light of a second wavelength band different from the first wavelength band,

[0249] A third filter among the third filters and a fourth filter among the fourth filters are associated with the fifth filter, and

[0250] Another third filter of the third filters and another fourth filter of the fourth filters are associated with the sixth filter. (7)

[0252] The light receiving device according to any one of (1) to (6), wherein each first filter is associated with the photoelectric conversion element on a one-to-one basis. (8)

[0254] The light receiving device according to any one of (1) to (7), further includes an on-chip lens that condenses a part of the incident light on any one of the photoelectric conversion elements. (9)

[0256] The light receiving device according to (8), wherein the first filter is located between the photoelectric conversion element and the on-chip lens. (10)

[0258] The light receiving device according to (9), wherein the second filter is located between the photoelectric conversion element and the first filter or between the first filter and the on-chip lens. (11)

[0260] A solid-state imaging device comprising:

[0261] The light receiving device according to any one of (1) to (10); and

[0262] The pixel circuit reads a pixel signal of a voltage value corresponding to the amount of charge accumulated in each photoelectric conversion element. (12)

[0264] An electronic device, comprising:

[0265] The solid-state imaging device according to (11); and

[0266] The data processing unit performs predetermined processing on the data output from the solid-state imaging device. (13)

[0268] The electronic device according to (12), wherein the data processing unit performs machine learning processing using a learning model on the data read from the solid-state imaging device. (14)

[0270] An electronic device according to (13), wherein the data processing unit performs processing from a pooling layer of a second layer in a convolutional neural network (CNN). (15)

[0272] An information processing system, comprising:

[0273] The electronic device according to any one of (12) to (14); and

[0274] The data processing device is connected to the electronic device via a predetermined network. (16)

[0276] An information processing system according to (15), wherein the data processing device performs processing from a pooling layer of a second layer in a CNN.

[0277] Reference Mark List

[0278] 1 Electronic equipment

[0279] 10 Imaging equipment

[0280] 11 Imaging unit

[0281] 11a Optical system

[0282] 12 Control Unit

[0283] 13 Signal Processing Unit

[0284] 14 DSP

[0285] 15 Memory

[0286] 16 Output Units

[0287] 20 Application Processor

[0288] 30 Cloud Servers

[0289] 40 Network

[0290] 50, 50R, 50G, 50B, 950, 1050 frame data

[0291] 51 Filter

[0292] 54, 54-1 to 54-25, 954-1 to 954-25 characteristic diagram

[0293] 100, 100A solid-state imaging device (image sensor)

[0294] 101 pixel array unit

[0295] 102 Vertical drive circuit

[0296] 103 column processing circuit

[0297] 104 Horizontal drive circuit

[0298] 105 System control unit

[0299] 108 signal processing circuit

[0300] 109 Data storage unit

[0301] 110, 110A, 110B Unit pixel

[0302] 111 Transfer transistor

[0303] 112 Reset transistor

[0304] 113 Amplifier transistor

[0305] 114 Select transistor

[0306] 121 Semiconductor Chip

[0307] 122 convolution filter array

[0308] 123 Color Filter Array

[0309] 124 Microlens Array

[0310] 130, 130A, 130B, 130-0, 130-45, 130-90, 130-135 convolution filters

[0311] 131A, 131B, 132A, 132B Diffraction grating

[0312] 133 Convolutional filter unit

[0313] 150, 151R, 151G, 151B color filters

[0314] 152 Color filter units

[0315] 154 Combined filter units

[0316] 155R, 155G, 155B Combined Filter Units

[0317] 160 On-chip lens

[0318] LD pixel drive line

[0319] LD111 Transfer transistor driver line

[0320] LD112 reset transistor driver line

[0321] LD114 select transistor drive line

[0322] PD, PD-A, PD-B Photodiode

[0323] VSL Vertical Signal Line.

Claims

1. A light receiving device, comprising: A plurality of first filters, each of which transmits an edge component in a predetermined direction in an incident image; a plurality of second filters, each second filter transmitting light of a predetermined wavelength band in the incident light; as well as a plurality of photoelectric conversion elements, each of which performs photoelectric conversion on light transmitted through one of the plurality of first filters and one of the plurality of second filters, wherein each of the plurality of first filters comprises a third filter that transmits edge components in a first direction and a fourth filter that transmits edge components in a second direction different from the first direction, and Wherein, the plurality of first filters are Gabor filters.

2. The light receiving device according to claim 1, wherein Each of the plurality of first filters includes a diffraction grating.

3. The light receiving device according to claim 1, wherein Each of the plurality of first filters is a Talbot diffraction grating.

4. The light receiving device according to claim 1, wherein Each of the plurality of second filters includes at least two fifth filters transmitting light of a first wavelength band and at least two sixth filters transmitting light of a second wavelength band different from the first wavelength band. A fifth filter of the at least two fifth filters and a sixth filter of the at least two sixth filters are associated with the third filter, and Another fifth filter of the at least two fifth filters and another sixth filter of the at least two sixth filters are associated with the fourth filter.

5. The light receiving device according to claim 1, wherein Each of the plurality of first filters includes at least two third filters that transmit edge components in the first direction and at least two fourth filters that transmit edge components in the second direction different from the first direction. Each of the plurality of second filters includes a fifth filter that transmits light of a first wavelength band and a sixth filter that transmits light of a second wavelength band different from the first wavelength band. One of the at least two third filters and one of the at least two fourth filters are associated with the fifth filter, and Another third filter of the at least two third filters and another fourth filter of the at least two fourth filters are associated with the sixth filter.

6. The light receiving device according to claim 1, wherein Each of the plurality of first filters is associated one-to-one with each of the plurality of photoelectric conversion elements. 7 . The light receiving device according to claim 1 , further comprising an on-chip lens that focuses a portion of the incident light on any one of the plurality of photoelectric conversion elements.

8. The light receiving device according to claim 7, wherein: Each of the plurality of first filters is located between each of the plurality of photoelectric conversion elements and the on-chip lens.

9. The light receiving device according to claim 8, wherein: Each of the plurality of second filters is located between each of the plurality of photoelectric conversion elements and each of the plurality of first filters or between each of the plurality of first filters and the on-chip lens.

10. A solid-state imaging device comprising: The light receiving device according to claim 1; as well as The pixel circuit reads a pixel signal of a voltage value corresponding to the amount of charge accumulated in each photoelectric conversion element.

11. An electronic device, comprising: The solid-state imaging device according to claim 10; as well as A data processing unit performs predetermined processing on the data output from the solid-state imaging device.

12. The electronic device according to claim 11, wherein: The data processing unit performs machine learning processing using a learning model on the data read from the solid-state imaging device.

13. The electronic device according to claim 12, wherein: The data processing unit performs processing from a pooling layer of a second layer in a convolutional neural network.

14. An information processing system comprising: The electronic device according to claim 11; as well as The data processing device is connected to the electronic device via a predetermined network.

15. The information processing system according to claim 14, wherein: The data processing device performs processing from the pooling layer of the second layer in the convolutional neural network.

Images