System

Abstract

To provide a system with high processing speed and low power consumption.SOLUTION: The system includes an imaging device and an arithmetic circuit. The imaging device includes an imaging unit, a first storage unit, and an arithmetic unit, and the arithmetic circuit includes a second storage unit. The imaging unit has a function of converting light reflected by an external subject into image data, and the first storage unit has a function of storing the image data and a first filter for performing first convolutional processing in a first layer of a neural network. The arithmetic unit has a function of performing the first convolutional processing using the image data and the first filter to generate first data. The second storage unit has a function of storing the first data and a plurality of filters for performing convolutional processing in the second and subsequent layers of the neural network. The arithmetic circuit has a function of using the first data to perform the processing in the second and subsequent layers of the neural network and generating a depth map of the image data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present invention relates to a system.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. The technical field of the invention disclosed in this specification relates to an object, a driving method, or a manufacturing method. Alternatively, one embodiment of the present invention relates to a process, a machine, a manufacture, or a composition of matter. Therefore, examples of the technical field of one embodiment of the present invention disclosed in this specification more specifically include a semiconductor device, a display device, a liquid crystal display device, a light-emitting device, a power storage device, an imaging device, a storage device, a signal processing device, a processor, an electronic device, a system, a driving method thereof, a manufacturing method thereof, or an inspection method thereof. [Background technology]

[0003] Currently, the development of integrated circuits that mimic the mechanisms of the human brain is actively progressing. Such integrated circuits incorporate the mechanisms of the brain as electronic circuits, and have circuits that correspond to the "neurons" and "synapses" of the human brain. For this reason, such integrated circuits are sometimes called "neuromorphic," "brain-morphic," or "brain-inspired." Such integrated circuits have a non-von Neumann architecture, and are expected to perform parallel processing with extremely low power consumption compared to the von Neumann architecture, which consumes more power as the processing speed increases.

[0004] An information processing model that mimics a neural network with "neurons" and "synapses" is called an artificial neural network (ANN). By using an artificial neural network, it is possible to make inferences with an accuracy comparable to or even exceeding that of humans. In an artificial neural network, the main operation is the weighted sum of neuron outputs, i.e., the product-sum operation.

[0005] By the way, by using a TOF (Time Of Flight) camera or a stereo camera, it is possible to obtain an image having a distance in the depth direction (referred to as depth in this specification, etc.), that is, an image (stereoscopic image) that allows spatial understanding. Currently, there is active development of a technology (referred to as depth estimation technology in this specification, etc.) that predicts depth from an image and adds the depth to the image by using the above-mentioned artificial neural network without using a TOF camera or a stereo camera, etc. For example, Non-Patent Document 1 discloses a technology that estimates depth from an image by using two networks, a Global Coarse-Scale Network and a Local Fine-Scale Network. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] D.Eigen et al., “Depth Map Prediction from a Single Image using a Multi-Scale Deep Network”, (Submitted on 9 Jun 2014) [online], [Retrieved July 26, 2019], Internet<URL:https: / / arxiv.org / pdf / 1406.2283v1.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] In the case of a TOF camera, for example, a light source that irradiates near-infrared light must be provided, and in the case of a stereo camera, for example, two or more lenses must be provided. In other words, since TOF cameras and stereo cameras have components for acquiring depth, they may be larger in size than normal cameras.

[0008] In addition, when the calculation of the artificial neural network used for image depth estimation is performed using, for example, a calculator configured with a digital circuit, it is necessary to multiply the digital data (multiplier data) serving as the multiplier by the digital data (multiplicand data) serving as the multiplicand in a digital multiplication circuit, add the digital data (product data) obtained by the multiplication in a digital addition circuit, and obtain digital data (product-sum data) as the result of the product-sum calculation. It is preferable that the digital multiplication circuit and the digital addition circuit are capable of handling multi-bit calculations, but in that case, it is necessary to increase the circuit scale of each of the digital multiplication circuit and the digital addition circuit, which may increase the circuit area and power consumption. Furthermore, if the circuit scale increases, the processing speed of the entire calculation may also decrease.

[0009] An object of one embodiment of the present invention is to provide a system capable of performing multiply-and-accumulate operations, or to provide a system with low power consumption, or to provide a system with high processing speed.

[0010] Another object of one embodiment of the present invention is to provide a novel system. Another object of one embodiment of the present invention is to provide an operation method of the novel system.

[0011] The problems of one embodiment of the present invention are not limited to the problems listed above. The problems listed above do not preclude the existence of other problems. The other problems are problems not mentioned in this section, which will be described below. Problems not mentioned in this section can be derived by a person skilled in the art from the description in the specification or drawings, and can be appropriately extracted from these descriptions. One embodiment of the present invention solves at least one of the problems listed above and other problems. One embodiment of the present invention does not need to solve all of the problems listed above and other problems. [Means for solving the problem]

[0012] (1) One embodiment of the present invention is a system including an imaging device and an arithmetic circuit. The imaging device includes an imaging unit, a first storage unit, and an arithmetic circuit, and the arithmetic circuit includes a second storage unit. The imaging unit has a function of converting light reflected from an external subject into image data. The first storage unit has a function of storing the image data and a first filter for performing a first convolution process in a first layer of a first neural network. The arithmetic unit has a function of performing a first convolution process of the image data and the first filter to generate first data. The second storage unit has a function of storing the first data and a plurality of filters for performing a convolution process in a second layer or later of the first neural network. The arithmetic circuit has a function of performing a process in a second layer or later of the first neural network using the first data to generate a depth map in the image data.

[0013] (2) Alternatively, in one aspect of the present invention, the device according to (1) above may further include a storage device. In particular, the storage device preferably has a function of storing the first filter and the plurality of filters, a function of transmitting the first filter to the first storage unit, and a function of transmitting the plurality of filters to the second storage unit.

[0014] (3) Alternatively, one embodiment of the present invention is a system including an imaging device and an arithmetic circuit. The imaging device includes an imaging unit, a first storage unit, and an arithmetic circuit, and the arithmetic circuit includes a second storage unit. The imaging unit has a function of converting light reflected from an external subject into image data. The first storage unit has a function of storing image data, a first filter for performing a first convolution process in a first layer of a first neural network, and a second filter for performing a second convolution process in a first layer of a second neural network. The arithmetic unit has a function of performing a first convolution process on the image data and the first filter to generate first data, and a function of performing a second convolution process on the image data and the second filter to generate second data. The second storage unit has a function of storing the first data, the second data, and a plurality of filters for performing convolution processes on the second and subsequent layers of the first neural network and convolution processes on the fourth and subsequent layers of the second neural network. The arithmetic circuit has a function of using the first data to perform processing from the second layer onwards of the first neural network and outputting third data from the output layer of the first neural network, a function of performing pooling processing on the second data as processing for the second layer of the second neural network to generate fourth data, a function of combining the third data and the fourth data as processing for the third layer of the second neural network to generate fifth data, and a function of using the fifth data to perform processing from the fourth layer onwards of the second neural network and outputting a depth map in the image data from the output layer of the second neural network.

[0015] (4) Alternatively, in the above embodiment (3), the device may further include a storage device. In particular, the storage device preferably has a function of storing the first filter, the second filter, and the plurality of filters, a function of transmitting the first filter and the second filter to the first storage unit, and a function of transmitting the plurality of filters to the second storage unit.

[0016] In this specification and the like, a semiconductor device is a device that utilizes semiconductor characteristics, and refers to a circuit including a semiconductor element (transistor, diode, photodiode, etc.), a device having such a circuit, etc. Also, refers to any device that can function by utilizing semiconductor characteristics. For example, an integrated circuit, a chip including an integrated circuit, and an electronic component that houses a chip in a package are examples of semiconductor devices. Also, memory devices, display devices, light-emitting devices, lighting devices, electronic devices, etc. are themselves semiconductor devices and may have semiconductor devices.

[0017] In addition, when it is stated in this specification that X and Y are connected, the following cases are also disclosed in this specification: when X and Y are electrically connected, when X and Y are functionally connected, and when X and Y are directly connected. Therefore, it is not limited to a specific connection relationship, for example, a connection relationship shown in a figure or text, and it is also disclosed in a figure or text other than the connection relationship shown in the figure or text. X and Y are objects (for example, a device, an element, a circuit, a wiring, an electrode, a terminal, a conductive film, a layer, etc.).

[0018] As an example of a case where X and Y are electrically connected, one or more elements (e.g., a switch, a transistor, a capacitive element, an inductor, a resistive element, a diode, a display device, a light-emitting device, a load, etc.) that enable the electrical connection between X and Y can be connected between X and Y. The switch has a function of controlling on / off. In other words, the switch has a function of being in a conductive state (on state) or a non-conductive state (off state) and controls whether or not a current flows.

[0019] As an example of a case where X and Y are functionally connected, one or more circuits that enable the functional connection between X and Y (for example, logic circuits (inverters, NAND circuits, NOR circuits, etc.), signal conversion circuits (digital-analog conversion circuits, analog-digital conversion circuits, gamma correction circuits, etc.), potential level conversion circuits (power supply circuits (boosting circuits, step-down circuits, etc.), level shifter circuits that change the potential level of a signal, etc.), voltage sources, current sources, switching circuits, amplifier circuits (circuits that can increase the signal amplitude or current amount, operational amplifiers, differential amplifier circuits, source follower circuits, buffer circuits, etc.), signal generation circuits, memory circuits, control circuits, etc.) can be connected between X and Y. As an example, even if another circuit is sandwiched between X and Y, if a signal output from X is transmitted to Y, X and Y are considered to be functionally connected.

[0020] In addition, when it is explicitly stated that X and Y are electrically connected, this includes the case where X and Y are electrically connected (i.e., the case where X and Y are connected with another element or circuit between them) and the case where X and Y are directly connected (i.e., the case where X and Y are connected without having another element or circuit between them).

[0021] Also, for example, it can be expressed as "X, Y, and the source (or first terminal, etc.) and drain (or second terminal, etc.) of the transistor are electrically connected to each other, and are electrically connected in the order of X, the source (or first terminal, etc.) of the transistor, the drain (or second terminal, etc.) of the transistor, and Y." Or, it can be expressed as "The source (or first terminal, etc.) of the transistor is electrically connected to X, the drain (or second terminal, etc.) of the transistor is electrically connected to Y, and X, the source (or first terminal, etc.) of the transistor, the drain (or second terminal, etc.) of the transistor, and Y are electrically connected in this order." Or, it can be expressed as "X is electrically connected to Y via the source (or first terminal, etc.) and drain (or second terminal, etc.) of the transistor, and X, the source (or first terminal, etc.) of the transistor, the drain (or second terminal, etc.) of the transistor, and Y are provided in this connection order." By using expressions similar to these examples to specify the order of connections in a circuit configuration, the source (or first terminal, etc.) and drain (or second terminal, etc.) of a transistor can be distinguished and the technical scope can be determined. Note that these expressions are merely examples and are not limited to these expressions. Here, X and Y are objects (e.g., a device, an element, a circuit, a wiring, an electrode, a terminal, a conductive film, a layer, etc.).

[0022] In addition, even when components that are independent on a circuit diagram are shown as being electrically connected to each other, one component may have the functions of multiple components. For example, when a part of a wiring also functions as an electrode, one conductive film has both the functions of a wiring and an electrode. Therefore, the term "electrical connection" in this specification also includes such a case where one conductive film has the functions of multiple components.

[0023] In addition, in this specification, the term "resistance element" may be, for example, a circuit element or wiring having a resistance value higher than 0Ω. Therefore, in this specification, the term "resistance element" includes wiring having a resistance value, a transistor in which a current flows between the source and drain, a diode, a coil, and the like. Therefore, the term "resistance element" can be rephrased as "resistance", "load", "region having a resistance value", and the like, and conversely, the terms "resistance", "load", and "region having a resistance value" can be rephrased as "resistance element". The resistance value can be, for example, preferably 1 mΩ or more and 10 Ω or less, more preferably 5 mΩ or more and 5 Ω or less, and even more preferably 10 mΩ or more and 1 Ω or less. In addition, for example, 1 Ω or more and 1×10 9 It may be set to Ω or less.

[0024] In addition, in this specification, the term "capacitive element" may refer to, for example, a circuit element having a capacitance value higher than 0F, a region of a wiring having a capacitance value, a parasitic capacitance, a gate capacitance of a transistor, and the like. Therefore, in this specification, the term "capacitive element" refers not only to a circuit element including a pair of electrodes and a dielectric contained between the electrodes, but also to a parasitic capacitance appearing between wirings, a gate capacitance appearing between one of the source or drain of a transistor and the gate, and the like. In addition, the terms "capacitive element", "parasitic capacitance", "gate capacitance", and the like can be rephrased as "capacitance", and conversely, the term "capacitance" can be rephrased as "capacitive element", "parasitic capacitance", "gate capacitance", and the like. In addition, the term "pair of electrodes" in "capacitance" can be rephrased as "pair of conductors", "pair of conductive regions", "pair of regions", and the like. The value of the capacitance can be, for example, 0.05 fF or more and 10 pF or less. In addition, it may be, for example, 1 pF or more and 10 μF or less.

[0025] In addition, in this specification, a transistor has three terminals called a gate, a source, and a drain. The gate is a control terminal that controls the conductive state of the transistor. The two terminals that function as a source or a drain are input / output terminals of the transistor. One of the two input / output terminals becomes a source and the other becomes a drain depending on the conductivity type (n-channel type, p-channel type) of the transistor and the level of the potential applied to the three terminals of the transistor. For this reason, in this specification, the terms source and drain can be replaced. In addition, in this specification, when describing the connection relationship of a transistor, the terms "one of the source or drain" (or the first electrode or the first terminal) and "the other of the source or drain" (or the second electrode or the second terminal) are used. Note that, depending on the structure of the transistor, a backgate may be included in addition to the above-mentioned three terminals. In this case, in this specification, one of the gate or the backgate of the transistor may be referred to as the first gate, and the other of the gate or the backgate of the transistor may be referred to as the second gate. Furthermore, in the same transistor, the terms "gate" and "backgate" may be interchangeable. Furthermore, when a transistor has three or more gates, in this specification and the like, the respective gates may be referred to as a first gate, a second gate, a third gate, and so on.

[0026] In this specification and the like, a node can be referred to as a terminal, a wiring, an electrode, a conductive layer, a conductor, an impurity region, etc. depending on a circuit configuration, a device structure, etc. Also, a terminal, a wiring, etc. can be referred to as a node.

[0027] In addition, in this specification and the like, "voltage" and "potential" can be interchanged as appropriate. "Voltage" refers to a potential difference from a reference potential, and if the reference potential is the ground potential (earth potential), for example, "voltage" can be interchanged as "potential." Note that ground potential does not necessarily mean 0V. Furthermore, potential is relative, and as the reference potential changes, the potential applied to wiring, the potential applied to a circuit, etc., the potential output from a circuit, etc. also change.

[0028] In addition, in this specification and the like, the terms "high-level potential" and "low-level potential" do not mean specific potentials. For example, when two wirings are both described as "functioning as wirings that supply a high-level potential," the high-level potentials provided by both wirings do not have to be equal to each other. Similarly, when two wirings are both described as "functioning as wirings that supply a low-level potential," the low-level potentials provided by both wirings do not have to be equal to each other.

[0029] "Current" refers to the phenomenon of charge transfer (electrical conduction). For example, the statement "electrical conduction of a positively charged body is occurring" can be rephrased as "electrical conduction of a negatively charged body is occurring in the opposite direction." Therefore, in this specification, unless otherwise specified, "current" refers to the phenomenon of charge transfer (electrical conduction) accompanying the movement of carriers. The carriers referred to here include electrons, holes, anions, cations, complex ions, etc., and the carriers differ depending on the system through which the current flows (for example, semiconductors, metals, electrolytes, vacuum, etc.). In addition, the "direction of current" in wiring, etc. is the direction in which positively charged carriers move, and is described as a positive current amount. In other words, the direction in which negatively charged carriers move is the opposite direction to the direction of current, and is expressed as a negative current amount. Therefore, in this specification, etc., unless otherwise specified regarding the positive and negative (or current direction) of the current, a statement such as "current flows from element A to element B" can be rephrased as "current flows from element B to element A" etc. Furthermore, statements such as "current is input to element A" can be rephrased as "current is output from element A" or the like.

[0030] In addition, in this specification, the ordinal numbers "first," "second," and "third" are used to avoid confusion of components. Therefore, they do not limit the number of components. Furthermore, they do not limit the order of the components. For example, a component referred to as "first" in one embodiment of this specification may be a component referred to as "second" in another embodiment or in the claims. Also, for example, a component referred to as "first" in one embodiment of this specification may be omitted in another embodiment or in the claims.

[0031] In addition, in this specification, the terms indicating the arrangement, such as "above" and "below", may be used for convenience in order to explain the positional relationship between the components with reference to the drawings. Furthermore, the positional relationship between the components changes as appropriate depending on the direction in which each component is depicted. Therefore, the terms are not limited to those described in the specification, and can be rephrased appropriately depending on the situation. For example, the expression "insulator located on the upper surface of a conductor" can be rephrased as "insulator located on the lower surface of a conductor" by rotating the orientation of the drawing shown by 180 degrees.

[0032] In addition, the terms "above" and "below" do not limit the positional relationship of components to being directly above or below and in direct contact with each other. For example, the expression "electrode B on insulating layer A" does not require that electrode B be formed in direct contact with insulating layer A, and does not exclude the inclusion of other components between insulating layer A and electrode B.

[0033] In addition, in this specification and the like, the terms "film" and "layer" can be interchanged depending on the situation. For example, the term "conductive layer" may be changed to the term "conductive film". Or, for example, the term "insulating film" may be changed to the term "insulating layer". Or, depending on the situation, it is possible to replace the terms "film" and "layer" with other terms without using them. For example, the terms "conductive layer" or "conductive film" may be changed to the term "conductor". Or, for example, the terms "insulating layer" and "insulating film" may be changed to the term "insulating body".

[0034] In addition, the terms "electrode", "wiring", "terminal" and the like in this specification do not limit the functions of these components. For example, "electrode" may be used as a part of "wiring", and vice versa. Furthermore, the terms "electrode" and "wiring" include cases where a plurality of "electrodes" and "wiring" are integrally formed. Furthermore, for example, "terminal" may be used as a part of "wiring" or "electrode", and vice versa. Furthermore, the term "terminal" includes cases where a plurality of "electrodes", "wiring", "terminals", and the like are integrally formed. Therefore, for example, an "electrode" can be a part of a "wiring" or "terminal", and for example, a "terminal" can be a part of a "wiring" or "electrode". Furthermore, the terms "electrode", "wiring", "terminal" and the like may be replaced with terms such as "region" depending on the circumstances.

[0035] In addition, in this specification and the like, terms such as "wiring", "signal line", and "power line" can be interchanged with each other depending on the circumstances. For example, the term "wiring" may be changed to the term "signal line". For example, the term "wiring" may be changed to the term "power line". The opposite is also true, and terms such as "signal line" and "power line" may be changed to the term "wiring". The term "power line" may be changed to the term "signal line". The opposite is also true, and terms such as "signal line" may be changed to the term "power line". The term "potential" applied to the wiring may be changed to the term "signal" depending on the circumstances. The opposite is also true, and terms such as "signal" may be changed to the term "potential".

[0036] In this specification and the like, the impurity of a semiconductor refers to, for example, other than the main component constituting the semiconductor layer. For example, an element with a concentration of less than 0.1 atomic % is an impurity. When an impurity is contained, for example, defect level density may be formed in the semiconductor, carrier mobility may decrease, or crystallinity may decrease. When the semiconductor is an oxide semiconductor, the impurity that changes the characteristics of the semiconductor may be, for example, a Group 1 element, a Group 2 element, a Group 13 element, a Group 14 element, a Group 15 element, or a transition metal other than the main component, and in particular, for example, hydrogen (including water), lithium, sodium, silicon, boron, phosphorus, carbon, nitrogen, and the like. Specifically, when the semiconductor is a silicon layer, the impurity that changes the characteristics of the semiconductor may be, for example, a Group 1 element, a Group 2 element, a Group 13 element, or a Group 15 element (however, oxygen and hydrogen are not included).

[0037] In this specification and the like, a switch refers to a device that has a function of being in a conductive state (on state) or a non-conductive state (off state) and controlling whether or not a current flows. Alternatively, a switch refers to a device that has a function of selecting and switching a path through which a current flows. As an example, an electrical switch, a mechanical switch, or the like can be used. In other words, the switch is not limited to a specific one as long as it can control a current.

[0038] Examples of electrical switches include transistors (e.g., bipolar transistors, MOS transistors, etc.), diodes (e.g., PN diodes, PIN diodes, Schottky diodes, MIM (Metal Insulator Metal) diodes, MIS (Metal Insulator Semiconductor) diodes, diode-connected transistors, etc.), and logic circuits combining these. When a transistor is used as a switch, the "conductive state" of the transistor refers to a state in which the source electrode and drain electrode of the transistor can be considered to be electrically short-circuited. Also, the "non-conductive state" of the transistor refers to a state in which the source electrode and drain electrode of the transistor can be considered to be electrically cut off. When a transistor is operated simply as a switch, the polarity (conductivity type) of the transistor is not particularly limited.

[0039] An example of a mechanical switch is a switch that uses MEMS (microelectromechanical system) technology. The switch has an electrode that can be moved mechanically, and the movement of the electrode controls whether the switch is conductive or non-conductive.

[0040] In this specification, "parallel" refers to a state in which two straight lines are arranged at an angle of -10° or more and 10° or less. Therefore, it also includes the case of -5° or more and 5° or less. Furthermore, "substantially parallel" or "roughly parallel" refers to a state in which two straight lines are arranged at an angle of -30° or more and 30° or less. Furthermore, "perpendicular" refers to a state in which two straight lines are arranged at an angle of 80° or more and 100° or less. Therefore, it also includes the case of 85° or more and 95° or less. Furthermore, "substantially perpendicular" or "approximately perpendicular" refers to a state in which two straight lines are arranged at an angle of 60° or more and 120° or less. Effect of the Invention

[0041] According to one embodiment of the present invention, a system capable of performing a multiply-and-accumulate operation can be provided. Alternatively, according to one embodiment of the present invention, a system with low power consumption can be provided. Alternatively, according to one embodiment of the present invention, a system with high processing speed can be provided.

[0042] Alternatively, according to one aspect of the present invention, a novel system may be provided, or a method of operating the novel system may be provided.

[0043] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from the description of the specification or drawings, and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0044] [Figure 1] FIG. 1 is a block diagram showing an example of the system configuration. [Diagram 2] FIG. 2 is a flowchart showing an example of the operation of the system. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a neural network that can be computed by the system. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of a CNN. [Diagram 5] FIG. 5A is a diagram showing an example of the configuration of a pixel, and FIG. 5B is a diagram showing an example of the configuration of a filter. [Figure 6] 6A and 6B are diagrams illustrating an example of the convolution process. [Figure 7] FIG. 7 is a diagram illustrating an example of the convolution process. [Figure 8]FIG. 8 is a diagram illustrating an example of the configuration of a feature map. [Figure 9] FIG. 9A is a diagram illustrating a pixel circuit, and FIG. 9B is a timing chart illustrating an imaging operation. [Figure 10] FIG. 10A is a schematic diagram showing the configuration of a pixel of an imaging device, and FIG. 10B is a block diagram of the imaging device. [Figure 11] 11A to 11C are cross-sectional views showing the configuration of an imaging device. [Figure 12] 12A to 12C are cross-sectional views showing the configuration of an imaging device. [Figure 13] 13A to 13C are cross-sectional views showing the configuration of an imaging device. [Figure 14] 14A to 14F are perspective views of a package that houses an imaging device. [Figure 15] FIG. 15 is a perspective view illustrating an example of an electronic device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] In an artificial neural network (hereafter referred to as a neural network), the strength of synapses can be changed by providing existing information to the neural network. This process of providing existing information to a neural network and determining the strength of connections is sometimes called "learning."

[0046] In addition, by providing some information to a neural network that has undergone "learning" (with connection strengths determined), it is possible for the network to output new information based on the connection strengths. In this way, the process of outputting new information based on given information and connection strengths in a neural network is sometimes called "inference" or "cognition."

[0047] Examples of neural network models include the Hopfield type and hierarchical type. In particular, a neural network with a multi-layer structure is sometimes called a "deep neural network" (DNN), and machine learning using a deep neural network is sometimes called "deep learning."

[0048] In this specification and the like, a metal oxide is an oxide of a metal in a broad sense. Metal oxides are classified into oxide insulators, oxide conductors (including transparent oxide conductors), oxide semiconductors (also referred to as oxide semiconductors or simply OSs), and the like. For example, when a metal oxide is included in a channel formation region of a transistor, the metal oxide may be referred to as an oxide semiconductor. In other words, when a metal oxide can constitute a channel formation region of a transistor having at least one of an amplifying function, a rectifying function, and a switching function, the metal oxide can be referred to as a metal oxide semiconductor. In addition, when an OS transistor is described, it can be rephrased as a transistor having a metal oxide or an oxide semiconductor.

[0049] In this specification and the like, nitrogen-containing metal oxides may also be collectively referred to as metal oxides. Nitrogen-containing metal oxides may also be referred to as metal oxynitrides.

[0050] In this specification and the like, the configurations shown in each embodiment can be appropriately combined with the configurations shown in other embodiments to form one aspect of the present invention. In addition, when multiple configuration examples are shown in one embodiment, the configuration examples can be appropriately combined with each other.

[0051] In addition, the content (or a part of the content) described in one embodiment may be applied, combined, or replaced with at least one of another content (or a part of the content) described in that embodiment and the content (or a part of the content) described in one or more other embodiments.

[0052] In addition, the contents described in the embodiments refer to the contents described in each embodiment (or example) using various figures, or the contents described using text in the specification.

[0053] Furthermore, a figure (or a part thereof) described in one embodiment can be combined with another part of that figure, with another figure (or a part thereof) described in that embodiment, and / or with at least one figure (or a part thereof) described in one or more other embodiments to form even more figures.

[0054] The embodiments described in this specification will be described with reference to the drawings. However, it is possible to implement the embodiments in many different ways, and it is easily understood by those skilled in the art that the form and details can be changed in various ways without departing from the spirit and scope of the invention. Therefore, the present invention is not interpreted as being limited to the description of the embodiments. In the configuration of the invention of the embodiments, the same reference numerals are used in common between different drawings for the same parts or parts having similar functions, and repeated explanations may be omitted. In addition, in perspective views and the like, the description of some components may be omitted in order to ensure the clarity of the drawings.

[0055] In this specification, when the same symbol is used for multiple elements, particularly when it is necessary to distinguish between them, an identification symbol such as “_1”, "[n]”, "[m,n]”, etc. may be added to the symbol.

[0056] In addition, in the drawings of this specification, the size, layer thickness, or area may be exaggerated for clarity. Therefore, it is not necessarily limited to the scale. Note that the drawings are schematic illustrations of ideal examples, and are not limited to the shapes or values ​​shown in the drawings. For example, it is possible to include variations in signals, voltages, or currents due to noise, or variations in signals, voltages, or currents due to timing deviations.

[0057] (Embodiment 1) In this embodiment, a system according to one embodiment of the present invention and an operation method thereof will be described.

[0058] The system according to one embodiment of the present invention is a system that predicts the depth of each pixel of an input image using a neural network, and generates a depth map corresponding to the image. The system according to one embodiment of the present invention can also generate a three-dimensional image by adding depth to each pixel of the image. Note that in this specification and the like, the system according to one embodiment of the present invention may be referred to as an AI system.

[0059] The neural network may be, for example, a hierarchical neural network with a total of Z layers (Z is an integer equal to or greater than 3). Furthermore, the first layer of the neural network performs convolution processing of image data. The convolution processing will be described in detail in the second embodiment.

[0060] <Configuration example> First, a configuration example of an AI system according to one embodiment of the present invention will be described.

[0061] 1 is a block diagram showing an example of an AI system. The AI ​​system includes, as an example, an image capture device 100, an arithmetic circuit 200, a control circuit 300, and a storage device 400.

[0062] The imaging device 100 includes, for example, an imaging section 110, a processing section 120, a storage section 130, and a calculation section 140.

[0063] The arithmetic circuit 200 includes, for example, a multiplier 210, an adder 220, an activation function circuit 230, a pooling processing unit 240, and a storage unit 250.

[0064] In the imaging device 100, the imaging section 110 is electrically connected to a processing section 120. In addition, the processing section 120 is electrically connected to a storage section 130. In addition, the storage section 130 is electrically connected to a calculation section 140.

[0065] 1, the imaging device 100 is electrically connected to a storage unit 130. Note that in the imaging device 100, the calculation unit 140 may be electrically connected to the calculation circuit 200.

[0066] In addition, the control circuit 300 is electrically connected to the imaging device 100, the arithmetic circuit 200, and the storage device 400. In addition, the storage device 400 is electrically connected to the imaging device 100 and the arithmetic circuit 200.

[0067] The imaging unit 110 has a function of acquiring light 10 reflected by an external subject and generating image data. Specifically, for example, the imaging unit 110 converts the acquired light 10 into an electrical signal (e.g., current, voltage, etc.), and the electrical signal is determined according to the image data. The imaging unit 110 can be a circuit having a CCD (Charge Coupled Device) image sensor with a color filter, a black and white CCD image sensor, etc.

[0068] The processing unit 120 has a function of processing the electrical signal generated by the imaging unit 110. For example, the processing unit 120 has an amplifier for amplifying the electrical signal, a correlated double sampling circuit for reducing noise, and the like.

[0069] The storage unit 130 has a function of acquiring the electrical signal processed by the processing unit 120 and storing image data based on the electrical signal. The storage unit 130 also has a function of storing not only the image data but also parameters (e.g., the size of a filter, a filter value included in a filter, a stride, etc.) to be input to the calculation unit 140, the results of calculations performed by the calculation unit 140, etc. The storage unit 130 also has a function of reading out the stored information and transmitting the information to a desired circuit.

[0070] For example, a volatile memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM) can be used as the storage unit 130. In addition, for example, a non-volatile memory such as a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), or a phase change memory can be used as the storage unit 130.

[0071] The arithmetic unit 140 has a function of performing multiplication using multiplier data and multiplicand data, and a function of performing addition on the results of multiple multiplications. That is, the arithmetic unit 140 has a function of performing product-sum calculations. For this reason, the arithmetic unit 140 may have a multiplier, an adder, etc.

[0072] The multiplier data may be, for example, either an arbitrary parameter (e.g., a filter value included in a filter) or image data, and the multiplicand data may be, for example, either an arbitrary parameter (e.g., a filter value included in a filter value) or the other image data.

[0073] The calculation unit 140 may also have a function circuit similar to the activation function circuit 230 described later. The function circuit of the calculation unit 140 has a function of outputting a function value with, for example, a result of a product-sum operation as an input value. The function calculated by the function circuit may be, for example, a ReLU (Rectified Linear Unit) function, a sigmoid function, a tanh function, etc. In particular, the types of ReLU functions include, for example, a softplus function, a leaky ReLU function, a parametarized ReLU function, and an ELU (Exponetial Linear Unit) function. In some cases, the function calculated by the function circuit may be a Softmax function, an identity function, etc.

[0074] The arithmetic circuit 200 has a function of performing a product-sum operation using a multiplier 210 and an adder 220, similar to the arithmetic unit 140. The arithmetic circuit 200 also has a function of outputting a function value using an activation function circuit 230, with the result of the product-sum operation as an input value. Note that the function operated by the activation function circuit 230 may be, for example, a ReLU function, a sigmoid function, a tanh function, or the like, similar to the function circuit included in the above-mentioned arithmetic unit 140. In some cases, the function operated by the function circuit may be a Softmax function, an identity function, or the like.

[0075] The pooling processing unit 240 included in the arithmetic circuit 200 has a function of performing a pooling process on image data calculated and output by the arithmetic circuit 200. The pooling process may be, for example, maximum pooling, average pooling, Lp pooling, etc. The pooling process will be described in detail in the second embodiment.

[0076] Further, the memory unit 250 included in the arithmetic circuit 200 has a function of storing, for example, image data sent from the imaging device 100, data calculated by the arithmetic unit 140, any parameters to be input to the multiplier 210 (for example, a filter value included in the filter value), the result of the product-sum operation by the multiplier 210 and the adder 220, the result of the pooling process, the output result of the function by the activation function circuit 230, and the like.

[0077] The control circuit 300 has a function of controlling the imaging device 100, the arithmetic circuit 200, and the storage device 400. Specifically, the control circuit 300 generates an electrical signal according to, for example, command information, and transmits the electrical signal to the imaging device 100, the arithmetic circuit 200, or the storage device 400. Upon receiving the electrical signal, the imaging device 100, the arithmetic circuit 200, and the storage device 400 perform driving according to the command information. This allows the control circuit 300 to control the imaging device 100, the arithmetic circuit 200, and the storage device 400.

[0078] The memory device 400 has a function of storing, for example, data calculated by the arithmetic circuit 200, parameters used in the convolution process for input to the imaging device 100 or the arithmetic circuit 200 (e.g., the size of the filter, the filter value included in the filter, the stride, etc.).

[0079] <Example of operation> Next, a description will be given of a method of operation of the AI ​​system in Fig. 1. The flowchart shown in Fig. 2 shows an example of a method of operation of the AI ​​system in Fig. 1. The method of operation of the flowchart in Fig. 2 includes steps ST1 to ST8.

[0080] The flowchart in Fig. 2 shows, as an example, an operation method in which the convolution processing of the first layer of the neural network of the AI ​​system is performed by the imaging unit 110, and the calculations of the second to Zth layers of the neural network are performed by the arithmetic circuit 200. Note that in Fig. 2, the start of the operation method of the AI ​​system is marked as "START" and the end of the operation method of the AI ​​system is marked as "END."

[0081] <<Step ST1>> Step ST1 includes a step of acquiring light 10 reflected from an external subject and generating image data by the imaging unit 110. Specifically, in step ST1, for example, the acquired light 10 is converted into an electric signal (for example, a current, a voltage, etc.) as image data.

[0082] In addition, in step ST1, various processes may be performed on the converted electrical signal by the processing unit 120. Specifically, for example, in step ST1, the electrical signal may be amplified. In addition, for example, in step ST1, the electrical signal may be subjected to correlated double sampling in order to reduce noise contained in the electrical signal.

[0083] <<Step ST2>> Step ST2 includes a step of writing the image data (electrical signals) generated in step ST1 into the storage unit .

[0084] <<Step ST3>> Step ST3 includes a step of reading out a first-layer filter of the neural network of the AI ​​system from the storage device 400, and inputting the filter to the imaging device 100. Specifically, the filter is inputted, for example, to the calculation unit 140 of the imaging device 100. Note that the filter may be written once in the storage unit 130 of the imaging device 100, and then the filter may be read out from the storage unit 130 and inputted to the calculation unit 140.

[0085] <<Step ST4>> In step ST4, convolution processing of the first layer of the AI ​​system's neural network is performed.

[0086] Step ST4 includes a step of performing a convolution process on the image data generated in step ST1 using the first layer filter read out in step ST3. Specifically, step ST4 includes, for example, a step of reading out a partial area of ​​the image data from the storage unit 130 and inputting it to the calculation unit 140, and a step of reading out the first layer filter from the storage unit 130 and inputting it to the calculation unit 140. At this time, the calculation unit 140 performs a step of performing a convolution process using the input filter as multiplier data and a partial area of ​​the image data as multiplicand data. Alternatively, the convolution process may be performed using the filter input to the calculation unit 140 in step ST3 as multiplicand data and a partial area of ​​the image data as multiplier data. In addition, a value obtained by the convolution process is written to the storage unit 130 or the storage unit 250 of the calculation circuit 200.

[0087] In step ST4, after the convolution process between the partial region of the image data read from the storage unit 130 and the filter is completed, the convolution process between the partial region of the image data different from the previous time and the filter is performed. In this way, partial regions are sequentially selected from the image data acquired by the imaging unit 110, and the convolution process between the filter and each selected region is performed, whereby the calculation value of the convolution process for each region is obtained.

[0088] The calculated values ​​obtained for each region are arranged, for example, in a matrix, and the calculated values ​​arranged in a matrix form become image data obtained by convolution processing of the image data acquired by the imaging unit 110 and the filter. That is, in step ST4, image data in which only characteristic parts are extracted from the image data acquired by the imaging unit 110 by the filter (hereinafter referred to as the first feature extraction image data) is generated. In addition, the first feature extraction image data may be written to the storage unit 130 as described above, or may be written to the storage unit 250 of the arithmetic circuit 200.

[0089] Furthermore, when the calculation unit 140 includes a function circuit, the calculated values ​​obtained for each region of the image data may be input to the function circuit to calculate the function value as part of the convolution process. In that case, for example, by arranging the function values ​​in a matrix instead of the calculated values ​​obtained for each region, the function values ​​arranged in a matrix can be treated as the first feature extraction image data instead of the calculated values ​​arranged in a matrix.

[0090] Step ST4 may further include a step of performing a pooling process on the first feature extraction image data using the pooling processing unit 240. In this case, the pooling process may be a second layer process of the neural network of the AI ​​system.

[0091] In addition, the user may set multiple regions in the image data so that the regions do not overlap each other. Also, the user may set multiple regions in the image data so that they partially overlap each other. In other words, in the convolution process, parameters such as filter size, filter value, stride, etc. can be determined according to the situation.

[0092] <<Step ST5>> Step ST5 includes a step in which the arithmetic circuit 200 receives command information sent from the control circuit 300 or the like. The command information includes the content of processing for the xth layer (where x is an integer between 2 and Z) of the neural network of the AI ​​system. For example, if step ST5 is the first time and calculations up to the first layer of the neural network of the AI ​​system have been completed in step ST4, then x can be set to 2. Also, for example, if step ST5 is the first time and calculations up to the second layer of the neural network of the AI ​​system have been completed in step ST4, then x can be set to 3.

[0093] The x-th layer processing included in the command information may be, for example, the same convolution processing as in step ST4, pooling processing, arithmetic processing in the fully connected layer, etc. The arithmetic processing in the fully connected layer will be described in the second embodiment.

[0094] In particular, when performing convolution processing, it is preferable that step ST 5 includes a step of reading out a filter used in the convolution processing from the storage device 400 and writing it into the storage unit 250 of the arithmetic circuit 200 .

[0095] In addition, in step ST5, a process may be performed in which image data output in another neural network is combined with image data output in the processing of the x-1th layer to generate new image data from which features have been extracted.

[0096] <<Step ST6>> In step ST6, processing of the xth layer of the neural network of the AI ​​system is performed.

[0097] Step ST6 includes a step of performing the processing included in the command information sent to the arithmetic circuit 200 in step ST5 on the image data processed and output in the (x-1)th layer. For example, when x=2, that is, in the processing of the second layer, the first feature extraction image data generated in step ST4 is read from the storage unit 130 or 250, and one of convolution processing, pooling processing, etc. is performed on the feature extraction image data by a circuit included in the arithmetic circuit 200.

[0098] In particular, when performing convolution processing, in step ST6, the filter used in the convolution processing read from the storage unit 250 is used as multiplier data, and the first feature extraction image data read from the storage unit 130 or the storage unit 250 is used as multiplicand data, and the convolution processing is performed.

[0099] By the second layer processing, image data in which features have been further extracted from the first feature extracted image data (second feature extracted image data) can be output.

[0100] Furthermore, in step ST6, for example, when x is 3 or more, the image data processed and output in the (x-1)th layer is subjected to x-th layer processing (convolution processing by the multiplier 210, adder 220, etc., pooling processing by the pooling processing unit 240, etc.). Therefore, the x-th layer can output an image with further features extracted from the image output in the (x-1)th layer.

[0101] As described above, by performing step ST6, the image data generated at the x-1th layer of the neural network of the AI ​​system can be converted into image data from which features have been further extracted by processing at the xth layer of the neural network of the AI ​​system.

[0102] <<Step ST7>> In step ST7, it is determined whether the calculation of the hierarchical neural network has been performed up to the Z-1th layer. If the calculation of the hierarchical neural network has been performed up to the Z-1th layer, the operation proceeds to step ST8. If the calculation of the hierarchical neural network has not reached the Z-1th layer, the operation returns to step ST5 to perform calculation of the next intermediate layer. In this case, if the last step ST5 performed is the processing of the xth layer of the neural network of the AI ​​system, the next step ST5 performed can be considered as the processing of the x+1th layer of the neural network of the AI ​​system.

[0103] <<Step ST8>> Step ST8 includes a step of performing calculations for the Zth layer (sometimes called the output layer) of the neural network of the AI ​​system. The Zth layer processing of the neural network of the AI ​​system may be, for example, convolution processing, pooling processing, arithmetic processing in the fully connected layer, etc.

[0104] By performing step ST8, the image data generated at the Z-1th layer of the neural network of the AI ​​system can be converted into image data from which features have been further extracted by processing at the Zth layer of the neural network of the AI ​​system (hereinafter referred to as the final feature-extracted image data).

[0105] The final feature extracted image data may then be stored, for example, in storage device 400 . This allows, for example, when using software such as image analysis software or image editing software, the last feature extraction image data can be read from storage device 400 and used by the software.

[0106] In addition, the final feature extraction image data can be used for the next calculation of another neural network. This can be applied to, for example, the Coarse / Refined model of <Depth Estimation> described later.

[0107] By performing the operation example shown in the flowchart of Fig. 2 using the AI ​​system of Fig. 1, the calculation (convolution process) of the first layer of the hierarchical neural network can be performed by the imaging device 100, and the calculation of the second layer and subsequent layers can be performed by the arithmetic circuit 200. As a result, since there is no need to perform the calculation of the first layer of the hierarchical neural network in the arithmetic circuit 200, it may be possible to increase the processing speed of the arithmetic circuit 200. Also, the power consumption of the arithmetic circuit 200 can be reduced.

[0108] The operation method of the configuration example shown in this embodiment is not limited to steps ST1 to ST8 shown in FIG. 2. In this specification, the processing shown in the flowchart is classified by function and shown as independent steps. However, in actual processing, it is difficult to separate the processing shown in the flowchart by function, and there may be cases where one step is involved in multiple steps, or one step is involved across multiple steps. Therefore, the processing shown in the flowchart is not limited to each step described in the specification, and can be appropriately replaced depending on the situation. Specifically, the order of steps can be changed, steps can be added, and steps can be deleted depending on the situation.

[0109] <Depth estimation> Here, we will explain an example of a method for performing depth estimation on input image data using the AI ​​system of Figure 1.

[0110] Fig. 3 shows a Coarse / Refined model, which is an example of a hierarchical neural network. The Coarse / Refined model is a model consisting of two neural networks, a Global Coarse-Scale Network and a Local Fine-Scale Network, and is used for applications such as image depth estimation. Specifically, the Global Coarse-Scale Network generates a feature map from input image data, and the Local Fine-Scale Network generates a depth map from the input image data and the feature map.

[0111] In FIG. 3, the network CNT is an example of a Global Coarse-Scale Network, and the network RNT is an example of a Local Fine-Scale Network.

[0112] First, the network CNT will be described.

[0113] The network CNT is a neural network that extracts global features from an input image by performing, for example, processes CP1 to CP6 on the input image. When an input image is input to the input layer of the network CNT, the network CNT can output image data including the global features of the input image (hereinafter referred to as a feature map) from the output layer of the network CNT.

[0114] In Fig. 3, image data IPD is illustrated as an input image. The image data IPD may be, for example, image data generated by the imaging unit 110 of the AI ​​system in Fig. 1 or image data processed by the processing unit 120.

[0115] When image data IPD is input to the network CNT, the network CNT performs a process CP1 on the image data IPD. The process CP1 corresponds to the first layer process of the network CNT, and may be a convolution process, for example. FIG. 3 shows a state in which image data CD1 is output by performing the process CP1 on the image data IPD. Note that parameters such as the filter size, filter value, and stride in the convolution process may be freely determined.

[0116] In addition, the convolution process outputs image data for the same number of channels as the number of filters, but in this embodiment, all channels are collectively referred to as image data. For example, when the number of filters in the convolution process in the process CP1 is 96, image data CD1 for 96 channels is generated by performing the process CP1.

[0117] The process CP1 can be a process performed by the calculation unit 140 of the AI ​​system in Fig. 1. That is, the process CP1 corresponds to the operation performed in step ST4 in the above-mentioned operation example.

[0118] Next, the image data CD1 is subjected to a process CP2. The process CP2 corresponds to a second-layer process of the network CNT, and may be a pooling process, for example. Figure 3 shows how the image data CD1 is subjected to the process CP2 to output the image data CD2.

[0119] Moreover, the image data CD2 is subjected to the process CP3. The process CP3 corresponds to the process of the third layer of the network CNT, and can be, for example, a convolution process. The image data output by the process CP3 is further subjected to the convolution process to generate new image data. In this way, the convolution process is performed multiple times after the process CP3. FIG. 3 shows a state in which the convolution process is performed up to the process CP4 and the image data CD3 is output. Note that the parameters such as the filter size, the filter value, and the stride in the convolution process between the process CP3 and the process CP4 may be freely determined. Also, between the process CP3 and the process CP4, another process such as a pooling process may be included instead of the convolution process. Note that, between the process CP3 and the process CP4, the number of channels of the image data to be output may be increased for each process.

[0120] Next, the image data CD3 is subjected to the process CP5. The process CP5 can be, for example, a calculation process in the fully connected layer. FIG. 3 shows a state in which the image data CD3 is subjected to the process CP5 to output the image data CD4. The number of channels of the image data CD4 may be greater than the number of channels of the image data CD3.

[0121] Note that the processes CP3 to CP5 can be processes performed by the arithmetic circuit 200 of the AI ​​system in Fig. 1. In other words, the processes CP3 to CP5 correspond to the operations performed in steps ST5 to ST7 in the above-mentioned operational example.

[0122] As the final process of the network CNT, a process CP6 is performed on the image data CD4. The process CP6 can be, for example, a process in a fully connected layer. Here, the process CP6 can output, for example, one channel of image data CD5. FIG. 3 shows how the image data CD5 is output by performing the process CP6 on the image data CD4.

[0123] The image data CD5 is a feature map of the image data IPD obtained by inputting the image data IPD to the network CNT. The feature map is preferably stored in the storage device 400. By storing the feature map in the storage device 400, it is possible to perform calculations using the feature map in the network RNT described next.

[0124] The process CP6 can be a process performed by the arithmetic circuit 200 of the AI ​​system in FIG. 1, and corresponds to the operation performed in step ST8 in the above-mentioned operational example.

[0125] Next, the network RNT will be described.

[0126] The network RNT is a neural network that estimates the depth of an input image by performing processes RP1 to RP5 using an input image and a feature map of the input image. The network RNT can output an image including information on the depth of the input image (hereinafter referred to as a depth map) from the output layer of the network RNT by inputting the input image and the feature map of the input image to the network RNT.

[0127] When image data IPD is input to the network RNT, the network RNT performs a process RP1 on the image data IPD. The process RP1 corresponds to the first layer process of the network RNT, and can be, for example, a convolution process. FIG. 3 shows a state in which image data RD1 is output by performing the process RP1 on the image data IPD. Note that parameters such as the filter size, filter value, and stride in the convolution process may be freely determined by the user. Here, for example, it is assumed that image data RD1 having image data of multiple channels is output by the process RP1.

[0128] The process RP1 can be a process performed by the calculation unit 140 of the AI ​​system in Fig. 1. That is, the process RP1 corresponds to the operation performed in step ST4 in the above-mentioned operation example.

[0129] Next, the image data RD1 is subjected to a process RP2. The process RP2 corresponds to a second-layer process of the network RNT, and may be a pooling process, for example. FIG. 3 shows a state in which the image data RD1 is subjected to the process RP2 to output the image data RD2.

[0130] In the process RP3, the image data RD2 is combined with the image data CD5 (feature map) generated by the network CNT. Specifically, the channel of the image data CD5 generated in the process CP6 and the channel of the image data RD2 generated in the process RP2 are combined and the image data RD3 is output. For example, if the image data CD5 has one channel of image data and the image data RD2 has 63 channels of image data, the image data RD3 is output as 64 channels of image data in the process CP6. Therefore, in order to perform the process RP3, the condition is that the size of the image data of one channel of the image data CD5 is equal to the size of the image data of all channels of the image data RD2. The process RP3 corresponds to the third layer process of the network RNT. FIG. 3 shows the state in which the image data RD3 is output by combining the image data RD2 and the image data CD5.

[0131] For example, at this time, image data CD5, which is the feature map previously output by the network CNT, is read out from the storage device 400 and input to the arithmetic circuit 200. Then, by processing RP3, the image data RD2 generated by processing RP2 and the image data CD5 (feature map) generated by the network CNT are combined, and image data RD3 is output.

[0132] Moreover, the image data RD3 is subjected to the process RP4. The process RP4 corresponds to the process of the fourth layer of the network RNT, and may be, for example, a convolution process. The image data output by the process CP4 may be further subjected to a convolution process to generate new image data. In this way, the convolution process may be performed once or multiple times after the process RP4. FIG. 3 shows a state in which the convolution process is performed up to the process RP5 and the image data RD4 is output. Here, for example, the image data of multiple channels may be output by the processes after the process RP4 (excluding the process RP5), and the image data RD4 of one channel may be output by the process RP5. Note that the parameters such as the filter size, the filter value, and the stride in the convolution process between the process RP4 and the process RP5 may be freely determined. Also, between the process RP4 and the process RP5, another process such as a pooling process may be included instead of the convolution process.

[0133] Note that the processes RP2 to RP4 can be processes performed by the arithmetic circuit 200 of the AI ​​system in Fig. 1. That is, the processes RP2 to RP4 correspond to the operations performed in steps ST5 to ST7 in the above-mentioned operation example. Also, the process RP5 corresponds to the operation performed in step ST8 in the above-mentioned operation example.

[0134] The image data RD4 output by the process RP5 is output from the network RNT as image data OPD. The image data OPD is a depth map of the image data IPD obtained by inputting the image data IPD to the Coarse / Refined model.

[0135] As described above, calculations of the Coarse / Refined model used for depth estimation, etc. can be performed using the AI ​​system shown in Figure 1.

[0136] In addition, in the above, an example of performing calculations of the Coarse / Refined model using the AI ​​system in Fig. 1 has been described, but one aspect of the present invention is not limited to this. For example, the AI ​​system in Fig. 1 may be used to perform calculations of FCN (Fully-Convolutional Network), U-NET, GAN (Generative Adversarial Network), etc.

[0137] For example, FCN and U-NET may be composed of one neural network. Therefore, the first layer of the neural network is processed by the imaging device 100 of the AI ​​system in Fig. 1, and the second and subsequent layers of the neural network are processed by the arithmetic circuit 200, thereby obtaining a depth map for the input image.

[0138] In addition, after generating a depth map according to input image data using the AI ​​system of Fig. 1, additional processing using the depth map may be performed. For example, a three-dimensional image may be generated from the image data and the depth map using the arithmetic circuit 200 or the like.

[0139] Note that this embodiment mode can be appropriately combined with other embodiment modes described in this specification.

[0140] (Embodiment 2) In this embodiment, a convolutional neural network (hereinafter referred to as CNN) used in the AI ​​system described in the above embodiment will be described.

[0141] <Convolutional Neural Network> CNN is one of the computational models used for extracting features from images, etc. Fig. 4 shows an example of the configuration of CNN. CNN is composed of a convolutional layer CL, a pooling layer PL, and a fully connected layer FCL. Image data IPD captured by the imaging unit 110 is input to CNN, and feature extraction is performed.

[0142] The convolution layer CL has the function of performing convolution processing on image data. The convolution processing is performed by repeating product-sum calculations between a partial area of ​​the image data and the filter value of the weight filter. The convolution in the convolution layer CL extracts image features.

[0143] In the convolution process, one or more weight filters can be used. When multiple weight filters are used, it is possible to extract multiple features contained in the image data. In FIG. 4, three filters (filters fil a , filter fil b , filter fil c ) is used. The image data input to the convolution layer CL is filtered using the filter fil a ,fil b ,fil c The data D a , D b , D c is generated.

[0144] The convolved data D a , D b , D c is converted by an activation function and then output to, for example, the pooling layer PL. As an example of the activation function, ReLU (Rectified Linear Unit) or the like can be used. ReLU is a function that outputs "0" when the input value is negative, and outputs the input value as is when the input value is "0" or more. In addition, as another activation function, a sigmoid function, a tanh function, or the like can also be used.

[0145] The pooling layer PL has a function of performing pooling on the image data input from the convolution layer CL. Pooling is a process of dividing the image data into multiple regions, extracting predetermined data for each region, and arranging the data in a new matrix. Pooling can reduce the image data while retaining the features extracted by the convolution layer CL. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as the pooling method.

[0146] The CNN extracts features by, for example, the above-mentioned convolution process and pooling process. The CNN may have multiple convolution layers CL and / or multiple pooling layers PL. In FIG. 4, as an example, a layer L composed of a convolution layer CL and a pooling layer PL is provided in z layers (where z is an integer equal to or greater than 1) (layer L 1 to layer L z ), convolution processing and pooling processing are performed z times. In this case, feature extraction can be performed in each layer L, making more advanced feature extraction possible. 1 , layer L 2 , layer L z is shown, and other layers L are omitted.

[0147] The fully connected layer FCL has a function of judging an image, for example, using image data that has been subjected to convolution and pooling. The fully connected layer FCL has a configuration in which all nodes in a certain layer are connected to all nodes in the next layer. The image data output from the convolution layer CL or the pooling layer PL is a two-dimensional feature map, which is expanded into one dimension when input to the fully connected layer FCL. Then, the image data OPD obtained by inference by the fully connected layer FCL is output.

[0148] The configuration of the CNN is not limited to that shown in Fig. 4. For example, a pooling layer PL may be provided for each of a plurality of convolution layers CL. In addition, when it is desired to retain as much position information of extracted features as possible, the pooling layer PL may be omitted.

[0149] In addition, when classifying images from the output data of the fully connected layer FCL, an output layer electrically connected to the fully connected layer FCL may be provided. The output layer can output classification classes using a softmax function or the like as a likelihood function.

[0150] In addition, CNN can perform supervised learning using image data as training data and teacher data. For example, backpropagation can be used for supervised learning. CNN learning can optimize the filter values ​​of the weight filter, the weight coefficients of the fully connected layer, and the like.

[0151] <Convolution processing> Next, a specific example of the convolution process performed in the convolution layer CL will be described.

[0152] 5A shows a plurality of pixels pix arranged in a matrix of n rows and m columns (where n and m are integers equal to or greater than 1) in the imaging unit 110. The pixels pix[1,1] to pix[n,m] respectively receive image data g[1,1] to g[n,m].

[0153] Convolution is performed by a multiplication and addition operation between image data g and the filter value of a weighting filter. FIG. 5B shows a filter fil with t rows and s columns (where t is an integer between 1 and n, and s is an integer between 1 and m). a Filter fil a Each address in the a [1,1] to filter value f a [t,s] is assigned.

[0154] When feature extraction is performed by convolution, the filter value f a [1,1] to filter value f aData indicating a specific feature (called feature data) can be stored as [t,s]. When extracting features, the feature data is compared with the image data. When performing image processing such as edge processing or blurring by convolution, the filter value f a [1,1] to filter value f a Parameters required for image processing can be stored as [t,s]. As an example, the operation for feature extraction will be described below in detail.

[0155] FIG. 6A shows a pixel region P[1,1] having corners of pixels pix[1,1], pix[1,s], pix[t,1], and pix[t,s], where a filter fil a By filtering the data D a 6B, the filter process obtains data included in one pixel pix in the pixel area P[1,1] and a filter fil corresponding to the address of the pixel pix. a Filter value f a In other words, for all pixels pix in the pixel region P[1,1], image data g[v,w] (where v is an integer between 1 and t, and w is an integer between 1 and s) and filter value f a A multiplication and accumulation operation is performed using [v,w]. Data D a [1,1] can be expressed as follows:

[0156]

number

[0157] After that, the above multiplication and accumulation operation is performed on other pixel regions in sequence. Specifically, as shown in FIG. 7, a filter process is performed on a pixel region P[1,2] having corners of pixels pix[1,2], pix[1,s+1], pix[t,2], and pix[t,s+1] to obtain data D a[1,2] is obtained. After that, the pixel area P is moved by one pixel column at a time, and data D a Get the.

[0158] Then, data D is obtained from a pixel region P[1,m-s+1] having corners of pixels pix[1,m], pix[t,m-s+1], and pix[t,m]. a [1, m-s+1] is obtained. In each of the pixel regions of one row, that is, the pixel regions P[1,1] to P[1, m-s+1], data D a After the acquisition of is completed, the pixel area P is moved by one pixel row, and the data D for one row is similarly acquired. a In FIG. 7, data D a [2,1] to Data D a This shows how [2,m-s+1] is obtained.

[0159] By repeating the above operations, data D is obtained from a pixel region P[n-t+1,m-s+1] having corners of pixels pix[n-t+1,m], pix[n,m-s+1], and pix[n,m]. a Once [n-t+1,m-s+1] is obtained, the filter fil a The filtering process using is completed.

[0160] In this manner, a pixel region P in a matrix shape with t rows and s columns is selected from the pixels pix[1,1] to pix[n,m], and a filter fil is applied to the pixel region P. a A filter process using the filter fil is performed on a pixel region P having corners of pixel pix[x,y] (where x is an integer between 1 and n-t+1, and y is an integer between 1 and m-s+1), pixel pix[x,y+s-1], pixel pix[x+t-1,y], and pixel pix[x+t-1,y+s-1]. a The data D obtained by filtering using a[x,y] can be expressed as follows:

[0161]

number

[0162] As described above, for all pixel regions P of t rows and s columns that can be selected from pixels pix[1,1] to pix[n,m], the filter fil a By filtering the data D a [1,1] to D a [n-t+1,m-s+1] can be obtained. Then, the data D a [1,1] to Data D a By arranging [n-t+1,m-s+1] in a matrix according to the addresses, the feature map (or depth map in some cases) shown in FIG. 8 is obtained.

[0163] As described above, convolution processing is performed by a product-sum operation using image data and filter values, and image features are extracted.

[0164] 4, when multiple filters fil are provided in the convolution layer CL, the above convolution process is performed for each filter fil. In addition, although an example in which the pixel region P is moved by one pixel column or one pixel row at a time has been described here, the moving distance of the pixel region P can be set freely.

[0165] Note that this embodiment mode can be appropriately combined with other embodiment modes described in this specification.

[0166] (Embodiment 3) In this embodiment, an example of an imaging device of one embodiment of the present invention will be described with reference to drawings.

[0167] 9A is a diagram illustrating a pixel circuit of an imaging device. The pixel circuit includes a photoelectric conversion element 1050, a transistor 1051, a transistor 1052, a transistor 1053, and a transistor 1054, for example.

[0168] One electrode (anode) of the photoelectric conversion element 1050 is electrically connected to one of the source and drain of the transistor 1051. One electrode of the photoelectric conversion element 1050 is electrically connected to one of the source and drain of the transistor 1052. The other of the source and drain of the transistor 1051 is electrically connected to the gate of the transistor 1053. The one of the source and drain of the transistor 1053 is electrically connected to one of the source and drain of the transistor 1054. Note that a capacitor may be electrically connected to the gate of the transistor 1053.

[0169] The other electrode (cathode) of the photoelectric conversion element 1050 is electrically connected to a wiring 1072. The gate of the transistor 1051 is electrically connected to a wiring 1075. The other of the source and the drain of the transistor 1053 is electrically connected to a wiring 1079. The gate of the transistor 1052 is electrically connected to a wiring 1076. The other of the source and the drain of the transistor 1052 is electrically connected to a wiring 1073. The other of the source and the drain of the transistor 1054 is electrically connected to a wiring 1071. The gate of the transistor 1054 is electrically connected to a wiring 1078. The wiring 1072 is electrically connected to one terminal of a power supply 1056, and the other terminal of the power supply 1056 is electrically connected to a wiring 1077.

[0170] Here, the wiring 1071 functions, for example, as an output line that outputs a signal from a pixel. The wirings 1073, 1077, and 1079 function, for example, as power supply lines. Specifically, for example, the wirings 1073 and 1077 may function as low-potential power supply lines, and the wiring 1079 may function as a high-potential power supply line. The wirings 1075, 1076, and 1078 function, for example, as signal lines that control switching between a conductive state and a non-conductive state of each transistor.

[0171] It is preferable to use a photoelectric conversion element that generates an avalanche multiplication effect for the photoelectric conversion element 1050 in order to increase the light detection sensitivity at low illuminance. A relatively high potential is required to generate the avalanche multiplication effect. Here, the power supply 1056 has a function of supplying HVDD as a relatively high potential. For this reason, the potential HVDD is supplied to the other electrode of the photoelectric conversion element 1050 via the wiring 1072. The photoelectric conversion element 1050 can also be used by applying a potential that does not generate the avalanche multiplication effect. Depending on the configuration of the pixel circuit of the imaging device, it is not necessary to apply a photoelectric conversion element that generates the avalanche multiplication effect to the photoelectric conversion element 1050.

[0172] The transistor 1051 can have a function of transferring the potential of the charge accumulation unit NR, which changes according to the output of the photoelectric conversion element 1050, to the charge detection unit ND. The transistor 1052 can have a function of initializing the potentials of the charge accumulation unit NR and the charge detection unit ND. The transistor 1053 can have a function of outputting a signal according to the potential of the charge detection unit ND. The transistor 1054 can have a function of selecting a pixel from which a signal is to be read out.

[0173] When a high voltage is applied to the photoelectric conversion element 1050, a high-voltage transistor that can withstand the high voltage needs to be used as a transistor connected to the photoelectric conversion element 1050. For example, an OS transistor can be used as the high-voltage transistor. Specifically, it is preferable to use an OS transistor as the transistor 1051 and the transistor 1052.

[0174] The transistors 1051 and 1052 are preferably excellent in switching characteristics, while the transistor 1053 is preferably excellent in amplification characteristics and is therefore preferably a transistor with a high on-state current. Therefore, for example, a transistor using silicon in an active layer or active region (hereinafter, referred to as a Si transistor) is preferably used for the transistors 1053 and 1054.

[0175] By using the transistors 1051 to 1054 in the above-described configuration, an imaging device that has high light detection sensitivity at low illuminance and can output a signal with little noise can be manufactured. In addition, since the light detection sensitivity is high, the time required to capture light can be shortened, and imaging can be performed at high speed.

[0176] Note that the present invention is not limited to the above configuration, and OS transistors may be used as the transistors 1053 and 1054. Alternatively, Si transistors may be used as the transistors 1051 and 1052. In any case, the pixel circuit can perform an imaging operation.

[0177] Next, an example of the operation of the pixel will be described with reference to the timing chart of Fig. 9B. In the operation of the example described below, it is assumed that a potential of HVDD is supplied as "H" and a potential of GND is supplied as "L" to a wiring 1076 connected to the gate of a transistor 1052. It is assumed that a potential of VDD is supplied as "H" and a potential of GND is supplied as "L" to a wiring 1075 connected to the gate of a transistor 1051 and a wiring 1078 connected to the gate of a transistor 1054. It is assumed that a potential of VDD is supplied to a wiring 1079 connected to the source of a transistor 1053. It is also possible to adopt a configuration in which a potential other than the above is supplied to each wiring.

[0178] At time T1, the wiring 1076 is set to "H", the wiring 1075 is set to "H", and the potentials of the charge storage unit NR and the charge detection unit ND are set to the reset potential (GND) (reset operation). Note that during the reset operation, the wiring 1076 may be set to "H" to supply a potential VDD.

[0179] At time T2, the wiring 1076 is set to “L” and the wiring 1075 is set to “L”, thereby changing the potential of the charge accumulation unit NR (accumulation operation). The potential of the charge accumulation unit NR changes from GND to a maximum of HVDD depending on the intensity of light incident on the photoelectric conversion element 1050.

[0180] At time T3, the wiring 1075 is set to "H" to transfer the charge in the charge storage unit NR to the charge detection unit ND (transfer operation).

[0181] At time T4, the line 1076 is set to "L" and the line 1075 is set to "L" to terminate the transfer operation. At this point, the potential of the charge detection unit ND is determined.

[0182] During the period from time T5 to time T6, the wiring 1076 is set to "L", the wiring 1075 is set to "L", and the wiring 1078 is set to "H", and a signal corresponding to the potential of the charge detection unit ND is output to the wiring 1071. That is, an output signal corresponding to the intensity of light incident on the photoelectric conversion element 1050 in the accumulation operation can be obtained.

[0183] 10A shows an example of a pixel configuration of an imaging device having the above-described pixel circuit. The pixel can have a layer 1061, a layer 1062, and a layer 1063, and can have a configuration in which the layers overlap each other.

[0184] The layer 1061 has components of a photoelectric conversion element 1050. The photoelectric conversion element 1050 has an electrode 1065 corresponding to a pixel electrode, a photoelectric conversion portion 1066, and an electrode 1067 corresponding to a common electrode.

[0185] It is preferable to use a low-resistance metal layer for the electrode 1065. For example, metals such as aluminum, titanium, tungsten, tantalum, and silver, or a laminate of a plurality of metals selected from these, can be used.

[0186] A conductive layer having a high light-transmitting property to visible light is preferably used for the electrode 1067. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, graphene, or the like can be used. Note that the electrode 1067 may be omitted.

[0187] For example, a pn junction photodiode with a photoelectric conversion layer made of a selenium-based material can be used for the photoelectric conversion unit 1066. It is preferable that the layer 1066a is made of a selenium-based material that is a p-type semiconductor, and the layer 1066b is made of gallium oxide or the like that is an n-type semiconductor.

[0188] Photoelectric conversion elements using selenium-based materials have the characteristic of high external quantum efficiency for visible light. In such photoelectric conversion elements, avalanche multiplication effect can be utilized to form highly sensitive sensors with a large amplification of electrons relative to the amount of incident light. In addition, selenium-based materials have a high light absorption coefficient, which is advantageous in terms of production, as the photoelectric conversion layer can be fabricated as a thin film. Thin films of selenium-based materials can be formed using vacuum deposition or sputtering methods.

[0189] As the selenium-based material, crystalline selenium such as single crystal selenium or polycrystalline selenium, amorphous selenium, a compound of copper, indium, and selenium (CIS), or a compound of copper, indium, gallium, and selenium (CIGS) can be used.

[0190] The n-type semiconductor is preferably made of a material that has a wide band gap and is transparent to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or a mixture of these oxides can be used. These materials also function as a hole injection blocking layer, and can reduce dark current.

[0191] The layer 1061 is not limited to the above configuration, and may be a pn junction photodiode using one of a p-type silicon semiconductor or an n-type silicon semiconductor for the layer 1066a and the other of a p-type silicon semiconductor or an n-type silicon semiconductor for the layer 1066b. Alternatively, the layer 1061 may be a pin junction photodiode in which an i-type silicon semiconductor layer is provided between the layers 1066a and 1066b.

[0192] The pn junction photodiode or pin junction photodiode can be formed using single crystal silicon. In this case, it is preferable to obtain an electrical junction between the layer 1061 and the layer 1062 by a bonding process. The pin junction photodiode can also be formed using a thin film of amorphous silicon, microcrystalline silicon, polycrystalline silicon, or the like.

[0193] The layer 1062 may be, for example, a layer having OS transistors (transistors 1051 and 1052). In the circuit configuration of the pixel shown in FIG. 9A, when the intensity of light incident on the photoelectric conversion element 1050 is low, the potential of the charge detection unit ND is low. Since the OS transistor has an extremely low off-current, even when the gate potential is extremely low, the current according to the gate potential can be accurately output. Therefore, the range of illuminance that can be detected, that is, the dynamic range, can be expanded.

[0194] In addition, the period during which charges can be held in the charge detection unit ND and the charge accumulation unit NR can be made extremely long by the low off-state current characteristics of the transistors 1051 and 1052. Therefore, a global shutter system in which charges are accumulated simultaneously in all pixels can be applied without complicating the circuit configuration or operation method.

[0195] The layer 1063 can be a supporting substrate or a layer having Si transistors (transistors 1053 and 1054). The Si transistor can have an active region in a single crystal silicon substrate, or a crystalline silicon active layer on an insulating surface. When a single crystal silicon substrate is used for the layer 1063, a pn junction photodiode or a pin junction photodiode may be formed on the single crystal silicon substrate. In this case, the layer 1061 can be omitted.

[0196] 10B is a block diagram illustrating a circuit configuration of an imaging device according to one embodiment of the present invention. The imaging device includes a pixel array 1081 having pixels 1080 arranged in a matrix, a circuit 1082 (row driver) having a function of selecting a row of the pixel array 1081, a circuit 1083 (CDS circuit) for performing correlated double sampling on output signals of the pixels 1080, a circuit 1084 (A / D conversion circuit or the like) having a function of converting analog data output from the circuit 1083 into digital data, and a circuit 1085 (column driver) having a function of selecting and reading out the data converted by the circuit 1084. Note that a configuration without the circuit 1083 is also possible.

[0197] For example, elements of a pixel array 1081 except for a photoelectric conversion element can be provided in a layer 1062 shown in Fig. 10A. Elements of circuits 1082 to 1085 can be provided in a layer 1063. These circuits can be configured using CMOS circuits using silicon transistors.

[0198] With this structure, transistors suitable for each circuit can be used, and the area of ​​the imaging device can be reduced.

[0199] 11A to 11C are diagrams for explaining a specific configuration of the imaging device shown in FIG. 10A. FIG. 11A is a cross-sectional view showing the channel length direction of a transistor 1051, a transistor 1052, a transistor 1053, and a transistor 1054. FIG. 11B is a cross-sectional view taken along dashed line A1-A2 shown in FIG. 11A, showing a cross-section in the channel width direction of the transistor 1052. FIG. 11C is a cross-sectional view taken along dashed line B1-B2 shown in FIG. 11A, showing a cross-section in the channel width direction of the transistor 1054.

[0200] The imaging device can be a stack of layers 1061 to 1063. The layer 1061 can include a photoelectric conversion element 1050 having a selenium layer and a partition wall 1092. The partition wall 1092 is provided so as to cover a step of the electrode 1065. The selenium layer used for the photoelectric conversion element 1050 has high resistance and can be configured not to be separated between pixels.

[0201] The layer 1062 includes OS transistors 1051 and 1052. Although both the transistors 1051 and 1052 have a back gate 1091, either one of them may have a back gate, or neither may have a back gate. The back gate 1091 may be electrically connected to the front gate of a transistor disposed opposite to the back gate 1091 as shown in FIG. 11B. Alternatively, a fixed potential different from that of the front gate may be supplied to the back gate 1091.

[0202] Although a self-aligned top-gate transistor is illustrated as an example of the OS transistor in FIG. 11A, a non-self-aligned transistor may be used as the OS transistor as shown in FIG. 12A.

[0203] The layer 1063 is provided with transistors 1053 and 1054, which are Si transistors. In FIG. 11A, the Si transistor is illustrated as having a fin-type semiconductor layer provided on a silicon substrate 1200, but as shown in FIG. 12B, the Si transistor may be a planar type having an active region in a silicon substrate 1201. Alternatively, as shown in FIG. 12C, the Si transistor may be a transistor having a semiconductor layer 1210 of a silicon thin film. The semiconductor layer 1210 may be, for example, single crystal silicon (SOI (Silicon On Insulator)) formed on an insulating layer 1220 on a silicon substrate 1202. Alternatively, the semiconductor layer 1210 may be polycrystalline silicon formed on an insulating surface such as a glass substrate. In addition, the layer 1063 may be provided with a circuit for driving pixels.

[0204] An insulating layer 1093 having a function of preventing diffusion of hydrogen is provided between a region where an OS transistor is formed and a region where a Si transistor is formed. Hydrogen in the insulating layer provided near the active regions of the transistors 1053 and 1054 terminates dangling bonds of silicon. Meanwhile, hydrogen in the insulating layer provided near the oxide semiconductor layer that is the active layer of the transistors 1051 and 1052 is one of factors that generate carriers in the oxide semiconductor layer.

[0205] The insulating layer 1093 confines hydrogen in one layer, which can improve the reliability of the transistors 1053 and 1054. In addition, the insulating layer 1093 can suppress diffusion of hydrogen from one layer to the other layer, which can improve the reliability of the transistors 1051 and 1052.

[0206] The insulating layer 1093 can be formed using, for example, aluminum oxide, aluminum oxynitride, gallium oxide, gallium oxynitride, yttrium oxide, yttrium oxynitride, hafnium oxide, hafnium oxynitride, yttria-stabilized zirconia (YSZ), or the like.

[0207] 13A is a cross-sectional view showing an example in which a color filter or the like is added to an imaging device of one embodiment of the present invention. The cross-sectional view shows a part of a region having pixel circuits for three pixels. An insulating layer 1300 is formed over a layer 1061 in which a photoelectric conversion element 1050 is formed. The insulating layer 1300 can be a silicon oxide film or the like that has high light-transmitting properties in the visible light region. A silicon nitride film may be stacked as a passivation film. A dielectric film such as hafnium oxide may be stacked as an antireflection film.

[0208] A light-shielding layer 1310 may be formed on the insulating layer 1300. The light-shielding layer 1310 has a function of preventing color mixing of light passing through the color filter above. The light-shielding layer 1310 may be a metal layer such as aluminum or tungsten. In addition, the metal layer may be laminated with a dielectric film that functions as an anti-reflection film.

[0209] An organic resin layer 1320 may be provided as a planarizing film on the insulating layer 1300 and the light-shielding layer 1310. In addition, color filters 1330 (color filters 1330a, 1330b, and 1330c) are formed for each pixel. For example, a color image can be obtained by assigning colors such as R (red), G (green), B (blue), Y (yellow), C (cyan), and M (magenta) to the color filters 1330a, 1330b, and 1330c.

[0210] On the color filter 1330, an insulating layer 1360 having a light-transmitting property to visible light and the like can be provided.

[0211] 13B, an optical conversion layer 1350 may be used instead of the color filter 1330. With such a configuration, an imaging device capable of obtaining images in various wavelength regions can be obtained.

[0212] For example, an infrared imaging device can be formed by using a filter that blocks light with wavelengths equal to or shorter than visible light in the optical conversion layer 1350. A far-infrared imaging device can be formed by using a filter that blocks light with wavelengths equal to or shorter than near-infrared light in the optical conversion layer 1350. An ultraviolet imaging device can be formed by using a filter that blocks light with wavelengths equal to or longer than visible light in the optical conversion layer 1350.

[0213] Furthermore, if a scintillator is used for the optical conversion layer 1350, an imaging device can be used for an X-ray imaging device or the like to obtain an image that visualizes the intensity of radiation. When radiation such as X-rays that has passed through a subject is incident on the scintillator, it is converted into light (fluorescence) such as visible light or ultraviolet light by the photoluminescence phenomenon. Then, image data is obtained by detecting the light with the photoelectric conversion element 1050. An imaging device having this configuration may also be used for a radiation detector or the like.

[0214] Scintillators contain materials that absorb the energy of radiation such as X-rays or gamma rays and emit visible or ultraviolet light when irradiated with the radiation. For example, Gd 2 O 2 S: Tb, Gd 2 O 2 S: Pr, Gd 2 O 2 S:Eu, BaFCl:Eu, NaI, CsI, CaF 2 , BaF 2 , CeF 3 , LiF, LiI, ZnO, etc. dispersed in resin or ceramics can be used.

[0215] In addition, in the photoelectric conversion element 1050 using a selenium-based material, radiation such as X-rays can be directly converted into an electric charge, so that a scintillator may not be required.

[0216] 13C, a microlens array 1340 may be provided on the color filters 1330a, 1330b, and 1330c. Light passing through each lens of the microlens array 1340 passes through the color filter directly below and is irradiated onto the photoelectric conversion element 1050. The microlens array 1340 may be provided on the optical conversion layer 1350 shown in FIG. 13B.

[0217] An example of a package containing an image sensor chip and a camera module will be described below. The image sensor chip can have the same configuration as the imaging device described above.

[0218] 14A is a perspective view of the upper surface of a package containing an image sensor chip. The package 1400A includes a package substrate 1410 for fixing an image sensor chip 1450, a cover glass 1420, and an adhesive 1430 for bonding the two together.

[0219] 14B is a perspective view of the exterior of the bottom surface of package 1400A. The bottom surface of the package has a BGA (Ball Grid Array) configuration with solder balls as bumps 1440. Note that the package is not limited to a BGA, and may be an LGA (Land Grid Array) or a PGA (Pin Grid Array), etc.

[0220] 14C is a perspective view of the package 1400A, with a portion of the cover glass 1420 and the adhesive 1430 omitted. Electrode pads 1460 are formed on the package substrate 1410, and the electrode pads 1460 and the bumps 1440 are electrically connected via through holes. The electrode pads 1460 are electrically connected to the image sensor chip 1450 by wires 1470.

[0221] 14D is a perspective view of the appearance of the upper surface side of a camera module in which an image sensor chip is housed in a lens-integrated package. Camera module 1400B has a package substrate 1411 for fixing image sensor chip 1451, a lens cover 1421, a lens 1435, and the like. Also, IC chip 1490 having functions such as a drive circuit and a signal conversion circuit of the imaging device is provided between package substrate 1411 and image sensor chip 1451, and has a configuration as SiP (System in package).

[0222] 14E is an external perspective view of the bottom surface side of camera module 1400B. The bottom surface and side surfaces of package substrate 1411 have a QFN (Quad Flat No-lead package) configuration in which lands 1441 for mounting are provided. Note that this configuration is merely an example, and may be a QFP (Quad Flat Package) or the above-mentioned BGA, etc.

[0223] 14F is a perspective view of camera module 1400B illustrated without a portion of lens cover 1421 and lens 1435. Land 1441 is electrically connected to electrode pad 1461, and electrode pad 1461 is electrically connected to image sensor chip 1451 or IC chip 1490 by wire 1471.

[0224] By housing the image sensor chip in a package or camera module of the above-mentioned form, mounting on a printed circuit board or the like becomes easy, and the image sensor chip can be incorporated into various semiconductor devices and electronic devices.

[0225] Note that this embodiment mode can be appropriately combined with other embodiment modes described in this specification.

[0226] (Embodiment 4) In this embodiment, an example of an electronic device having the configuration of the AI ​​system described in the above embodiment will be described. FIG. 15 illustrates a state in which an imaging device including a camera module 1400 is included in each electronic device. Note that in FIG. 15, the camera module 1400B of FIG. 14D to FIG. 14F described in the third embodiment is illustrated as the camera module 1400, but the camera module 1400 may be the package 1400A of FIG. 14A to FIG. 14C described in the third embodiment instead of the camera module 1400B.

[0227] [Mobile object] As an example, the camera module 1400 can be applied to an automobile, which is a moving body, and to an imaging device that can be installed around the driver's seat of the automobile.

[0228] Fig. 15 illustrates an automobile 5700, which is an example of a moving body. The automobile 5700 is equipped with an imaging device 5710. Note that, although the imaging device 5710 is equipped on the inside of the windshield in Fig. 15, it may also be equipped on the inside of the rear window, the bonnet, the roof, a pillar, a bumper, a side sill, or the like.

[0229] An instrument panel capable of displaying a speedometer, a tachometer, a mileage, a fuel gauge, a gear status, an air conditioner setting, etc. may be provided around the driver's seat of the automobile 5700. A display device showing such information may also be provided around the driver's seat.

[0230] In particular, the display device can display an image from an imaging device 5710 provided in an automobile 5700, thereby compensating for visibility blocked by pillars and blind spots around the driver's seat, thereby improving safety.

[0231] Since the imaging device described in the above embodiment can be applied as a component of artificial intelligence, the computer can be used, for example, in an automatic driving system for an automobile 5700. The computer can also be used in a system that performs road guidance, risk prediction, etc. The display device may be configured to display information such as road guidance and risk prediction.

[0232] In the above description, an automobile is described as an example of a moving body, but the moving body is not limited to an automobile. For example, the moving body may be a train, a monorail, a ship, an aircraft (helicopter, unmanned aerial vehicle (drone), airplane, rocket), etc., and an embodiment of the present invention may be applied to these moving bodies to provide a system using artificial intelligence.

[0233] [Video camera] The camera module 1400 can be applied to a video camera, for example.

[0234] 15 illustrates a video camera 6300, which is an example of an imaging device. The video camera 6300 includes a first housing 6301, a second housing 6302, a display unit 6303, an operation key 6304, a lens 6305, a connection unit 6306, and the like. The operation key 6304 and the lens 6305 are provided in the first housing 6301, and the display unit 6303 is provided in the second housing 6302. The camera module 1400 is provided inside the lens 6305. In particular, when the camera module 1400 is the camera module 1400B described in the above embodiment, the lens 6305 can be the lens 1435 of the camera module 1400B.

[0235] Moreover, the first housing 6301 and the second housing 6302 are connected by a connection unit 6306, and the angle between the first housing 6301 and the second housing 6302 can be changed by the connection unit 6306. A configuration may be used in which the image on the display unit 6303 is switched according to the angle between the first housing 6301 and the second housing 6302 at the connection unit 6306.

[0236] By applying the camera module 1400 described in the above embodiment to the video camera 6300, it is possible to add depth to an image captured by the video camera 6300. In addition, the video camera 6300 can have a function of automatically recognizing a subject such as a face or an object, a function of adjusting the focus according to the subject, a function of toning the captured image, and the like.

[0237] [camera] The camera module 1400 can be applied to a camera, as an example.

[0238] 15 illustrates a digital camera 6240, which is an example of an imaging device. The digital camera 6240 includes a housing 6241, a shutter button 6244, a light emitter 6245, a microphone 6246, a lens 6247, and the like. Note that the camera module 1400 is provided, for example, inside the lens 6247. In particular, when the camera module 1400 is the camera module 1400B described in the above embodiment, the lens 6247 can be the lens 1435 of the camera module 1400B.

[0239] The lens 6247 may be detachable from the digital camera 6240. Alternatively, the digital camera 6240 may be configured such that the lens 6247 is integrated with a housing 6241. The digital camera 6240 may be configured such that a viewfinder or the like can be separately attached.

[0240] A low power consumption digital camera 6240 can be realized by applying the semiconductor device described in the above embodiment to the digital camera 6240. In addition, low power consumption can reduce heat generation from a circuit, and therefore influence of heat generation on the circuit itself, peripheral circuits, and modules can be reduced.

[0241] Furthermore, by applying the camera module 1400 described in the above embodiment to the digital camera 6240, it is possible to realize a digital camera 6240 having artificial intelligence. By using artificial intelligence, the digital camera 6240 can add depth to a captured image by depth estimation. In addition, the digital camera 6240 can have a function of automatically recognizing a subject such as a face or an object, a function of adjusting the focus according to the subject, a function of automatically turning on a flash according to the environment, a function of adjusting the color of a captured image, and the like.

[0242] [Surveillance Camera] The camera module 1400 can be applied to a surveillance camera, for example.

[0243] 15 illustrates a surveillance camera 6400, which is an example of an imaging device. The surveillance camera 6400 includes a housing 6451, a lens 6452, a support 6453, and the like. Note that the camera module 1400 is provided, for example, inside the lens 6452. In particular, when the camera module 1400 is the camera module 1400B described in the above embodiment, the lens 6452 can be the lens 1435 of the camera module 1400B.

[0244] Note that surveillance camera is a common name and does not limit the use. For example, a device that has the function of a surveillance camera is also called a camera or a video camera.

[0245] [Wearable devices] The camera module 1400 can be applied to a wearable terminal, for example.

[0246] 15 illustrates a wearable terminal 5900 as an example of an information terminal. The wearable terminal 5900 includes a housing 5901, a display unit 5902, operation buttons 5903, a crown 5904, a band 5905, and a camera 5910. In particular, the camera 5910 includes the camera module 1400.

[0247] By applying the camera module 1400 described in the above embodiment, the wearable terminal 5900 can perform depth estimation using artificial intelligence on captured images.

[0248] [Desktop information terminal] As an example, the camera module 1400 can be applied to an imaging device that can be installed in a desktop information terminal. Note that such an imaging device may be called a web camera.

[0249] 15 illustrates a desktop information terminal 5300. The desktop information terminal 5300 includes a main body 5301 of the information terminal, a display 5302, a keyboard 5303, and a web camera 5310. In particular, the web camera 5310 includes the camera module 1400.

[0250] The webcam 5310 can perform depth estimation using artificial intelligence on a captured image by applying the camera module 1400 described in the above embodiment, similarly to the wearable terminal 5900 described above. The image to which depth has been added can be used for various applications by the desktop information terminal 5300.

[0251] [mobile phone] The camera module 1400 can be applied to an imaging device that can be installed in a mobile phone.

[0252] 15 illustrates an information terminal 5500 which is an example of a mobile phone (smartphone). The information terminal 5500 has a housing 5510, a display unit 5511, and a camera 5512. In particular, the camera 5512 includes a camera module 1400. In addition, as an input interface, a touch panel is provided on the display unit 5511, and buttons are provided on the housing 5510.

[0253] The information terminal 5500, like the wearable terminal 5900 and webcam 5310 described above, can apply depth estimation using artificial intelligence to captured images by applying the camera module 1400 described in the above embodiment.

[0254] [Game consoles] The camera module 1400 can be applied to an imaging device that can be installed in a game console.

[0255] 15 illustrates a portable game machine 5200, which is an example of a game machine. The portable game machine 5200 includes a housing 5201, a display unit 5202, a button 5203, a camera 5210, and the like. In particular, the camera 5210 includes a camera module 1400.

[0256] 15 illustrates a portable game machine as an example of a game machine, but the camera module 1400 may be provided in a game machine of another type. Examples of game machines of another type include a home-use stationary game machine, an arcade game machine installed in an entertainment facility (game center, amusement park, etc.), and a pitching machine for batting practice installed in a sports facility. In other words, these electronic devices can also be provided with an imaging device including the camera module 1400 described in the above embodiment.

[0257] [Other electronic devices] 15 shows various electronic devices, but an imaging device including a camera module 1400 can also be provided in electronic devices not shown in FIG 15. Specifically, the imaging device can be provided in, for example, display devices such as television receivers, electronic book terminals, goggle-type displays (head-mounted displays), copiers, facsimiles, printers, printer-multifunction machines, automated teller machines (ATMs), vending machines, etc. The imaging device can also be provided in, for example, electric refrigerator-freezers, vacuum cleaners, microwave ovens, electric ovens, rice cookers, water heaters, IH (Induction Heating) cookers, water servers, air conditioners and other heating and cooling appliances, washing machines, dryers, audiovisual equipment, and other electrical appliances.

[0258] Note that this embodiment mode can be appropriately combined with other embodiment modes described in this specification. [Explanation of symbols]

[0259] ST1: step, ST2: step, ST3: step, ST4: step, ST5: step, ST6: step, ST7: step, ST8: step, CD1: image data, CD2: image data, CD3: image data, CD4: image data, CD5: image data, RD1: image data, RD2: image data, RD3: image data, RD4: image data, CP1: processing, CP2: processing, CP3: processing, CP4: processing, CP5: processing, CP6: processing, RP1: processing, RP2: processing, RP3: processing, RP4: processing, RP5: processing, CL: convolution layer, PL: pooling layer, FCL: fully connected layer, L 1 :Layer, L 2 :Layer, L z: layer, IPD: image data, OPD: image data, 10: light, 100: imaging device, 110: imaging section, 120: processing section, 130: memory section, 140: calculation section, 200: calculation circuit, 210: multiplier, 220: adder, 230: activation function circuit, 240: pooling processing section, 250: memory section, 300: control circuit, 400: storage device, 1050: photoelectric conversion element, 1051: transistor, 1052: transistor, 1053: transistor, 1054: transistor, 1056: power supply, 1061: layer, 1062: layer, 1063: layer, 1065: electrode, 1066: photoelectric Conversion unit, 1066a: layer, 1066b: layer, 1067: electrode, 1071: wiring, 1072: wiring, 1073: wiring, 1075: wiring, 1076: wiring, 1077: wiring, 1078: wiring, 1079: wiring, 1080: pixel, 1082: circuit, 1081: pixel array, 1083: circuit, 1084: circuit, 1085: circuit, 1091: back gate, 1092: partition wall, 1093: insulating layer, 1200: silicon substrate, 1201: silicon substrate, 1202: silicon substrate, 1210: semiconductor layer, 1220: insulating layer, 1300: insulating layer, 1310: light shielding layer, 1320: Organic resin layer, 1330: color filter, 1330a: color filter, 1330b: color filter, 1330c: color filter, 1340: microlens array, 1350: optical conversion layer, 1360: insulating layer, 1400: camera module, 1400A: package, 1400B: camera module, 1410: package substrate, 1411: package substrate, 1420: cover glass, 1421: lens cover, 1430: adhesive, 1435: lens, 1440: bump, 1441: land, 1450: image sensor chip, 1451: image sensor sensor chip, 1460: electrode pad, 1461: electrode pad, 1470: wire, 1471: wire, 1490: IC chip, 5200: portable game machine, 5201: housing, 5202: display unit, 5203: button, 5210: camera, 5300: desktop information terminal, 5301: main body, 5302: display, 5303: keyboard, 5310: web camera, 5500: information terminal, 5510: housing, 5511: display unit, 5512: camera, 5700: automobile, 5710: imaging device, 5900: wearable terminal, 5901: housing, 5902: display unit,5903: Operation button, 5904: Crown, 5905: Band, 5910: Camera, 6240: Digital camera, 6241: Housing, 6244: Shutter button, 6245: Light emitter, 6246: Microphone, 6247: Lens, 6300: Video camera, 6301: First housing, 6302: Second housing, 6303: Display, 6304: Operation key, 6305: Lens, 6306: Connection, 6400: Surveillance camera, 6451: Housing, 6452: Lens, 6453: Support,

Claims

[Claim 1] The imaging device includes an arithmetic circuit and a storage device. The imaging device includes an imaging unit, a first storage unit, and a calculation unit, the arithmetic circuit has a second storage unit, The imaging unit has a function of converting light reflected from an external subject into image data, the first storage unit has a function of storing the image data and a first filter for performing a first convolution process in a first layer of a first neural network; the calculation unit has a function of performing the first convolution process of the image data and the first filter to generate first data; the second storage unit has a function of storing the first data and a plurality of filters for performing convolution processing of a second layer and subsequent layers of the first neural network; the arithmetic circuit has a function of performing processing of a second layer or later of the first neural network using the first data to generate a depth map in the image data; The storage device includes: storing the first filter and the plurality of filters; transmitting the first filter to the first storage unit; and transmitting the plurality of filters to the second storage unit. system.

Images