Learning method of neural network model

Abstract

To provide a neural network model that can perform highly accurate processing on input data.SOLUTION: A neural network model includes first and second neutral networks. The first neural network includes a first layer, a second layer, and a third layer. A feature map output from the first layer is input to the second layer and the second neural network, and a feature map output from the second neural network is input to the third layer. Given that the feature map output from the first layer when first data is input to the first neural network is a correct feature map and that the feature map output from the first layer when second data obtained by adding noise to the first data is input to the first neural network is a learning feature map, the second neural network is trained so that an output feature map matches the correct feature map when the learning feature map is input.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present invention relates to a neural network model and a learning method for a neural network model. [Background technology]

[0002] Image processing techniques using AI (Artificial Intelligence) are attracting attention. For example, Patent Document 1 discloses a segmentation technique that uses a neural network to detect the type of object contained in an image and the area of ​​the object. In addition, techniques have been developed that use AI to colorize black-and-white images and estimate depth.

[0003] Also, a neural network having a layer with a skip connection has been developed. Here, the skip connection indicates a configuration in which a feature map output from a layer of a neural network is input not only to the next layer but also to other layers closer to the output layer. Non-Patent Document 1 discloses U-Net as a neural network with a skip connection. With the skip connection, U-Net makes it easier to reflect, for example, information contained in image data input to the neural network in a feature map output from a layer near the output layer. This allows for highly accurate image processing. For example, U-Net makes it easier to reflect information such as the position and contour of an object contained in image data input to the neural network in a feature map output from a layer near the output layer, so that segmentation can be performed with high accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-82979 A [Non-patent literature]

[0005] [Non-Patent Document 1] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," Medical Image Computing and Computer-Assisted Intervention, 234-241 (2015). Summary of the Invention [Problem to be solved by the invention]

[0006] In a neural network with skip-connected layers, if the data input to the neural network (input data) contains noise, the noise is likely to be transmitted to layers close to the output layer. This may result in the data not being processed with high accuracy.

[0007] An object of one embodiment of the present invention is to provide a neural network model capable of removing noise contained in input data, to provide a neural network model capable of processing input data with high accuracy, or to provide a novel neural network model.

[0008] An object of one embodiment of the present invention is to provide a neural network model learning method capable of removing noise contained in input data, to provide a neural network model learning method capable of processing input data with high accuracy, or to provide a novel neural network model learning method.

[0009] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0010] One aspect of the present invention is a neural network model having a first neural network and a second neural network, the first neural network having a layer that is skip-connected via the second neural network, and a learning method thereof. The second neural network has a function of performing noise removal on the input feature map. This makes it possible to suppress the noise from being transmitted to a layer close to the output layer of the first neural network even if the data input to the first neural network contains noise.

[0011] When training the neural network model of one embodiment of the present invention, first, the initial values ​​of the weight coefficients of the first neural network are obtained. Next, the first training is performed to obtain the weight coefficients of the second neural network. After that, the second training is performed to obtain the weight coefficients of the first neural network. After the first training and the second training are performed, the inference accuracy of the neural network model of one embodiment of the present invention is evaluated. The first training and the second training are performed until the inference accuracy becomes equal to or greater than a specified value.

[0012] One aspect of the present invention is a neural network model having a first neural network and a second neural network, the first neural network having a first layer, a second layer, and a third layer, a feature map output from the first layer being input to the second layer and the second neural network, a feature map output from the second neural network being input to the third layer, and the second neural network being trained by a first learning such that a feature map output from the first layer when first data is input to the first neural network is a correct answer feature map, a feature map output from the first layer when second data, which is the first data plus noise, is input to the first neural network is a trained feature map, and a feature map output from the second neural network when the trained feature map is input to the second neural network is identical to the correct answer feature map.

[0013] Alternatively, in the above aspect, the weighting coefficients of the second neural network may be obtained by a first learning process after inputting data for obtaining initial values ​​into the first neural network to obtain initial values ​​of the weighting coefficients of the first neural network.

[0014] Alternatively, in the above aspect, the weighting coefficients of the first neural network may be obtained by a second learning process that is performed by obtaining the weighting coefficients of the second neural network and then inputting second data into the first neural network.

[0015] Alternatively, in the above aspect, the first neural network may have a fourth layer, a fifth layer, and a sixth layer, and the feature map output from the fourth layer may be input to the fifth layer and the sixth layer that is skip-connected to the fourth layer, and the fourth layer may be closer to the output layer of the first neural network than the first layer, and the third layer may be closer to the output layer of the first neural network than the sixth layer.

[0016] Alternatively, one embodiment of the present invention includes a first neural network and a second neural network, the first neural network having a first layer, a second layer, a third layer, and a fourth layer, the fourth layer being closer to an output layer of the first neural network in the order of the third layer, the second layer, and the first layer, the feature map output from the first layer being input to the second layer and the second neural network, the feature map output from the third layer and the feature map output from the second neural network being input to the fourth layer, and the second neural network being input to the fourth layer. The neural network is a neural network model that is trained by a first learning such that when first data is input to the first neural network, a feature map output from the first layer is defined as a correct feature map, and when second data, which is the first data plus noise, is input to the first neural network, a feature map output from the first layer is defined as a trained feature map, and when the trained feature map is input to the second neural network, the feature map output from the second neural network matches the correct feature map.

[0017] Alternatively, in the above aspect, the weighting coefficients of the second neural network may be obtained by a first learning process after inputting data for obtaining initial values ​​into the first neural network to obtain initial values ​​of the weighting coefficients of the first neural network.

[0018] Alternatively, in the above aspect, the weighting coefficients of the first neural network may be obtained by a second learning process that is performed by obtaining the weighting coefficients of the second neural network and then inputting second data into the first neural network.

[0019] Alternatively, in the above aspect, the first neural network model may have a fifth layer, a sixth layer, and a seventh layer, and the feature map output from the fifth layer may be input to the sixth layer and the seventh layer that is skip-connected to the fifth layer, and the fifth layer may be closer to the output layer of the first neural network than the first layer, and the fourth layer may be closer to the output layer of the first neural network than the seventh layer.

[0020] Alternatively, one aspect of the present invention is a method for learning a neural network model, the method including a first neural network and a second neural network, the first neural network having a first layer, a second layer, and a third layer, a feature map output from the first layer being input to the second layer and the second neural network, and a feature map output from the second neural network being input to the third layer, the method including a first step of acquiring initial values ​​of weight coefficients of the first neural network by inputting data for acquiring initial values ​​to the first neural network, a feature map output from the first layer when the first data is input to the first neural network being called a correct feature map, and a feature map output from the first layer when second data, in which noise has been added to the first data, is input to the first neural network being called a correct feature map. The method for learning a neural network model includes a second step of acquiring weight coefficients for the second neural network by performing a first learning process such that a feature map output from the first layer is defined as a learning feature map and a feature map output from the second neural network matches a correct answer feature map when the learning feature map is input to the second neural network; a third step of acquiring weight coefficients for the first neural network by inputting the second data into the first neural network and performing a second learning process; and a fourth step of inputting test data into the first neural network and evaluating the inference accuracy of the neural network model based on the output data output from the first neural network, and the second to fourth steps are performed until the inference accuracy becomes equal to or greater than a specified value.

[0021] Alternatively, in the above aspect, the first neural network may have a fourth layer, a fifth layer, and a sixth layer, and the feature map output from the fourth layer may be input to the fifth layer and the sixth layer that is skip-connected to the fourth layer, and the fourth layer may be closer to the output layer of the first neural network than the first layer, and the third layer may be closer to the output layer of the first neural network than the sixth layer. Effect of the Invention

[0022] According to one aspect of the present invention, it is possible to provide a neural network model capable of removing noise contained in input data, or to provide a neural network model capable of processing input data with high accuracy, or to provide a novel neural network model.

[0023] According to one aspect of the present invention, it is possible to provide a neural network model training method capable of removing noise contained in input data, a neural network model training method capable of processing input data with high accuracy, or a novel neural network model training method.

[0024] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from the descriptions in the specification, drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0025] [Figure 1]1A and 1B are diagrams showing an example of the configuration of a neural network model. [Diagram 2] FIG. 2 is a flowchart showing an example of a method for learning a neural network model. [Diagram 3] 3A and 3B are diagrams showing an example of a learning method for a neural network model. [Figure 4] 4A, 4B1, 4B2, and 4C are diagrams showing an example of a learning method for a neural network model. [Diagram 5] FIG. 5 is a diagram illustrating an example of a learning method for a neural network model. [Figure 6] FIG. 6 is a diagram illustrating an example of a learning method for a neural network model. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a neural network model. [Figure 8] 8A and 8B are block diagrams showing examples of the configuration of a computation system. [Figure 9] FIG. 9 is a block diagram illustrating an example of the configuration of the imaging unit. [Figure 10] FIG. 10 is a diagram showing an example of the configuration of a pixel block 200 and a circuit 201. As shown in FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of a pixel configuration. [Figure 12] 12A to 12C are diagrams showing filters. [Figure 13] Fig. 13A is a diagram showing a configuration example of a pixel, and Figs. 13B to 13D are diagrams showing configuration examples of a photoelectric conversion device. [Figure 14] FIG. 14 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 15] 15A to 15C are cross-sectional views showing examples of the configuration of a transistor. [Figure 16] FIG. 16 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 17] FIG. 17 is a cross-sectional view showing an example of the configuration of an imaging device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it is easily understood by those skilled in the art that the form and details of the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention is not interpreted as being limited to the description of the embodiments shown below. In the configuration of the invention described below, the same reference numerals are used in common between different drawings for the same parts or parts having similar functions, and repeated explanations may be omitted. In addition, hatching of the same elements constituting the drawings may be omitted or changed as appropriate between different drawings.

[0027] In addition, the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and do not limit the numbers.

[0028] In addition, even if a circuit diagram shows a single element, the element may be configured as a plurality of elements as long as there is no functional problem. For example, a plurality of transistors operating as a switch may be connected in series or parallel. A capacitor may also be divided and placed in multiple positions.

[0029] In addition, one conductor may have multiple functions such as wiring, an electrode, and a terminal, and in this specification, multiple names may be used for the same element. Even if elements are shown as being directly connected to each other on a circuit diagram, the elements may actually be connected to each other via multiple conductors, and in this specification, such a configuration is also included in the category of direct connection.

[0030] (Embodiment) In this embodiment, a neural network model of one embodiment of the present invention and an example of a learning method for the neural network model will be described.

[0031] <Neural network model_1> 1A is a diagram showing an example of the configuration of a neural network model 10 according to one embodiment of the present invention. The neural network model 10 includes a neural network NN1 and a neural network NN2.

[0032] The neural network NN1 has a plurality of layers each composed of neurons, and the neurons in each layer are connected to each other. Therefore, the neural network NN1 can be said to be a hierarchical neural network. Each layer composed of neurons has a function of extracting a feature from data input to the layer, and outputting data representing the feature as a feature map. The neural network NN2 can also be a hierarchical neural network.

[0033] 1A shows layers L_1, L_2, and L_3 among the layers of the neural network NN1. Also, the flow of data between the layers is shown by arrows.

[0034] In this specification and the like, when the same reference numeral is used for multiple elements, and when it is particularly necessary to distinguish between them, a distinguishing reference numeral such as “_1”, “[1]”, etc. may be added to the reference numeral. For example, in FIG. 1A and the like, three layers L are distinguished by being described as layer L_1, layer L_2, and layer L_3, respectively.

[0035] A neural network has an input layer and an output layer, and an intermediate layer is provided between the input layer and the output layer. A plurality of intermediate layers can be provided for one neural network. Layers L_1, L_2, and L_3 shown in FIG. 1A can be intermediate layers. Layer L_1 can be an input layer, and layer L_3 can be an output layer.

[0036] In this specification, inputting data into an input layer of a neural network may be referred to as "inputting data into a neural network," and outputting data from an output layer of a neural network may be referred to as "outputting data from a neural network."

[0037] The feature map output from layer L_1 is input to layer L_2, and the feature map output from layer L_2 is input to layer L_3. That is, data is transmitted in the order of layer L_1, layer L_2, and layer L_3. Therefore, it can be said that the layer next to layer L_1 is layer L_2, and the layer next to layer L_2 is layer L_3.

[0038] The feature map output from the layer L_1 is also input to the neural network NN2. Specifically, when the neural network NN2 is a hierarchical neural network having an input layer, an intermediate layer, and an output layer, the feature map output from the layer L_1 is input to the input layer of the neural network NN2. Then, the feature map output from the neural network NN2 is input to the layer L_3.

[0039] Here, the neural network NN2 has a function of removing noise contained in the feature map input to the neural network NN2. Therefore, it can be said that the feature map input from the layer L_1 to the neural network NN2 and from which noise has been removed by the neural network NN2 is input to the layer L_3. From the above, it can be said that the feature map output from the layer L_1 is input to the layer L_3 via the neural network NN2.

[0040] As described above, the layer next to the layer L_1 is the layer L_2. On the other hand, the feature map output from the layer L_1 and input to the neural network NN2 is input to the layer L_3 via the neural network NN2. That is, the feature map output from the layer L_1 and input to the neural network NN2 is input to the layer L_3, skipping the layer L_2, which is the layer next to the layer L_1. Therefore, it can be said that the layer L_1 and the layer L_3 are skip-connected via the neural network NN2. Also, it can be said that the layer L_3 is a combination layer that combines the feature map output from the layer L_1 and the feature map output from the layer L_2.

[0041] In this specification, layers that are skip-connected are shown connected by dashed lines. For example, in Fig. 1A, layers L_1 and L_3 are skip-connected via the neural network NN2, which is shown by connecting layers L_1 and NN2, and layers L_3 and NN2 with dashed lines.

[0042] The neural network NN1 has a function of outputting a predetermined inference result as a feature map from the output layer for data input to the input layer. For example, when image data is input to the input layer of the neural network NN1, the neural network NN1 has a function of performing segmentation or depth estimation on the image data. Or, when black and white or grayscale image data is input to the input layer of the neural network NN1, the neural network NN1 has a function of colorizing the image data.

[0043] In this specification and the like, the feature map output from the output layer may be referred to as output data output from a neural network having the output layer.

[0044] In a neural network having a skip-connected layer, if the data input to the neural network contains noise, the noise is likely to be transmitted to a layer close to the output layer. This may result in a failure to perform inference on the data with high accuracy. On the other hand, the neural network NN1 of the neural network model 10 has a layer that is skip-connected via a neural network NN2 that has a function of removing noise. This makes it possible to suppress the noise from being transmitted to a layer close to the output layer even if the data input to the neural network NN1 contains noise. This makes it possible to perform inference on the data with high accuracy.

[0045] In the neural network model 10 shown in Fig. 1A, the layer L provided between the layer L_1 and the layer L_3, which are layers L skip-connected via the neural network NN2, is one layer, but the present invention is not limited to this. As shown in Fig. 1B, m layers (m is an integer of 2 or more) of layers L_2 may be provided between the layer L_1 and the layer L_3. In Fig. 1B, the m layers of layers L_2 are distinguished from each other by being described as layers L_2[1] to L_2[m] in order from the layer closest to the input layer of the neural network NN1.

[0046] In the neural network model 10 shown in Fig. 1B, the feature map output from the layer L_1 is input to the layer L_2[1] and the neural network NN2. The feature map output from the layer L_2[m] and the feature map output from the neural network NN2 are input to the layer L_3.

[0047] The neural network NN1 of the neural network model 10 can be, for example, a convolutional neural network (CNN). When the neural network NN1 has the configuration shown in FIG. 1B and is a CNN, the layer L_1 and the layer L_2[m] can be, for example, a pooling layer.

[0048] The neural network NN2 may be, for example, an autoencoder. For example, it may be a generative adversarial network (GAN). When the neural network NN2 is a GAN, it may be, for example, a conditional GAN. When the neural network NN2 is a GAN, it is preferable because inference by the neural network NN2, for example, noise removal for a feature map input to the neural network NN2, can be performed with high accuracy.

[0049] <Learning Method> Next, an example of a learning method of the neural network model 10 will be described with reference to Figs. 2 to 6. Fig. 2 is a flowchart showing an example of the learning method of the neural network model 10. Figs. 3 to 6 are schematic diagrams showing an example of processing in each step shown in Fig. 2. Note that Figs. 3 to 6 assume that the neural network model 10 has the configuration shown in Fig. 1A. Also, Figs. 3 to 6 show, as an example, an example of a learning method of the neural network model 10 for endowing the neural network NN1 with a function of performing segmentation on image data when the image data is input to the neural network NN1.

[0050] [Step S1] First, data for obtaining initial values ​​is input to the neural network NN1 to obtain a weighting coefficient 11_1, which is an initial value of the weighting coefficient 11 of the neural network NN1 (step S1). Figures 3A and 3B are schematic diagrams showing an example of the process in step S1. Specifically, Figure 3A is a schematic diagram showing an example of the process in step S1_1, and Figure 3B is a schematic diagram showing an example of the process in step S1_2.

[0051] As shown in FIG. 3A, in step S1_1, data 21 is prepared. The data 21 can be represented by, for example, a matrix. Here, when the data 21 is image data acquired by imaging, for example, the elements of the matrix can be gradation values ​​representing the illuminance of light irradiated to pixels of an imaging device. Also, the elements can be gradation values ​​representing the luminance of light emitted from pixels of a display device that displays an image corresponding to the data 21. That is, the number of rows and columns of the matrix representing the data 21 can be the same as the resolution of the image corresponding to the data 21, for example. Note that data other than the data 21 can also be represented by a matrix in some cases. For example, the data can be represented by a matrix with the same number of rows and columns as the data 21. Or, the data can be represented by a matrix with fewer rows and columns than the data 21.

[0052] Data 22 is obtained by adding data assuming noise to data 21. For example, when image data acquired by imaging with a camera is assumed as data input to the neural network NN1, the noise may assume, for example, scratches on the lens of the camera or water droplets on the camera. Or, the noise may assume shot noise that occurs when imaging in a dark place. When image data acquired by imaging an image displayed on a display device is assumed as data input to the neural network NN1, the noise may assume defects such as point defects and line defects. Note that the data added to the data 21 does not necessarily have to assume noise as long as it is data to be removed by the neural network NN2.

[0053] In the method described above, in step S1_1, the data 22 is created from the data 21, but one aspect of the present invention is not limited to this. For example, the data 21 may be created from the data 22. In this case, for example, the data 22 is obtained by capturing an image in a rainy environment, and the data 21 can be obtained by performing image processing on the data 22 to remove water droplets.

[0054] Next, as shown in Fig. 3B, in step S1_2, data 22 and data 23 are input to the neural network NN1. Here, the data 23 can be data that represents a desired result of segmentation of the data 22, for example. This makes it possible to obtain a weighting factor 11_1, which is an initial value of the weighting factor 11 of the neural network NN1. For example, the neural network NN1 can obtain a weighting factor 11_1 that minimizes the error between the output data output from the neural network NN1 and the data 23 when the data 22 is input to the neural network NN1.

[0055] In step S1_1 shown in Fig. 3A, one piece of data 21 is prepared, but two or more pieces of data 21 may be prepared. In step S1_2 shown in Fig. 3B, one each of data 22 and data 23 is input to the neural network NN1, but two or more pieces of data 22 and data 23 may be input to the neural network NN1. In step S1_2, data 23 does not have to be input to the neural network NN1. For example, when learning of the neural network NN1, which will be described later, is performed by unsupervised learning, data 23 does not have to be input to the neural network NN1.

[0056] 3A and 3B, the data 22 can be said to be data for obtaining initial values. In addition, both the data 22 and the data 23 can be said to be data for obtaining initial values.

[0057] At the stage of performing step S1, the neural network NN2 has not yet acquired the weighting coefficients, and therefore, no processing is performed by the neural network NN2 at step S1.

[0058] [Step S2] Next, the neural network model 10 performs a first learning to obtain the weight coefficients 12 of the neural network NN2 (step S2). Figures 4A, 4B1, 4B2, and 4C are schematic diagrams showing an example of the process in step S2. Specifically, Figure 4A is a schematic diagram showing an example of the process in step S2_1, Figure 4B1 is a schematic diagram showing an example of the process in step S2_2a, Figure 4B2 is a schematic diagram showing an example of the process in step S2_2b, and Figure 4C is a schematic diagram showing an example of the process in step S2_3.

[0059] In step S2, weighting coefficient 11_i (i is an integer equal to or greater than 1) is held in each layer of neural network NN1. When step S2 is performed immediately after step S1, i can be set to 1. In addition, in Fig. 4B1 and Fig. 4B2, weighting coefficient 11_i held in layer L_1 is indicated as weighting coefficient 11_i[1].

[0060] Details of step S2 for performing the first learning will be described below. First, as shown in Fig. 4A, in step S2_1, a data set 25 including a plurality of data 24 is prepared, and data assuming noise is added to each data 24 to obtain data 26. In this way, a data set 27 including a plurality of data 26 is obtained. The noise added to the data 24 can be the same type of noise as the noise that can be added to the data 21 shown in Fig. 3A. Note that the data added to the data 24 does not necessarily have to assume noise as long as it is data to be removed by the neural network NN2, similar to the data added to the data 21.

[0061] In the method described above, data 26 is created from data 24 in step S2_1, but one aspect of the present invention is not limited to this. The method of acquiring data 24 and data 26 can be the same as the method of acquiring data 21 and data 22 in step S1_1. For example, if data 21 is created from data 22 in step S1_1, data 24 can be created from data 26 in step S2_1, thereby acquiring data 24 and data 26.

[0062] Next, as shown in FIG. 4B1, in step S2_2a, data 24 is input to the neural network NN1, and a feature map 34 output from the layer L_1 is obtained. For example, a feature map 34 is obtained for each of the multiple data 24 included in the dataset 25. As a result, a feature map set 35 including the multiple feature maps 34 is obtained. Also, as shown in FIG. 4B2, in step S2_2b, data 26 is input to the neural network NN1, and a feature map 36 output from the layer L_1 is obtained. For example, a feature map 36 is obtained for each of the multiple data 26 included in the dataset 27. As a result, a feature map set 37 including the multiple feature maps 36 is obtained.

[0063] Thereafter, as shown in FIG. 4C, in step S2_3, the feature map 36 is input as learning data and the feature map 34 is input as correct data to the neural network NN2. This makes it possible to obtain the weighting coefficient 12 of the neural network NN2. For example, the neural network NN2 can obtain the weighting coefficient 12 such that when the feature map 36 is input to the neural network NN2, the feature map output from the neural network NN2 matches the feature map 34 corresponding to the feature map 36. Specifically, for example, by using a gradient descent method using an error function (also called a loss function), it is possible to obtain the weighting coefficient 12 such that the error between the feature map output from the neural network NN2 when the feature map 36 is input to the neural network NN2 and the feature map 34, which is the correct data, is minimized. As the error function, a square sum error, a cross entropy error, or the like can be used. In the gradient descent method, a weighting coefficient is obtained such that the error function is minimized. The minimum value can be obtained, for example, by differentiating the error function. The differentiation of the error function can be obtained, for example, by an error backpropagation method.

[0064] In addition, when the neural network NN2 is a Conditional GAN, the neural network NN2 can perform learning by inputting the feature map 34 to the generator and the feature map 36 to the discriminator.

[0065] In this specification, a feature map that can be used as learning data may be referred to as a learning feature map, and a feature map that can be used as correct data may be referred to as a correct feature map. For example, feature map 36 may be referred to as a learning feature map, and feature map 34 may be referred to as a correct feature map.

[0066] [Step S3] Next, the neural network model 10 performs a second learning process to obtain the weighting coefficients 11 of the neural network NN1 (step S3). Figure 5 is a schematic diagram showing an example of the process in step S3.

[0067] In step S3, weight coefficients 11_i are held in each layer of the neural network NN1. As described above, for example, when the initial value of the weight coefficient 11 is held in each layer of the neural network NN1, i can be set to 1. In addition, the weight coefficient 12 acquired in the previous step S2 is held in the neural network NN2. In FIG. 5, the weight coefficient 11_i held in the layer L_1 is written as weight coefficient 11_i[1], the weight coefficient 11_i held in the layer L_2 is written as weight coefficient 11_i[2], and the weight coefficient 11_i held in the layer L_3 is written as weight coefficient 11_i[3].

[0068] In step S3, the data 26 is input to the neural network NN1 as training data and the data 28 is input as correct answer data. Here, the data 26 can be the same data as the data 26 acquired in step S2_1 shown in FIG. 4A and input to the neural network NN1 in step S2_2b shown in FIG. 4B2. That is, in step S2, when training the neural network NN2, the same data as the data input to the neural network NN1 to acquire the feature map 36, which is the training feature map, can be used as training data in training the neural network NN1. In addition, the data 28 can be data that represents a desired result as a result of segmentation of the data 26, for example. Here, a set of the data 28 used as correct answer data in training the neural network NN1 is set as a data set 29. Note that the data 26 input to the neural network NN1 in step S2_2b and the data 26 input to the neural network NN1 in step S3 may be different.

[0069] In this manner, the weighting coefficient 11_i+1 of the neural network NN1 can be obtained. Specifically, the weighting coefficient 11_i+1 can be obtained by optimizing the weighting coefficient 11_i. For example, the neural network NN1 can obtain the weighting coefficient 11_i+1 such that when data 26 is input to the neural network NN1, the output data output from the neural network NN1 matches the data 28 corresponding to the data 26. Specifically, for example, by using a gradient descent method using an error function, the weighting coefficient 11_i+1 can be obtained such that when data 26 is input to the neural network NN1, the error between the output data output from the neural network NN1 and the correct data 28 is minimized.

[0070] As described above, the weighting factor 12 acquired in the immediately preceding step S2 is held in the neural network NN2. Therefore, the data 26 input to the neural network NN1 is also processed by the neural network NN2. Therefore, the weighting factor 11_i+1 acquired by the neural network NN1 in step S3 can be made to reflect the weighting factor 12 held in the neural network NN2. Note that, although it has been described that the weighting factor 12 is not acquired in step S3, that is, the optimization of the weighting factor 12 acquired in the immediately preceding step S2 is not performed in step S3, one aspect of the present invention is not limited to this. For example, both the weighting factor 11 and the weighting factor 12 may be optimized in step S3.

[0071] 5 shows a case where the neural network NN1 is trained by supervised learning, but the neural network NN1 may be trained by unsupervised learning. When the neural network NN1 is trained by unsupervised learning, the data 28, which is the correct answer data, does not need to be input to the neural network NN1.

[0072] [Step S4] Next, data 41 is input as test data to the neural network NN1, and the inference accuracy of the neural network model 10 is evaluated based on the output data 43 output from the neural network NN1 (step S4). Figure 6 is a schematic diagram showing an example of the process in step S4.

[0073] For example, the output data 43 is compared with data 45 that represents a desired result of segmentation of the data 41, and the inference accuracy of the neural network model 10 is evaluated based on the difference between the output data 43 and the data 45. For example, if the output data 43 and the data 45 are represented by matrices with the same number of rows and columns, the inference accuracy of the neural network model 10 is evaluated based on the difference between each element of the matrix constituting the output data 43 and each element of the matrix constituting the data 45.

[0074] For example, assuming that output data 43 and data 45 are each represented by a matrix with p rows and q columns (p and q are integers equal to or greater than 1), first, it is determined whether the element in row 1, column 1 of output data 43 matches the element in row 1, column 1 of data 45. Similarly, it is determined whether output data 43 matches data 45 for all elements. After that, the number of matching elements is divided by the total number of elements in the matrix (p×q), and this is taken as the accuracy rate. The inference accuracy of neural network model 10 is evaluated based on the accuracy rate. It can be evaluated that the higher the accuracy rate, the higher the inference accuracy.

[0075] Alternatively, the inference accuracy of the neural network model 10 may be evaluated based on an error function. In addition, when learning the neural network model 10 so that the neural network NN1 has a function of performing segmentation on data input to the neural network NN1, the inference accuracy of the neural network model 10 can be evaluated by mIoU (mean intersection over union).

[0076] Alternatively, for example, assuming that the output data 43 and the data 45 are each represented by a matrix of p rows and q columns, first, the absolute value of the difference between the element in the first row and first column of the output data 43 and the element in the first row and first column of the data 45 is calculated. Similarly, the absolute value of the difference between the output data 43 and the data 45 is calculated for all elements. Then, the sum of the absolute values ​​of the calculated p×q differences is calculated, and the inference accuracy of the neural network model 10 is evaluated based on the sum. The smaller the sum, the smaller the error between the output data 43 and the data 45, that is, the higher the inference accuracy can be evaluated. This method can be suitably used when learning the neural network model 10 so that the neural network NN1 has a function of performing depth estimation on data input to the neural network NN1.

[0077] Alternatively, when the neural network model 10 is trained so that the neural network NN1 has a function of colorizing black-and-white image data input to the neural network NN1, the inference accuracy can be evaluated based on the color space. For example, the inference accuracy of the neural network model 10 can be evaluated by comparing the HSV color space of the output data 43 and the data 45. Here, the HSV color space is a numerical representation of the hue, saturation, and brightness of the image represented by the image data. The inference accuracy may also be evaluated using the RGB color space, the CMY color space, the HLS color space, or the like.

[0078] Note that a plurality of pieces of data 41, which are test data, can be prepared. In this case, a set of the data 41 is set as a data set 42. Furthermore, when a plurality of pieces of data 41 are prepared, a plurality of pieces of output data 43 and data 45 are prepared. For example, the same number of pieces of output data 43 as the data 41 and the same number of pieces of data 45 as the data 41 are prepared. When a plurality of pieces of output data 43 and data 45 are prepared, a set of the output data 43 is set as a data set 44, and a set of the data 45 is set as data 46. When the inference accuracy of the neural network model 10 is evaluated using a plurality of pieces of data 41, for example, each of the plurality of pieces of data 41 is input to the neural network NN1, and the inference accuracy for each piece of data 41 is calculated. Then, the average value of the inference accuracy for each piece of data 41 can be set as the inference accuracy of the neural network model 10 used in the next step.

[0079] [Step S5] Thereafter, it is determined whether the inference accuracy of the neural network model 10 evaluated in step S4 is equal to or greater than a specified value (step S5). If it is equal to or greater than the specified value, the learning of the neural network model 10 is terminated. If it is less than the specified value, 1 is added to i, and steps S2 to S5 are performed again. That is, the weighting coefficient 12 is optimized by the first learning, and the weighting coefficient 11 is optimized by the second learning until the inference accuracy becomes equal to or greater than the specified value. The above is an example of a learning method of the neural network model 10. Note that in step S5, the increase in the inference accuracy due to the first learning and the second learning may be evaluated, and the learning of the neural network model 10 may be terminated when the increase becomes equal to or less than a specified value. For example, the difference in inference accuracy between the case where the first learning and the second learning are performed j times (j is an integer equal to or greater than 1) and the case where the first learning and the second learning are performed j+1 times may be evaluated, and the learning of the neural network model 10 may be terminated when the difference becomes equal to or less than a specified value.

[0080] <Neural network model_2> 7 is a diagram showing a configuration example of a neural network model 10a, which is a neural network model according to one embodiment of the present invention. The neural network model 10a includes a neural network NN1a and a neural network NN2a. The neural network NN1a has a similar function to the neural network NN1, and the neural network NN2a has a similar function to the neural network NN2. The neural network model 10a can learn in a similar procedure to the neural network model 10.

[0081] In this specification and the like, the neural network model 10 may include the neural network model 10a. In addition, the neural network NN1 may include the neural network NN1a, and the neural network NN2 may include the neural network NN2a.

[0082] The neural network NN1a has an input layer IL, intermediate layers ML_1 to ML_17, and an output layer OL. The architecture of the neural network NN1a may be, for example, U-Net. Alternatively, it may be, for example, DenseNet.

[0083] In the neural network NN1a having the configuration shown in Fig. 7, the feature map output from the input layer IL is input to the intermediate layer ML_1, and the feature map output from the intermediate layer ML_1 is input to the intermediate layer ML_2. In this way, the data input to the input layer IL is transmitted sequentially to the intermediate layers ML_1 to ML_17. Then, the feature map output from the intermediate layer ML_17 is input to the output layer OL. From the above, it can be said that, among the intermediate layers ML_1 to ML_17, the intermediate layer ML_1 is closest to the input layer IL, the intermediate layers ML_2 to ML_16 are farther from the input layer IL in this order, that is, closer to the output layer OL, and the intermediate layer ML_17 is closest to the output layer OL.

[0084] The feature map output from the intermediate layer ML_1 is input not only to the intermediate layer ML_2 but also to the neural network NN2a. The feature map output from the neural network NN2a is input to the intermediate layer ML_16. That is, in addition to the feature map output from the intermediate layer ML_15, the feature map output from the neural network NN2a is input to the intermediate layer ML_16. From the above, it can be said that the intermediate layer ML_1 and the intermediate layer ML_16 are skip-connected via the neural network NN2a. In addition, it can be said that the intermediate layer ML_16 has a function as a connection layer that connects the feature map output from the intermediate layer ML_15 and the feature map output from the neural network NN2a.

[0085] The feature map output from the intermediate layer ML_3 is input not only to the intermediate layer ML_4 but also to the intermediate layer ML_14. That is, the feature map output from the intermediate layer ML_13 as well as the feature map output from the intermediate layer ML_3 are input to the intermediate layer ML_14. Therefore, it can be said that the intermediate layer ML_3 and the intermediate layer ML_14 are skip-connected. It can also be said that the intermediate layer ML_14 functions as a combination layer that combines the feature map output from the intermediate layer ML_13 and the feature map output from the intermediate layer ML_3.

[0086] The feature map output from the intermediate layer ML_5 is input not only to the intermediate layer ML_6 but also to the intermediate layer ML_12. That is, the feature map output from the intermediate layer ML_11 as well as the feature map output from the intermediate layer ML_5 are input to the intermediate layer ML_12. Therefore, it can be said that the intermediate layer ML_5 and the intermediate layer ML_12 are skip-connected. It can also be said that the intermediate layer ML_12 functions as a combination layer that combines the feature map output from the intermediate layer ML_11 and the feature map output from the intermediate layer ML_5.

[0087] The feature map output from the intermediate layer ML_7 is input not only to the intermediate layer ML_8 but also to the intermediate layer ML_10. That is, the feature map output from the intermediate layer ML_9 as well as the feature map output from the intermediate layer ML_7 are input to the intermediate layer ML_10. Therefore, it can be said that the intermediate layer ML_7 and the intermediate layer ML_10 are skip-connected. It can also be said that the intermediate layer ML_10 functions as a combination layer that combines the feature map output from the intermediate layer ML_9 and the feature map output from the intermediate layer ML_7.

[0088] From the above, it can be said that the neural network NN1a shown in Fig. 7 has four pairs of skip-connected layers. In addition, among the hidden layers ML with skip connections, it can be said that the hidden layer ML_1 closest to the input layer IL is skip-connected to the hidden layer ML_16 closest to the output layer OL via the neural network NN2a.

[0089] For example, in U-Net, a feature map output from an intermediate layer close to the input layer is input to an intermediate layer close to the output layer by skip connection. Therefore, if the data input to the input layer contains noise, the noise is likely to be transmitted to a layer close to the output layer. This may result in a high level of inference on the data. On the other hand, for example, in the neural network NN1a configured as shown in FIG. 7, the intermediate layer ML_1 closest to the input layer IL among the intermediate layers ML that are skip-connected is skip-connected to the intermediate layer ML_16 closest to the output layer OL via the neural network NN2a that has a function of removing noise. This makes it possible to suppress the noise from being transmitted to a layer close to the output layer OL even if the data input to the input layer IL contains noise. This allows the neural network NN1a to perform high-precision inference on the data.

[0090] In addition, as feature extraction is performed by the intermediate layer ML, the influence of noise contained in the data input to the input layer IL becomes smaller. For example, when data containing noise is input to the input layer IL, the influence of the noise on the feature map output by the intermediate layer ML_7 becomes smaller than the influence of the noise on the feature map output by the intermediate layer ML_1. Therefore, even if a feature map output from an intermediate layer ML far from the input layer IL is input to another intermediate layer ML by skip connection, noise removal does not need to be performed on the feature map. For example, the intermediate layer ML_3, the intermediate layer ML_5, and the intermediate layer ML_7, which are layers farther from the input layer IL than the intermediate layer ML_1, can be skip-connected to the intermediate layer ML_14, the intermediate layer ML_12, and the intermediate layer ML_10, respectively, without going through a neural network having a function of removing noise. Therefore, even if the number of layers to be skip-connected increases, the accuracy of inference on the data input to the input layer IL can be maintained while suppressing the expansion of the scale of the neural network model 10.

[0091] FIG. 7 shows a configuration in which the neural network NN1a has 17 hidden layers ML and has four pairs of layers connected in a skip manner, but an embodiment of the present invention is not limited to this. For example, the number of hidden layers ML may be more than 17. Also, FIG. 7 shows a configuration in which only the hidden layers ML_1 and ML_16 are connected in a skip manner via a neural network having a function of removing noise, but an embodiment of the present invention is not limited to this. For example, in addition to the hidden layers ML_1 and ML_16, the hidden layers ML_3 and ML_14 may be connected in a skip manner via a neural network having a function of removing noise. By increasing the number of layers connected in a skip manner via a neural network having a function of removing noise, inference on data input to the input layer IL can be performed with high accuracy.

[0092] <Example of the configuration of the computation processing system> The neural network model of one embodiment of the present invention can be applied to, for example, a processing system. Fig. 8A is a block diagram showing a configuration example of a processing system 100 to which the neural network model of one embodiment of the present invention can be applied. The processing system 100 includes an information processing device 120.

[0093] The information processing device 120 includes an imaging unit 121, an output unit 122, a calculation unit 123, a main storage unit 124, and an auxiliary storage unit 125. Data and the like can be transmitted between the components of the information processing device 120 via a transmission path 127. Note that in this specification and the like, the imaging unit can be referred to as an imaging device.

[0094] The imaging unit 121 has a function of capturing an image and acquiring image data. The image data can be the data 21 shown in Fig. 3A. The output unit 122 has a function of supplying information.

[0095] The calculation unit 123 has a function of performing calculation processing. The calculation unit 123 has a function of performing predetermined calculation processing on data transmitted to the calculation unit 123 from, for example, the imaging unit 121, the main storage unit 124, the auxiliary storage unit 125, etc. via the transmission path 127. The calculation unit 123 can have, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0096] The main memory unit 124 has a function of storing data, programs, etc. The calculation unit 123 can read the data, programs, etc. stored in the main memory unit 124 and execute calculation processing. For example, the calculation unit 123 can execute a predetermined calculation processing on the data read from the main memory unit 124 by executing a program read from the main memory unit 124.

[0097] It is preferable that the main memory unit 124 operates faster than the auxiliary memory unit 125. The main memory unit 124 can include, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0098] The auxiliary storage unit 125 has a function of storing data, programs, etc. for a longer period of time than the main storage unit 124. The auxiliary storage unit 125 may have, for example, a hard disk drive (HDD), a solid state drive (SSD), etc. The auxiliary storage unit 125 may also have a non-volatile memory such as a resistive random access memory (ReRAM, also called a resistance change memory), a phase change random access memory (PRAM), a ferroelectric random access memory (FeRAM), a magnetoresistive random access memory (MRAM, also called a magnetoresistive memory), or a flash memory.

[0099] The weighting coefficient 11 and the weighting coefficient 12 acquired by learning the neural network model of one embodiment of the present invention are stored in the auxiliary storage unit 125. Note that the weighting coefficient 11 and the weighting coefficient 12 may be stored in the main storage unit 124.

[0100] The information processing device 120 can be provided in an information terminal such as a smartphone, a tablet, or a personal computer.

[0101] It should be noted that the configuration of the data processing system 100 is not limited to the above. For example, the data processing system 100 may have an information processing device 110 in addition to the information processing device 120, as shown in FIG.

[0102] The information processing device 110 includes an imaging unit 111, an output unit 112, a calculation unit 113, a main memory unit 114, an auxiliary memory unit 115, and a communication unit 116. Data and the like can be transmitted between the components of the information processing device 110 via a transmission path 117.

[0103] The imaging unit 111 has a function of capturing an image and acquiring image data. The image data can be the data 21 shown in Fig. 3A. The output unit 112 has a function of supplying information.

[0104] The calculation unit 113 has a function of performing calculation processing. The calculation unit 113 has a function of performing predetermined calculation processing on data transmitted to the calculation unit 113 from, for example, the imaging unit 111, the main storage unit 114, the auxiliary storage unit 115, the communication unit 116, etc. via the transmission path 117. The calculation unit 113 can have, for example, a CPU, a GPU, etc.

[0105] The main memory unit 114 has a function of storing data, programs, etc. The calculation unit 113 can read the data, programs, etc. stored in the main memory unit 114 and execute calculation processing. For example, the calculation unit 113 can execute a predetermined calculation processing on the data read from the main memory unit 114 by executing a program read from the main memory unit 114.

[0106] It is preferable that the main memory unit 114 operates faster than the auxiliary memory unit 115. The main memory unit 114 can include, for example, a DRAM, an SRAM, or the like.

[0107] The auxiliary storage unit 115 has a function of storing data, programs, and the like for a longer period of time than the main storage unit 114. The auxiliary storage unit 115 may have, for example, an HDD or an SSD. The auxiliary storage unit 115 may also have a non-volatile memory such as ReRAM, PRAM, FeRAM, MRAM, or a flash memory.

[0108] The weighting coefficient 11 and the weighting coefficient 12 obtained by learning the neural network model of one embodiment of the present invention are stored in the auxiliary storage unit 115. Note that the weighting coefficient 11 and the weighting coefficient 12 may be stored in the main storage unit 114.

[0109] The communication unit 116 has a function of transmitting and receiving data to and from devices provided outside the information processing device 110. The communication unit 116 can also have a function of supplying data to a network and a function of acquiring data from the network.

[0110] The information processing device 120 includes an imaging unit 121, an output unit 122, a calculation unit 123, a main memory unit 124, an auxiliary memory unit 125, and a communication unit 126. Note that the above description can be taken into consideration for the description of the information processing device 120 and the components included in the information processing device 120.

[0111] The communication unit 126 has a function of transmitting and receiving data to and from devices provided outside the information processing device 120. The communication unit 126 can also have a function of supplying data to a network and a function of acquiring data from the network.

[0112] The calculation unit 123 has a function of performing predetermined calculation processing on data transmitted to the calculation unit 123 via a transmission path 127 from, for example, the imaging unit 121, the main memory unit 124, the auxiliary memory unit 125, the communication unit 126, etc.

[0113] Here, when the neural network model of one embodiment of the present invention is applied to the calculation unit 113 and the calculation unit 123, for example, the calculation unit 113 can acquire weighting coefficients 11 and 12 by learning, and can supply them from the information processing device 110 to the information processing device 120. As a result, even if the calculation unit 123 provided in the information processing device 120 does not learn, it can perform calculation processing using the neural network model on the data input to the calculation unit 123 based on the weighting coefficients 11 and 12 acquired by the calculation unit 113. Therefore, the calculation processing capacity of the calculation unit 123 can be made lower than that of the calculation unit 113.

[0114] When the calculation unit 113 acquires the weighting coefficients 11 and 12 and supplies them from the information processing device 110 to the information processing device 120, the information processing device 110 can be provided in, for example, a server. When the information processing device 110 is provided in the server, the information processing device 110 does not need to be provided with the imaging unit 111 and the output unit 112. In other words, the imaging unit 111 and the output unit 112 may be provided outside the information processing device 110.

[0115] Furthermore, the information processing device 120 can be provided in an information terminal such as a smartphone, tablet, or personal computer, as described above. Furthermore, both at least a part of the components of the information processing device 110 and at least a part of the components of the information processing device 120 may be provided in a server. For example, the calculation unit 113 and the calculation unit 123 may be provided in a server. In this case, for example, data acquired by an information terminal is supplied to the calculation unit 123 via a network, and the calculation unit 123 provided in the server performs estimation or the like on the data. Then, the result of the estimation is supplied to the information terminal via the network, so that the information terminal can acquire the result of the estimation.

[0116] 8B and the calculation unit 113 performs learning to obtain the weighting coefficients 11 and 12, the data 21 shown in Fig. 3A does not need to be obtained by the imaging unit 111. For example, the imaging unit 121 may perform imaging to obtain image data that becomes the data 21, and the image data may be supplied from the information processing device 120 to the information processing device 110. In this case, the imaging unit 111 does not need to be provided in the information processing device 110.

[0117] <Example of imaging unit configuration> 9 is a block diagram for explaining a configuration example of the imaging unit 111 and the imaging unit 121. The imaging unit includes a pixel array 300, a circuit 201, a circuit 302, a circuit 303, a circuit 304, a circuit 305, and a circuit 306. Note that each of the circuit 201 and the circuits 301 to 306 is not limited to a single circuit configuration, and may be configured by a combination of multiple circuits. Alternatively, any of multiple circuits described above may be integrated. Furthermore, a circuit other than the above may be connected. Furthermore, the circuit 201, the circuit 301, the circuit 302, the circuit 303, the circuit 304, the circuit 305, and the circuit 306 may be provided in the calculation unit 113 or the calculation unit 123.

[0118] The pixel array 300 has an imaging function and a calculation function. The circuits 201 and 302 have a calculation function. The circuit 302 has a calculation function or a data conversion function. The circuits 303, 304, and 306 have a selection function. The circuit 303 is electrically connected to the pixel block 200 through a wiring 424. The circuit 304 is electrically connected to the pixel block 200 through a wiring 423. The circuit 305 has a function of supplying a potential for a product-sum calculation to the pixel. A shift register, a decoder, or the like can be used as the circuit having the selection function. The circuit 306 is electrically connected to the pixel block 200 through a wiring 413. Note that the circuits 301 and 302 may be provided outside the pixel block 200.

[0119] 10, the pixel array 300 includes a plurality of pixel blocks 200. The pixel block 200 includes a plurality of pixels 400 arranged in a matrix, and each of the pixels 400 is electrically connected to a circuit 201 via a wiring 412. Note that the circuit 201 can also be provided within the pixel block 200.

[0120] In addition, the pixel 400 is electrically connected to the adjacent pixel 400 through a transistor 450 (transistors 450a to 450f). The function of the transistor 450 will be described later.

[0121] The pixel 400 can acquire image data and generate data by adding the image data and a weighting factor. In FIG. 10, the number of pixels in the pixel block 200 is 3×3 as an example, but is not limited to this. For example, it can be 2×2, 4×4, etc. Alternatively, the number of pixels in the horizontal direction and the vertical direction may be different. Also, some pixels may be shared between adjacent pixel blocks.

[0122] The pixel block 200 and the circuit 201 can be operated as a multiply-accumulate circuit.

[0123] As shown in FIG. 11, a pixel 400 can include a photoelectric conversion device 401, a transistor 402, a transistor 403, a transistor 404, a transistor 405, a transistor 406, and a capacitor 407.

[0124] One electrode of the photoelectric conversion device 401 is electrically connected to one of the source or drain of a transistor 402. The other of the source or drain of the transistor 402 is electrically connected to one of the source or drain of a transistor 403, the gate of a transistor 404, and one electrode of a capacitor 407. One of the source or drain of the transistor 404 is electrically connected to one of the source or drain of a transistor 405. The other electrode of the capacitor 407 is electrically connected to one of the source or drain of a transistor 406.

[0125] The other electrode of the photoelectric conversion device 401 is electrically connected to a wiring 414. The other of the source and drain of the transistor 403 is electrically connected to a wiring 415. The other of the source and drain of the transistor 405 is electrically connected to a wiring 412. The other of the source and drain of the transistor 404 is electrically connected to a GND wiring or the like. The other of the source and drain of the transistor 406 is electrically connected to a wiring 411. The other electrode of the capacitor 407 is electrically connected to a wiring 417.

[0126] A gate of the transistor 402 is electrically connected to a wiring 421. A gate of the transistor 403 is electrically connected to a wiring 422. A gate of the transistor 405 is electrically connected to a wiring 423. A gate of the transistor 406 is electrically connected to a wiring 424.

[0127] Here, an electrical connection point between the other of the source or drain of the transistor 402, one of the source or drain of the transistor 403, one electrode of the capacitor 407, and the gate of the transistor 404 is defined as a node FD. Also, an electrical connection point between the other electrode of the capacitor 407 and one of the source or drain of the transistor 406 is defined as a node FDW.

[0128] The wiring 414 and the wiring 415 can function as power supply lines. For example, the wiring 414 can function as a high-potential power supply line, and the wiring 415 can function as a low-potential power supply line. The wirings 421, 422, 423, and 424 can function as signal lines for controlling the conduction of each transistor. The wiring 411 can function as a wiring for supplying a potential corresponding to a weighting coefficient to the pixel 400. The wiring 412 can function as a wiring for electrically connecting the pixel 400 and the circuit 201. The wiring 417 can function as a wiring for electrically connecting the other electrode of the capacitor 407 of the pixel to the other electrode of the capacitor 407 of another pixel through a transistor 450 (see FIG. 10).

[0129] Note that the wiring 412 may be electrically connected to an amplifier circuit or a gain adjustment circuit.

[0130] A photodiode can be used as the photoelectric conversion device 401. Any type of photodiode can be used, including a Si photodiode having silicon in the photoelectric conversion layer, an organic photodiode having an organic photoconductive film in the photoelectric conversion layer, etc. Note that, when it is desired to increase the light detection sensitivity at low illuminance, it is preferable to use an avalanche photodiode.

[0131] The transistor 402 can have a function of controlling the potential of the node FD. The transistor 403 can have a function of initializing the potential of the node FD. The transistor 404 can have a function of controlling a current flowing from the circuit 201 in accordance with the potential of the node FD. The transistor 405 can have a function of selecting a pixel. The transistor 406 can have a function of supplying a potential corresponding to a weighting coefficient to the node FDW.

[0132] When an avalanche photodiode is used for the photoelectric conversion device 401, a high voltage may be applied, and therefore a high-voltage transistor is preferably used for the transistor connected to the photoelectric conversion device 401. For example, a transistor using a metal oxide in a channel formation region (hereinafter, referred to as an OS transistor) can be used for the high-voltage transistor. Specifically, an OS transistor is preferably used for the transistor 402.

[0133] In addition, the off-state current of the OS transistor is extremely low. By using OS transistors for the transistors 402, 403, and 406, the period during which charges can be held at the nodes FD and FDW can be extremely long. Therefore, a global shutter system in which charges are accumulated simultaneously in all pixels can be applied without complicating the circuit configuration or operation method. In addition, while image data is held in the node FD, multiple calculations can be performed using the image data.

[0134] On the other hand, it may be desirable for the transistor 404 to have excellent amplification characteristics. In addition, it may be preferable to use a transistor with high mobility that can operate at high speed as the transistor 406. Therefore, a transistor using silicon for a channel formation region (hereinafter, referred to as a Si transistor) may be used as the transistor 404 and the transistor 406.

[0135] Note that the present invention is not limited to the above, and any combination of OS transistors and Si transistors may be used. All the transistors may be OS transistors. Alternatively, all the transistors may be Si transistors. Examples of Si transistors include transistors having amorphous silicon and transistors having crystalline silicon (microcrystalline silicon, low-temperature polysilicon, and single crystal silicon).

[0136] The potential of the node FD in the pixel 400 is determined by a potential obtained by adding a reset potential supplied from the wiring 415 and a potential (image data) generated by photoelectric conversion by the photoelectric conversion device 401. Alternatively, the potential is determined by further capacitive coupling with a potential corresponding to a weighting factor supplied from the wiring 411. Therefore, a current corresponding to data obtained by adding an arbitrary weighting factor to image data can flow through the transistor 405.

[0137] It should be noted that the above is only one example of the circuit configuration of the pixel 400, and the photoelectric conversion operation can be performed using other circuit configurations.

[0138] 10, the pixels 400 are electrically connected to each other through wirings 412. The circuit 201 can perform calculations using the sum of currents flowing through the transistors 404 of the pixels 400.

[0139] The circuit 201 includes a capacitor 202 , a transistor 203 , a transistor 204 , a transistor 205 , a transistor 206 , and a resistor 207 .

[0140] One electrode of the capacitor 202 is electrically connected to one of the source or drain of the transistor 203. One of the source or drain of the transistor 203 is electrically connected to the gate of the transistor 204. One of the source or drain of the transistor 204 is electrically connected to one of the source or drain of the transistor 205. One of the source or drain of the transistor 205 is electrically connected to one of the source or drain of the transistor 206. One electrode of the resistor 207 is electrically connected to the other electrode of the capacitor 202.

[0141] The other electrode of the capacitor 202 is electrically connected to a wiring 412. The other of the source and drain of the transistor 203 is electrically connected to a wiring 218. The other of the source and drain of the transistor 204 is electrically connected to a wiring 219. The other of the source and drain of the transistor 205 is electrically connected to a reference power supply line such as a GND wiring. The other of the source and drain of the transistor 206 is electrically connected to a wiring 212. The other electrode of the resistor 207 is electrically connected to a wiring 217.

[0142] The wiring 217, the wiring 218, and the wiring 219 can function as power supply lines. For example, the wiring 218 can function as a wiring that supplies a dedicated potential for reading. The wiring 217 and the wiring 219 can function as high-potential power supply lines. The wiring 213, the wiring 215, and the wiring 216 can function as signal lines that control the conduction of each transistor. The wiring 212 is an output line and can be electrically connected to, for example, the circuit 301 shown in FIG. 9.

[0143] The transistor 203 can have a function of resetting the potential of the wiring 211 to the potential of the wiring 218. The wiring 211 is a wiring connected to one electrode of the capacitor 202, one of the source and the drain of the transistor 203, and the gate of the transistor 204. The transistors 204 and 205 can function as source follower circuits. The transistor 206 can have a function of controlling reading. Note that the circuit 201 has a function as a correlated double sampling circuit (CDS circuit) and can be replaced with a circuit having another structure having the function.

[0144] In one embodiment of the present invention, offset components other than the product of image data (X) and weighting coefficient (W) are removed to extract the target WX. WX can be calculated using data with and without imaging for the same pixel, and data when weights are applied to each of the data.

[0145] The current (I p ) is the sum of kΣ(XV th ) 2 , the current (I p ) is kΣ(W+XV th ) 2 In addition, the current (I ref ) is the sum of kΣ(0-V th ) 2 , the current (I ref ) is the sum of kΣ(WVth ) 2 Here, k is a constant, V th is the threshold voltage of transistor 405.

[0146] First, calculate the difference (data A) between the captured data and the weighted data. kΣ((XV th ) 2 -(W+XV th ) 2 )=kΣ(-W 2 -2W·X+2W·V th )

[0147] Next, the difference between the no-image data and the weighted data (data B) is calculated. kΣ((0-V th ) 2 -(WV th ) 2 )=kΣ(-W 2 +2W·V th )

[0148] Then, take the difference between data A and data B. kΣ(-W 2 -2W·X+2W·V th -(-W 2 +2W·V th ))=kΣ(-2W·X). In other words, it is possible to remove offset components other than the product of the image data (X) and the weighting coefficient (W).

[0149] The circuit 201 can read out data A and data B. Note that the difference between data A and data B can be calculated by the circuit 301, for example.

[0150] Here, the weights supplied to the entire pixel block 200 function as a filter. For example, a convolution filter of a convolutional neural network (CNN) can be used as the filter. Alternatively, an image processing filter such as an edge extraction filter can be used. Examples of edge extraction filters include a Laplacian filter shown in FIG. 12A, a Prewitt filter shown in FIG. 12B, and a Sobel filter shown in FIG. 12C.

[0151] When the pixel block 200 has 3×3 pixels 400, the elements of the edge extraction filter can be assigned as weights to each pixel 400 and supplied. As described above, data A and data B can be calculated using data with and without imaging and data when weights are added to each of the data. Here, the data with and without imaging is data with no weight added, and can be rephrased as data with a weight of 0 added to all pixels 400.

[0152] 12A to 12C are filters in which the sum (ΣΔW / N, N is the number of elements) of the filter elements (weight: ΔW) is 0. Therefore, even if an operation of newly supplying ΔW=0 from another circuit is not performed, by performing an operation of acquiring ΣΔW / N, it is possible to acquire data in which ΔW=0 has been added to all pixels 400.

[0153] This operation corresponds to turning on the transistors 450 (transistors 450a to 450f) provided between the pixels 400 (see FIG. 10). By turning on the transistors 450, the nodes FDW of the pixels 400 are all short-circuited via the wiring 417. At this time, the charge accumulated in the nodes FDW of the pixels 400 is redistributed, and when the edge extraction filters illustrated in FIGS. 12A to 12C are used, the potential (ΔW) of the nodes FDW becomes 0 or approximately 0. Therefore, data with the equivalent of ΔW=0 added can be acquired.

[0154] When rewriting the weight (ΔW) by supplying charge from a circuit outside the pixel array 300, it takes time to complete the rewriting due to the capacitance of the long-distance wiring 411. On the other hand, the pixel block 200 is a very small area, and the wiring 417 is short and has a small capacity. Therefore, in the operation of redistributing the charge accumulated in the node FDW in the pixel block 200, the weight (ΔW) can be rewritten at high speed.

[0155] 10 illustrates a configuration in which the transistors 450a to 450f are electrically connected to different gate lines (wirings 413a to 413f), respectively. In this configuration, the conduction of the transistors 450a to 450f can be controlled independently, and an operation for obtaining ΣΔW / N can be selectively performed.

[0156] 12B or 12C is used, there is a pixel to which ΔW=0 is initially supplied. On the assumption that ΣΔW / N=0, the pixel to which ΔW=0 is supplied may be excluded from the pixels to be subjected to the summation. By excluding the pixel, it is not necessary to supply a potential for operating some of the transistors 450a to 450f, and therefore power consumption can be reduced.

[0157] The data resulting from the product-sum operation output from the circuit 201 is sequentially input to the circuit 301. The circuit 301 may have various calculation functions in addition to the above-mentioned function of calculating the difference between data A and data B. For example, the circuit 301 may have the same configuration as the circuit 201. Alternatively, the function of the circuit 301 may be replaced by software processing.

[0158] The circuit 301 may also include a circuit that performs an activation function calculation. For example, a comparator circuit can be used for the circuit. The comparator circuit compares input data with a set threshold value and outputs the result as binary data. That is, the pixel block 200 and the circuit 301 can act as a part of a neural network.

[0159] Data output from the circuit 301 is sequentially input to the circuit 302. The circuit 302 can have a configuration including, for example, a latch circuit and a shift register. With this configuration, parallel-serial conversion can be performed, and data input in parallel can be output to the wiring 311 as serial data.

[0160] For example, if the neural network model 10 having the configuration shown in FIG. 1A or FIG. 1B is applied to the calculation unit 113 and the calculation unit 123, the data output from the wiring 311 can be input to the layer L_1. Therefore, the layer L_1 can be used as the input layer of the neural network NN1. In addition, when the neural network model 10 having the configuration shown in FIG. 1A or FIG. 1B performs learning, the data output from the wiring 311 can be the data 21 shown in FIG. 3A, and data obtained by adding noise to the data 21 can be the data 22. The noise can be, for example, Gaussian noise. As described above, even if the data output from the wiring 311 contains noise, it is possible to suppress a decrease in the accuracy of inference of the neural network model applied to the calculation unit 113 and the calculation unit 123 due to the noise.

[0161] [Pixel configuration example] 13A is a diagram showing an example of the configuration of a pixel 400. The pixel 400 can have a laminated structure of a layer 561 and a layer 563.

[0162] The layer 561 includes the photoelectric conversion device 401. The photoelectric conversion device 401 can include a layer 565a and a layer 565b as shown in Fig. 13B. Note that the layer may be referred to as a region in some cases.

[0163] 13B is a pn junction photodiode, and may use, for example, a p-type semiconductor for the layer 565a and an n-type semiconductor for the layer 565b. Alternatively, the layer 565a may be an n-type semiconductor, and the layer 565b may be a p-type semiconductor.

[0164] The pn junction photodiode can be typically formed using single crystal silicon.

[0165] Furthermore, the photoelectric conversion device 401 included in the layer 561 may be a laminate of layers 566a, 566b, 566c, and 566d as shown in Fig. 13C. The photoelectric conversion device 401 shown in Fig. 13C is an example of an avalanche photodiode, in which the layers 566a and 566d correspond to electrodes, and the layers 566b and 566c correspond to a photoelectric conversion unit.

[0166] Layer 566a is preferably a low resistance metal layer, etc. For example, aluminum, titanium, tungsten, tantalum, silver, or a laminate of these may be used.

[0167] The layer 566d is preferably a conductive layer having a high light-transmitting property to visible light. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, graphene, or the like can be used. Note that the layer 566d may be omitted.

[0168] The layers 566b and 566c of the photoelectric conversion unit may be configured as a pn junction photodiode with a photoelectric conversion layer made of, for example, a selenium-based material. It is preferable that the layer 566b is made of a selenium-based material that is a p-type semiconductor, and the layer 566c is made of gallium oxide or the like that is an n-type semiconductor.

[0169] Photoelectric conversion devices using selenium-based materials have the characteristic of high external quantum efficiency for visible light. In such photoelectric conversion devices, avalanche multiplication can be used to increase the amplification of electrons relative to the amount of incident light. In addition, selenium-based materials have a high optical absorption coefficient, which offers the advantage in terms of production, such as the ability to fabricate a thin photoelectric conversion layer. Thin films of selenium-based materials can be formed using vacuum deposition, sputtering, or the like.

[0170] As the selenium-based material, crystalline selenium such as single crystal selenium or polycrystalline selenium, amorphous selenium, a compound of copper, indium, and selenium (CIS), or a compound of copper, indium, gallium, and selenium (CIGS), or the like can be used.

[0171] The n-type semiconductor is preferably made of a material that has a wide band gap and is transparent to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or a mixture of these oxides can be used. These materials also function as a hole injection blocking layer, and can reduce dark current.

[0172] Furthermore, the photoelectric conversion device 401 included in the layer 561 may be a laminate of layers 567a, 567b, 567c, 567d, and 567e as shown in Fig. 13D. The photoelectric conversion device 401 shown in Fig. 13D is an example of an organic photoconductive film, in which the layer 567a is a lower electrode, the layer 567e is a light-transmitting upper electrode, and the layers 567b, 567c, and 567d correspond to photoelectric conversion units.

[0173] In the photoelectric conversion portion, one of the layer 567b and the layer 567d can be a hole-transporting layer, the other of the layer 567b and the layer 567d can be an electron-transporting layer, and the layer 567c can be a photoelectric conversion layer.

[0174] For the hole transport layer, for example, molybdenum oxide can be used. For the electron transport layer, for example, C 60 , C70 or derivatives thereof can be used.

[0175] As the photoelectric conversion layer, a mixed layer (bulk heterojunction structure) of an n-type organic semiconductor and a p-type organic semiconductor can be used.

[0176] 13A includes, for example, a silicon substrate. Si transistors and the like are provided over the silicon substrate. The pixel 400 can be formed using the Si transistors. In addition, the circuit 201 and the circuits 301 to 306 shown in FIG. 9 can be formed.

[0177] Next, the laminated structure of the imaging device will be described using a cross-sectional view. Note that the elements such as the insulating layer and the conductive layer shown below are examples, and other elements may be further included. Alternatively, some of the elements shown below may be omitted. Also, the laminated structure shown below can be formed using a bonding process, a polishing process, etc., as necessary.

[0178] 14 includes layers 560, 561, and 563. In FIG. 14, the transistors 402 and 403 are shown as elements provided in the layer 563; however, other elements such as transistors 404 to 406 may also be provided in the layer 563.

[0179] The layer 563 includes a silicon substrate 632, an insulating layer 633, an insulating layer 634, an insulating layer 635, and an insulating layer 637. Also, a conductive layer 636 is provided.

[0180] The insulating layers 634, 635, and 637 function as an interlayer insulating film and a planarizing film. The insulating layer 633 functions as a protective film. The conductive layer 636 is electrically connected to the wiring 414 shown in FIG.

[0181] The interlayer insulating film and the planarizing film may be, for example, an inorganic insulating film such as a silicon oxide film, or an organic insulating film such as an acrylic resin or a polyimide resin. The protective film may be, for example, a silicon nitride film, a silicon oxide film, an aluminum oxide film, or the like.

[0182] The conductive layer may be made of a metal element selected from aluminum, chromium, copper, silver, gold, platinum, tantalum, nickel, titanium, molybdenum, tungsten, hafnium, vanadium, niobium, manganese, magnesium, zirconium, beryllium, indium, ruthenium, iridium, strontium, lanthanum, etc., or an alloy containing the above-mentioned metal element as a component, or an alloy combining the above-mentioned metal elements, etc. The conductor is not limited to a single layer, and may be a plurality of layers made of different materials.

[0183] The Si transistor shown in Fig. 14 is a fin type having a channel formation region in a silicon substrate. A cross section in the channel width direction (a cross section of A1-A2 shown in layer 563 in Fig. 14) is shown in Fig. 15A. The Si transistor may be a planar type as shown in Fig. 15B.

[0184] 15C, the transistor may have a silicon thin-film semiconductor layer 545. The semiconductor layer 545 may be, for example, single crystal silicon (SOI: Silicon on Insulator) formed on an insulating layer 546 on a silicon substrate 632.

[0185] The layer 561 is provided with a photoelectric conversion device 401. The photoelectric conversion device 401 can be formed on the layer 563. Fig. 14 shows a configuration in which the organic photoconductive film shown in Fig. 13D is used as the photoelectric conversion layer for the photoelectric conversion device 401. Note that here, the layer 567a is a cathode, and the layer 567e is an anode.

[0186] The layer 561 includes an insulating layer 651, an insulating layer 652, an insulating layer 653, an insulating layer 654, and a conductive layer 655.

[0187] The insulating layers 651, 653, and 654 function as an interlayer insulating film and a planarizing film. The insulating layer 654 is provided to cover an end of the photoelectric conversion device 401 and also functions to prevent a short circuit between the layer 567e and the layer 567a. The insulating layer 652 functions as an element isolation layer. An organic insulating film or the like is preferably used as the element isolation layer.

[0188] The layer 567a corresponding to the cathode of the photoelectric conversion device 401 is electrically connected to one of the source and the drain of the transistor 402 included in the layer 563. The layer 567e corresponding to the anode of the photoelectric conversion device 401 is electrically connected to the conductive layer 636 provided in the layer 563 through the conductive layer 655.

[0189] The layer 560 is formed on the layer 561. The layer 560 includes a light-shielding layer 671, an optical conversion layer 672, and a microlens array 673.

[0190] The light-shielding layer 671 can suppress the inflow of light into adjacent pixels. A metal layer such as aluminum or tungsten can be used for the light-shielding layer 671. In addition, the metal layer and a dielectric film having a function as an anti-reflection film may be laminated.

[0191] A color filter can be used for the optical conversion layer 672. A color image can be obtained by assigning colors such as red (red), green (G), blue (B), yellow (Y), cyan (C), and magenta (M) to the color filter for each pixel. In addition, if a wavelength cut filter is used for the optical conversion layer 672, an imaging device capable of obtaining images in various wavelength regions can be obtained.

[0192] For example, an infrared imaging device can be formed by using an infrared filter that blocks light with wavelengths equal to or shorter than visible light in the optical conversion layer 672. A far-infrared imaging device can be formed by using a filter that blocks light with wavelengths equal to or shorter than near-infrared light in the optical conversion layer 672. An ultraviolet imaging device can be formed by using an ultraviolet filter that blocks light with wavelengths equal to or longer than visible light in the optical conversion layer 672.

[0193] Furthermore, if a scintillator is used for the optical conversion layer 672, it can be an imaging device that obtains an image that visualizes the intensity of radiation used in an X-ray imaging device or the like. When radiation such as X-rays that has passed through a subject is incident on the scintillator, it is converted into light (fluorescence) such as visible light or ultraviolet light by the photoluminescence phenomenon. Then, image data is obtained by detecting the light with the photoelectric conversion device 401. Also, an imaging device having this configuration may be used for a radiation detector or the like.

[0194] A scintillator contains a substance that absorbs the energy of radiation such as X-rays or gamma rays and emits visible or ultraviolet light when irradiated with the radiation. For example, Gd 2 O 2 S: Tb, Gd 2 O 2 S: Pr, Gd 2 O 2 S:Eu, BaFCl:Eu, NaI, CsI, CaF 2 , BaF 2 , CeF 3 , LiF, LiI, ZnO, etc. dispersed in a resin or ceramics can be used.

[0195] A microlens array 673 is provided on the optical conversion layer 672. Light passing through each lens of the microlens array 673 passes through the optical conversion layer 672 directly below, and is irradiated onto the photoelectric conversion device 401. By providing the microlens array 673, concentrated light can be incident on the photoelectric conversion device 401, so that photoelectric conversion can be performed efficiently. The microlens array 673 is preferably formed of resin, glass, or the like that is highly translucent to light of the wavelength of the object of imaging.

[0196] 16 is a modification of the stacked structure shown in FIG 14, and is different in the configuration of the photoelectric conversion device 401 included in the layer 561 and in a portion of the configuration of the layer 563. In the configuration shown in FIG 16, a bonding surface is provided between the layer 561 and the layer 563.

[0197] Layer 561 includes photovoltaic device 401 , insulating layer 661 , insulating layer 662 , insulating layer 664 , insulating layer 665 , conductive layer 685 , and conductive layer 686 .

[0198] The photoelectric conversion device 401 is a pn junction type photodiode formed on a silicon substrate, and has a layer 565b corresponding to a p-type region and a layer 565a corresponding to an n-type region. The photoelectric conversion device 401 is a buried type photodiode, and a thin p-type region (part of the layer 565b) provided on the surface side (current extraction side) of the layer 565a can suppress dark current and reduce noise.

[0199] The insulating layer 661, the conductive layer 685, and the conductive layer 686 function as bonding layers. The insulating layer 662 functions as an interlayer insulating film and a planarizing film. The insulating layer 664 functions as an element isolation layer. The insulating layer 665 functions to suppress the outflow of carriers.

[0200] The silicon substrate is provided with grooves for separating pixels, and the insulating layer 665 is provided on the upper surface of the silicon substrate and in the grooves. By providing the insulating layer 665, it is possible to prevent carriers generated in the photoelectric conversion device 401 from flowing out to adjacent pixels. The insulating layer 665 also has a function of preventing the intrusion of stray light. Therefore, the insulating layer 665 can prevent color mixing. An anti-reflection film may be provided between the upper surface of the silicon substrate and the insulating layer 665.

[0201] The element isolation layer can be formed by using a LOCOS (LOCal Oxidation of Silicon) method. Alternatively, it may be formed by using a STI (Shallow Trench Isolation) method or the like. For example, an inorganic insulating film such as silicon oxide or silicon nitride, or an organic insulating film such as polyimide or acrylic can be used as the insulating layer 665. The insulating layer 665 may have a multi-layer structure. It is also possible to adopt a structure in which no element isolation layer is provided.

[0202] A layer 565a (n-type region, corresponding to a cathode) of the photoelectric conversion device 401 is electrically connected to a conductive layer 685. A layer 565b (p-type region, corresponding to an anode) is electrically connected to a conductive layer 686. The conductive layers 685 and 686 have regions embedded in an insulating layer 661. The surfaces of the insulating layer 661, and the conductive layers 685 and 686 are flattened so that they are at the same height.

[0203] In the layer 563, an insulating layer 638 is formed over the insulating layer 637. In addition, a conductive layer 683 electrically connected to one of the source and the drain of the transistor 402, and a conductive layer 684 electrically connected to the conductive layer 636 are formed.

[0204] The insulating layer 638, and the conductive layers 683 and 684 function as bonding layers. The conductive layers 683 and 684 have regions buried in the insulating layer 638. The surfaces of the insulating layer 638, the conductive layers 683, and the conductive layers 684 are flattened to be at the same height.

[0205] Here, it is preferable that the conductive layers 683 and 685 are mainly composed of the same metal element, and it is preferable that the conductive layers 684 and 686 are mainly composed of the same metal element. It is also preferable that the insulating layers 638 and 661 are mainly composed of the same metal element.

[0206] For example, Cu, Al, Sn, Zn, W, Ag, Pt, Au, or the like can be used for the conductive layers 683 to 686. In particular, Cu, Al, W, or Au is preferably used because of ease of bonding. Silicon oxide, silicon oxynitride, silicon nitride oxide, silicon nitride, titanium nitride, or the like can be used for the insulating layer 638 and the insulating layer 661.

[0207] That is, the same metal material as described above is preferably used for each of the conductive layers 683 to 686. The same insulating material as described above is preferably used for each of the insulating layers 638 and 661. With this structure, the layer 563 and the layer 561 can be bonded to each other at a boundary therebetween.

[0208] Note that the conductive layers 683 to 686 may each have a multilayer structure including a plurality of layers, in which case the surface layers (bonding surfaces) may be made of the same metal material. The insulating layer 638 and the insulating layer 661 may also have a multilayer structure including a plurality of layers, in which case the surface layers (bonding surfaces) may be made of the same insulating material.

[0209] By this bonding, the conductive layer 683 and the conductive layer 685 can be electrically connected to each other, and the conductive layer 684 and the conductive layer 686 can be electrically connected to each other. In addition, the insulating layer 661 and the insulating layer 638 can be connected to each other with sufficient mechanical strength.

[0210] To bond metal layers together, a surface activation bonding method can be used, in which oxide films and adsorbed layers of impurities on the surfaces are removed by a sputtering process or the like, and cleaned and activated surfaces are brought into contact with each other to bond them. Alternatively, a diffusion bonding method can be used, in which surfaces are bonded together using a combination of temperature and pressure. In either case, bonding occurs at the atomic level, so a bond that is excellent not only electrically but also mechanically can be obtained.

[0211] In addition, for bonding insulating layers, a hydrophilic bonding method can be used in which high flatness is obtained by polishing, etc., and then surfaces that have been hydrophilically treated with oxygen plasma, etc. are brought into contact with each other to form a temporary bond, and then the final bond is performed by dehydrating them through heat treatment. Hydrophilic bonding also produces bonds at the atomic level, so mechanically excellent bonds can be obtained.

[0212] When the layer 563 and the layer 561 are bonded to each other, a metal layer and an insulating layer are mixed on each bonding surface, so that, for example, a surface activated bonding method and a hydrophilic bonding method may be combined.

[0213] For example, a method of cleaning the surface after polishing, subjecting the surface of the metal layer to an anti-oxidation treatment, and then subjecting it to a hydrophilic treatment and bonding can be used. The surface of the metal layer may be made of a resistant metal such as Au and then subjected to a hydrophilic treatment. Note that bonding methods other than the above-mentioned methods may also be used.

[0214] By the above-mentioned bonding, the elements included in the layer 563 and the elements included in the layer 561 can be electrically connected to each other.

[0215] FIG. 17 shows a modification of the laminated structure shown in FIG.

[0216] In this modified example, the transistor 402 included in the pixel 400 is provided in a layer 561. In the layer 561, the transistor 402 is formed using a Si transistor. One of a source or a drain of the transistor 402 is directly connected to one electrode of the photoelectric conversion device 401. The other of the source or the drain of the transistor 402 is electrically connected to a node FD.

[0217] 17, the layer 563 includes transistors other than at least the transistor 402 among the transistors included in the imaging device. In FIG. 17, the transistors 404 and 405 are shown as elements provided in the layer 563, but other elements such as the transistor 403 and the transistor 406 can also be provided in the layer 563. In addition, an insulating layer 647 is provided between the insulating layer 635 and the insulating layer 637 in the layer 563 of the imaging device illustrated in FIG. 17. The insulating layer 647 functions as an interlayer insulating film and a planarization film. [Explanation of symbols]

[0218] 10: neural network model, 10a: neural network model, 21: data, 22: data, 23: data, 24: data, 25: data set, 26: data, 27: data set, 28: data, 29: data set, 34: feature map, 35: feature map set, 36: feature map, 37: feature map set, 41: data, 42: data set, 43: output data, 44: data set, 45: data, 46: data, 100: calculation processing system, 110: information processing device, 111: imaging unit, 112: output unit, 113: calculation unit, 114: main memory unit, 115: auxiliary memory unit, 116: communication unit, 117: transmission path, 120: information processing device, 121: imaging unit, 122: output unit, 123: calculation unit, 124: main memory unit, 125: auxiliary memory unit, 126: communication unit, 127: transmission path, 200: pixel block, 201: circuit, 202: capacitor, 203: transistor, 204: transistor, 205: transistor, 206: transistor, 207: resistor, 211: wiring, 212: wiring, 213: wiring, 215: wiring, 216: wiring, 217: wiring, 218: wiring, 219: wiring, 300: pixel array, 301: circuit, 302: circuit, 303: circuit, 304: circuit, 305: circuit, 306: circuit, 311: wiring, 400: pixel, 401: photoelectric conversion device, 402: transistor, 403: transistor, 404: transistor, 405: transistor, 406: transistor, 407: capacitor, 411: wiring, 412: wiring, 413: wiring, 414: wiring, 415: wiring, 417: wiring, 421: wiring, 422: wiring, 423: wiring, 424: wiring, 450: transistor, 545: semiconductor layer, 546: insulating layer, 560: layer, 561: layer, 563: layer , 565a: layer, 565b: layer, 566a: layer, 566b: layer, 566c: layer, 566d: layer, 567a: layer, 567b: layer, 567c: layer, 567d: layer, 567e: layer, 632: silicon substrate, 633: insulating layer, 634: insulating layer, 635: insulating layer, 636: conductive layer, 637: insulating layer, 638: insulating layer, 647: insulating layer, 651: insulating layer, 652: insulating layer, 653: insulating layer, 654: insulating layer, 655: conductive layer, 661: insulating layer, 662: insulating layer, 664: insulating layer, 665: insulating layer, 671: light shielding layer, 672: optical conversion layer, 673: microlens array,683: conductive layer, 684: conductive layer, 685: conductive layer, 686: conductive layer,

Claims

1. A first neural network and a second neural network, the first neural network has a first layer, a second layer, and a third layer; The feature map output from the first layer is input to the second layer and the second neural network; The feature map output from the second neural network is input to the third layer; A method for training a neural network model, which causes a computer to function to output output image data in which noise has been removed from input image data, comprising the steps of: a first step of inputting data for obtaining initial values ​​to the first neural network to obtain initial values ​​of weight coefficients of the first neural network; a second step of acquiring weight coefficients of the second neural network by performing a first learning process such that a feature map output from the first layer when first data is input to the first neural network is defined as a correct feature map, a feature map output from the first layer when second data obtained by adding noise to the first data is input to the first neural network is defined as a training feature map, and a feature map output from the second neural network when the training feature map is input to the second neural network is identical to the correct feature map; a third step of inputting the second data into the first neural network to perform second learning, thereby acquiring weight coefficients of the first neural network; a fourth step of inputting test data to the first neural network and evaluating the inference accuracy of the neural network model based on output data output from the first neural network; A method for learning a neural network model, comprising the steps of: performing the second to fourth steps until the inference accuracy reaches a specified value or higher.

2. In claim 1, the first neural network has a fourth layer, a fifth layer, and a sixth layer; The feature map output from the fourth layer is input to the fifth layer and the sixth layer which is skip-connected to the fourth layer; the fourth layer is closer to an output layer of the first neural network than the first layer; A method for training a neural network model, wherein the third layer is closer to an output layer of the first neural network than the sixth layer.

Images