Computation system

Abstract

This provides a neural network model that processes input data with high accuracy. [Solution] The neural network model 10 has neural networks NN1 and NN2, and neural network NN1 has layers L_1 to L_3. The feature map output from layer L_1 is input to SOUL_2 and neural network NN2, and the feature map output by neural network NN2 is input to layer L_3. Here, if we consider the feature map output from layer L_1 when the first data is input to neural network NN1 as the ground truth feature map, and the feature map output from layer L_1 when the second data, which has noise added to the first data, is input to neural network NN1 as the trained feature map, then neural network NN2 is trained so that the feature map output when the trained feature map is input matches the ground truth feature map.

Description

[Technical Field]

[0001] One aspect of the present invention relates to a neural network model. Another aspect of the present invention is This is about the training methods for neural network models. [Background technology]

[0002] A technology that uses AI (Artificial Intelligence) to perform image processing. This is attracting attention. For example, Patent Document 1 describes the type of object included in the image and the type of the object. A segmentation technique that uses a neural network to detect regions has been disclosed. Furthermore, technologies have been developed that use AI to colorize black and white images and estimate depth. It is.

[0003] Furthermore, neural networks with skip-connected layers have been developed. Here, skip connections refer to feature maps output from layers of a neural network. However, this configuration shows that the signal is input not only to the next layer but also to other layers closer to the output layer. Non-patent literature In section 1, U-Net is disclosed as a neural network with skip connections. Yes. U-Net uses skip connections to input data into, for example, a neural network. This makes it easier to reflect the information contained in image data into the feature map output from layers near the output layer. This allows for highly accurate image processing. For example, U-Net is... The image data input to the neural network contains information such as the position and contour of objects. This makes it easier to reflect the features in the feature map output from layers near the output layer, thus improving segmentation. It can perform the operation with high precision. [Prior art documents]

Patent Document

[0004]

Patent Document 1

Non-Patent Document

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In a neural network having layers connected by skip connections, when noise is included in the data (input data) input to the neural network, the noise is likely to be transmitted to layers closer to the output layer. As a result, it may become impossible to perform processing on the data with high accuracy. When noise is included in the data (input data) input to the neural network, the noise is likely to be transmitted to layers closer to the output layer. As a result, it may become impossible to perform processing on the data with high accuracy. This may make it impossible to perform processing on the data with high accuracy. There is a possibility that it becomes impossible to perform processing on the data with high accuracy.

[0007] One aspect of the present invention is to provide a neural network model capable of removing noise included in input data. Or, one of the problems is to provide a neural network model capable of performing processing on input data with high accuracy. Or, one of the problems is to provide a novel neural network model. Or, one of the problems is to provide a neural network model capable of performing processing on input data with high accuracy. Or, one of the problems is to provide a novel neural network model. Or, one of the problems is to provide a neural network model capable of performing processing on input data with high accuracy. Or, one of the problems is to provide a novel neural network model.

[0008] One aspect of the present invention is to provide a learning method for a neural network model that can remove noise included in input data. Or, one of the problems is to provide a learning method for a neural network model that can perform processing with high accuracy on input data. Or, one of the problems is to provide a learning method for a novel neural network model. Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will be naturally revealed from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc.

Means for Solving the Problems

[0009] Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will be naturally revealed from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc. Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will be naturally revealed from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc. Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will be naturally revealed from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc. Note that the description of these problems does not prevent the existence of other problems. Note that one aspect of the present invention does not need to solve all of these problems. Note that other problems will be naturally revealed from the descriptions in the specification, drawings, claims, etc., and it is possible to extract these other problems from the descriptions in the specification, drawings, claims, etc.

Means for Solving the Problems

[0010] One aspect of the present invention is a neural network model having a first neural network and a second neural network, and a learning method therefor. The first neural network has a layer skip-connected via the second neural network. The second neural network has a function of removing noise from the input feature map. Thereby, even when the data input to the first neural network includes noise, it is possible to suppress the transmission of the noise to a layer close to the output layer of the first neural network. The first neural network has a layer skip-connected via the second neural network. The second neural network has a function of removing noise from the input feature map. Thereby, even when the data input to the first neural network includes noise, it is possible to suppress the transmission of the noise to a layer close to the output layer of the first neural network. When learning the neural network model according to one aspect of the present invention, first, the first neural network... When learning the neural network model according to one aspect of the present invention, first, the first neural network... When learning the neural network model according to one aspect of the present invention, first, the first neural network...

[0011] When learning the neural network model according to one aspect of the present invention, first, the first neural network... We obtain the initial values ​​of the weight coefficients of the model network. Next, we perform the first training and the second training. Obtain the weight coefficients of the neural network. Then, perform a second training and the first neural network. Obtain the weight coefficients of the multi-network. After performing the first and second training, the final output is obtained. This evaluates the inference accuracy of one embodiment of a neural network model. Continue with the first and second learning sessions until you reach the above level.

[0012] One aspect of the present invention comprises a first neural network and a second neural network The first neural network has a first layer, a second layer, a third layer, and The feature map output from the first layer is used in the second layer and the second neural network. The feature map that is input to and output from the second neural network is the third The data is input to the first layer, and the second neural network processes the first data into the first neural network. The feature map output from the first layer when input to the twerk is called the ground truth feature map. When a second set of data, with noise added to the original data, is input into the first neural network... The feature map output from the first layer is used as the trained feature map, and the second neural network When a learning feature map is input to the first neural network, the second neural network outputs the feature map. The feature map was trained through the first learning process so that it matches the ground truth feature map. It is a neural network model.

[0013] Alternatively, in the above embodiment, the weight coefficients of the second neural network are obtained using initial value acquisition data The data is input to the first neural network and the weights of the first neural network are calculated. After obtaining the initial values ​​of the numbers, they may also be those obtained through the first learning process.

[0014] Alternatively, in the above embodiment, the weight coefficients of the first neural network are the weight coefficients of the second neural network After obtaining the weight coefficients of the neural network, the second data is fed into the first neural network. It may also be obtained through a second learning process performed by inputting data into the system.

[0015] Alternatively, in the above embodiment, the first neural network comprises a fourth layer, a fifth layer, The sixth layer has, and the feature map output from the fourth layer has, the fifth layer and the fourth layer and The sixth layer, which is connected by a ticket, is input to the fourth layer, which is the first new layer from the first layer. The third layer is close to the output layer of the neural network, and is closer to the sixth layer than the first neural network. It can be located close to the output layer of the amp.

[0016] Alternatively, one aspect of the present invention comprises a first neural network and a second neural network The first neural network has a first layer, a second layer, and a third layer. It has a fourth layer and a first layer in the order of the fourth layer, third layer, second layer, and first layer. The feature map, which is close to the output layer of the multi-network, is output from the first layer and the second layer , the second neural network, and the feature map that is input and output from the third layer, The feature maps output from the second neural network are input to the fourth layer. The second neural network inputs the data from the first neural network. If this is the case, the feature map output from the first layer is called the ground truth feature map, and the first data is called noise. When the second data with the added information is input to the first neural network, from the first layer... The output feature map is used as the trained feature map, and the trained features are transferred to the second neural network. When a map is input, the feature map output from the second neural network is correct. The neural network trained in the first training is set to match the solution feature map. This is a twerk model.

[0017] Alternatively, in the above embodiment, the weight coefficients of the second neural network are obtained using initial value acquisition data The data is input to the first neural network and the weights of the first neural network are calculated. After obtaining the initial values ​​of the numbers, they may also be those obtained through the first learning process.

[0018] Alternatively, in the above embodiment, the weight coefficients of the first neural network are the weight coefficients of the second neural network After obtaining the weight coefficients of the neural network, the second data is fed into the first neural network. It may also be obtained through a second learning process performed by inputting data into the system.

[0019] Alternatively, in the above embodiment, the first neural network model includes a fifth layer and a sixth layer It has a 7th layer and a 6th layer, and the feature map output from the 5th layer is the 6th layer and the 5th The seventh layer is connected to the layer and skipped, and the fifth layer is input to the first layer from the first layer. The fourth layer is closer to the output layer of the neural network than the seventh layer. It can be located close to the output layer of the network.

[0020] Alternatively, one aspect of the present invention comprises a first neural network and a second neural network The first neural network has a first layer, a second layer, and a third layer. The first layer has a feature map output from the second layer and a second neural network. The twerk and the feature map output from the second neural network are, A method for learning a neural network model input to the third layer, for obtaining initial values. By inputting the data into the first neural network, the first neural network The first step is to obtain the initial values ​​of the weight coefficients of the workpiece, and the first data is used in the first neural network. The feature map output from the first layer when input into a network is called the ground truth feature map. The second data, which has noise added to the first data, is input into the first neural network. In this case, the feature map output from the first layer is used as the learned feature map, and the second neural network When a learned feature map is input to the network, the output from the second neural network is... By performing the first learning process so that the resulting feature map matches the correct feature map, the second learning process is performed. The second step is to obtain the weight coefficients of the neural network, and the second data is used in the first step By inputting this into the neural network and performing a second learning process, the first neural network The third step is to obtain the network weight coefficients and then apply them to the first neural network. Based on input data and output data from the first neural network, , a fourth step to evaluate the inference accuracy of the neural network model, and The neural network model performs steps 2 through 4 until the accuracy exceeds the specified value. This is Dell's learning method.

[0021] Alternatively, in the above embodiment, the first neural network comprises a fourth layer, a fifth layer, The sixth layer has, and the feature map output from the fourth layer has, the fifth layer and the fourth layer and The sixth layer, which is connected by a ticket, is input to the fourth layer, which is the first new layer from the first layer. The third layer is close to the output layer of the neural network, and is closer to the sixth layer than the first neural network. It can be located close to the output layer of the amp. [Effects of the Invention]

[0022] According to one aspect of the present invention, a neural network can remove noise contained in the input data. It can provide a network model, or process input data with high precision. We can provide a neural network model that can perform this task. Alternatively, a new It can provide a neural network model.

[0023] According to one aspect of the present invention, a neural network can remove noise contained in the input data. We can provide a method for training network models, or a method that provides high performance relative to the input data. To provide a method for training neural network models that can perform processing with high accuracy. This can be done. Or, it can provide a novel method for training neural network models. ru.

[0024] Furthermore, the effects of one embodiment of the present invention are not limited to the effects listed above. This does not preclude the existence of other effects. These other effects are described in the following section. This is an effect not mentioned in the description. Effects not mentioned in this section can be understood by those skilled in the art through the details. This information can be derived from descriptions in documents, drawings, etc., and can be extracted appropriately from these descriptions. Furthermore, one aspect of the present invention includes at least the effects listed above and / or other effects. It also has another effect. Therefore, one aspect of the present invention may, in some cases, be described above It may not always have the effects listed. [Brief explanation of the drawing]

[0025] [Figure 1] Figures 1A and 1B show examples of neural network model configurations. [Figure 2] Figure 2 is a flowchart illustrating an example of a neural network model training method. [Figure 3] Figures 3A and 3B illustrate an example of a neural network model training method. [Figure 4] Figures 4A, 4B1, 4B2, and 4C illustrate an example of a neural network model training method. [Figure 5] Figure 5 shows an example of a neural network model training method. [Figure 6] Figure 6 shows an example of a neural network model training method. [Figure 7] Figure 7 shows an example of a neural network model configuration. [Figure 8] Figures 8A and 8B are block diagrams showing examples of the configuration of a computing system. [Figure 9] Figure 9 is a block diagram showing an example of the configuration of the imaging unit. [Figure 10] Figure 10 shows an example configuration of the pixel block 200 and circuit 201. [Figure 11] Figure 11 shows an example of a pixel configuration. [Figure 12] Figures 12A to 12C show filters. [Figure 13] Figure 13A shows an example of a pixel configuration. Figures 13B to 13D show examples of photoelectric conversion device configurations. [Figure 14] Figure 14 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 15] Figures 15A to 15C are cross-sectional views showing examples of transistor configurations. [Figure 16] Figure 16 is a cross-sectional view showing an example of the configuration of an imaging device. [Figure 17]Figure 17 is a cross-sectional view showing an example of the configuration of an imaging device. [Modes for carrying out the invention]

[0026] Embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description. Without departing from the spirit and scope of the present invention, its form and details may be modified in various ways. It will be easily understood by those skilled in the art. Therefore, the present invention is in the form shown below. The interpretation is not limited to the description of the state. Furthermore, the structure of the invention described below... Furthermore, the same reference numeral is used across different drawings for parts that are identical or have similar functions. The explanation of its use and repetition may be omitted. Note that the hatching of the same elements that make up the figure is also omitted. The drawing may be omitted or modified as appropriate between different drawings.

[0027] Furthermore, the ordinal numbers "1st," "2nd," and "3rd" used in this specification, etc., refer to the mixed elements of the constituent elements. This was added to avoid ambiguity and does not limit the number.

[0028] Furthermore, even if it is shown as a single element in the circuit diagram, there may be functional inconveniences. If not, the element may consist of multiple units. For example, a transistor that acts as a switch. Multiple zistas may be connected in series or parallel. Also, the capacitor may be divided In some cases, the elements may be divided and placed in multiple locations.

[0029] Furthermore, when a single conductor has multiple functions such as wiring, electrodes, and terminals. In this specification, multiple names may be used for the same element. Even if the elements are shown to be directly connected in the circuit diagram, in reality In this specification, the elements may be connected via multiple conductors, and this is how it is described. Even in such configurations, it falls under the category of direct connection.

[0030] (Embodiment) In this embodiment, a neural network model according to one aspect of the present invention, and the neural This section explains an example of a network model training method.

[0031] <Neural Network Model_1> Figure 1A shows a neural network model according to one aspect of the present invention. This is a diagram showing an example configuration of the neural network model 10. The neural network model 10 is a neural network. It has a neural network NN1 and a neural network NN2.

[0032] The neural network NN1 has multiple layers composed of neurons, and each layer The neurons located there are connected to each other. Therefore, a neural network N N1 can be described as a hierarchical neural network. Each layer in the structure extracts features from the data input to that layer and displays those features. It has the function to output the data as a feature map. Option 2 can also be a hierarchical neural network.

[0033] Figure 1A shows the layers of the neural network NN1, specifically layer L_1, layer L_2, and Layer L_3 is shown. The data flow between layers is indicated by arrows.

[0034] In this specification, when the same reference numeral is used for multiple elements, there is no particular need to distinguish them. In some cases, the code may be written with an additional identifier such as "_1" or "[1]". Yes. For example, in Figure 1A, the three layers L are called layer L_1, layer L_2, and layer L_ It is distinguished by being labeled with "3".

[0035] A neural network has an input layer and an output layer, with an intermediate layer between the input and output layers. It is provided. Multiple hidden layers can be provided in a single neural network. Layers L_1, L_2, and L_3 shown in Figure 1A can be intermediate layers. Layer L_1 may be an input layer, and layer L_3 may be an output layer.

[0036] In this specification, inputting data into the input layer of a neural network is referred to as "nu There are cases where data is input into a neural network. Data is output from the output layer of the neural network. There are cases where this happens.

[0037] The feature map output from layer L_1 is input to layer L_2, and the feature map output from layer L_2 The characteristic map is input to layer L_3. In other words, the data is entered in the order of layer L_1, layer L_2, and layer L_3. The signal is transmitted. Therefore, the layer after layer L_1 is layer L_2, and the layer after layer L_2 is layer L It can be said that it is _3.

[0038] Furthermore, the feature map output from layer L_1 is also input to the neural network NN2. Specifically, the neural network NN2 has an input layer, a hidden layer, and an output layer. If it is a hierarchical neural network having, then the neural network NN2 The feature map output from layer L_1 is input to the input layer. Then, the neural network is formed. The feature map output from the NN2 workpiece is input to layer L_3.

[0039] Here, the neural network NN2 receives input to the neural network NN2. It has the function of removing noise contained in the feature map. Therefore, layer L_3 has the same function as layer L_1. The data is input to the neural network NN2, and the neural network NN2 performs the following actions It can be said that a feature map with noise removed is input. Therefore, in layer L_3 The feature map output from layer L_1 is input via the neural network NN2. It can be said that it is done.

[0040] As mentioned above, the layer following layer L_1 is layer L_2. On the other hand, the output from layer L_1 is, The feature map input to the neural network NN2 is used by the neural network NN2 It is input to layer L_3 via [this]. In other words, it is output from layer L_1 and the neural network The feature map input to NN2 skips layer L_2, which is the next layer after layer L_1. This is input to layer L_3. Therefore, layer L_1 and layer L_3 are part of the neural network N. It can be said that it is skipped via N2. Also, layer L_3 is connected to layer L_1 The feature map output from layer L_2 is combined with the feature map output from layer L_2. It can be described as a layer.

[0041] In this specification, layers that are connected by a skip connection are indicated by a dashed line. For example, in Figure 1A, layers L_1 and L_3 are connected via the neural network NN2. The KIP connection is between layer L_1 and the neural network NN2, and the neural This is shown by connecting the NN2 network and layer L_3 with dashed lines.

[0042] The neural network NN1 processes the data input to the input layer and then processes the output layer to determine It has the function to output the inference results as a feature map. For example, a neural network. When image data is input to the input layer of NN1, the neural network NN1 will process the image It has the function of performing segmentation or depth estimation on image data. Input black and white or grayscale image data into the input layer of the NN1 web network. In this case, the neural network NN1 performs a function to colorize the image data. It holds.

[0043] In this specification, the feature map output from the output layer is used in the neural network having the output layer. Sometimes it refers to output data generated from a network.

[0044] In a neural network with skip-connected layers, the neural network If the data input to the workpiece contains noise, that noise will propagate to layers closer to the output layer. It becomes easier to reach. This makes it possible to perform inferences on the data with high accuracy. There is a possibility that it will become less effective. On the other hand, the neural network model 10 has The network NN1, via the neural network NN2 which has the function of denoising, It has skipped layers. This allows the neural network NN1 to receive input. Even if the data contains noise, that noise is transmitted to layers close to the output layer. This can suppress the process of making inferences about the data with high accuracy. It is possible.

[0045] In the neural network model 10 shown in Figure 1A, the neural network NN2 is Layer L is provided between layer L_1 and layer L_3, which are layers L connected via a skip connection. Although the present invention is described as a layer, it is not limited to this. As shown in Figure 1B, layer L_1 and layer L_3 A layer L_2 of m layers (where m is an integer greater than or equal to 2) may be placed between them. In Figure 1B, the m layer L_2 is ordered from the layer closest to the input layer of the neural network NN1, starting with layer L_2[1] to It is distinguished by being written as layer L_2[m].

[0046] In the neural network model 10 shown in Figure 1B, the feature matrix output from layer L_1 is... The input is passed to layer L_2[1] and the neural network NN2. _3 contains the feature map output from layer L_2[m] and the neural network NN2 The feature map output from and are input.

[0047] The neural network NN1 in neural network model 10 is, for example, a tatami mat. Convolutional Neural Network (CNN) It can be a neural network as shown in Figure 1B. If it is a CNN, then layers L_1 and L_2[m] are, for example, pooling It can be made into layers.

[0048] The neural network NN2 can be used as an autoencoder, for example. Generative Adversarial Networks (GANs) (etworks) can be used. For example, it can be called a Conditional GAN. If we consider NN2 as a GAN, then inference using the neural network NN2, for example, To perform high-precision noise reduction on feature maps input to a NN2 network. This is preferable because it allows for this.

[0049] <Learning Method> Next, using Figures 2 to 6, we will explain an example of a learning method for neural network model 10. To clarify, Figure 2 shows a flowchart illustrating an example of how neural network model 10 is trained. Figures 3 to 6 are schematic diagrams showing an example of the process in each step shown in Figure 2. Yes. In Figures 3 to 6, the neural network model 10 is shown in the structure shown in Figure 1A. It is stated that it is a success. Furthermore, in Figures 3 to 6, a neural network N is shown as an example. When image data is input to N1, segmentation is performed on that image data. A neural network model for giving functionality to the neural network NN1. Here are 10 examples of learning methods.

[0050] [Step S1] First, the neural network NN1 is input with data for acquiring initial values, and the neural network Obtain the initial value of weight coefficient 11 of the twerk NN1, which is weight coefficient 11_1 (step S1). Figures 3A and 3B are schematic diagrams showing an example of the process in step S1. Specifically, Figure 3A is a schematic diagram showing an example of the process in step S1_1, and Figure 3 B is a schematic diagram showing an example of the process in step S1_2.

[0051] As shown in Figure 3A, in step S1_1, data 21 is prepared. Data 21 is For example, it can be represented by a matrix. Here, the elements of the matrix are, for example, the data 21 acquired during imaging. When image data is obtained using this method, the illuminance of the light shining on the pixels of the imaging device is expressed. It can be set to a grayscale value. Also, there is a display device that displays the image corresponding to data 21. This can be used as a grayscale value representing the brightness of the light emitted from the pixel. In other words, data 21 The number of rows and columns in the matrix representing the data shall be the same as the resolution of the image corresponding to data 21, for example. It is possible to do so. Furthermore, data other than data 21 can also be represented by matrices in some cases. For example, it can be represented by a matrix with the same number of rows and columns as data 21. Yes, it can be represented by a matrix with fewer rows and fewer columns than data 21. There are cases where this is the case.

[0052] Data 22 is obtained by adding data that simulates noise to data 21. For example, New The data input to the NN1 network is obtained from imaging by a camera. If we consider image data, the noise could be, for example, a scratch on the camera lens or something attached to the camera. This can be assumed to be a water droplet. Alternatively, it can be assumed to be a shock that occurs when imaging in a dark place. It can be assumed that this is a net noise. Also, the neural network NN1 The input data is an image acquired by capturing an image of the display device. When considering image data, the above noise assumes defects such as point defects and line defects. This can be done. The data to be added to data 21 is a neural network NN. If the data you want to remove using method 2, it doesn't necessarily have to be data that is assumed to contain noise.

[0053] In the method described above, in step S1_1, data 22 is created from data 21. However, the present invention is not limited to this. For example, data 21 can be created from data 22. This is also acceptable. In this case, for example, if imaging is performed in a rainy environment to acquire data 22, then data 2 By performing image processing on 2 to remove water droplets, data 21 can be obtained. Cut.

[0054] Next, as shown in Figure 3B, in step S1_2, data 22 and data 23 are Input to the neural network NN1. Here, data 23 is, for example, the sequence of data 22. The result of the annotation can be data that represents the desired item. The initial value of the weight coefficient 11 of the neural network NN1 is weight coefficient 11_1. It can be obtained. For example, when data 22 is input to the neural network NN1 The error between the output data from neural network NN1 and data 23. The neural network NN1 obtains the weight coefficient 11_1 that minimizes [the specified value]. It is possible.

[0055] Note that in step S1_1 shown in Figure 3A, one data 21 is prepared, but data 2 You may prepare two or more 1s. Also, in step S1_2 shown in Figure 3B, data 22 and Each of the data 23 is input into the neural network NN1, but the data You can input two or more data points each of 22 and 23 into the neural network NN1. Also, in step S1_2, data 23 is input into the neural network NN1. It is not necessary to force it. For example, the training of the neural network NN1, which will be described later, is unsupervised. If it is done by learning, you do not need to input data 23 into the neural network NN1. That's good too.

[0056] When the process in step S1 is performed according to the procedure shown in Figures 3A and 3B, data 22 However, it can be said that this is data for obtaining initial values. Also, data 22 and data 23 and Both can be considered data used to obtain initial values.

[0057] At the stage of performing step S1, the neural network NN2 has not yet obtained the weight coefficients. Therefore, in step S1, processing by the neural network NN2 is not performed. I can't.

[0058] [Step S2] Next, the neural network model 10 performs its first learning process, Obtain the weight coefficient 12 of the network NN2 (step S2). Figures 4A, 4B1 and Figure Figures 4B2 and 4C are schematic diagrams illustrating an example of the process in step S2. Figure 4A is a schematic diagram showing an example of the process in step S2_1, and Figure 4B1 is Figure 4B2 is a schematic diagram showing an example of the process in step S2_2a, and step S Figure 4C is a schematic diagram showing an example of the process in 2_2b, and in step S2_3 This is a schematic diagram illustrating an example of the process.

[0059] In step S2, each layer of the neural network NN1 has a weight coefficient of 11 _i (where i is an integer greater than or equal to 1) is held. Step S2 is performed immediately after step S1. When performing this, i can be set to 1. Note that in Figures 4B1 and 4B2, layer L The weight coefficient 11_i held in _1 is shown as weight coefficient 11_i[1].

[0060] The details of step S2, which involves the first learning step, are described below. First, as shown in Figure 4A, In step S2_1, a dataset 25 containing multiple data points 24 is prepared, and Noi Data assuming the size is added to each data 24 to obtain data 26. Then, obtain a dataset 27 containing multiple data points 26. Add the following to data point 24. The noise is of the same type as the noise that can be added to the data 21 shown in Figure 3A. This is possible. Note that the data to be added to data 24 is the same as the data to be added to data 21. Similarly, if the data is to be removed by the neural network NN2, it is not necessarily the case that It doesn't have to be designed with the assumption of a specific type of character.

[0061] In the method described above, in step S2_1, data 26 is created from data 24. However, the present invention is not limited to this. The method for acquiring data 24 and data 26 is The method for obtaining data 21 and data 22 in step S1_1 is the same. Yes, it is possible. For example, if in step S1_1 data 21 is created from data 22. In step S2_1, data 24 is created from data 26. And data 26 can be obtained.

[0062] Next, as shown in Figure 4B1, in step S2_2a, the neural network N Input data 24 into N1 and obtain the feature map 34 output from layer L_1. For example, For each of the multiple data points 24 included in dataset 25, a feature map 34 is generated. Obtain it. This will obtain a feature mapset 35 that contains multiple feature maps 34. Furthermore, as shown in Figure 4B2, in step S2_2b, the neural network Input data 26 into NN1 and obtain the feature map 36 output from layer L_1. Example For example, for each of the multiple data points 26 included in dataset 27, feature map 3 6 is obtained. This gives a feature map set 37 which contains multiple feature maps 36. It's advantageous.

[0063] Subsequently, as shown in Figure 4C, in step S2_3, the feature map 36 is used as training data. Then, the feature map 34 is input into the neural network NN2 as ground truth data. This allows us to obtain the weight coefficient 12 of the neural network NN2. For example, When feature map 36 is input to neural network NN2, the neural network - The feature map output from NN2 corresponds to feature map 34 of the feature map 36. The neural network NN2 can obtain a weight coefficient 12 that matches this. Specifically, for example, by using gradient descent with an error function (also called a loss function), When feature map 36 is input to neural network NN2, the neural network N The error between the feature map output from N2 and the ground truth feature map 34 is minimized. A weight coefficient of 12 can be obtained. The error function is the sum of squared errors, crossover error. An tropy error can be used. In gradient descent, the error function is minimized. Obtain the weight coefficients. The minimum value can be found, for example, by differentiating the error function. Yes, it's possible. The derivative of the error function can be found, for example, by backpropagation.

[0064] Furthermore, when using the neural network NN2 as a Conditional GAN, Input the feature map 34 into the generator, and input the feature map 36 into the discriminator. This allows the neural network NN2 to perform training.

[0065] In this specification, a feature map that can be used as training data is referred to as a training feature map. A feature map that can be used as ground truth data is sometimes called a ground truth feature map. Feature map 36 can be called a learned feature map, and feature map 34 is a ground truth feature map. It can be said that...

[0066] [Step S3] Next, the neural network model 10 performs a second learning process, Obtain the weight coefficient 11 of the network NN1 (step S3). Figure 5 shows step S3. This is a schematic diagram showing an example of the process.

[0067] In step S3, each layer of the neural network NN1 has a weight coefficient of 11 _i is retained. As mentioned above, for example, each of the neural network NN1 has If the layer retains an initial value of weight coefficient 11, then i can be set to 1. The neural network NN2 retains the weight coefficient 12 obtained in the previous step S2. It is held. In Figure 5, the weight coefficient 11_i held in layer L_1 is the weight coefficient Denoted as 11_i[1], the weight coefficient 11_i held in layer L_2 is the weight coefficient 11_i[ [2] is written, and the weight coefficient 11_i held in layer L_3 is written as weight coefficient 11_i[3]. This is how it is shown.

[0068] In step S3, data 26 is used as training data and data 28 as ground truth data, The data is input to the multi-network NN1. Here, data 26 is input to step S shown in Figure 4A. The data obtained in 2_1 is then processed in step S2_2b shown in Figure 4B2, and the neural network It can be the same data as data 26 input to work NN1. In other words, In step S2, when training the neural network NN2, the learning feature map is The same data was input to the neural network NN1 to obtain feature map 36. This data will be used as training data for training the neural network NN1. This can be done. Also, data 28 is, for example, the result of segmentation of data 26. This can be used as data to represent what is desired. Here, the neural network NN1 The set of data 28 used as the correct answer data in the learning process will be referred to as dataset 29. In step S2_2b, the data 26 input to the neural network NN1 and In step S3, the data 26 input to the neural network NN1 is different. You may do so.

[0069] Based on the above, we can obtain the weight coefficient 11_i+1 of the neural network NN1. Specifically, by optimizing the weight coefficient 11_i, the weight coefficient 11_i+1 can be calculated. It can be obtained. For example, input data 26 into neural network NN1. In this case, the output data from the neural network NN1 corresponds to the data 26. A weight coefficient 11_i+1 that matches the corresponding data 28 is used in the neural network N N1 can be obtained. Specifically, for example, by gradient descent using an error function, When data 26 is input to neural network NN1, the neural network N The error between the output data from N1 and the correct data, data 28, is minimized. A weight coefficient 11_i+1 can be obtained.

[0070] As mentioned above, the neural network NN2 uses the weight obtained in the previous step S2. The coefficient 12 is retained. Therefore, the data input to the neural network NN1 For step 26, processing is also performed by the neural network NN2. Therefore, step In step S3, the weight coefficient 11_i+1 obtained by the neural network NN1 is, This can be made to reflect the weight coefficient 12 held in the NN2 network. It is possible. Note that in step S3, the weight coefficient 12 is not obtained, that is, in the preceding step S2 Although it was explained that the optimization of the acquired weight coefficient 12 would not be performed in step S3, The manifestation is not limited to this. For example, in step S3, the weight coefficient 11 and the weight coefficient Both of the numbers 1 and 2 may be optimized.

[0071] Furthermore, Figure 5 shows the case where the neural network NN1 is trained using supervised learning. As shown, the neural network NN1 can also be trained using unsupervised learning. When training a neural network NN1 using unsupervised learning, the correct data is Data 28 does not need to be input into the neural network NN1.

[0072] [Step S4] Next, input data 41 as test data into the neural network NN1, Based on the output data 43 from the neural network NN1, the neural network Evaluate the inference accuracy of model 10 (step S4). Figure 6 shows the process in step S4. This is a schematic diagram illustrating an example of the principle.

[0073] For example, the output data 43 can be used as the desired result of segmentation of data 41. By comparing it with the represented data 45, the neural network is generated based on the difference between output data 43 and data 45. Evaluate the inference accuracy of the work model 10. For example, if output data 43 and data 45 have a row count and Assuming that the output data 43 is represented by matrices where both the number of columns and the number of rows are equal to each other, the output data 43 is constructed Based on the difference between each element of the matrix and each element of the matrix that makes up data 45, the neural network Evaluate the inference accuracy of network model 10.

[0074] For example, output data 43 and data 45 are p rows and q columns (where p and q are integers greater than or equal to 1) Assuming it is represented by a matrix, first, the element in the first row and first column of the output data 43, and Determine whether the element in the first row and first column of Ta45 matches. Similarly, check all elements. Then, it is determined whether output data 43 and data 45 match. The number of elements present is divided by the total number of elements in the matrix (p × q), and this is taken as the accuracy rate. The inference accuracy of neural network model 10 is evaluated based on the rate. The higher the accuracy rate, The inference accuracy can be evaluated as high.

[0075] Alternatively, the inference accuracy of the neural network model 10 may be evaluated based on the error function. Furthermore, segmentation is performed on the data input to the neural network NN1. The neural network NN1 will be given the function to perform this task. When training Model 10, the inference accuracy of the neural network Model 10 is mIo Evaluate using U (mean Intersection over Union). It is possible.

[0076] Alternatively, for example, output data 43 and data 45 are each represented by a p x q matrix. Assuming this is the case, first, the element in the first row and first column of output data 43 and the element in the first row and first column of data 45 Calculate the absolute difference between the elements of and . Similarly, for all elements, output data 43 and Calculate the absolute value of the difference between data 45 and the result. Then, calculate the absolute value of the p × q differences calculated above. The total is calculated, and the inference accuracy of the neural network model 10 is evaluated based on the total value. The smaller the total value, the smaller the error between output data 43 and data 45, i.e., the smaller the inference. This method can be evaluated as having high accuracy. For example, the neural network NN1 The neural network NN1 has a function to perform depth estimation on the input data. This method is suitable for use when training the neural network model 10. can.

[0077] Alternatively, for example, for a black and white image data input to a neural network NN1, color To give the neural network NN1 the function to perform -ization, When training the 10th model, for example, the inference accuracy can be evaluated based on the color space. For example, by comparing the HSV color spaces of output data 43 and data 45, The inference accuracy of the neural network model 10 can be evaluated. Here, HSV color Space refers to the numerical representation of the hue, saturation, and brightness of an image data. The inference accuracy may be evaluated using the RGB color space, CMY color space, or HLS color space, etc. .

[0078] Note that multiple copies of the test data, Data 41, can be prepared. In this case, Data 4 Let the set of 1s be dataset 42. Also, if multiple data 41s are prepared, output data 4 Multiple copies of data 3 and data 45 will be prepared. For example, the same number of output data 4 as data 41. 3. Prepare the same number of data 45 as data 41. Output data 43 And if multiple data 45s are prepared, the set of output data 43 will be set to dataset 44, and The set of data 45 is called data 46. A neural network is constructed using multiple data 41. When evaluating the inference accuracy of Model 10, for example, each of the multiple data 41 is nucleotides The data is input into the NN1 network, and the inference accuracy is calculated for each of the 41 data points. Then, the average inference accuracy for each data point 41 is used in the next step. This can be used to determine the inference accuracy of the neural network model 10.

[0079] [Step S5] Subsequently, the inference accuracy of the neural network model 10, which was evaluated in step S4, was... Determine whether the degree is above the specified value (step S5). If it is above the specified value, End training of the neural network model 10. If i is below the specified value, set i to 1. In addition, steps S2 through S5 are repeated. That is, the inference accuracy is greater than or equal to the specified value. Up to the point of optimizing the weight coefficient 12 through the first learning process, and optimizing the weight coefficient 11 through the second learning process. Optimization is performed. The above is one example of a learning method for neural network model 10. Note that, In step S5, the increase in inference accuracy resulting from the first and second learning processes is evaluated. The training of the neural network model 10 is terminated when the increase falls below a specified value. It may be allowed to complete. For example, the first learning and the second learning can each be performed j times (where j is 1 or more). The difference in inference accuracy between performing the operation a certain number of times and performing it j+1 times is evaluated, and the difference is... The training of neural network model 10 may be terminated if the value falls below a specified level. .

[0080] <Neural Network Model_2> Figure 7 shows a neural network model according to one aspect of the present invention. This is a diagram showing an example configuration of model 10a. The neural network model 10a is a neural network. It has a neural network NN1a and a neural network NN2a. The neural network NN1a has the same functionality as the neural network NN1, The neural network NN2a has the same functionality as the neural network NN2. Furthermore, neural network model 10a is the same as neural network model 10. You can learn by following the instructions provided.

[0081] In this specification, when referring to neural network model 10, it means neural network Network model 10a may be included. Also, neural network NN1 and In such cases, the neural network NN1a may be included, and the neural network The term "NN2" sometimes includes the neural network NN2a.

[0082] The neural network NN1a consists of an input layer IL and an intermediate layer ML_1 to intermediate layer ML_1 7 and the output layer OL. The architecture of the neural network NN1a is, for example It can be called U-Net. Alternatively, it could be called DenseNet, for example.

[0083] In the neural network NN1a with the configuration shown in Figure 7, the features output from the input layer IL are... The map is input to the hidden layer ML_1, and the feature map output from the hidden layer ML_1 is... It is input to the hidden layer ML_2. In this way, the data input to the input layer IL is input to the hidden layer ML It is transmitted sequentially from _1 to the intermediate layer ML_17. Then, it is output from the intermediate layer ML_17. The feature map is input to the output layer OL. Therefore, the intermediate layer ML_1 to the intermediate layer ML_1 Among the 7 layers, the hidden layer ML_1 is closest to the input layer IL, and the hidden layers ML_2 through the hidden layer ML_ The order from 16 is from the input layer IL, meaning it moves closer to the output layer OL, with the hidden layer ML_17 being the furthest away. It can also be said that it is close to the output layer OL.

[0084] The feature map output from the hidden layer ML_1 is not only from the hidden layer ML_2, but also from the neural network It is also input to the neural network NN2a. The characteristic map is input to the hidden layer ML_16. In other words, the hidden layer ML_16 contains the hidden layer M In addition to the feature map output from L_15, the neural network NN2a also outputs... A feature map is input. Therefore, the hidden layer ML_1 and hidden layer ML_16 are neural It can be said that the connection is skipped via the NN2a network. The intermediate layer ML_16 combines the feature map output from the hidden layer ML_15 with the neural network It has the function of a coupling layer that combines the feature maps output from NN2a. It is possible.

[0085] The feature map output from the hidden layer ML_3 is not only from the hidden layer ML_4, but also from the hidden layer ML_ It is also input to 14. In other words, the output from the hidden layer ML_13 is sent to the hidden layer ML_14. In addition to the feature map, the feature map output from the intermediate layer ML_3 is also input. Layer ML_3 and the intermediate layer ML_14 can be said to be connected via a skip connection. The hidden layer ML_14 uses the feature map output from the hidden layer ML_13 and the hidden layer ML_3 It can be said that it functions as a coupling layer, combining the feature maps output from it. .

[0086] The feature map output from the hidden layer ML_5 is not only from the hidden layer ML_6, but also from the hidden layer ML_ It is also input to 12. In other words, the output from the hidden layer ML_11 is sent to the hidden layer ML_12. In addition to the feature map, the feature map output from the intermediate layer ML_5 is also input. Layer ML_5 and the intermediate layer ML_12 can be said to be connected via a skip connection. The hidden layer ML_12 uses the feature map output from the hidden layer ML_11 and the hidden layer ML_5 It can be said that it functions as a coupling layer, combining the feature maps output from it. .

[0087] The feature map output from the hidden layer ML_7 is not only from the hidden layer ML_8, but also from the hidden layer ML_ It is also input to 10. In other words, the special output from the hidden layer ML_9 is input to the hidden layer ML_10. In addition to the feature map, the feature map output from the intermediate layer ML_7 is input. Therefore, the intermediate layer It can be said that ML_7 and the hidden layer ML_10 are connected via a skip connection. Hidden layer ML_10 uses the feature map output from hidden layer ML_9 and from hidden layer ML_7. It can be said that it functions as a coupling layer, combining the output feature maps.

[0088] Based on the above, the neural network NN1a shown in Figure 7 has four sets of skipped layers. It can be said that it has. Also, among the skip-connected intermediate layer MLs, the most input Hidden layer ML_1, which is closest to layer IL, and hidden layer ML_16, which is closest to the output layer OL, and neural It can be said that the connection is skipped via the NN2a network.

[0089] For example, in U-Net, feature maps output from hidden layers close to the input layer are skipped. Therefore, the data is input to the intermediate layer, which is closer to the output layer. As a result, noise is introduced into the data input to the input layer. If present, the noise is more likely to be transmitted to layers closer to the output layer. This could potentially prevent the ability to perform inferences on data with high accuracy. On the other hand, for example, Figure The neural network NN1a with the configuration shown in 7 has skipped hidden layers ML Of these, the hidden layer ML_1 closest to the input layer IL is the same as the hidden layer ML_1 closest to the output layer OL. 6. Skip via the neural network NN2a, which has a noise reduction function. It is connected. This means that if the data input to the input layer IL contains noise... However, this can suppress the transmission of the noise to layers close to the output layer (OL). As a result, the neural network NN1a can perform inferences on the data with high accuracy. It is possible to do so.

[0090] Furthermore, as feature extraction is performed by the hidden layer ML, the data input to the input layer IL includes... The effect of noise is reduced. For example, if data containing noise is input to the input layer IL In this case, the effect of the noise on the feature map output by the hidden layer ML_7 is as follows: The impact of this noise on the feature map output by 1 becomes smaller. Therefore, the input layer IL Feature maps output from a distant intermediate layer ML are connected to other intermediate layer MLs via skip connections. Even if the feature map is input, it is not necessary to perform noise reduction on it. For example, layers that are further from the input layer IL than hidden layer ML_1, such as hidden layer ML_3 and hidden layer ML _5, and the intermediate layer ML_7 are the intermediate layers ML_14, ML_12, and intermediate layers, respectively. Layer ML_10 and a neural network with noise reduction capabilities are used to create the desired result. It can be connected via skip connections. Therefore, even if there are many layers that are skipped, the neural While suppressing the scaling of the network model 10, the data input to the input layer IL This allows us to maintain the accuracy of our reasoning.

[0091] In Figure 7, the neural network NN1a has 17 hidden layers ML, and also skip connections. Although the present invention shows a configuration having four sets of consecutive layers, the present invention is not limited to this. For example, The number of hidden layers ML may be more than 17. Also, in Figure 7, hidden layer ML_1 and hidden layer M Only L_16 is skipped through a neural network that has the function of removing noise. The present invention is shown as a connected configuration, but is not limited to this. For example, an intermediate layer M In addition to L_1 and the intermediate layer ML_16, the intermediate layers ML_3 and ML_14 remove noise. The configuration may also involve skip connections via a neural network that has the necessary functionality. Layers that are skipped through a neural network with noise reduction capabilities. By increasing the number of elements, inference on the data input to the input layer (IL) can be performed with higher accuracy. It is possible.

[0092] <Example of a arithmetic processing system configuration> One aspect of the present invention is a neural network model that can be applied, for example, to a computing system. This is possible. Figure 8A shows the application of a neural network model according to one embodiment of the present invention. This is a block diagram showing an example configuration of a computing system 100 that can perform the following. 100 has an information processing device 120.

[0093] The information processing device 120 includes an imaging unit 121, an output unit 122, a calculation unit 123, and a main memory unit 1 It has 24 and an auxiliary storage unit 125. Data between components of the information processing device 120 Data and other information can be transmitted via the transmission path 127. The imaging unit can be rephrased as an imaging device.

[0094] The imaging unit 121 has the function of taking images and acquiring image data. The image data is The data 21 shown in Figure 3A can be obtained. The output unit 122 has the function of supplying information. do.

[0095] The calculation unit 123 has the function of performing calculations. For example, the calculation unit 123 is connected to the imaging unit 121 The data is transmitted from the main memory unit 124, auxiliary memory unit 125, etc., to the arithmetic unit 123 via the transmission line 127. It has the function of performing predetermined calculations on the data. The calculation unit 123 is, for example, CP U (Central Processing Unit), and GPU (Graphics) It may have a Processing Unit, etc.

[0096] The main memory unit 124 has the function of storing data and programs, etc. The arithmetic unit 123 is The system reads data and programs stored in the main memory unit 124 and executes arithmetic processing. For example, the arithmetic unit 123 reads a program from the main memory unit 124. By executing this, predetermined arithmetic processing is performed on the data read from the main memory unit 124. It can be executed.

[0097] The main memory unit 124 preferably operates at a higher speed than the auxiliary memory unit 125. 4 is, for example, DRAM (Dynamic Random Access Memory) , possessing SRAM (Static Random Access Memory), etc. It is possible.

[0098] The auxiliary storage unit 125 stores data and programs for a longer period than the main storage unit 124. It has the function of an auxiliary storage unit 125, for example, an HDD (Hard Disk Drive). It can have, ), SSD (Solid State Drive), etc. The auxiliary storage unit 125 is ReRAM (Resistive Random Access Memory (also called resistive random-access memory), PRAM (Phase change R) andom Access Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresis (also known as magnetically resistive random access memory) It may also have a non-volatile memory such as a flash memory.

[0099] The weight coefficient 11 obtained by learning the neural network model according to one aspect of the present invention and the weight coefficient 12 are stored in the auxiliary storage unit 125. Note that the weight coefficient 11 and the weight coefficient 12 may be stored in the main storage unit 124.

[0100] The information processing apparatus 120 can be provided in an information terminal such as, for example, a smartphone, a tablet, or a personal computer.

[0101] Note that the configuration of the arithmetic processing system 100 is not limited to the above. For example, as shown in FIG. 8B, the arithmetic processing system 100 may have an information processing apparatus 110 in addition to the information processing apparatus 120.

[0102] The information processing apparatus 110 includes an imaging unit 111, an output unit 112, an arithmetic unit 113, a main storage unit 1 14, an auxiliary storage unit 115, and a communication unit 116. Transmission of data and the like between the components included in the information processing apparatus 110 can be performed via a transmission path 117.

[0103] The imaging unit 111 has a function of performing imaging and acquiring image data. The image data can be the data 21 shown in FIG. 3A. The output unit 112 has a function of supplying information.

[0104] The arithmetic unit 113 has a function of performing arithmetic processing. The arithmetic unit 113 has a function of performing predetermined arithmetic processing on the data transmitted to the arithmetic unit 113 via the transmission path 117 from, for example, the imaging unit 111, the main storage unit 114, the auxiliary storage unit 115, the communication unit 116, and the like. The arithmetic unit 113 ​​​​​​For example, it may have a CPU and a GPU.

[0105] The main memory unit 114 has the function of storing data and programs, etc. The arithmetic unit 113 is The system reads data and programs stored in the main memory unit 114 and executes arithmetic processing. For example, the arithmetic unit 113 reads a program from the main memory unit 114. By executing this, predetermined arithmetic processing is performed on the data read from the main memory unit 114. It can be executed.

[0106] The main memory unit 114 preferably operates at a higher speed than the auxiliary memory unit 115. 4 may include, for example, DRAM, SRAM, etc.

[0107] The auxiliary storage unit 115 stores data, programs, etc., for a longer period than the main storage unit 114. It has the function of having an auxiliary storage unit 115, for example, an HDD or an SSD. Furthermore, the auxiliary storage unit 115 includes ReRAM, PRAM, FeRAM, MRAM, or flash memory. It may have non-volatile memory such as volatile memory.

[0108] Weight coefficients 11 obtained by training a neural network model according to one aspect of the present invention, The weight coefficients 11 and 12 are stored in the auxiliary storage unit 115. Number 12 may be stored in the main memory unit 114.

[0109] The communication unit 116 transmits data to devices located outside the information processing device 110. It has a receiving function. Furthermore, the communication unit 116 is a device that supplies data to the network. It can have the ability to acquire data from a network, etc.

[0110] The information processing apparatus 120 includes an imaging unit 121, an output unit 122, an arithmetic unit 123, a main memory unit 1 24, an auxiliary storage unit 125, and a communication unit 126. Regarding the description of the information processing apparatus 120 , and the description of the components included in the information processing apparatus 120, the above description can be referred to .

[0111] The communication unit 126 has a function of transmitting and receiving data and the like to and from devices and the like provided outside the information processing apparatus 120 . Further, the communication unit 126 can have a function of supplying data and the like to a network , and a function of acquiring data and the like from the network

[0112] The arithmetic unit 123 has a function of performing predetermined arithmetic processing on the data transmitted to the arithmetic unit 123 via the transmission path 127 from, for example, the imaging unit 121, the main memory unit 124, the auxiliary storage unit 125, the communication unit 1 26, etc .

[0113] Here, when applying the neural network model of one aspect of the present invention to the arithmetic unit 113 and the arithmetic unit 123 , for example, the arithmetic unit 113 can perform learning to obtain the weight coefficients 11 and the weight coefficients 1 2 and supply them from the information processing apparatus 110 to the information processing apparatus 120. Thus , even if the arithmetic unit 123 provided in the information processing apparatus 120 does not perform learning, with respect to the data input to the arithmetic unit 1 23, arithmetic processing by the neural network model can be performed based on the weight coefficients 11 and the weight coefficients 12 obtained by the arithmetic unit 113 . Therefore, the arithmetic processing ability of the arithmetic unit 123 can be made lower than that of the arithmetic unit 113 .

[0114] The calculation unit 113 obtains the weight coefficients 11 and 12, and the information processing device 110 receives the information When supplying to the processing unit 120, the information processing unit 110 can be installed, for example, in a server. It is possible. Furthermore, if the information processing device 110 is installed in the server, the information processing device 110 will have an imaging unit. It is not necessary to provide the imaging unit 111 and the output unit 112. The device 2 may be located outside the information processing device 110.

[0115] Furthermore, as mentioned above, the information processing device 120 can be used for smartphones, tablets, personal computers, etc. It can be installed in information terminals such as personal computers. Also, the configuration of the information processing device 110 Both at least a portion of the element and at least a portion of the components of the information processing device 120 It may be installed on the server. For example, the arithmetic unit 113 and the arithmetic unit 123 may be installed on the server. Good. In this case, for example, the data acquired by the information terminal is transmitted to the processing unit 12 via the network. The data is supplied to unit 3, and the arithmetic unit 123 located in the server performs estimations and other operations on the data. Then, by supplying the estimation results to the information terminal via the network, the information terminal This allows us to obtain the estimation results.

[0116] The arithmetic processing system 100 has the configuration shown in Figure 8B, and the arithmetic unit 113 performs learning and weight assignment When acquiring the number 11 and the weight coefficient 12, the data 21 shown in Figure 3A is sent to the imaging unit 111. It is not necessary to acquire more data. For example, the imaging unit 121 performs imaging and the resulting image data 21 is The data is acquired, and the image data is supplied from the information processing device 120 to the information processing device 110. Alternatively, the imaging unit 111 may not be provided in the information processing device 110.

[0117] <Example of the configuration of the imaging unit> Figure 9 is a block diagram illustrating an example configuration of the imaging unit 111 and imaging unit 121. The image unit consists of a pixel array 300, a circuit 201, a circuit 301, a circuit 302, and a circuit 303 It also has circuits 304, 305, and 306. Note that circuits 201 and 30 Each of circuits 1 through 306 is not limited to a single circuit configuration, but can be configured as a combination of multiple circuits. This may be done. Alternatively, any or more of the above circuits may be integrated. Also, Other circuits may be connected. Also, circuits 201, 301, 302, and 3 Circuits 304, 305, and 306 are connected to the calculation unit 113 or calculation unit 123. It may be established.

[0118] The pixel array 300 has imaging and processing functions. Circuits 201 and 301 are, It has a calculation function. Circuit 302 has a calculation function or a data conversion function. Circuit 303, Circuits 304 and 306 have a selection function. Circuit 303 is divided via wiring 424. It is electrically connected to the base block 200. Circuit 304 is connected to the pixel block via wiring 423. It is electrically connected to the 200. Circuit 305 supplies potential to the pixels for multiply-accumulate operations. It has the function of selecting. Circuits with a selection function use a shift register or decoder, etc. This is possible. Circuit 306 is electrically connected to pixel block 200 via wiring 413. Circuits 301 and 302 may be located externally.

[0119] The pixel array 300 has a plurality of pixel blocks 200. The pixel blocks 200 are shown in Figure 1. As shown in 0, it has multiple pixels 400 arranged in a matrix, and each pixel 4 00 is electrically connected to circuit 201 via wiring 412. Note that circuit 201 is a pixel It can also be installed within block 200.

[0120] Furthermore, pixel 400 is connected to the adjacent pixel 400 and transistor 450 (transistor 450a The transistor 450 is electrically connected via transistor 450f. More details will follow.

[0121] At 400 pixels, image data is acquired and data is generated by adding the image data to a weight coefficient. This is possible. In Figure 10, as an example, the pixel block 200 has The example uses 3x3 prime numbers, but it's not limited to this. For example, it could be 2x2, 4x4, etc. Alternatively, the number of pixels in the horizontal and vertical directions may be different. Also, some pixels may be adjacent to each other. They can be shared among the corresponding pixel blocks.

[0122] The pixel block 200 and circuit 201 can be operated as a multiply-accumulate circuit.

[0123] Pixel 400, as shown in Figure 11, includes a photoelectric conversion device 401 and a transistor 402. , transistor 403, transistor 404, transistor 405, transistor It may have 406 and a capacitor 407.

[0124] One electrode of the photoelectric conversion device 401 is either the source or drain of the transistor 402. It is electrically connected to one side. The other side of the source or drain of transistor 402 is connected to the transistor Either the source or drain of transistor 403, the gate of transistor 404, and capacitor 4 It is electrically connected to one electrode of 07. One of the source or drain of transistor 404. One side is electrically connected to either the source or drain of transistor 405. The other electrode of transistor 407 is electrically connected to either the source or the drain of transistor 406. It will continue.

[0125] The other electrode of the photoelectric conversion device 401 is electrically connected to the wiring 414. The source or drain of terminal 403 is electrically connected to wiring 415. Transistor The source or drain of terminal 405 is electrically connected to wiring 412. Transistor The source or drain of the TA404 is electrically connected to the GND wiring, etc. The source or drain of station 406 is electrically connected to wiring 411. The other electrode of terminal 407 is electrically connected to wiring 417.

[0126] The gate of transistor 402 is electrically connected to wiring 421. Transistor 403 The gate of transistor 405 is electrically connected to wiring 422. The gate of transistor 405 is connected to wiring 4 It is electrically connected to 23. The gate of transistor 406 is electrically connected to wiring 424. It will be done.

[0127] Here, the other of the source or drain of transistor 402 and the source of transistor 403 The drain or one of the electrodes of capacitor 407 and the gate of transistor 404 The electrical connection point with the terminal is defined as node FD. Furthermore, the other electrode of capacitor 407 and the terminal are connected. The electrical connection point between either the source or drain of the transistor 406 is defined as node FDW. .

[0128] Wiring 414 and wiring 415 can function as power lines. For example, Line 414 can function as a high-potential power line, and wiring 415 can function as a low-potential power line. Wires 421, 422, 423, and 424 control the conduction of each transistor. It can function as a signal line. Wiring 411 corresponds to a weighting coefficient for pixel 400. It can function as wiring that supplies potential. Wiring 412 is connected to pixel 400 and It can function as wiring that electrically connects to path 201. Wiring 417 is said The other electrode of the capacitor 407 of one pixel and the other electrode of the capacitor 407 of another pixel It can function as wiring that is electrically connected via transistor 450 (Figure 10). reference).

[0129] Note that wiring 412 may also be electrically connected to an amplification circuit or a gain adjustment circuit. .

[0130] A photodiode can be used as the photoelectric conversion device 401. Regardless of the type of diode, Si photodiodes and organic photodiodes have silicon as their photoelectric conversion layer. Organic photodiodes and the like, which have an electrical film in their photoelectric conversion layer, can be used. If you want to increase the light detection sensitivity, it is preferable to use an avalanche photodiode. stomach.

[0131] Transistor 402 can have the function of controlling the potential of node FD. The ZISTA 403 can have the function of initializing the potential of node FD. Transition The 404 has the function of controlling the current that the circuit 201 flows according to the potential of node FD. Transistor 405 can have the function of selecting pixels. ZISTA 406 has the function of supplying a potential equivalent to the weighting coefficient to node FDW. can.

[0132] When an avalanche photodiode is used in the photoelectric conversion device 401, a high voltage is applied. In some cases, the transistor connected to the photoelectric conversion device 401 is a high-voltage transistor. It is preferable to use a zista. High-voltage transistors have, for example, a channel formation region. Transistors using metal oxides (hereinafter referred to as OS transistors) can be used. Specifically, it is preferable to apply an OS transistor to transistor 402.

[0133] Furthermore, OS transistors also possess the characteristic of extremely low off-current. Transistor 402, By using OS transistors for transistors 403 and 406 This allows for an extremely long period during which charge can be retained at nodes FD and FDW. Therefore, charge accumulation operation can be performed simultaneously at all pixels without complicating the circuit configuration or operating method. A global shutter method can be applied to perform this operation. Additionally, image data can be stored in the node FD. It is also possible to perform multiple calculations using the image data while retaining the data.

[0134] On the other hand, transistor 404 may sometimes be required to have superior amplification characteristics. Also, Transistor 406 is preferably a high-mobility transistor capable of high-speed operation. There are cases where this is difficult. Therefore, transistors 404 and 406 have a series Even if a transistor using a capacitor in the channel formation region (hereinafter referred to as a Si transistor) is applied... good.

[0135] Furthermore, the above is not limited to the application of any combination of OS transistors and Si transistors. You may do so. Alternatively, all transistors may be OS transistors. Or, all The transistor may be a Si transistor. As for the Si transistor, amorphous Transistors containing silicon, crystalline silicon (microcrystalline silicon, low-temperature polysilicon) Examples include transistors having (single-crystal silicon).

[0136] The potential of node FD at pixel 400 is determined by the reset potential supplied from wiring 415 and the optical potential. The potential (image data) generated by photoelectric conversion by the power conversion device 401 is added to the total potential. This is determined. Alternatively, the potential corresponding to the weighting coefficient supplied from wiring 411 is capacitively determined. The data is combined and finalized. Therefore, transistor 405 assigns an arbitrary weight to the image data. It can supply current according to the data that has been added.

[0137] Note that the above is just one example of a circuit configuration for 400 pixels; other circuit configurations are available for the photoelectric conversion operation. It can also be done this way.

[0138] As shown in Figure 10, each pixel 400 is electrically connected to one another by wiring 412. Circuit 2 01 performs calculations using the sum of the currents flowing through the transistors 404 of each pixel 400. can.

[0139] Circuit 201 includes capacitor 202, transistor 203, transistor 204, and It has a transistor 205, a transistor 206, and a resistor 207.

[0140] One electrode of capacitor 202 is connected to either the source or drain of transistor 203. They are electrically connected. Either the source or drain of transistor 203 is connected to transistor 2 It is electrically connected to the gate of 04. Either the source or the drain of transistor 204 is It is electrically connected to either the source or drain of transistor 205. Either the source or drain of transistor 205 is connected to either the source or drain of transistor 206. It is electrically connected to the other electrode of the capacitor 202. They are connected by air.

[0141] The other electrode of capacitor 202 is electrically connected to wiring 412. Transistor 20 The source or drain of transistor 3 is electrically connected to wiring 218. The source or drain of transistor 4 is electrically connected to wiring 219. The other end of the source or drain of 5 is electrically connected to a reference power line such as a GND wire. The source or drain of the transistor 206 is electrically connected to the wiring 212. The other electrode of electrode 207 is electrically connected to wiring 217.

[0142] Wires 217, 218, and 219 can function as power lines. For example, wiring 218 has the function of being wiring that supplies a dedicated potential for reading. Wiring 217 and wiring 219 can be made to function as high-potential power lines. Wires 213, 215, and 216 control the conduction of each transistor. It can function as a line. Wiring 212 is an output line, for example, in the circuit shown in Figure 9. It can be electrically connected to 301.

[0143] Transistor 203 has a function of resetting the potential of wiring 211 to the potential of wiring 218. It can. Wiring 211 is a wiring connected to one electrode of capacitor 202, one of the source or drain of transistor 203, and the gate of transistor 204. The transistors 204 and 205 can have a function as a source follower circuit. Transistor 206 can have a function of controlling reading. Note that circuit 201 has a function as a correlated double sampling circuit (CDS circuit), and can be replaced with a circuit of another configuration having the same function.

[0144] In one aspect of the present invention, an offset component other than the product of image data (X) and weight coefficient (W) is removed, and the target WX is extracted. WX can be calculated using data with and without imaging for the same pixel, and data when weights are added to each of them.

[0145] The sum of the currents (I ,

[0144] , , ,

[0145] , , , , 2 ) flowing through pixel 400 when imaging is performed is kΣ(X - V th ) 2 , and the sum of the currents (I ) flowing through pixel 400 when weights are added is kΣ(W + X - V p ) th and becomes 2 . Also, the sum of the currents (I ) flowing through pixel 400 when no imaging is performed is kΣ(0 - V ref ) th ), and the sum of the currents (I 2 ) flowing through pixel 400 when weights are added is kΣ(W - ref V V th ) 2 and becomes. Here, k is a constant, and V th is the threshold voltage of transistor 405. . ​

[0146] First, calculate the difference (Data A) between the data with imaging and the data with weights added to that data. Output. kΣ((XV th ) 2 -(W+XV th ) 2 )=kΣ(-W 2 -2W·X+ 2W·V th )

[0147] Next, we calculate the difference (Data B) between the data without imaging and the data with weights added to that data. Output. kΣ((0-V th ) 2 -(WV th ) 2 )=kΣ(-W 2 +2W·V th ) This is the result.

[0148] Then, we take the difference between data A and data B. kΣ(-W 2 -2W·X+2W·V th - (-W 2 +2W·V th )) = kΣ(-2W·X). That is, image data (X) Offset components other than the product of the product and the weight coefficient (W) can be removed.

[0149] Circuit 201 can read out data A and data B. The difference calculation with respect to B can be performed, for example, by circuit 301.

[0150] Here, the weights supplied to the entire pixel block 200 function as a filter. For example, a convolutional filter in a convolutional neural network (CNN) Alternatively, you can use image processing filters such as edge detection filters. Yes, it is possible. As an edge extraction filter, for example, the Laplacian filter shown in Figure 12A, The Prewitt filter shown in Figure 12B and the Sobel filter shown in Figure 12C are examples of the above. It can be listed.

[0151] If the number of pixels 400 in the pixel block 200 is 3x3, then the edge extraction filter described above The elements can be assigned as weights to each of the 400 pixels and supplied. As mentioned above, To calculate data A and data B, we need data with and without imaging, and the corresponding data for each. Then, it can be calculated using the data when weights are added. Here, with imaging, Data without weights is data with no weights added, and all 400 pixels are weighted to 0. It can also be rephrased as "ta".

[0152] The edge extraction filters illustrated in Figures 12A to 12C have filter elements (weights: ΔW) This is a filter whose sum (ΣΔW / N, where N is the number of elements) is 0. Therefore, a new other Even without performing the operation to supply ΔW=0 from the circuit, if the operation to obtain ΣΔW / N is performed, It is possible to acquire data with the equivalent of ΔW=0 added to all 400 pixels.

[0153] This operation is performed by the transistor 450 (transistor 450a to transistor 450a) located between the pixels 400. This is equivalent to making transistor 450f conduct (see Figure 10). Transistor 450 By enabling conductivity, all node FDWs of each pixel 400 are short-circuited via wiring 417. At this time, the charge accumulated in the node FDW of each pixel 400 is redistributed, as shown in Figure 12A. Alternatively, if the edge extraction filter exemplified in Figure 12C is used, the potential of node FDW (Δ W) will be 0 or approximately 0. Therefore, it is possible to obtain data with the equivalent of ΔW=0 added. Cut.

[0154] Furthermore, the weights (ΔW) are rewritten by supplying charge from a circuit located outside the pixel array 300. In this case, the long distance of the wiring 411 may cause a delay in the rewriting process. On the other hand, pixel block 200 is a very small area, and the distance of the wiring 417 is short, resulting in a small capacitance. Therefore, the charge accumulated in node FDW within pixel block 200 is redistributed. In operation, the weights (ΔW) can be rewritten at high speed.

[0155] In the pixel block 200 shown in Figure 10, transistors 450a to 450f Each of these is electrically connected to a different gate wire (wiring 413a to wiring 413f). This indicates that the conduction of transistors 450a to 450f is It can be controlled independently, and the operation to acquire ΣΔW / N can be selectively performed.

[0156] For example, when using the filter shown in Figure 12B or Figure 12C, ΔW=0 is initially provided. There are pixels being supplied. Assuming that ΣΔW / N=0, then ΔW=0 is supplied. Pixels that are being treated may be excluded from the pixels that are subject to summation. By excluding such pixels, The supply of potential to operate a portion of transistors 450a to 450f is Since it becomes unnecessary, power consumption can be reduced.

[0157] The data resulting from the sum-of-accumulate operation output from circuit 201 is sequentially input to circuit 301. In addition to the function to calculate the difference between data A and data B mentioned above, 301 also performs various other calculations. It may have a function. For example, circuit 301 may have the same configuration as circuit 201. Yes, it is possible. Alternatively, the function of circuit 301 may be replaced by software processing.

[0158] Furthermore, circuit 301 may have a circuit that performs calculations on the activation function. This circuit may include, For example, a comparator circuit can be used. In a comparator circuit, the input data The result of comparing the image with a set threshold is output as binary data. The basic block 200 and circuit 301 act as elements of a neural network. It is possible.

[0159] The data output from circuit 301 is sequentially input to circuit 302. Circuit 302 is, for example The configuration may include a latch circuit and a shift register, etc. It can perform parallel-to-serial conversion, and the data input in parallel is connected to wiring 311. It can be output as real data.

[0160] For example, the arithmetic unit 113 and the arithmetic unit 123 have the neural network configuration shown in Figure 1A or Figure 1B. Assuming network model 10 is applied, the data output from wiring 311 is layer It can be input to L_1. Therefore, layer L_1 can be used in the neural network NN1. It can be used as an input layer. Also, a neural network with the configuration shown in Figure 1A or Figure 1B When the learning model 10 performs learning, the data output from wiring 311 is shown in Figure 3A. Data 21 can be used as the data itself, and data 22 can be obtained by adding noise to data 21. The noise in question can be, for example, Gaussian noise. Thus, the output from wiring 311 is... Even if the processed data contains noise, the calculation unit 113 and the calculation unit will be affected by that noise. The accuracy of the inference of the neural network model applied to the calculation unit 123 decreases. It can be suppressed.

[0161] [Example of pixel configuration] Figure 13A shows an example of the configuration of pixel 400. Pixel 400 is made up of layers 561 and 563. It can be made into a layered structure.

[0162] Layer 561 has a photoelectric conversion device 401. The photoelectric conversion device 401 is shown in Figure 13B. As shown, it can have layers 565a and 565b. In some cases, the layers may be It can also be described as a domain or area.

[0163] The photoelectric conversion device 401 shown in Figure 13B is a pn junction type photodiode, for example, A p-type semiconductor can be used in layer 565a and an n-type semiconductor in layer 565b. Alternatively, layer 56 An n-type semiconductor may be used for layer 5a and a p-type semiconductor for layer 565b.

[0164] The above pn junction type photodiode is typically formed using single-crystal silicon. can.

[0165] Furthermore, the photoelectric conversion device 401 in layer 561 is located in layer 566a, as shown in Figure 13C. Alternatively, the layers 566b, 566c, and 566d may be stacked. The light shown in Figure 13C The power conversion device 401 is an example of an avalanche photodiode, with layer 566a and Layer 566d corresponds to an electrode, and layers 566b and 566c correspond to a photoelectric conversion section.

[0166] Layer 566a is preferably a low-resistance metal layer, such as aluminum or titanium. Tungsten, tantalum, silver, or alloys thereof can be used.

[0167] It is preferable to use a conductive layer with high light transmittance to visible light for layer 566d. For example, Indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc Oxides, indium-gallium-zinc oxide, or graphene can be used. It is also possible to omit layer 566d.

[0168] Layers 566b and 566c of the photoelectric conversion section are, for example, p, which uses a selenium-based material as the photoelectric conversion layer. This can be configured as an n-junction photodiode. Layer 566b is a p-type semiconductor. Using a certain selenium-based material, the layer 566c is an n-type semiconductor such as gallium oxide. It is preferable to do so.

[0169] Photoelectric conversion devices using selenium-based materials have the characteristic of high external quantum efficiency for visible light. In this photoelectric conversion device, by utilizing avalanche multiplication, the incident light is The amplification of electrons in relation to the amount of light can be greatly increased. Also, selenium-based materials have a light absorption coefficient. Because of its high viscosity, it offers production advantages such as the ability to fabricate the photoelectric conversion layer as a thin film. Thin films can be formed using methods such as vacuum deposition or sputtering.

[0170] Selenium-based materials include crystalline selenium such as single-crystal selenium or polycrystalline selenium, and amorphous selenium. , copper, indium, selenium compounds (CIS), or copper, indium, gallium, selenium Compounds such as CIGS can be used.

[0171] n-type semiconductors are formed from materials that have a wide band gap and are transparent to visible light. This is preferable. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or These materials can be used in the form of oxides containing a mixture of these substances. Furthermore, these materials can block hole injection. It also functions as a layer and can reduce dark current.

[0172] Furthermore, the photoelectric conversion device 401 in layer 561 is located in layer 567a, as shown in Figure 13D. Alternatively, the layers 567b, 567c, 567d, and 567e may be stacked. (Figure) The photoelectric conversion device 401 shown in 13D is an example of an organic photoconductive film, and layer 567a is the lower layer The electrode, layer 567e is a translucent upper electrode, and layers 567b, 567c, and 56 7d corresponds to the photoelectric conversion unit.

[0173] One of the layers 567b or 567d in the photoelectric conversion section can be a hole transport layer. The other of layer 567b or layer 567d may be an electron transport layer. Also, layer 567c This can be used as a photoelectric conversion layer.

[0174] For the hole transport layer, for example, molybdenum oxide can be used. For example, C 60 , C 70 Fullerenes such as these, or their derivatives, can be used. Cut.

[0175] The photoelectric conversion layer is a mixed layer of n-type organic semiconductor and p-type organic semiconductor (bulk heterojunction structure). (The construction method) can be used.

[0176] The layer 563 shown in Figure 13A includes, for example, a silicon substrate. Si transistors and the like are provided. Pixels 400 are formed using these Si transistors. This is possible. In addition, circuits 201 and 301 to 306 shown in Figure 9 are formed. It is possible.

[0177] Next, the stacked structure of the imaging device will be explained using a cross-sectional diagram. Note that the insulating layer shown below... The elements such as conductive layers are just examples, and other elements may also be included. Or, as follows: Some of the elements shown may be omitted. Also, the laminated structure shown below may be attached as needed. It can be formed using processes such as joining and polishing.

[0178] The imaging device with the configuration shown in Figure 14 has layers 560, 561, and 563. The elements provided in layer 563 are transistor 402 and transistor 40 Although it shows 3, other elements such as transistors 404 to 406 are also on layer 563. It can be installed in [location].

[0179] Layer 563 includes a silicon substrate 632, an insulating layer 633, an insulating layer 634, an insulating layer 635, and An insulating layer 637 is provided. A conductive layer 636 is also provided.

[0180] The insulating layer 634, insulating layer 635, and insulating layer 637 function as an interlayer insulating film and a planarizing film. It has the function of a protective film. The insulating layer 633 has the function of a protective film. The conductive layer 636 is shown in Figure 11. It is electrically connected to the indicated wiring 414.

[0181] Examples of interlayer insulating films and planarizing films include inorganic insulating films such as silicon oxide films and acrylic resin films. Organic insulating films such as lipids and polyimide resins can be used. For example, nitrogen Silicon oxide films, silicon dioxide films, aluminum oxide films, etc., can be used.

[0182] The conductive layer can be made of aluminum, chromium, copper, silver, gold, platinum, tantalum, nickel, or titanium. Molybdenum, tungsten, hafnium, vanadium, niobium, manganese, magnesium Indium, zirconium, beryllium, indium, ruthenium, iridium, strontium A metallic element selected from lanthanum, etc., or an alloy containing the aforementioned metallic elements, or the above Alloys and the like, which combine the aforementioned metal elements, can be appropriately selected and used. The conductor is not limited to a single layer. Alternatively, it may consist of multiple layers made of different materials.

[0183] The Si transistor shown in Figure 14 is a fin-type transistor with a channel formation region on the silicon substrate. The cross-section in the channel width direction (the cross-section of A1-A2 shown in layer 563 of Figure 14) is shown in Figure 15A. The Si transistor may also be of the planar type, as shown in Figure 15B.

[0184] Alternatively, as shown in Figure 15C, the transistor has a silicon thin-film semiconductor layer 545. It is also acceptable to do so. The semiconductor layer 545 is formed, for example, on the insulating layer 546 on the silicon substrate 632. This is a single-crystal silicon (SOI: Silicon on Insulator) that has been processed. It is possible.

[0185] A photoelectric conversion device 401 is provided in layer 561. It can be formed on 63. In Figure 14, the photoelectric conversion device 401 is shown as in Figure 13D. The diagram shows a configuration in which the organic photoconductive film shown is used as the photoelectric conversion layer. Note that here, layer 567 Let a be the cathode and layer 567e be the anode.

[0186] Layer 561 includes insulating layer 651, insulating layer 652, insulating layer 653, insulating layer 654, and conductive layer 655 will be established.

[0187] The insulating layer 651, insulating layer 653, and insulating layer 654 function as an interlayer insulating film and a planarizing film. It has the ability. In addition, the insulating layer 654 is provided covering the end of the photoelectric conversion device 401, and the layer It also has the function of preventing short circuits between layer 567e and layer 567a. The insulating layer 652 is the element isolation layer and It has the function of [doing something]. It is preferable to use an organic insulating film or the like as the element isolation layer.

[0188] The layer 567a, which corresponds to the cathode of the photoelectric conversion device 401, has the transient of layer 563 The photoelectric conversion device 401 is electrically connected to either the source or drain of the sta 402. The layer 567e, which corresponds to the anode, is connected to the conductive layer 563 via the conductive layer 655. It is electrically connected to layer 636.

[0189] Layer 560 is formed on layer 561. Layer 560 consists of a light-shielding layer 671 and an optical conversion layer 672. and has a microlens array 673.

[0190] The light-shielding layer 671 can suppress the inflow of light to adjacent pixels. The light-shielding layer 671 has, A metal layer such as aluminum or tungsten can be used. Furthermore, the metal layer and reflection A dielectric film having the function of a protective film may be laminated.

[0191] A color filter can be used in the optical conversion layer 672. (Red) Colors such as G (green), B (blue), Y (yellow), C (cyan), and M (magenta) are assigned to each pixel. By applying it, a color image can be obtained. Also, the optical conversion layer 672 has wavelength cutting By using a filter, it is possible to create an imaging device that can obtain images in various wavelength ranges. Cut.

[0192] For example, if an infrared filter that blocks light with wavelengths below visible light is used in the optical conversion layer 672, It can be used as an infrared imaging device. In addition, the optical conversion layer 672 can emit light with a wavelength less than or equal to near-infrared light. By using a filter that blocks it, it can be made into a far-infrared imaging device. Also, optical conversion layer 6 If an ultraviolet filter that blocks light with wavelengths greater than visible light is used in 72, it can be made into an ultraviolet imaging device. It is possible.

[0193] Furthermore, if a scintillator is used in the optical conversion layer 672, the intensity of the radiation used in X-ray imaging devices, etc. It can be used as an imaging device to obtain images that visualize weak radiation. When a beam of light enters a scintillator, the photoluminescence phenomenon causes visible light or ultraviolet light to be emitted. It is converted into light (fluorescence) such as lines. Then, this light is detected by the photoelectric conversion device 401. Image data is acquired by this method. Furthermore, even if the imaging device with this configuration is used with a radiation detector, etc. good.

[0194] When a scintillator is irradiated with radiation such as X-rays or gamma rays, it absorbs that energy. It contains substances that emit visible light or ultraviolet light. For example, Gd2O2S:Tb, Gd2O2S: Pr, Gd2O2S:Eu, BaFCl:Eu, NaI, CsI, CaF2, BaF2, This method uses a mixture of CeF3, LiF, LiI, ZnO, etc., dispersed in a resin or ceramic. It is possible.

[0195] A microlens array 673 is provided on the optical conversion layer 672. Light passing through each lens of I-673 passes through the optical conversion layer 672 directly below it, and the photoelectric conversion device The light will be directed to chair 401. By providing the microlens array 673, Since the focused light can be injected into the photoelectric conversion device 401, photoelectric conversion can be performed efficiently. This can be done. The microlens array 673 transmits light of the wavelength of the object to be imaged. It is preferable to form it with a high-quality resin or glass.

[0196] Figure 16 shows a modified example of the laminated structure shown in Figure 14, and the photoelectric conversion device 4 in layer 561 The configuration of 01 and some of the configuration of layer 563 are different. In the configuration shown in Figure 16, layer 561 and layer 5 It has a bonding surface between it and 63.

[0197] Layer 561 is the photoelectric conversion device 401, insulating layer 661, insulating layer 662, insulating layer 664, and It has an insulating layer 665, a conductive layer 685, and a conductive layer 686.

[0198] The photoelectric conversion device 401 is a pn junction type photodiode formed on a silicon substrate. It has a layer 565b corresponding to the p-type region and a layer 565a corresponding to the n-type region. The conversion device 401 is an embedded photodiode, and the surface side of layer 565a (current A thin p-shaped region (part of layer 565b) is provided on the extraction side to suppress dark current. It can reduce noise.

[0199] The insulating layer 661, and the conductive layers 685 and 686, function as bonding layers. The insulating layer 662 functions as an interlayer insulating film and a planarizing film. The insulating layer 664 is The insulating layer 665 has the function of suppressing carrier outflow. To possess.

[0200] The silicon substrate is provided with grooves to separate pixels, and the insulating layer 665 is located on the upper surface of the silicon substrate and the surrounding area. The groove is provided. The provision of the insulating layer 665 allows the photoelectric conversion device 401 to This can suppress the outflow of generated carriers to adjacent pixels. Also, the insulating layer 6 Layer 65 also has the function of suppressing the intrusion of stray light. Therefore, the insulating layer 665 prevents color mixing. This can be suppressed. Furthermore, an anti-reflective film is applied between the upper surface of the silicon substrate and the insulating layer 665. It may be provided.

[0201] The device isolation layer is LOCOS (LOCal Oxidation of Silicon). It can be formed using the law. Or, STI (Shallow Trench Is It may be formed using methods such as the olation method. As for the insulating layer 665, for example, sulfate By using inorganic insulating films such as silicon nitride and organic insulating films such as polyimide and acrylic. This can be done. The insulating layer 665 may have a multilayer configuration. Note that an element isolation layer is not provided. It can also be configured in a different way.

[0202] Layer 565a (n-type region, corresponding to the cathode) of the photoelectric conversion device 401 is connected to the conductive layer 685. They are electrically connected. Layer 565b (p-type region, corresponding to the anode) is electrically connected to conductive layer 686. They are connected precisely. The conductive layer 685 and conductive layer 686 are connected to the region embedded in the insulating layer 661. It has. In addition, the surfaces of the insulating layer 661, and the conductive layers 685 and 686 are, It has been flattened so that the heights match.

[0203] In layer 563, an insulating layer 638 is formed on the insulating layer 637. Also, Transis A conductive layer 683 and conductive layer 6 are electrically connected to either the source or drain of the 402. A conductive layer 684 is formed which is electrically connected to 36.

[0204] The insulating layer 638, and the conductive layers 683 and 684, function as bonding layers. The conductive layer 683 and the conductive layer 684 have regions embedded in the insulating layer 638. Furthermore, the surfaces of the insulating layer 638, and the conductive layers 683 and 684 each have a height of 1 It has been flattened in such a way.

[0205] Here, conductive layer 683 and conductive layer 685 are composed of the same metal element as their main component. Preferably, conductive layer 684 and conductive layer 686 have the same metal element as their main component. It is preferable that the insulating layer 638 and the insulating layer 661 have the same main components as each other. It is preferable that it be one.

[0206] For example, conductive layers 683 to 686 contain Cu, Al, Sn, Zn, W, Ag, and Pt. Alternatively, materials such as Au can be used. Due to their ease of bonding, Cu, Al, W, or Au are particularly suitable. It is preferable to use silicon oxide in the insulating layer 638 and insulating layer 661. Silicon oxide nitride, silicon nitride, silicon nitride, titanium nitride, etc. can be used. ru.

[0207] In other words, the same metal material as described above is used for each of the conductive layers 683 to 686. It is preferable that the insulating layer 638 and the insulating layer 661 each have the same as described above. It is preferable to use an insulating material. With this configuration, the boundary between layer 563 and layer 561 The joint can be used to bond the parts together.

[0208] Furthermore, conductive layers 683 to 686 may have a multilayer structure consisting of multiple layers, In this case, the surface layer (joint surface) only needs to be made of the same metal material. Also, insulating layer 638 and insulating layer 661 may also have a multilayer structure with multiple layers, in which case the surface layer (bonding surface) is the same insulation. Any material will do.

[0209] This bonding process electrically connects the conductive layer 683 and the conductive layer 685 to each other. This allows conductive layer 684 and conductive layer 686 to be electrically connected to each other. A connection with mechanical strength can be obtained between layer 661 and the insulating layer 638.

[0210] For bonding metal layers, the surface oxide film and adsorbed impurity layer are removed by sputtering or other processes. This method uses a surface activation bonding technique in which cleaned, purified, and activated surfaces are brought into contact and bonded together. This can be done. Alternatively, a diffusion bonding method that uses both temperature and pressure to join surfaces together can be used. This is possible. Since bonding occurs at the atomic level in both cases, they are superior not only electrically but also mechanically. A joint can be obtained.

[0211] Furthermore, for bonding the insulating layers, high flatness is achieved by polishing, etc., followed by oxygen plasma, etc. Hydrophilic treatment involves bringing hydrophilic treated surfaces into contact to create a temporary bond, followed by dehydration through heat treatment to create a permanent bond. Hydrophilic bonding methods can be used. Hydrophilic bonding methods also involve bonding at the atomic level, so Mechanically superior bonding can be achieved.

[0212] When layer 563 and layer 561 are bonded together, the bonding surfaces of each layer contain a mixture of metal and insulating layers. Therefore, for example, a combination of surface activation bonding and hydrophilic bonding methods can be used.

[0213] For example, after polishing, the surface is cleaned, an anti-oxidation treatment is applied to the surface of the metal layer, and then a hydrophilic treatment is applied. Methods such as performing a process to join the metal layer can be used. A hydrophilic metal may be used, and hydrophilic treatment may be performed. Furthermore, joining methods other than those described above may also be used. good.

[0214] The above bonding process electrically connects the elements of layer 563 and the elements of layer 561. It can be continued.

[0215] Figure 17 shows a modified example of the laminated structure shown in Figure 16, in which part of layer 561 and layer 563 are different.

[0216] This modified configuration has a transistor 402 in the pixel 400 that is located on layer 561. In layer 561, transistor 402 is formed from a Si transistor. Either the source or drain of the 402 is directly connected to one electrode of the photoelectric conversion device 401. Furthermore, the source or drain of transistor 402 is electrically connected to node FD. Connected.

[0217] In the imaging device shown in Figure 17, layer 563 contains a small number of transistors among those that make up the imaging device. At the very least, transistors other than transistor 402 are provided. In Figure 17, layer 563 The elements provided are shown to be transistor 404 and transistor 405. However, other elements such as transistor 403 and transistor 406 are also provided in layer 563. This is possible. Also, layer 563 of the imaging device shown in Figure 17 has an insulating layer 635 and an insulating layer 637 An insulating layer 647 is provided between them. The insulating layer 647 acts as an interlayer insulating film and a planarizing film. To have the ability. [Explanation of Symbols]

[0218] 10: Neural network model, 10a: Neural network model, 21: Data, 22: Data, 23: Data, 24: Data, 25: Dataset, 26: Data T, 27: Dataset, 28: Data, 29: Dataset, 34: Feature map, 35 : Feature map set, 36: Feature map, 37: Feature map set, 41: Data, 42 :Dataset, 43:Output data, 44:Dataset, 45:Data, 46:Data , 100: Processing system, 110: Information processing device, 111: Imaging unit, 112: Output unit , 113: Arithmetic unit, 114: Main memory unit, 115: Auxiliary memory unit, 116: Communication unit, 117: Transmission line, 120: Information processing device, 121: Imaging unit, 122: Output unit, 123: Calculation unit, 1 24: Main memory unit, 125: Auxiliary memory unit, 126: Communication unit, 127: Transmission line, 200: Pixel Blocks: 201: Circuit, 202: Capacitor, 203: Transistor, 204: Transistor Zista, 205: Transistor, 206: Transistor, 207: Resistor, 211: Wiring, 212: Wiring, 213: Wiring, 215: Wiring, 216: Wiring, 217: Wiring, 218: Distribution Line, 219: wiring, 300: pixel array, 301: circuit, 302: circuit, 303: circuit, 304: circuit, 305: circuit, 306: circuit, 311: wiring, 400: pixel, 401: light Electrical conversion device, 402: transistor, 403: transistor, 404: transistor , 405: Transistor, 406: Transistor, 407: Capacitor, 411: Wiring, 412: Wiring, 413: Wiring, 414: Wiring, 415: Wiring, 417: Wiring, 421: Distribution Wire, 422: Wiring, 423: Wiring, 424: Wiring, 450: Transistor, 545: Semiconductor Body layer, 546: insulating layer, 560: layer, 561: layer, 563: layer, 565a: layer, 565b : layer, 566a: layer, 566b: layer, 566c: layer, 566d: layer, 567a: layer, 56 7b: layer, 567c: layer, 567d: layer, 567e: layer, 632: silicon substrate, 633 : insulating layer, 634: insulating layer, 635: insulating layer, 636: conductive layer, 637: insulating layer, 638 : insulating layer, 647: insulating layer, 651: insulating layer, 652: insulating layer, 653: insulating layer, 654 : insulating layer, 655: conductive layer, 661: insulating layer, 662: insulating layer, 664: insulating layer, 665 : insulating layer, 671: light-shielding layer, 672: optical conversion layer, 673: microlens array, 68 3: conductive layer, 684: conductive layer, 685: conductive layer, 686: conductive layer

Claims

[Claim 1] A computing system that uses a neural network model to output output image data from which noise has been removed from input image data, The aforementioned neural network model comprises a first neural network and a second neural network. The first neural network described above has a first layer, a second layer, and a third layer. The feature map output from the first layer is input to the second layer and the second neural network. The feature map output from the second neural network is input to the third layer. The second neural network is a computing system that has been trained through a first learning process so that when the second data, which is the first data to be input to the first neural network, is output by the first learning process, the feature map output by the second neural network is the first feature map to be input by the first neural network, using a ground truth feature map, which is the feature map output by the first layer when the first data is input to the first neural network, and a learning feature map, which is the second data to which noise has been added to the first data.

Images