LEARNING DEVICE, LEARNING METHOD, PROGRAM, TRAINED MODEL, AND ENDOSCOPE SYSTEM

Abstract

Provided are a learning device with which efficient learning can be carried out even when learning is carried out using data obtained under different conditions, a learning method, a program, a learned model, and an endoscope system. A learning device 10 comprises a hierarchical network including a first input layer 301, a second input layer 302, a first intermediate layer 303, a first normalization layer 311, a second normalization layer 312, a second intermediate layer 313, and an output layer 304, wherein a learning control unit enables a first learning for learning a learning model on the basis of a first error between a first recognition result and a correct answer of first data, and a second learning for learning a learning model on the basis of a second error between a second recognition result and a correct answer of second data.

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, a program, a trained model, and an endoscope system, and more particularly to a learning device, a learning method, a program, a trained model, and an endoscope system that perform learning using a hierarchical network. [Background technology]

[0002] In the field of machine learning, it is known that learning is performed using a hierarchical network. A hierarchical network is generally composed of multiple layers that perform feature extraction, recognition, etc., but there are various types of specific network configurations and learning methods.

[0003] For example, Patent Document 1 describes a learning device that aims to appropriately learn a first data group and a second data group acquired under mutually different conditions. Specifically, the device describes a hierarchical network in which a first data group and a second data group acquired under mutually different conditions are input to a first input layer and a second input layer that are independent of each other, and a common intermediate layer is provided for the first input layer and the second input layer.

[0004] Furthermore, in the advancement of machine learning, a technique is known that improves the accuracy of a recognizer by normalizing calculated features, as described in Non-Patent Document 1. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 022027 [Non-patent literature]

[0006] [Non-Patent Document 1] Sergey Ioffe, Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", [online], March 2, 2015, Cornell University, arXiv:1502.03167v3[cs.LG], (Retrieved August 17, 2020), Internet<URL : https: / / arxiv.org / abs / 1502.03167> Summary of the Invention [Problem to be solved by the invention]

[0007] However, as in the hierarchical network described in Patent Document 1, there is a problem that normalization such as that described in the technique described in Non-Patent Document 1 cannot be appropriately applied to the respective feature amounts obtained from the first data group and the second data group acquired under mutually different conditions. When normalizing the feature amount, different normalization should be performed for each data group acquired under the same conditions. However, in the hierarchical network described in Patent Document 1, since the intermediate layer is configured to be common, normalization cannot be performed under different conditions for each data input to the feature amount output from the intermediate layer, and efficient learning may not be performed.

[0008] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a learning device, a learning method, a program, a trained model, and an endoscopic system that are capable of efficient learning even when learning is performed using data acquired under different conditions. [Means for solving the problem]

[0009] A learning device according to one aspect of the present invention for achieving the above object includes a learning device including a processor constituting a learning model of a recognizer and a learning control unit for training the learning model, the learning model including a first input layer receiving first data selected from a first data group composed of a plurality of data acquired under a first condition and outputting a first feature amount, a second input layer independent of the first input layer receiving second data selected from a second data group composed of a plurality of data that belong to the same category as the data constituting the first data group and acquired under a second condition different from the first condition and outputting a second feature amount, a first intermediate layer common to the first input layer and the second input layer, outputting a first intermediate feature amount when the first feature amount is input and outputting a second intermediate feature amount when the second feature amount is input, and a first normalized feature amount based on the first intermediate feature amount. a first normalization layer that outputs a characteristic feature of a first intermediate feature, a second normalization layer that receives a second intermediate feature and outputs a second normalized feature based on the second intermediate feature, a second intermediate layer that is common to the first normalization layer and the second normalization layer and outputs a third intermediate feature when the first normalized feature is input and outputs a fourth intermediate feature when the second normalized feature is input, and an output layer that receives the third intermediate feature or a fourth intermediate feature, and outputs a first recognition result based on the third intermediate feature when the third intermediate feature is input and outputs a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, wherein the learning control unit performs first learning to train a learning model based on a first error between the first recognition result and a correct answer for the first data, and second learning to train a learning model based on a second error between the second recognition result and a correct answer for the second data.

[0010] In this embodiment, the first intermediate layer outputs a first intermediate feature when a first feature based on the first data is input, and outputs a second intermediate feature when a second feature based on the second data is input. The first normalization layer inputs the first intermediate feature and outputs a first normalized feature, and the second normalization layer inputs the second intermediate feature and outputs a second normalized feature. The second intermediate layer inputs the first normalized feature and the second normalized feature. As a result, in this embodiment, the first intermediate feature derived from the first data and the second intermediate feature derived from the second data can be normalized under different conditions, and therefore the first intermediate feature and the second intermediate feature can be properly normalized, and efficient learning can be performed.

[0011] In this embodiment, the first and second data are input to the independent first and second input layers, respectively, and the features are calculated in the first and second input layers, respectively, so that the feature calculation in one of the first and second input layers is not affected by the feature calculation in the other input layer. In this embodiment, in addition to the feature extraction in the input layers (the first input layer and the second input layer), the first intermediate feature and the second intermediate feature are calculated in the first intermediate layer common to the first and second input layers, so that the feature calculated from the first and second data in the input layer can be reflected in the intermediate feature calculation in the first intermediate layer. The second intermediate layer is also common to the first normalization layer and the second normalization layer, so that the first normalization feature and the second normalization feature can be reflected in the intermediate feature calculation in the second intermediate layer in the same manner. In addition, the hierarchical network has a large number of parameters and is prone to overlearning, but overlearning can be avoided by providing a large amount of data. In the learning device according to this embodiment, the intermediate layer is trained with a large amount of data, which is the combination of the first and second data, and is therefore unlikely to overfit, while the input layer is independent of the first and second input layers, and the parameters of each input layer are small, so that overfitting is unlikely to occur even with a small amount of data. According to this embodiment, data belonging to the same category and acquired under different conditions can be appropriately trained in this way.

[0012] In this embodiment and the following embodiments, with regard to "first and second features based on features output from the first and second input layers," the features output from the first and second input layers may be directly input as the first and second features, or features output from the first and second input layers that have been subjected to some processing may be input as the first and second features. Furthermore, "belonging to the same category" refers to combinations such as image and image, text and text, and voice and voice, and "the first condition and the second condition are different" does not include "splitting data acquired under the same conditions into two."

[0013] In this embodiment and the following embodiments, the first and second input layers and the intermediate layer may be composed of one layer or multiple layers. The number of layers constituting the first and second input layers may be the same or different. The hierarchical network may include an output layer, a recognition layer, etc. in addition to the first and second input layers and intermediate layers.

[0014] In this and the following aspects, it is preferable to adjust the number of layers of the first and second input layers and the parameters of each layer in consideration of the learning results (e.g., error between the recognition result and the correct data, loss, etc.) so that the feature quantities output from the first and second input layers can appropriately express the features of the first and second data. It is also preferable to adjust the number of layers of the intermediate layers and the parameters of each layer in consideration of the learning results in the same manner.

[0015] Preferably, the learning control unit causes the first learning to be performed at least twice, and the second intermediate layer outputs the fourth intermediate feature in the second learning during a period after the third intermediate feature in the first learning is outputted in the first learning and before the third intermediate feature in the second learning is outputted.

[0016] When the first learning is performed many times in succession and then the second learning is performed, the feature amount calculated in the intermediate layer may be strongly influenced by the first data, and learning (calculation of feature amount) for the second data may not be performed appropriately (the same applies in the opposite case). For this reason, in this embodiment, the calculation of the fourth intermediate feature amount is performed during the period from the end of the calculation of the third intermediate feature amount to the start of the calculation of another third intermediate feature amount, so that the feature amount calculated when calculating the fourth intermediate feature amount is prevented from being excessively influenced by the first data, and learning can be performed appropriately for the first and second data.

[0017] Preferably, the learning control unit causes the first learning to be performed at least twice, and the second intermediate layer outputs the fourth intermediate feature in the second learning after completion of output of the third intermediate feature in the first learning in a first round and completion of output of the third intermediate feature in the second learning in a second round.

[0018] In this aspect, as described above, it is possible to prevent the feature values ​​calculated when calculating the third intermediate feature values ​​from being excessively affected by the first data, and to appropriately learn the first and second data.

[0019] Preferably, the hierarchical network is a convolutional neural network.

[0020] Preferably, the first normalization layer calculates the first normalized feature by a batch normalization process, and the second normalization layer calculates the second normalized feature by a batch normalization process.

[0021] Preferably, the first input layer outputs the first feature by an operation including any one of a convolution operation, a pooling operation, a batch normalization operation, and an activation operation.

[0022] Preferably, the second input layer outputs the second feature by an operation including any one of a convolution operation, a pooling operation, a batch normalization operation, and an activation operation.

[0023] Preferably, the first intermediate layer outputs the first intermediate feature or the second intermediate feature by an operation including any one of a convolution operation, a pooling operation, and an activation operation.

[0024] Preferably, the second intermediate layer outputs the third intermediate feature or the fourth intermediate feature by an operation including any one of a convolution operation, a pooling operation, and an activation operation.

[0025] Preferably, the first input layer inputs first image data acquired under first conditions as the first data, and the second input layer inputs second image data acquired under second conditions different from the first conditions as the second data.

[0026] Preferably, the first condition and the second condition differ in at least one of the imaging device, the wavelength balance of the observation light, the resolution, and the image processing applied to the image.

[0027] In this embodiment, "imaging devices are different" means "modalities are the same, but models, model numbers, performance, etc. are different." For example, an endoscope device and a CT device have different modalities. Furthermore, "wavelength balance of observation light is different" means that the wavelength band of the observation light and / or the relative relationship of the intensity of each wavelength band in the observation light are different. Furthermore, "image processing applied to the image is different" includes, but is not limited to, processing to emphasize or reduce the influence of a specific wavelength component, or processing to emphasize or make less noticeable a specific target or area.

[0028] Preferably, the first input layer inputs data of a first medical image acquired by a first observation light as first image data, and the second input layer inputs data of a second medical image acquired by a second observation light having a wavelength balance different from that of the first observation light as second image data.

[0029] Since "what structure of the subject is clearly (or unclearly) shown in the captured image" depends on the wavelength balance of the observation light used for the image capture, images may be captured using multiple observation lights with different wavelength balances in diagnostic and medical examination situations, but in this embodiment, image learning can be performed appropriately even in such cases. Note that in this embodiment and in each of the following embodiments, "medical image" is also called "medical image".

[0030] Preferably, the first input layer inputs data of a first medical image acquired using white light as the first observation light as first image data, and the second input layer inputs data of a second medical image acquired using narrowband light as the second observation light as second image data.

[0031] When acquiring medical images, images are often acquired using white light as observation light for visual confirmation by the user. On the other hand, in the case of narrowband light, it is possible to observe structures different from those in white light images, such as fine details and deep parts of the subject, depending on the wavelength, but since narrowband light is not suitable for visual observation, the number of images acquired is smaller than that of white light images. In this embodiment, learning can be performed appropriately even in such cases. In this embodiment, the "narrowband light" may be observation light with a short wavelength, such as blue light or purple light, or observation light with a long wavelength, such as red light or infrared light.

[0032] Preferably, the first input layer inputs data of a first medical image acquired using a first narrowband light as the first observation light as first image data, and the second input layer inputs data of a second medical image acquired using a second narrowband light different from the first narrowband light as the second observation light as second image data.

[0033] When acquiring a medical image, multiple narrowband lights may be used as observation light depending on the purpose of the image, and according to this embodiment, learning can be performed appropriately even in such a case. Note that "a second narrowband light different from the first narrowband light" means that the wavelength band and / or the intensity of the observation light are different between the first narrowband light and the second narrowband light.

[0034] A learning method according to another aspect of the present invention is a learning method for a learning device including a processor constituting a learning model of a recognizer and a learning control unit for learning the learning model, the learning model including a first input layer receiving first data selected from a first data group composed of a plurality of data acquired under a first condition and outputting a first feature amount; a second input layer independent of the first input layer receiving second data selected from a second data group composed of a plurality of data that belong to the same category as the data constituting the first data group and that are acquired under a second condition different from the first condition and outputting a second feature amount; a first intermediate layer common to the first input layer and the second input layer, outputting a first intermediate feature amount when the first feature amount is inputted and outputting a second intermediate feature amount when the second feature amount is inputted; and a first intermediate layer receiving the first intermediate feature amount and outputting a first normalized feature amount based on the first intermediate feature amount. a second normalization layer receiving a second intermediate feature and outputting a second normalized feature based on the second intermediate feature; a second intermediate layer that is common to the first normalization layer and the second normalization layer and outputs a third intermediate feature when the first normalized feature is input and outputs a fourth intermediate feature when the second normalized feature is input; and an output layer receiving the third intermediate feature or a fourth intermediate feature and outputting a first recognition result based on the third intermediate feature when the third intermediate feature is input and outputting a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, the learning method for a learning device having a hierarchical network including: a first normalization layer,

[0035] A program according to another aspect of the present invention is a program for executing a learning method of a learning device including a processor constituting a learning model of a recognizer and a learning control unit for learning the learning model, the learning model including a first input layer receiving first data selected from a first data group composed of a plurality of data acquired under a first condition and outputting a first feature amount; a second input layer independent of the first input layer receiving second data selected from a second data group composed of a plurality of data that belong to the same category as the data constituting the first data group and acquired under a second condition different from the first condition and outputting a second feature amount; a first intermediate layer common to the first input layer and the second input layer, the first intermediate layer outputting a first intermediate feature amount when the first feature amount is inputted and outputting a second intermediate feature amount when the second feature amount is inputted; and a first normalization layer receiving the first intermediate feature amount and outputting a first normalized feature amount based on the first intermediate feature amount. a second normalization layer receiving an input of a second intermediate feature and outputting a second normalized feature based on the second intermediate feature; a second intermediate layer that is common to the first normalization layer and the second normalization layer and outputs a third intermediate feature when the first normalized feature is input and outputs a fourth intermediate feature when the second normalized feature is input; and an output layer receiving an input of the third intermediate feature or a fourth intermediate feature and outputting a first recognition result based on the third intermediate feature when the third intermediate feature is input and outputting a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, the program executing the learning method for a learning device having a hierarchical network including:

[0036] The trained model of the recognizer, which is another aspect of the present invention, is obtained by the training method described above.

[0037] Another aspect of the present invention is an endoscope system equipped with a trained model of the above-mentioned recognizer.

[0038] Preferably, the first condition and the second condition differ in at least one of the imaging device, the wavelength balance of the observation light, the resolution, and the image processing applied to the image. Effect of the Invention

[0039] According to the present invention, even when learning is performed using data acquired under mutually different conditions, efficient learning can be performed. [Brief description of the drawings]

[0040] [Figure 1] FIG. 1 is a block diagram showing the configuration of a learning device. [Diagram 2] FIG. 2 is a diagram showing an example of a layer structure of a CNN. [Diagram 3] FIG. 3 is a diagram showing input and output data and feature amounts in each layer of the CNN shown in FIG. [Figure 4] FIG. 4 is a flow chart showing the learning method executed by the learning device. [Diagram 5] FIG. 5 is a diagram illustrating the first learning. [Figure 6] FIG. 6 is a diagram illustrating the second learning. [Figure 7] FIG. 7 is a diagram showing how features to be input to the first hidden layer are switched. [Figure 8] FIG. 8 is a diagram showing a state of convolution when features are input from the first input layer and the second input layer to the first hidden layer. [Figure 9] FIG. 9 is a diagram showing the first learning pattern and the second learning pattern. [Figure 10] FIG. 10 is a diagram showing other patterns of the first learning and the second learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Hereinafter, preferred embodiments of a learning device, a learning method, a program, a trained model, and an endoscope system according to the present invention will be described with reference to the accompanying drawings.

[0042] <Learning device configuration> FIG. 1 is a block diagram showing the configuration of a learning device 10 according to this embodiment. The learning device 10 includes a recognizer 100 that performs recognition processing based on an image captured by an endoscope inserted into a subject, a first image database 201 that records a plurality of endoscopic images acquired using normal light (white light) as observation light, and a second image database 202 that records a plurality of endoscopic images acquired using special light (narrowband light) as observation light. In the following description, an image acquired using normal light (white light) as observation light is referred to as a "normal light image" (or a "white light image"), and an image acquired using special light (narrowband light) as observation light is referred to as a "special light image" (or a "narrowband light image"). The endoscopic images recorded in the first image database 201 and the second image database 202 are examples of medical images.

[0043] <First and second image databases> <Normal light images and special light images> The first image database 201 and the second image database 202 are composed of a recording medium such as a hard disk. The first image database 201 records a plurality of normal light images (first data group, first data, first image data, first medical image) captured using normal light as observation light (first observation light), and the second image database 202 records a plurality of special light images (second data group, second data, second image data, second medical image) captured using special light as observation light (second observation light). That is, the plurality of normal light images recorded in the first image database 201 are one aspect of the "plurality of data acquired under a first condition" in the present invention, and the plurality of special light images recorded in the second image database 202 are one aspect of the "plurality of data acquired under a second condition different from the first condition" in the present invention. The special light (narrowband light) for capturing the special light images can be, for example, narrowband blue light, but may be other wavelengths such as narrowband red light. In addition, in the above example, the first and second observation lights are white light and narrowband light, but medical images such as endoscopic images acquired using first and second narrowband lights having different wavelength bands and / or intensities as observation lights may also be used.

[0044] In this way, the normal light image acquisition condition (first condition) and the special light image acquisition condition (second condition) have different wavelength balances of the observation light, but the normal light image and the special light image may also have different imaging devices, resolutions, and image processing applied to the images. That is, at least one of the imaging devices, the wavelength balance of the observation light, the resolution, and the image processing applied to the images may be different between the first condition and the second condition. "Different imaging devices" includes, but is not limited to, the use of endoscopes with different optical system characteristics and processor performance. "Different image processing applied to the images" includes, but is not limited to, the presence and / or degree of processing that emphasizes or makes inconspicuous a specific area such as a region of interest, and processing that emphasizes or reduces the influence of a specific wavelength component.

[0045] <Difference in the amount of data depending on data acquisition conditions> When performing observation or inspection using an endoscope, the user often checks images acquired using normal light (white light) as the observation light by displaying them on a monitor. In some cases, images are acquired using special light such as narrowband light as the observation light depending on the purpose and situation of the observation or inspection (for example, it is difficult to observe the structure of a lesion using normal light), but it is used less frequently as the observation light compared to normal light, and therefore the number of special light images is often significantly smaller than that of normal light images. When learning and / or recognizing images using machine learning, it is necessary to learn and / or recognize special light images as well, but if the number of data is small, there is a risk that the accuracy of learning and / or recognition will be reduced compared to normal light images. In view of this situation, the present embodiment employs a hierarchical network configuration described later to enable appropriate learning and / or recognition even in a situation where there is a difference in the number of data.

[0046] <Correct data for endoscopic images> The first image database 201 and the second image database 202 store "correct answer data" for identifying a region of interest (ROI) in association with the image in addition to the above-mentioned endoscopic images. Specifically, the first image database 201 stores a plurality of correct answer data corresponding to a plurality of normal light images, and the second image database 202 stores a plurality of correct answer data corresponding to a plurality of special light images. The correct answer data is preferably a region of interest or a discrimination result designated by a doctor for the endoscopic image.

[0047] <Recognizer configuration> The recognizer 100 is composed of an image acquisition unit 110 , an operation unit 120 , a control unit 130 , a display unit 140 , a recording unit 150 , and a processing unit 160 .

[0048] The image acquisition unit 110 is configured by a device that communicates with an external server, a database, etc. via a network, and acquires endoscopic images and correct answer data used for learning and recognition from a first image database 201 and a second image database 202. The image acquisition unit 110 can also acquire endoscopic images from an endoscope system connected to the learning device 10 via a network not shown, a hospital server, etc. The operation unit 120 has input devices such as a keyboard and a mouse not shown, and a user can perform operations required for image acquisition, learning, recognition, and other processing via these devices. The control unit 130 reads various programs recorded in the recording unit 150 and controls the operation of the entire learning device 10 according to commands input from the operation unit 120. The control unit 130 also updates the weight parameters of the CNN 162 (CNN: Convolutional Neural Network) by backpropagating the error (loss) calculated by the error calculation unit 164 described later to the CNN 162. That is, the control unit 130 has a function as a learning control unit that causes the CNN 162 to learn. Moreover, the CNN 162 is a learning model of the recognizer 100. When the first learning and the second learning described below are performed in the CNN 162, the CNN 162 becomes a trained model of the recognizer 100.

[0049] The display unit 140 includes a monitor 142 (display device) and displays an endoscopic image, learning results, recognition results, a processing condition setting screen, etc. The recording unit 150 includes a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, etc. (not shown), and records data acquired by the image acquisition unit 110, learning results and recognition results in the processing unit 160, etc. The recording unit 150 also records a program (including a program for causing the learning device 10 to execute the learning method of the present invention) for learning and recognizing endoscopic images (medical images). The processing unit 160 includes a hierarchical network, CNN 162, and an error calculation unit 164 that calculates a loss (error) based on the output (recognition result) of CNN 162 and the above-mentioned "correct answer data".

[0050] <Realization of functions using various processors> The functions of the image acquisition unit 110, the control unit 130, and the processing unit 160 (CNN 162, error calculation unit 164) described above can be realized by using various processors. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). Furthermore, the various processors described above also include dedicated electric circuits, which are processors having a circuit configuration designed specifically for executing specific processing, such as an ASIC (Application Specific Integrated Circuit).

[0051] The functions of each part may be realized by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by one processor. As an example of configuring multiple functions by one processor, first, as represented by a computer, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor realizes multiple functions. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system by one IC (Integrated Circuit) chip. In this way, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure.

[0052] Furthermore, the hardware structure of each of these various processors is, more specifically, an electrical circuitry that combines circuit elements such as semiconductor elements.

[0053] When the above-described processor or electrical circuitry executes software (program), the processor-readable code of the software to be executed is stored in a non-transitory recording medium such as a ROM (Read Only Memory), and the processor refers to the software. The software stored in the non-transitory recording medium includes a program for executing the learning method according to the present invention. The code may be recorded in non-transitory recording media such as various magneto-optical recording devices and semiconductor memories instead of the ROM. For example, a RAM (Random Access Memory) is used as a temporary storage area during processing using software, and data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory) (not shown) can also be referred to. These ROM, RAM, EEPROM, etc. can be those provided in the recording unit 150.

[0054] <Layer configuration of CNN> Next, regarding the layer configuration of the CNN 162, an explanation will be given with reference to FIGS. 2 and 3.

[0055] FIG. 2 is a diagram showing an example of the layer configuration of the CNN 162. FIG. 3 is a diagram showing the data, features, etc. input and output in each layer of the CNN 162 shown in FIG. 2. In the example shown in FIGS. 2 and 3, the CNN 162 includes a first input layer 301 (first input layer), a second input layer 302 (second input layer), a first intermediate layer 303 (intermediate layer), a first normalization layer 311 (first normalization layer), a second normalization layer 312 (second normalization layer), a second intermediate layer 313 (second intermediate layer), and an output layer 304 (output layer).

[0056] The first input layer 301 receives an image (first data) selected from normal light images (first data group) stored in the first image database 201, and outputs a feature amount (first feature amount).

[0057] The second input layer 302 is an input layer independent of the first input layer 301, and inputs an image (second data) selected from the special light images (second data group) stored in the second image database 202, and outputs a feature (second feature).

[0058] The first intermediate layer 303 is a common intermediate layer for the first input layer 301 and the second input layer 302. When the first feature (A1) output by the first input layer 301 is input to the first intermediate layer 303, the first intermediate layer 303 outputs a first intermediate feature (B1). When the second feature (A2) output by the second input layer 302 is input to the first intermediate layer 303, the first intermediate layer 303 outputs a second intermediate feature (B2). Note that switching between the features output by the first intermediate layer 303 and the second intermediate layer 313 will be described later.

[0059] The first normalization layer 311 receives the first intermediate feature (B1) output from the first hidden layer 303, and outputs a first normalized feature (C1) based on the first intermediate feature.

[0060] The second normalization layer 312 receives the second intermediate features (B2) output from the first intermediate layer 303, and outputs second normalized features (C2) based on the second intermediate features.

[0061] The second intermediate layer 313 is a common intermediate layer for the first normalization layer 311 and the second normalization layer 312. The second intermediate layer 313 outputs a third intermediate feature (D1) when the first normalization feature (C1) output from the first normalization layer 311 is input. Moreover, the second intermediate layer 313 outputs a fourth feature (D2) when the second feature (C2) output from the second normalization layer 312 is input.

[0062] The output layer 304 receives the feature from the second intermediate layer 313 and outputs the recognition result for the image input to the first input layer 301 or the second input layer 302. Specifically, when the output layer 304 receives the third intermediate feature (D1) output from the second intermediate layer 313, the output layer 304 outputs a first recognition result (E1) based on the third feature (D1). When the output layer 304 receives the fourth intermediate feature (D2) output from the second intermediate layer 313, the output layer 304 outputs a second recognition result (E2) based on the fourth intermediate feature (D2). Here, the first recognition result (E1) is the recognition result of the first data, and the second recognition result (E2) is the recognition result of the second data.

[0063] The first input layer 301, the first intermediate layer 303, the first normalization layer 311, the second intermediate layer 313, and the output layer 304 have a structure in which a plurality of "nodes" are connected by "edges", and each of them holds a plurality of weight parameters. The second input layer 302, the first intermediate layer 303, the second normalization layer 312, the second intermediate layer 313, and the output layer 304 have a structure in which a plurality of "nodes" are connected by "edges", and each of them holds a plurality of weight parameters. The values ​​of these weight parameters change as learning progresses.

[0064] Next, the processing in each layer constituting the CNN 162 will be explained.

[0065] <Processing in the input layer and intermediate layer> Each of the first input layer 301 and the second input layer 302 outputs a feature by an operation including any one of a convolution operation, a pooling process, an activation process, and a batch normalization process. Each of the first hidden layer 303 and the second hidden layer 313 outputs a feature by an operation including any one of a convolution operation, a pooling process, and an activation process. For example, each of the first input layer 301 and the second input layer 302 is a layered combination of a convolution operation, a pooling process, an activation process, and a batch normalization process, and outputs a feature. For example, each of the first hidden layer 303 and the second hidden layer 313 is a layered combination of a convolution operation, a pooling process, and an activation process, and outputs a feature.

[0066] Convolution is a process that obtains a feature map by performing a convolution operation using a filter on input data (e.g. an image). Convolution is responsible for extracting features such as edges from images. A feature map with one channel (one image) is generated for one filter by performing a convolution operation using this filter. The size of the feature map is downscaled by the convolution, becoming smaller as convolution is performed at each layer.

[0067] Pooling is a process that shrinks (or expands) the feature map output by the convolution operation to create a new feature map. Pooling provides robustness to the extracted features so that they are not affected by parallel translation, etc.

[0068] The activation process is performed on the feature map using an activation function, such as a sigmoid function or ReLU (Rectified Linear Unit).

[0069] Batch normalization is a process that normalizes the distribution of data in units of mini-batches during learning, and plays a role in speeding up learning, reducing dependency on initial values, and suppressing overlearning.

[0070] The first input layer 301, the second input layer 302, the first hidden layer 303, and the second hidden layer 313 can be configured with one or more layers that perform these processes. Note that the layer configuration is not limited to including one layer each of the convolution operation, the pooling process, the activation process, and the batch normalization process, and may include multiple layers of any of these processes.

[0071] Of these layers, the first input layer 301, the second input layer 302, the first intermediate layer 303, and the second intermediate layer 313, low-level feature extraction (such as edge extraction) is performed in the layers closer to the input side, while higher-level feature extraction (extraction of features related to the shape, structure, etc. of the object) is performed as the layers approach the output side.

[0072] <Processing in normalization layer> The first normalization layer 311 and the second normalization layer 312 normalize the input feature. Specifically, the first normalization layer 311 and the second normalization layer 312 normalize the input feature distribution and output the normalized feature. Here, the first normalization layer 311 normalizes the first intermediate feature (B1) based on the first data, and the second normalization layer 312 normalizes the second intermediate feature (B2) based on the second data. In this way, in the CNN 162, the first normalization layer 311 dedicated to the first intermediate feature (B1) and the second normalization layer 312 dedicated to the second intermediate feature (B2) are independently provided. As a result, the first intermediate feature (B1) and the second intermediate feature (B2) are normalized under appropriate conditions that are independent of each other. If the first intermediate feature (B1) and the second intermediate feature (B2) are normalized under the same conditions in a common normalization layer, the effect of the normalization process may be reduced, or the normalization process may cause the learning of the CNN 162 to proceed less efficiently. This is because, when two features derived from the first data and the second data acquired under different conditions are normalized, normalization to an intermediate feature is performed. Therefore, in the CNN 162, a first normalization layer 311 dedicated to the first intermediate feature (B1) and a second normalization layer 312 dedicated to the second intermediate feature (B2) are provided between the first hidden layer 303 and the second hidden layer 313, thereby realizing normalization processes suitable for the first data and the second data, respectively. In addition, the first normalization layer 311 and the second normalization layer 312 are provided in parallel at positions sandwiched between the first hidden layer 303 and the second hidden layer 313. As a result, the first intermediate feature (B1) and the second intermediate feature (B2) output in the first intermediate layer 303 are each normalized, and the normalized features (first normalized feature and second normalized feature) can be further output to the second intermediate layer 313. The normalization process performed in the first normalization layer 311 and the second normalization layer 312 is, for example, a batch normalization process. For example, the batch normalization process performs normalization so that the distribution of the first intermediate feature (B1) has a mean of 0 and a variance of 1, and the distribution of the second intermediate feature (B2) has a mean of 0 and a variance of 1.As a specific example, when a medical image under normal light is used as the first data and a medical image under special light is used as the second data, the first normalization layer 311 and the second normalization layer 312 may perform normalization under different conditions for color. In this way, by providing the first normalization layer 311 and the second normalization layer 312, even when learning is performed using the first data and the second data acquired under different conditions, the CNN 162 can perform appropriate normalization and perform efficient learning. Note that the batch normalization process is also performed in the first input layer 301 and the second input layer 302 described above, but since only the first data or the second data is input to the first input layer 301 and the second input layer 302, respectively, the batch normalization process is dedicated to the first data or the second data. On the other hand, since the first hidden layer 303 receives features derived from the first data and the second data, which have different properties, the first hidden layer 303 is branched into a first normalization layer 311 and a second normalization layer 312 to perform normalization correctly.

[0073] <Processing in the output layer> The output layer 304 is a layer that detects the position of the area of ​​interest in the input image (normal light image, special light image) based on the feature amount output from the second intermediate layer 313, and outputs the result. The output layer 304 grasps the position of the area of ​​interest in the image at the pixel level using the "feature map" obtained from the second intermediate layer 313. In other words, it can detect whether or not each pixel of the endoscopic image belongs to the area of ​​interest, and output the detection result.

[0074] The output layer 304 may perform lesion discrimination and output discrimination results. For example, the output layer 304 may classify endoscopic images into three categories, "neoplastic," "non-neoplastic," and "other," and output three scores (the sum of the three scores is 100%) corresponding to "neoplastic," "non-neoplastic," and "other" as discrimination results, or may output a classification result when a clear classification can be made from the three scores. Note that when outputting discrimination results, it is preferable that the output layer 304 has a fully connected layer as the last layer or layers.

[0075] <Learning Method> Next, a description will be given of a learning method executed by the above-mentioned learning device 10. FIG.

[0076] First, the control unit 130 of the learning device 10 performs a first learning process (see step S106), and then performs a second learning process (see step S112).

[0077] First, the first learning will be described. The processing unit 160 performs a calculation process (step S101) of a first feature amount in the first input layer 301. Next, the processing unit 160 performs a calculation process (step S102) of a first intermediate feature amount in the first hidden layer 303. Next, the processing unit 160 performs a calculation process (step S103) of a first normalized feature amount in the first normalization layer 311. Next, the processing unit 160 performs a calculation process (step S104) of a third intermediate feature amount in the second hidden layer 313. Next, the processing unit 160 performs a first recognition result output process (step S105) in the output layer 304. After that, the control unit 130 causes the CNN 162 to perform the first learning (step S106).

[0078] Next, the second learning will be described. The second learning is performed after the above-mentioned first learning. The processing unit 160 performs a second feature amount calculation process (step S107) in the second input layer 302. Next, the processing unit 160 performs a second intermediate feature amount calculation process (step S108) in the first hidden layer 303. Next, the processing unit 160 performs a second normalized feature amount calculation process (step S109) in the second normalized layer 312. Next, the processing unit 160 performs a fourth intermediate feature amount calculation process (step S110) in the second hidden layer 313. Next, the processing unit 160 performs a second recognition result output process (step S111) in the output layer 304. After that, the control unit 130 causes the CNN 162 to perform the second learning (step S112).

[0079] Next, each process in the first learning and the second learning will be described in detail.

[0080] <First lesson> Fig. 5 is a diagram for explaining the first learning in the CNN 162. In Fig. 5, a downward arrow indicates that information is transmitted in a direction (learning direction) from the first input layer 301 to the output layer 304 via the first hidden layer 303, the first normalization layer 311, and the second hidden layer 313, and an upward arrow opposite to the learning direction indicates that information is transmitted from the output layer 304 to the second hidden layer 313, the first normalization layer 311, the first hidden layer 303, and the first input layer 301 (error backpropagation, described later).

[0081] [First feature calculation process] In the first learning, a mini-batch is formed of a plurality of images (first data) selected from a plurality of normal light images recorded in the first image database 201, and input to the first input layer 301. Then, in the first input layer 301, a first feature calculation process (step S101) is performed to calculate the first feature.

[0082] [First intermediate feature calculation process] As described above, the first input layer 301 and the second input layer 302 are connected to the first intermediate layer 303, and therefore, during learning, the output of the first input layer 301 and the output of the second input layer 302 are switched and input. As shown in Fig. 5, when the first feature output from the first input layer 301 is input to the first intermediate layer 303, the first intermediate layer 303 calculates a first intermediate feature (step S102).

[0083] FIG. 7 is a diagram showing a state in which the feature to be input to the first hidden layer 303 is switched. FIG. 7(a) shows a state in which the first feature is input to the first hidden layer 303 (the output from the node 301A constituting the layer included in the first input layer 301 is input to the node 303A constituting the first hidden layer 303). At the time of input, the feature output from the first input layer 301 may be input to the first hidden layer 303 as the first feature as it is, or the feature multiplied by an appropriate weight may be input to the first hidden layer 303 as the first feature (see FIG. 8). Note that the solid lines in the figure show a state in which data is output or input from the node by the above-mentioned output switching, and the dotted lines in the figure show a state in which data is not output or input from the node. The nodes 301A and 303A are shown conceptually, and the number is not particularly limited. These points are the same in FIG. 8.

[0084] 8 is a diagram showing the state of convolution when inputting features from the first input layer 301 and the second input layer 302 to the first hidden layer 303. Part (a) of FIG. 8 shows the state in which the outputs of nodes X11, X12, and X13 of the first input layer 301 are multiplied by weight parameters W11, W12, and W13, respectively, and input to node Y11 of the first hidden layer 303 (in the state shown in the figure, no input is made from node X10 to node Y11). Although the figure shows the input relationship between nodes X11, X12, and X13 and node Y11, the same relationship also holds for other nodes Y10, Y12, and Y13 of the first hidden layer 303.

[0085] [First normalized feature calculation process] The processing unit 160 performs a first normalized feature calculation process in the first normalization layer 311. Specifically, the processing unit 160 calculates the first normalized feature based on the first intermediate feature output from the first intermediate layer 303 (step S103).

[0086] [Third intermediate feature calculation process] The processing unit 160 performs a third intermediate feature calculation process in the second intermediate layer 313 (step S104). Specifically, the processing unit 160 calculates a third intermediate feature based on the second normalized feature output from the first normalization layer 311. Note that, in the second intermediate layer 313, similar to the above-described first intermediate feature calculation process, the first normalized feature output from the first normalization layer 311 and the second normalized feature output from the second normalization layer 312 are switched and input. A detailed description of the third intermediate feature calculation process is omitted because it is similar to the first intermediate feature calculation process.

[0087] [First recognition result output process] The output layer 304 receives the third intermediate feature calculated in the second intermediate layer 313, performs a first recognition result output process, and outputs the first recognition result (step S105).

[0088] [First learning process (updating weight parameters by backpropagation)] The error calculation unit 164 compares the first recognition result output by the output layer 304 with the correct answer for the first data to calculate a loss (first error). In the second learning described later, the error calculation unit 164 compares the second recognition result output by the output layer 304 with the correct answer for the second data to calculate a loss (second error). Then, as shown in FIG. 5, the error calculation unit 164 updates the weight parameters in the first input layer 301, the first hidden layer 303, the first normalization layer 311, the second hidden layer 313, and the output layer 304 from the layers on the output side to the layers on the input side so as to reduce the calculated loss (error backpropagation). The update of these parameters is the first learning (step S106).

[0089] <Second Study> Fig. 6 is a diagram for explaining the second learning in the CNN 162. In Fig. 6, a downward arrow indicates that information is transmitted in a direction (learning direction) from the second input layer 302 to the output layer 304 via the first hidden layer 303, the second normalization layer 312, and the second hidden layer 313, and an upward arrow opposite to the learning direction indicates that information is transmitted from the output layer 304 to the second hidden layer 313, the second normalization layer 312, the first hidden layer 303, and the second input layer 302 (error backpropagation, described later).

[0090] [Second feature calculation process] In the second learning, a mini-batch is formed of a plurality of images (second data) selected from a plurality of special light images recorded in the second image database 202, and input to the second input layer 302. Then, in the second input layer 302, a second feature calculation process (step S107) is performed to calculate the second feature.

[0091] [Second intermediate feature calculation process] The first intermediate layer 303 receives the second feature and performs a second intermediate feature calculation process to calculate the second intermediate feature (step S108). Since the first input layer 301 and the second input layer 302 are connected to the first intermediate layer 303 as described above, the output of the first input layer 301 and the output of the second input layer 302 are switched and input during learning.

[0092] In the second learning, the output is switched as shown in FIG. 6, and the output from the second input layer 302 is input to the first intermediate layer 303. FIG. 7(b) is a diagram showing a state in which the second feature is input to the first intermediate layer 303 (the output from the node 302A constituting the second input layer 302 is input to the node 303A constituting the first intermediate layer 303). In the state shown in FIG. 6, the second feature based on the feature output from the second input layer 302 is input to the first intermediate layer 303, and the first intermediate feature is calculated in the first intermediate layer 303. FIG. 7(b) shows a state in which the second feature is input to the first intermediate layer 303.

[0093] 8(b), like part (a) of the figure, shows how the outputs of nodes X21, X22, and X23 in the second input layer 302 are multiplied by weight parameters W21, W22, and W23, respectively, and input to node Y11 in the first hidden layer 303 (in the state shown in the figure, no input is made from node X20 to node Y11). Although the figure shows the input relationship between nodes X21, X22, and X23 and node Y11, similar relationships also hold for the other nodes Y10, Y12, and Y13 in the first hidden layer 303.

[0094] Note that the “second normalized feature calculation process (step S109)”, “fourth intermediate feature calculation process (step S110)”, “second recognition result output process (step S111)”, and “second learning (step S112)” in the second learning are similar to the “first normalized feature calculation process (step S103)”, “third intermediate feature calculation process (step S104)”, “first recognition result output process (step S105)”, and “first learning (step S106)” in the first learning, and therefore will not be described.

[0095] <Example of learning pattern> In the above description of the learning method, an example in which the first learning and the second learning are each performed once has been described, but the learning method performed by the learning device 10 is not limited to this. It is sufficient that the first learning and the second learning are each performed at least once, and various modes can be adopted. Below, examples of the number of times and the order of processing will be described.

[0096] (First example) In the first example, the second intermediate layer 313 outputs a fourth intermediate feature in the second learning during a period after the third intermediate feature in the first learning is outputted in the first round and before the third intermediate feature in the second learning is outputted.

[0097] For example, the process is repeated in the order shown in Fig. 9(a). In the figure, "A" and "B" respectively mean "calculation of the third intermediate feature in the second hidden layer 313" and "calculation of the fourth intermediate feature in the second hidden layer 313", and are counted once, twice, and so on, in mini-batch units.

[0098] (Second example) In the second example, the second intermediate layer 313 outputs a fourth intermediate feature in the second learning after completing the output of the third intermediate feature in the first learning in the first round and the output of the third intermediate feature in the second learning in the first round. For example, the process is repeated in the order shown in FIG. 9(b). In FIG. 9(b), "A" and "B" have the same meaning as in part (a) of the same figure. In this case, "B" may be performed twice in succession as shown in FIG. 9(c).

[0099] (Third example) In a third example, the learning device 10 performs the first learning multiple times in succession, and then performs the second learning multiple times in succession. For example, the learning device 10 performs learning in the order shown in FIG. 10. Note that "first" and "second" in FIG. 10 mean "first learning" and "second learning", respectively. Note that the patterns shown in FIGS. 9 and 10 are examples, and learning can be performed in various other patterns.

[0100] <Effects> In the learning device 10, the first intermediate layer 303 outputs a first intermediate feature when a first feature based on the first data is input, and outputs a second intermediate feature when a second feature based on the second data is input. The first normalization layer 311 inputs the first intermediate feature and outputs a first normalized feature, and the second normalization layer 312 inputs the second intermediate feature and outputs a second normalized feature. The second intermediate layer 313 inputs the first normalized feature and the second normalized feature. As a result, in this embodiment, the first intermediate feature derived from the first data and the second intermediate feature derived from the second data can be normalized under different conditions, and therefore the first intermediate feature and the second intermediate feature can be properly normalized, and efficient learning can be performed.

[0101] In addition, in the learning device 10, the first and second data are input to the independent first and second input layers, respectively, and features are calculated in the first and second input layers, respectively, so that feature calculation in one of the first and second input layers is not affected by feature calculation in the other input layer. In addition, in the learning device 10, in addition to feature extraction in the input layers (the first input layer 301 and the second input layer 302), the first intermediate feature and the second intermediate feature are calculated in the first intermediate layer 303 common to the first and second input layers, so that the feature calculated from the first and second data in the input layer can be reflected in the intermediate feature calculation in the first intermediate layer 303. The second intermediate layer 313 is also common to the first normalization layer 311 and the second normalization layer 312, so that the first normalized feature and the second normalized feature can be reflected in the intermediate feature calculation in the second intermediate layer 313 in the same manner. In addition, since a hierarchical network has many parameters, it is prone to overlearning, but by providing a large amount of data, overlearning can be avoided. In addition, in the learning device 10, the intermediate layer can learn with a large amount of data, which is the combination of the first and second data, so overlearning is unlikely to occur, while the input layer is independent of the first and second input layers, and the parameters of each input layer are small, so overlearning is unlikely to occur even with a small amount of data.

[0102] In this way, the learning device 10 can appropriately learn data that belong to the same category but that have been acquired under different conditions.

[0103] <Learning with synthetic mini-batches> In the above-described learning pattern, the feature values ​​are calculated for the first and second data separately for each mini-batch, but the first and second mini-batches may be combined into one mini-batch immediately before input to the first hidden layer 303. Specifically, a mini-batch (first mini-batch) is formed of a plurality of images (first data) selected from a plurality of normal light images recorded in the first image database 201, and input to the first input layer 301 to calculate the feature values. Also, a mini-batch (second mini-batch) is formed of a plurality of images (second data) selected from a plurality of special light images recorded in the second image database 202, and input to the second input layer 302 to calculate the feature values. The first and second mini-batches may be combined into one mini-batch immediately before input to the first hidden layer 303, and input to the first hidden layer 303 for these feature values.

[0104] <Recognition process> As the above-mentioned learning (first learning and second learning) progresses, the CNN 162 of the recognizer 100 becomes a trained model. In a recognition (inference) process using the CNN 162 that has become a trained model, recognition may be performed in a configuration in which the first input layer 301 or the second input layer 302 is removed. For example, as shown in FIG. 5, recognition can be performed on the first data in a state in which the second input layer 302 is removed and only the first input layer 301 is connected. Also, as shown in FIG. 6, recognition can be performed on the second data in a state in which the first input layer 301 is removed and only the second input layer 302 is connected.

[0105] <Learning using the first narrowband light image and the second narrowband light image> In the above example, learning using a normal light image (white light image) and a special light image (e.g., a blue special light image) has been described, but learning may also be performed using multiple narrowband light images with different wavelength balances of observation light. The first input layer 301 may input data of a first medical image acquired using a first narrowband light as the first observation light as the first image data, and the second input layer 302 may input data of a second medical image acquired using a second narrowband light different from the first narrowband light as the second observation light as the second image data. In this case, combinations of narrowband light may include multiple blue narrowband lights, blue narrowband light and purple narrowband light, multiple red narrowband lights, etc.

[0106] <Learning using other data> In the above embodiment, learning using endoscopic images acquired with different observation lights has been described, but the learning device and learning method according to the present invention can learn in the same way even when using medical images other than endoscopic images, such as images from a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device. Learning can also be done in the same way even when using images other than medical images (e.g., images of people, animals, landscapes, etc.). Furthermore, learning can also be done in the same way when the input data is not an image but text, audio, etc.

[0107] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0108] 10: Learning device 100: Recognizer 110: Image acquisition unit 120:Operation unit 130: Control section 140: Display section 142: Monitor 150: Recording section 160: Processing section 164:Error calculation section 201: First image database 202: Second image database 301: First input layer 302: Second input layer 303: First middle layer 304: Output layer 311: First normalization layer 312: Second normalization layer 313: The second middle class

Claims

1. A learning device including a processor constituting a learning model of a recognizer and a learning control unit that learns the learning model, The learning model is a first input layer that receives first data selected from a first data group composed of a plurality of data acquired under a first condition and outputs a first feature amount; a second input layer independent of the first input layer, which receives second data selected from a second data group composed of a plurality of data belonging to the same category as data constituting the first data group and acquired under second conditions different from the first conditions, and outputs a second feature amount; and a first intermediate layer that is common to the first input layer and the second input layer, the first intermediate layer outputting a first intermediate feature when the first feature is inputted, and outputting a second intermediate feature when the second feature is inputted; a first normalization layer that receives the first intermediate feature and outputs a first normalized feature based on the first intermediate feature; a second normalization layer that receives the second intermediate feature and outputs a second normalized feature based on the second intermediate feature; a second intermediate layer which is a common intermediate layer for the first normalization layer and the second normalization layer, and which outputs a third intermediate feature when the first normalization feature is input, and outputs a fourth intermediate feature when the second normalization feature is input; an output layer to which the third intermediate feature or the fourth intermediate feature is input, and which outputs a first recognition result based on the third intermediate feature when the third intermediate feature is input, and which outputs a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input; A hierarchical network including: the learning control unit is a learning device that performs a first learning to train the learning model based on a first error between the first recognition result and a correct answer of the first data, and a second learning to train the learning model based on a second error between the second recognition result and a correct answer of the second data, the first input layer receives as input first image data acquired under the first condition as the first data; the second input layer receives as the second data second image data acquired under the second condition different from the first condition; the first input layer inputs the first image data acquired with white light; the second input layer inputs the second image data acquired with light having a narrower band than the white light; A learning device in which the first normalization layer and the second normalization layer perform normalization under different conditions for color.

2. The learning control unit causes the first learning to be performed at least twice, 2. The learning device according to claim 1, wherein the second intermediate layer outputs the fourth intermediate feature in the second learning during a period after the third intermediate feature in the first learning for a first time is output and before the third intermediate feature in the first learning for a second time is output.

3. The learning control unit causes the first learning to be performed at least twice, 2. The learning device according to claim 1, wherein the second intermediate layer outputs the fourth intermediate feature in the second learning after a first round of output of the third intermediate feature in the first learning and a second round of output of the third intermediate feature in the first learning are completed.

4. The learning device according to claim 1 , wherein the hierarchical network is a convolutional neural network.

5. 5. The learning device according to claim 1, wherein the first normalization layer calculates the first normalized feature by a batch normalization process, and the second normalization layer calculates the second normalized feature by a batch normalization process.

6. The learning device according to claim 1 , wherein the first input layer outputs the first feature amount by an operation including any one of a convolution operation, a pooling process, a batch normalization process, and an activation process.

7. The learning device according to claim 1 , wherein the second input layer outputs the second feature amount by an operation including any one of a convolution operation, a pooling process, a batch normalization process, and an activation process.

8. The learning device according to claim 1 , wherein the first intermediate layer outputs the first intermediate feature or the second intermediate feature by an operation including any one of a convolution operation, a pooling process, and an activation process.

9. 9. The learning device according to claim 1, wherein the second intermediate layer outputs the third intermediate feature or the fourth intermediate feature by an operation including any one of a convolution operation, a pooling process, and an activation process.

10. The first input layer inputs image data acquired with a first light, which is light having a narrower band than the white light, as the first image data; The learning device according to claim 1 , wherein the second input layer inputs image data acquired with a second light different from the first light as the second image data.

11. A learning method for a learning device including a processor constituting a learning model of a recognizer and a learning control unit that learns the learning model, comprising: The learning model includes a first input layer that receives first data selected from a first data group consisting of a plurality of data acquired under a first condition and outputs a first feature amount; a second input layer independent of the first input layer that receives second data selected from a second data group consisting of a plurality of data that belong to the same category as data constituting the first data group and that are acquired under a second condition different from the first condition and outputs a second feature amount; a first intermediate layer that is common to the first input layer and the second input layer and outputs a first intermediate feature amount when the first feature amount is input, and outputs a second intermediate feature amount when the second feature amount is input; and a second intermediate layer that receives the first intermediate feature amount and outputs the first intermediate feature amount. a first normalization layer that outputs a first normalized feature based on an intermediate feature of the first normalization layer, a second normalization layer that receives the second intermediate feature and outputs a second normalized feature based on the second intermediate feature, a second intermediate layer that is common to the first normalization layer and the second normalization layer, the second intermediate layer outputting a third intermediate feature when the first normalized feature is input and outputting a fourth intermediate feature when the second normalized feature is input, and an output layer that receives the third intermediate feature or the fourth intermediate feature, outputs a first recognition result based on the third intermediate feature when the third intermediate feature is input, and outputs a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, The learning control unit a first learning step of learning the learning model based on a first error between the first recognition result and a correct answer for the first data; a second learning step of learning the learning model based on a second error between the second recognition result and a correct answer for the second data; A learning method comprising: the first input layer receives as input first image data acquired under the first condition as the first data; the second input layer receives as the second data second image data acquired under the second condition different from the first condition; the first input layer inputs the first image data acquired with white light; the second input layer inputs the second image data acquired with light having a narrower band than the white light; A learning method in which the first normalization layer and the second normalization layer perform normalization under different conditions with respect to color.

12. A program for executing a learning method of a learning device including a processor constituting a learning model of a recognizer and a learning control unit for learning the learning model, The learning model includes a first input layer that receives first data selected from a first data group consisting of a plurality of data acquired under a first condition and outputs a first feature amount; a second input layer independent of the first input layer that receives second data selected from a second data group consisting of a plurality of data that belong to the same category as data constituting the first data group and that are acquired under a second condition different from the first condition and outputs a second feature amount; a first intermediate layer that is common to the first input layer and the second input layer and outputs a first intermediate feature amount when the first feature amount is input, and outputs a second intermediate feature amount when the second feature amount is input; and a second intermediate layer that receives the first intermediate feature amount and outputs the first intermediate feature amount. a first normalization layer that outputs a first normalized feature based on an intermediate feature of the first normalization layer, a second normalization layer that receives the second intermediate feature and outputs a second normalized feature based on the second intermediate feature, a second intermediate layer that is common to the first normalization layer and the second normalization layer, the second intermediate layer outputting a third intermediate feature when the first normalized feature is input and outputting a fourth intermediate feature when the second normalized feature is input, and an output layer that receives the third intermediate feature or the fourth intermediate feature, outputs a first recognition result based on the third intermediate feature when the third intermediate feature is input, and outputs a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, The learning control unit a first learning step of learning the learning model based on a first error between the first recognition result and a correct answer for the first data; a second learning step of learning the learning model based on a second error between the second recognition result and a correct answer for the second data; A program for executing a learning method including: the first input layer receives as input first image data acquired under the first condition as the first data; the second input layer receives as the second data second image data acquired under the second condition different from the first condition; the first input layer inputs the first image data acquired with white light; the second input layer inputs the second image data acquired with light having a narrower band than the white light; The program, wherein the first normalization layer and the second normalization layer perform normalization under different conditions for color.

13. A trained model configured by a program obtained by executing a training method of a training device including a processor that configures a training model of a recognizer and a training control unit that trains the training model, The learning model includes a first input layer to which first data selected from a first data group consisting of a plurality of data acquired under a first condition is input, and which outputs a first feature amount; a second input layer independent of the first input layer, to which second data selected from a second data group consisting of a plurality of data belonging to the same category as data constituting the first data group and acquired under a second condition different from the first condition is input, and which outputs a second feature amount; a first intermediate layer common to the first input layer and the second input layer, which outputs a first intermediate feature amount when the first feature amount is input, and outputs a second intermediate feature amount when the second feature amount is input; a first normalization layer that outputs a first normalized feature based on a feature of the first normalization layer, a second normalization layer that receives the second intermediate feature and outputs a second normalized feature based on the second intermediate feature, a second intermediate layer that is common to the first normalization layer and the second normalization layer, and outputs a third intermediate feature when the first normalized feature is input, and outputs a fourth intermediate feature when the second normalized feature is input, and an output layer that receives the third intermediate feature or the fourth intermediate feature, and outputs a first recognition result based on the third intermediate feature when the third intermediate feature is input, and outputs a second recognition result based on the fourth intermediate feature when the fourth intermediate feature is input, The trained model is The learning control unit performs a first learning step of learning the learning model based on a first error between the first recognition result and a correct answer for the first data; a second learning step of learning the learning model based on a second error between the second recognition result and a correct answer for the second data, the first input layer receives as input first image data acquired under the first condition as the first data; the second input layer receives as the second data second image data acquired under the second condition different from the first condition; the first input layer inputs the first image data acquired with white light; the second input layer inputs the second image data acquired with light having a narrower band than the white light; A trained model for causing a computer to function such that the first normalization layer and the second normalization layer perform normalization under different conditions regarding color.

14. An endoscope system equipped with the trained model of the recognizer according to claim 13.

15. The endoscope system according to claim 14 , wherein the first condition and the second condition differ from each other in at least one of an imaging device, a wavelength balance of observation light, a resolution, and image processing applied to an image.

Images