Identification device, identification method, and program

Abstract

To accurately execute identification of an object using data of a plurality of different modalities.SOLUTION: A diagnosis supporting device, which is a diagnostic device, includes a multi-modal identifier (multi-modal identifier 30x) corresponding to a plurality of mutually different modalities, and a plurality of single modal identifiers (single modal identifiers 30a and single modal identifiers 30b) corresponding to each of the plurality of mutually different modalities. The multi-modal identifier is configured so as to identify an object on the basis of a plurality of identification scores calculated by the plurality of single modal identifiers and a plurality of feature amounts calculated by the mutually different single modal identifiers obtained in the process for calculating the plurality of identification scores.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The disclosure of this specification relates to an identification device, an identification method, and a program. [Background technology]

[0002] Conventionally, a multimodal classifier that classifies an object by integrating classification results based on data of different modalities has been known. Such a multimodal classifier is described in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-076913 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, a multimodal classifier is expected to have classification accuracy equal to or higher than that of a single-modal classifier, but there is room for improvement in the classification accuracy, and further improvements in accuracy are expected.

[0005] In view of the above circumstances, an object according to one aspect of the present invention is to accurately identify an object using data from a plurality of different modalities. [Means for solving the problem]

[0006] A classification device according to one aspect of the present invention includes a multimodal classifier corresponding to a plurality of mutually different modalities and a plurality of single-modal classifiers corresponding to each of the plurality of mutually different modalities, wherein the multimodal classifier is configured to classify an object based on a plurality of classification scores calculated by the plurality of single-modal classifiers and a plurality of feature amounts calculated by the mutually different single-modal classifiers obtained in the process of calculating the plurality of classification scores.

[0007] An identification method according to one aspect of the present invention includes: An identification method performed by an identification device, The method includes the steps of: inputting each of a plurality of input data having different modalities into a corresponding single-modal classifier to obtain a plurality of classification scores; acquiring a plurality of feature amounts calculated by the single-modal classifiers that are different from each other and obtained in the process of calculating the plurality of classification scores; and classifying an object using a multi-modal classifier that corresponds to a plurality of modalities corresponding to the plurality of input data based on the plurality of classification scores and the plurality of feature amounts.

[0008] A program according to one aspect of the present invention causes a computer to execute the steps of inputting each of a plurality of input data having different modalities into a corresponding single-modal classifier to obtain a plurality of classification scores, obtaining a plurality of feature amounts calculated by the single-modal classifiers that are different from each other and obtained in the process of calculating the plurality of classification scores, and classifying an object using a multi-modal classifier corresponding to a plurality of modalities that correspond to the plurality of input data, based on the plurality of classification scores and the plurality of feature amounts. [Effects of the Invention]

[0009] According to the above aspect, it is possible to accurately identify an object using data from a plurality of different modalities. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a functional configuration of a diagnosis support apparatus according to an embodiment. [Figure 2] 2 is a diagram illustrating an example of the functional configuration of a recognition unit of the diagnosis support device shown in FIG. 1. FIG. [Figure 3] 1. FIG. 4 is another diagram illustrating the detailed functional configuration of the identification unit of the diagnosis support device shown in FIG. [Figure 4] 2 is a diagram illustrating a network configuration of an identification unit of the diagnosis support device shown in FIG. 1. FIG. [Figure 5]FIG. 10 is a diagram illustrating a method for calculating a score of a multimodal classifier. [Figure 6] FIG. 10 is a diagram for explaining a parameter sharing range. [Figure 7] 2 is a flowchart showing an example of a learning process for generating the identification unit shown in FIG. [Figure 8] 2 is a flowchart showing an example of an inference process performed by the diagnosis support device shown in FIG. [Figure 9] 4 is a diagram showing a modified example of the functional configuration of the recognition unit shown in FIG. 3. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 is a diagram illustrating the functional configuration of a diagnosis support device according to one embodiment. The diagnosis support device 100 shown in FIG. 1 is a diagnosis support device used to diagnose the presence or absence of a skin disease and the type of disease. The diagnosis support device 100 outputs a classification result of a disease to be diagnosed in response to input skin image data. Therefore, the diagnosis support device 100 is an example of a classification device, and is configured, for example, by a general-purpose or dedicated computer. Note that the diagnosis support device 100 may be configured as a device including a classification device that outputs a classification result.

[0012] The skin image data input for diagnosis is not particularly limited, but may be, for example, dermoscopy image data acquired by an imaging device equipped with a dermoscope. A dermoscope is a magnifying glass that can non-invasively observe suspected skin lesions by magnifying them while reducing light reflected from the skin surface. Therefore, dermoscopy images acquired by a dermoscope are suitable for identifying skin diseases. However, the input skin image is not limited to a dermoscopy image.

[0013] Specific techniques for reducing the influence of light reflected from the skin surface in dermatoscopes include imaging methods using gels and polarized filters. Another known imaging method for dermatoscopes is the use of UV (UltraViolet) light. Imaging using UV light makes it possible to obtain dermoscopy images that reflect the properties of melanin, which absorbs UV light, and the fluorescent reaction caused by UV light. Similar to images in which the influence of light reflected from the skin surface is reduced, this method is suitable for identifying skin diseases. Note that imaging methods using gels can be used in combination with other imaging methods that use polarized filters or UV light.

[0014] In this way, an imaging device with a dermoscope can obtain skin image data of multiple different modalities using these imaging methods or combinations thereof. The diagnosis support device 100 supports the input of skin image data of multiple different modalities. That is, it can obtain a classification result from each of the multiple skin image data of different modalities. Furthermore, the diagnosis support device 100 can also use skin image data of two or more different modalities to obtain a single classification result that is expected to have higher classification accuracy than when each of the skin image data is used alone. That is, the diagnosis support device 100 is a classification device that can operate with both single-modality input and multi-modality input.

[0015] The following is a specific description of the configuration of the diagnosis support device 100. As shown in Fig. 1, the diagnosis support device 100 includes a control unit 10, a storage unit 50, an operation input unit 60, a data input unit 70, an output unit 80, and a communication unit 90.

[0016] The control unit 10 executes various types of arithmetic processing to control the operation of the diagnosis support device 100. The control unit 10 includes a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 10 may also include any electronic circuit that operates as a programmable device, such as an FPGA (Field-Programmable Gate Array).

[0017] The storage unit 50 stores various programs executed by the control unit 10, various data used when executing the programs, etc. The storage unit 50 includes, for example, semiconductor memory such as RAM (Random Access Memory) and ROM (Read Only Memory). The storage unit 50 may further include an external storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive).

[0018] The programs and data stored in the memory unit 50 include a machine learning model, input data used for inference using the machine learning model, etc. The input data to the machine learning model is specifically skin image data of one or more modalities as described above. The input data is input via the data input unit 70 or the communication unit 90 and stored in the memory unit 50. The data stored in the memory unit 50 is input to the machine learning model. Specifically, the machine learning model is a deep learning model (hereinafter referred to as a CNN model) that uses a convolutional neural network (CNN) that outputs a classification result in response to input image data (skin image data).

[0019] The storage unit 50 may further store training image data used for training the machine learning model, in which case the training image data is labeled with a label indicating the type of disease (or the absence of a disease). However, training of the machine learning model using the training image data may be performed by a device other than the diagnostic support device 100, and a machine learning model that has been trained by a device other than the diagnostic support device 100 may be stored in the storage unit 50 of the diagnostic support device 100.

[0020] In the diagnosis support device 100, the processor of the control unit 10 executes a program stored in the storage unit 50, causing the control unit 10 to operate as an input control unit 20, a classification unit 30, and an output control unit 40, as shown in Fig. 1, and output a classification result based on input data. That is, the diagnosis support device 100 operates as a classification device using a CNN model.

[0021] The operation input unit 60 accepts operation inputs from the user to the diagnosis support device 100. The operation input unit 60 may include, for example, a keyboard, a mouse, a touch panel, or the like, or may be an interface for connecting to these input devices.

[0022] The data input unit 70 acquires input data. The data input unit 70 is connected to, for example, a photographing device with a dermoscope, and acquires image data of a dermoscopy image (skin image) acquired by the photographing device. The data acquired by the data input unit 70 is stored in the storage unit 50.

[0023] The output unit 80 outputs the classification result obtained by the control unit 10. The output unit 80 may include a display device such as a liquid crystal display or an organic EL display, or may be an interface connected to a display device. The output unit 80 provides the classification result to a user of the diagnosis support device 100 by displaying the classification result on the display device.

[0024] The communication unit 90 exchanges data with devices other than the diagnosis support device 100. The communication unit 90 may receive image data of a skin image from a device other than the diagnosis support device 100. The image data acquired by the communication unit 90 is stored in the storage unit 50. The communication unit 90 may also transmit the identification result to a device other than the diagnosis support device 100. For example, the diagnosis support device 100 may transmit the identification result to a computer of a user who has requested the diagnosis support device 100 to perform diagnostic processing.

[0025] Fig. 2 is a diagram illustrating the functional configuration of the recognition unit of the diagnosis support device shown in Fig. 1. The control unit 10 will be described in detail with reference to Fig. 1 and Fig. 2. As described above, the control unit 10 includes the input control unit 20, the recognition unit 30, and the output control unit 40.

[0026] As shown in FIG. 2, the classifier 30 includes a plurality of single-modal classifiers (single-modal classifier 30a, single-modal classifier 30b, . . . ) and one or more multi-modal classifiers (multi-modal classifier 30x, . . . ).

[0027] A single-modal classifier is a classifier that corresponds to a single modality and is a CNN model that is trained specifically for input data of that modality. The multiple single-modal classifiers included in the classifier 30 correspond to multiple mutually different modalities, respectively.

[0028] The image data of the multiple different modalities that the multiple single-modal classifiers included in the classifier 30 correspond to are, for example, skin image data captured using multiple different imaging methods with a dermoscope. The imaging methods may be distinguished by a combination of factors such as the presence or absence of gel, the presence or absence of a polarizing filter, the presence or absence of UV light, and contact or non-contact imaging. Therefore, the single-modal classifier may be a classifier that corresponds to a specific imaging method using a dermoscope.

[0029] More specifically, the identification unit 30 may include six single-modal classifiers corresponding to the modalities of image data captured by a total of six imaging methods, combining, for example, two patterns: presence or absence of gel and three patterns: presence or absence of use of a polarizing filter or UV light (use of polarizing filter, use of UV light, and neither use of polarizing filter nor UV light). Furthermore, when no gel is used, the imaging methods may be further distinguished into a contact type in which imaging is performed while in contact with the skin and a non-contact type in which imaging is performed away from the skin. In this case, the identification unit 30 may include nine single-modal classifiers corresponding to the modalities of image data captured by a total of nine imaging methods. Note that it is desirable for the identification unit 30 to support two or more modalities of image data captured by the above-mentioned nine imaging methods.

[0030] A multimodal classifier is a classifier that corresponds to a plurality of mutually different modalities, and is a CNN model that is trained to specialize in a specific combination of input data of the plurality of mutually different modalities. The multimodal classifier included in the classifier 30 corresponds to a plurality of mutually different modalities. The classifier 30 is required to include at least one multimodal classifier, but may also include a multimodal classifier that corresponds to any combination of two or more of the plurality of mutually different modalities. In other words, when the classifier 30 includes n single-modal classifiers, it may include up to m multimodal classifiers, where m is Σ n C r (Σ is the number of times r is between 2 and n) n C r (the sum of

[0031] As will be described in detail later, the multimodal classifier included in the classification unit 30 is configured to classify an object based on a plurality of classification scores calculated by a plurality of single-modal classifiers included in the classification unit 30 and a plurality of feature amounts calculated by different single-modal classifiers obtained in the process of calculating the plurality of classification scores.

[0032] The input control unit 20 inputs input data to a single-modal classifier corresponding to the modality of the input data. For example, the input control unit 20 reads skin image data selected based on an image selection operation by a user on the operation input unit 60 from the storage unit 50 and inputs the skin image data to the classification unit 30 as input data.

[0033] At this time, the input control unit 20 may input the input data to a single-modal classifier corresponding to the modality of the input data based on information identifying the modality of the input data obtained from metadata attached to the input data. That is, the input control unit 20 may determine the modality of the image selected by the user and input it to an appropriate single-modal classifier of the classification unit 30. The input control unit 20 may also input the input data to a single-modal classifier corresponding to the modality specified by the user. That is, the user may themselves select the modality of the image selected by the user, and the input control unit 20 may input the input data to a single-modal classifier corresponding to the modality selected by the user.

[0034] The output control unit 40 outputs the classification result generated by the classification unit 30. For example, in response to a user's image selection operation on the operation input unit 60, the output control unit 40 outputs the classification result generated by the classification unit 30 to the output unit 80. At this time, if the input data is data having a single modality, the output control unit 40 obtains the classification result from a single-modality classifier corresponding to that single modality, and if the input data is multiple data having multiple modalities, the output control unit 40 obtains the classification result from a multi-modal classifier corresponding to those multiple modalities. Note that cases where the input data is data having a single modality include, for example, when the user selects only one image, or when the user selects multiple images but the modalities of those images are the same.

[0035] The diagnosis support device 100 having the control unit 10 configured as described above can handle input data of various modalities and combinations of input data of various modalities because the classification unit 30 has multiple single-modal classifiers and one or more multi-modal classifiers.

[0036] In particular, in the diagnosis support device 100, the multimodal classifier classifies an object by using not only multiple classification scores (corresponding to multiple classification results) obtained by multiple single-modal classifiers, but also multiple feature amounts (corresponding to intermediate output information compared to the classification scores) obtained by the multiple single-modal classifiers. These multiple feature amounts are feature amounts obtained by different single-modal classifiers. By using these multiple feature amounts, it is possible to reflect the interaction between the multiple feature amounts in the classification result. Therefore, the diagnosis support device 100 can obtain a classification result using the multimodal classifier with higher accuracy than a classification result obtained by a relatively simple ensemble of multiple single-modal classifiers (e.g., averaging, majority voting, linear combination, etc.).

[0037] Furthermore, the diagnosis support device 100 allows the input control unit 20 to input input data to a single-modal classifier corresponding to the modality of the input data, thereby allowing the classifier 30 to function appropriately according to the input data. In particular, in a configuration in which the input control unit 20 automatically selects a classifier to which input data should be input based on metadata, the diagnosis support device 100 can perform classification processing using an appropriate classifier without making the user aware of the modality of the image data.

[0038] Furthermore, the diagnosis support device 100 can provide the user with the result of the classification process performed by an appropriate classifier (classification result) by having the output control unit 40 acquire the classification result from a classifier corresponding to the modality of the input data. Furthermore, when input data of multiple modalities is input, the classification result of the multimodal classifier is acquired in preference to the classification result of the single-modal classifier, thereby making it possible to obtain a highly accurate classification result.

[0039] Furthermore, the diagnosis support device 100 supports multiple modalities contained in multiple image data captured using multiple different imaging methods with a dermatoscope. This allows it to support various dermoscopy images that have traditionally been used for diagnosis. Furthermore, it is possible to provide highly accurate diagnostic support by combining various dermoscopy images that have traditionally been used. In particular, a multimodal classifier that supports multiple modalities contained in multiple image data captured using imaging methods with significantly different characteristics (e.g., an imaging method using a polarizing filter and an imaging method using UV light) may be particularly effective in utilizing the interaction effect and contribute to highly accurate classification. Therefore, the diagnosis support device 100 can provide diagnostic support by effectively utilizing dermoscopy images more effectively than conventional diagnostic support.

[0040] Fig. 3 is a diagram illustrating the detailed functional configuration of the classification unit of the diagnosis support device shown in Fig. 1. Fig. 4 is a diagram illustrating the network configuration of the classification unit of the diagnosis support device shown in Fig. 1. Fig. 5 is a diagram for explaining a score calculation method of a multimodal classifier. The configuration of the control unit 10 will be described in further detail with reference to Figs. 3 to 5.

[0041] For simplicity of explanation, the following description will be given taking as an example a case where the classification unit 30 includes a single-modal classifier 30a and a single-modal classifier 30b, each corresponding to a different imaging method using a dermoscope, and a multi-modal classifier 30x corresponding to a combination of the imaging method supported by the single-modal classifier 30a and the imaging method supported by the single-modal classifier 30b. Note that the imaging method supported by the single-modal classifier 30a is, for example, an imaging method using gel and polarized light, and the imaging method supported by the single-modal classifier 30b is, for example, an imaging method using gel and UV light.

[0042] As shown in Fig. 3, the single-modal classifier 30a and the single-modal classifier 30b each include input units 31a and 31b, feature extraction units 32a and 32b, classification score calculation units 33a and 33b, and output units 37a and 37b. Furthermore, the single-modal classifier 30a and the single-modal classifier 30b are configured as CNN models, and each has a neural network including input layers L11 and L21, two or more hidden layers L12 to L18, hidden layers L22 to L28, and output layers L19 and L29, as shown in Fig. 4.

[0043] The input units 31a and 31b correspond to the input layer L11 and the input layer L21, respectively. The input control unit 20 inputs skin image data of the corresponding modality to the input units 31a and 31b.

[0044] The feature extraction units 32a and 32b and the classification score calculation units 33a and 33b correspond to the two or more intermediate layers L12 to L18 and the intermediate layers L22 to L28, respectively. The feature extraction units 32a and 32b extract features from skin image data by repeating convolution processing (convolution layer), pooling processing (pooling layer), fully connected processing (affine layer), etc.

[0045] Regarding the features extracted by the feature extraction unit 32, low-level features obtained from local regions of the image are extracted from layers located relatively early in the network. In contrast, high-level features obtained from wider regions are extracted from layers located later in the network. In the process of diagnosing using multiple images, doctors first look at each image one by one to obtain high-level representations that summarize the patterns of each image, and then integrate these representations based on their own experience to determine a diagnosis. Therefore, reflecting the interaction between the former features and the latter features in the classification results is thought to be closer to the thought process of doctors when diagnosing using multiple images.

[0046] The identification score calculation units 33a and 33b correspond to the final layer or multiple layers including the final layer among the two or more intermediate layers. The identification score calculation units 33a and 33b respectively calculate an identification score based on the features extracted by the feature extraction units 32a and 32b. The identification score is output information from the final intermediate layer. The identification score is a one-dimensional array consisting of N elements, the number of classes identified by the single-modal classifier, i.e., an N-dimensional vector, and each element is an identification value (scalar) for the corresponding class. Each class corresponds to a disease whose diagnosis is supported by the diagnosis support device 100. The identification value is a value that represents the likelihood of the corresponding class, that is, the likelihood of the corresponding disease. Figure 5 shows an example in which the identification score Sa calculated by the single-modal classifier 30a and the identification score Sb calculated by the single-modal classifier 30b are output as N-dimensional vectors.

[0047] The output units 37a and 37b correspond to the output layers L19 and L29, respectively. The output units 37a and 37b input the classification scores obtained by the classification score calculation units 33a and 33b to an activation function to calculate classification results. The activation function is, for example, a SoftMax function, and calculates classification results that represent probabilities.

[0048] 3, the multimodal classifier 30x includes a plurality of input units 31a and 31b, a plurality of feature extraction units 32a and 32b, a plurality of classification score calculation units 33a and 33b, a classification score combination unit 34, an interaction feature extraction unit 35, a correction unit 36, and an output unit 38. The multimodal classifier 30x is also configured as a CNN model, and has a neural network including input layers L11 and L21, two or more hidden layers L12 to L18, hidden layers L22 to L28, hidden layers L31 and L32, and an output layer L33, as shown in FIG.

[0049] Each of the input units 31a and 31b is as described for the single-modal classifier. The input layers L11 and L21 corresponding to the input units 31a and 31b, respectively, are shared between each of the multiple single-modal classifiers and the multi-modal classifier, as shown in FIG.

[0050] The feature extraction units 32a and 32b, the multiple identification score calculation units 33a and 33b, the identification score combination unit 34, the interaction feature extraction unit 35, and the correction unit 36 correspond to two or more hidden layers L12 to L18, L22 to L28, and L31 to L32.

[0051] The feature extraction units 32a and 32b and the classification score calculation units 33a and 33b are as described for the single-modal classifier. The hidden layers L12 to L18 and L22 to L28 corresponding to the feature extraction units 32a and 32b and the classification score calculation units 33a and 33b are shared between each of the multiple single-modal classifiers and the multi-modal classifier, as shown in FIG.

[0052] The classification score combining unit 34, the interaction feature extracting unit 35, and the correcting unit 36 correspond to the hidden layers L31 and L32. The interaction feature extracting unit 35 corresponds to the hidden layer L31, and the classification score combining unit 34 and the correcting unit 36 correspond to the hidden layer L32. As shown in FIG. 4, these hidden layers are not shared with the single-modal classifier. That is, the multi-modal classifier includes, as hidden layers, shared hidden layers (hidden layers L12 to L18, hidden layers L22 to L28) shared with each of the multiple single-modal classifiers, and specific hidden layers (hidden layers L31 to L32) that are unique to the multi-modal classifier.

[0053] As shown in FIG. 3, the identification score combination unit 34 calculates a total identification score based on the multiple identification scores output from the multiple identification score calculation units 33. Specifically, the identification score combination unit 34 calculates the total identification score by combining the multiple identification scores. More specifically, as shown in FIG. 5, the identification score combination unit 34 calculates the total identification score by weighted addition of the multiple identification scores. That is, the total identification score is calculated as a linear combination of the multiple identification scores calculated by the multiple single-modal classifiers (single-modal classifier 30a, single-modal classifier 30b). FIG. 5 shows an example in which the identification score Sa calculated by the single-modal classifier 30a and the identification score Sb calculated by the single-modal classifier 30b are linearly combined using a weight w to calculate the total identification score Ts. Note that in this example using two single-modal classifiers, the weight w is, for example, an N-dimensional vector. The total identification score Ts is also an N-dimensional vector, like the identification scores Sa and Sb.

[0054] The interaction feature extraction unit 35 extracts an interaction feature based on multiple features obtained in the process of calculating multiple classification scores. The multiple features input to the interaction feature extraction unit 35 are each extracted by a different single-modal classifier (single-modal classifier 30a, single-modal classifier 30b) and are represented as a matrix or tensor. The features are also referred to as a feature map. The interaction feature extraction unit 35 linearly combines the multiple features and further performs convolution processing (convolution layer) and fully connected processing (affine layer) to extract an interaction feature FI consisting of an N-dimensional vector as shown in FIG. 5.

[0055] Note that an interaction feature is a feature in which, when multiple feature quantities are regarded as a dependent variable with factors, the influence of each of the multiple feature quantities on the interaction feature, which is the dependent variable, changes depending on the levels of the other feature quantities. In other words, an interaction feature cannot be expressed as a linear combination of multiple feature quantities, but is calculated as a nonlinear combination of multiple feature quantities.

[0056] The correction unit 36 corrects the overall identification score based on a plurality of feature quantities. Specifically, the correction unit 36 corrects the overall identification score with an interaction feature quantity extracted based on the plurality of feature quantities. More specifically, as shown in Fig. 5, the correction unit 36 adds the interaction feature quantity FI as a bias to the overall identification score TS to calculate a final identification score Sx consisting of the same N-dimensional vector as the identification scores Sa and Sb.

[0057] The output unit 38 corresponds to the output layer (output layer L33). The output unit 38 inputs the final classification score obtained by the correction unit 36 into an activation function to calculate a classification result. The activation function is, for example, a SoftMax function, and calculates a classification result that represents a probability. Note that although an example is shown in FIGS. 3 and 4 in which the output layer and the output unit are not shared between each of the multiple single-modal classifiers and the multi-modal classifier, they may be shared.

[0058] In the diagnosis support device 100 having the control unit 10 configured as described above, the multimodal classifier corrects the overall classification score based on multiple feature quantities. By correcting the overall classification score, which corresponds to a relatively simple ensemble of multiple single-modal classifiers, based on multiple feature quantities obtained from the multiple single-modal classifiers, it is possible to reflect interactions not taken into account in the overall classification score. In particular, by extracting interaction features based on multiple feature quantities and correcting the overall classification score with the extracted interaction features, it is possible to more reliably reflect interactions through correction. Therefore, it is possible to obtain classification results with higher accuracy than conventional methods.

[0059] Furthermore, in the diagnosis support device 100, the multiple single-modal classifiers and the multi-modal classifier are each configured as a CNN model, and the input layer and part of the hidden layer (shared hidden layer) of the multi-modal classifier are shared with one of the multiple single-modal classifiers. Therefore, it is possible to configure a multi-modal classifier by extending an existing single-modal classifier, and the number of parameters required for the entire diagnosis support device 100 can be significantly reduced compared to when the multi-modal classifier is configured as a completely new classifier independent of the single-modal classifier. Reducing the number of parameters can reduce memory capacity accordingly, and also reduce the time and effort required for training a machine learning model.

[0060] Furthermore, the diagnosis support device 100, which configures a multimodal classifier by extending a single-modal classifier, can efficiently use training data, and in this respect is also superior to cases where a single-modal classifier and a multimodal classifier are configured independently.

[0061] The collected learning image data usually contains more image data of a specific modality than sets of image data of multiple modalities. For example, if images taken with a gel imaging method and images taken with a UV light imaging method are collected, the number of images taken with a gel imaging method and images taken with a UV light imaging method provided as a set (i.e., sets of images of the same subject taken with different imaging methods) is often less than one-tenth the number of each type provided individually.

[0062] Therefore, when a single-modal classifier and a multi-modal classifier are configured independently, the training data for the multi-modal classifier tends to be insufficient compared to the training data for the single-modal classifier, and a situation may arise in which the classification accuracy of the multi-modal classifier is inferior to that of the single-modal classifier due to insufficient training.

[0063] In contrast, the diagnosis support device 100 can first construct a single-modal classifier by training using a sufficient amount of training data. Alternatively, an existing single-modal classifier can be used as is. Then, the multi-modal classifier can be trained. At this time, the parameters of the layers shared with the single-modal classifier can be fixed while the multi-modal classifier is trained. As a result, in training the multi-modal classifier, only the parameters of the layers unique to the multi-modal classifier (intrinsic hidden layers) are optimized through training, so sufficient training is possible even with less training data than in training the single-modal classifier. In other words, it is possible to avoid overtraining the multi-modal classifier, which is caused by adapting only to a small amount of training data.

[0064] Furthermore, by providing a shared and unique hidden layer in the hidden layer of the multimodal classifier and inputting the outputs from multiple single-modal classifiers to the unique hidden layer, the classification score of the single-modal classifier can be highly corrected. In particular, by inputting the features obtained during the classification score calculation process in addition to the classification scores output by the single-modal classifiers, it is possible to calculate a score that reflects interaction effects that cannot be obtained by linearly combining classification scores. Furthermore, as shown in Figure 3, by extracting interaction features from multiple features in advance and using the extracted interaction features as bias terms in the weighted addition of classification scores, it is possible to correct the overall classification score, which corresponds to the linear combination of classification scores.

[0065] Fig. 6 is a diagram illustrating the parameter sharing range. A configuration that is more effective in reducing the number of parameters will be described below with reference to Fig. 6. Although the above describes the sharing of parameters between a multimodal classifier and a single-modal classifier, parameters may also be shared among multiple single-modal classifiers.

[0066] That is, each of the multiple single-modal classifiers may have a neural network with two or more hidden layers (hidden layers L12 to L18) as shown in Fig. 6, and one or more of the two or more hidden layers (hidden layers L12 to L14) may share parameters with the hidden layers of other single-modal classifiers. This allows one or more hidden layers to use the same parameters as the hidden layers of other single-modal classifiers.

[0067] The diagnosis support device 100 has multiple single-modal classifiers that correspond to input data of different modalities, and these single-modal classifiers are all configured as CNN models that handle skin image data. Even between single-modal classifiers that handle input data of different modalities, parameters can be shared as long as they handle input data of modalities that are somewhat similar, such as between images, rather than input data of completely different modalities, such as between images and speech.

[0068] The shared parameters are weight parameters, such as those in the convolution filter of the convolution layer, and this parameter sharing is also called weight shraring. However, the parameters are not limited to weights and other types of parameters may also be shared.

[0069] It is preferable to share parameters in the intermediate layers in the first half of the network rather than in the second half, because intermediate layers in the first half often extract local features (such as corners, edges, and colors) that are common across modalities, making it possible to effectively share parameters between neural networks targeting image data from different modalities.

[0070] As shown in Fig. 6, by sharing parameters among multiple single-modal classifiers, the number of parameters required for the entire diagnosis support device 100 can be further reduced. Reducing the number of parameters also reduces memory capacity. Furthermore, by sharing weight parameters, which are parameters to be learned, the time and effort required for learning a machine learning model can be reduced.

[0071] When parameters are partially shared among multiple single-modal classifiers, it is generally desirable to share parameters in layers in a relatively early part of the network, since lower-level features are more common between modalities. While Fig. 6 shows an example in which each of the multiple features used to extract interaction features is extracted in an intermediate layer that does not share parameters with the intermediate layers of other single-modal classifiers, each of the multiple features used to extract interaction features is not limited to an intermediate layer that does not share parameters with the intermediate layers of other single-modal classifiers, and may also be extracted in an intermediate layer that shares parameters (intermediate layers L12 to L14).

[0072] The parameters of a trained single-modal classifier may be copied and used when training another single-modal classifier, i.e., used for transfer learning, thereby improving the learning efficiency.

[0073] Fig. 7 is a flowchart showing an example of a learning process for generating the classifier shown in Fig. 1. The flow of the learning process will be described below using an example in which the learning process is performed by the diagnosis assistance device 100.

[0074] In the learning process, first, disease labels are assigned to image data for learning (step S1). The image data for learning is, for example, data of dermoscopy images taken using a dermoscope, and is collected in advance. These image data are known to have diseases corresponding to lesions shown in the images. In step S1, the diagnosis support device 100 assigns labels of known diseases to the image data according to input by the operator. Note that step S1 may be performed before the series of learning processes shown in FIG. 7.

[0075] Next, the diagnosis support device 100 classifies the labeled training image data by modality (step S2). The labeled training image data has various modalities. In step S2, the diagnosis support device 100 classifies the image data by modality. The classification may be performed based on metadata attached to the image data, or may be performed according to input by the operator, as in step S1. Note that, here, an example will be described in which the image data is classified into two types: image data of modality MA and image data of modality MB.

[0076] After completing the preparation of image data for training through the processes of steps S1 and S2, the diagnosis support device 100 first trains a model (CNN model) of the single-modal classifier 30a using image data of modality MA (step S3). Here, parameters are adjusted using backpropagation. That is, the image data for training is input to the single-modal classifier 30a to calculate a loss function, and the parameters are adjusted by calculating the gradient of the loss function (cross-entropy error) with respect to the parameters (weights and biases) using the chain rule.

[0077] When the learning of the model of the single-modal classifier 30a is completed, the diagnosis support device 100 learns a model (CNN model) of the single-modal classifier 30b using image data of modality MB (step S4). In step S4, learning of some parameters may be omitted by using parameters of the model already learned in step S3. That is, learning may be performed while fixing parameters of the model of the single-modal classifier 30b that are shared with the model of the single-modal classifier 30a, and other parameters may be adjusted.

[0078] Note that steps S3 and S4 may be performed simultaneously. In this case, the diagnosis support device 100 may calculate the sum of a loss function calculated by inputting image data of modality MA to the single-modal classifier 30a and a loss function calculated by inputting image data of modality MB to the single-modal classifier 30b. Both the model parameters of the single-modal classifier 30a and the model parameters of the single-modal classifier 30b (including parameters shared between models) may be optimized based on the gradient of the summed loss function.

[0079] When the training of the single-modal classifier is completed, the diagnosis support device 100 trains a model (CNN model) of the multimodal classifier 30x using a set of image data of modality MA and image data of modality MB (step S5). In step S5, parameters of the models trained in steps S3 and S4 are shared, thereby training only parameters specific to the multimodal classifier.

[0080] By performing learning according to the procedure shown in FIG. 7, it is possible to efficiently construct a classification device that is compatible with modalities MA and MB, or both.

[0081] Fig. 8 is a flowchart showing an example of inference processing performed by the diagnosis support device shown in Fig. 1. The following describes the flow of the inference processing performed by the diagnosis support device 100. Note that the inference processing performed by the diagnosis support device 100, which is an identification device, is identification processing.

[0082] In the inference process, first, the diagnosis support device 100 acquires unknown image data (step S11). The image data acquired in step S11 is, for example, a dermoscopy image acquired by the data input unit .

[0083] Next, the diagnosis support device 100 determines the modality of the unknown image data acquired in step S11 (step S12). As an example, the control unit 10 determines the modality based on metadata of the image data. Thereafter, the diagnosis support device 100 inputs the image data acquired in step S11 to a single-modal classifier corresponding to the modality determined in step S12 (step S13).

[0084] The diagnosis support device 100 determines whether all of the image data input in step S11 has been acquired (step S14). For example, if a set of image data consisting of image data of multiple different modalities has been input in step S11, the diagnosis support device 100 determines whether all of the multiple image data constituting the set have been acquired. The diagnosis support device 100 repeats the processes from step S11 to step S14 until all of the image data has been acquired.

[0085] When all the image data have been acquired, the diagnosis support device 100 executes inference (step S14). Here, the above-described classifier 30 executes the classification process. Note that, when a set of image data consisting of image data of a plurality of different modalities has been input in step S11, step S14 includes at least the steps of: acquiring a plurality of classification scores from a corresponding single-modal classifier to which each of the plurality of input data of different modalities has been input; acquiring a plurality of feature amounts calculated by the single-modal classifiers that are each different from each other and obtained in the process of calculating the plurality of classification scores; and classifying the object using a multi-modal classifier corresponding to a plurality of modalities corresponding to the plurality of input data, based on the plurality of classification scores and the plurality of feature amounts.

[0086] When the inference is completed, the diagnosis support device 100 acquires the classification result (steps S16 to S18). Specifically, the diagnosis support device 100 first determines whether the modality of the input image data is single (step S16). That is, it determines whether the modality of the image data acquired in step S11 is single.

[0087] If the modality is single (YES in step S16), the diagnosis support device 100 acquires a classification result from a single-modal classifier corresponding to the single modality (step S17). On the other hand, if the modality is not single (NO in step S16), the diagnosis support device 100 acquires a classification result from a multi-modal classifier corresponding to the non-single combination of modalities (step S18).

[0088] Finally, the diagnosis support device 100 outputs the classification result acquired in step S17 or step S18 (step S19). Here, the diagnosis support device 100 displays the classification result on a display device, for example.

[0089] By performing inference according to the procedure shown in Fig. 8, when the unknown image data has a single modality, the classification result obtained by the single-modal classifier can be provided to the user, and when the unknown image data has multiple modalities, the classification result obtained by the multi-modal classifier can be provided to the user. Therefore, it is possible to provide the user with the most accurate classification result corresponding to image data of various modalities and combinations of image data of various modalities.

[0090] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention. The present invention is not limited to the above-described embodiments, and should be understood to encompass various modifications and alternative forms of the above-described embodiments. For example, it will be understood that each embodiment can be embodied by modifying its components without departing from its spirit and scope. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-described embodiments. Furthermore, those skilled in the art will understand that various embodiments can be implemented by deleting some components from all of the components shown in the embodiments or by adding some components to the components shown in the embodiments. In other words, the identification device, identification method, and program can be variously modified and changed without departing from the scope of the claims.

[0091] In the above-described embodiment, image data of different imaging methods is exemplified as input data of different modalities, but the input data is not limited to image data. For example, other data such as audio data and text data may be included. Furthermore, in the above-described embodiment, skin image data is exemplified as image data, but the image data is not limited to skin image data, and image data of different parts may be used depending on the diagnostic target.

[0092] In the above-described embodiment, as shown in FIG. 1, an example was shown in which the input control unit 20 and the output control unit 40 are configured separately from the recognition unit 30 and each controls input and output to the recognition unit 30, but the functions of the input control unit 20 and the output control unit 40 may also be performed by an input unit or an output unit within the recognition unit 30.

[0093] Fig. 9 is a diagram showing a modified example of the functional configuration of the classifier shown in Fig. 3. For example, as shown in Fig. 9, the classifier 30 may have a single input unit 31, and may allocate input data acquired by the input unit 31 to an appropriate single-modal classifier 30c or 30d according to the modality by a switching process performed within the input unit 31. The classifier shown in Fig. 9 differs from the classifier 30 shown in Fig. 3 in that it has a single input unit instead of an input unit provided for each single-modal classifier.

[0094] 9 shows an example in which the input unit 31 is shared, but the classification unit 30 may include a single output unit instead of the output units provided for each of the single-modal classifiers and the multi-modal classifier. The single output unit may perform switching processing according to the modality of the input data, so that the classification result according to the modality can be output from the classification unit 30.

[0095] 9 shows an example in which the input unit 31 is shared, but not only the input layer corresponding to the input unit 31 but also the intermediate layers that share parameters may be shared. That is, the above-described switching process may be performed in the feature extraction unit (intermediate layer) rather than in the input unit (input layer). In this case, too, one or more intermediate layers of a single-modal classifier use the same parameters as the intermediate layers of other single-modal classifiers, just like when different intermediate layers share parameters.

[0096] In the above-described embodiment, an example has been shown in which a deep learning model, particularly a CNN model, is used as the machine learning model, but the machine learning model used in the identification unit 30 is not limited to the CNN model. Other different algorithms, such as a support vector machine (SVM), may be adopted, or may be configured by combining the CNN model with them. Furthermore, the other algorithms used in the process corresponding to the feature extraction unit are not limited to machine learning model algorithms such as CNN and SVM, and may also be existing image processing models.

[0097] When the processing equivalent to that of the feature extraction unit is performed using another algorithm, the processing equivalent to that of the feature extraction unit may be performed as preprocessing of the classification processing performed by the classification unit 30 using a CNN model, and may be performed, for example, by the input control unit 20. Examples of preprocessing include image reduction processing, brightness adjustment processing, filtering processing, etc. Furthermore, if the classification device is equipped with a photographing device, the preprocessing may include cropping processing performed during photographing.

[0098] In this specification, the expression "based on A" does not mean "based only on A," but means "based on at least A." That is, "based on A" may also be based on B in addition to A. [Explanation of symbols]

[0099] 10 Control Unit 20 Input control section 30 Identification unit 30a, 30b Single-modal classifier 30b Single-modal classifier 30x multimodal classifier 31a, 31b Input section 32a, 32b Feature extraction unit 33a, 33b Identification score calculation unit 34 Identification score combination 35 Interaction feature extraction unit 36 Correction unit 37a, 37b, 38, 80 output section 40 Output control section 50 Storage section 60 Operation input section 70 Data Entry Section 100 Diagnostic support device FI Interaction feature L11, L21 input layer L12~L18, L22~L28, L31, L32 middle layer L19, L29, L33 output layer Sa, Sb discrimination score Sx Final discrimination score TS Total Discrimination Score

Claims

1. a multimodal classifier corresponding to a plurality of different modalities; a plurality of single-modal classifiers corresponding to the plurality of different modalities, The multimodal classifier is configured to classify an object based on a plurality of classification scores calculated by the plurality of single-modal classifiers and a plurality of feature amounts calculated by the single-modal classifiers that are different from each other and obtained in the process of calculating the plurality of classification scores. An identification device characterized by:

2. The multimodal classifier includes a correction unit that corrects a total classification score calculated by combining the plurality of classification scores based on the plurality of feature amounts.

2. The identification device according to claim 1.

3. the multimodal classifier further includes an interaction feature extraction unit that extracts an interaction feature based on a plurality of feature quantities; The correction unit corrects the overall classification score using the interaction feature.

3. The identification device according to claim 2.

4. Each of the plurality of single-modal classifiers A neural network having two or more hidden layers, At least one of the two or more hidden layers uses the same parameters as those of the hidden layers of other single-modal classifiers.

4. The identification device according to claim 3.

5. Each of the plurality of feature quantities is extracted in an intermediate layer that does not share parameters with intermediate layers of other single-modal classifiers.

5. The identification device according to claim 4.

6. The interaction feature extraction unit extracts the interaction feature from the plurality of features in the intermediate layer that do not share the parameter.

6. The identification device according to claim 5.

7. The multimodal classifier an input layer shared with each of the plurality of single-modal classifiers; two or more hidden layers including a shared hidden layer shared with each of the plurality of single-modal classifiers and a specific hidden layer specific to the multi-modal classifier; and an output layer.

7. The identification device according to claim 3, wherein the first and second electrodes are electrically connected to each other.

8. The intrinsic intermediate layer is an identification score combining unit that calculates the overall identification score by weighted addition of the plurality of identification scores; the interaction feature extraction unit extracting the interaction feature based on the plurality of features output from the shared hidden layer; the correction unit adding the interaction feature to the overall classification score as a bias, The output layer inputs a final classification score obtained by adding the interaction feature to the overall classification score into an activation function to calculate a classification result.

8. The identification device according to claim 7.

9. further comprising an output control unit that outputs the classification result; The output control unit When the input data has a single modality, the classification result is obtained from a single-modal classifier corresponding to the single modality; When the input data is a plurality of data having the plurality of modalities, the classification result is obtained from the multimodal classifier corresponding to the plurality of modalities.

7. The identification device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

10. an input control unit that inputs input data to a single-modal classifier that corresponds to the modality of the input data; 6. The identification device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

11. The input control unit inputs the input data to a single-modal classifier corresponding to the modality of the input data based on information identifying the modality of the input data obtained from metadata attached to the input data.

11. The identification device according to claim 10.

12. the identification device constitutes a diagnosis support device that outputs an identification result of a disease to be diagnosed in response to input skin image data; the skin image data having the plurality of different modalities that the multimodal classifier corresponds to includes image data captured by a plurality of different imaging methods using a dermoscope; The multiple imaging methods are distinguished by the combination of factors such as the presence or absence of gel, the presence or absence of a polarizing filter, the presence or absence of UV light, and contact or non-contact imaging.

6. The identification device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

13. The multimodal classifier corresponds to a plurality of input data having a plurality of mutually different modalities, Each of the plurality of single-modal classifiers corresponds to a respective input data having the plurality of mutually different modalities.

7. The identification device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

14. An identification method performed by an identification device, comprising: inputting each of a plurality of input data having different modalities into a corresponding single-modal classifier to obtain a plurality of classification scores; acquiring a plurality of feature amounts calculated by different single-modal classifiers obtained in the process of calculating the plurality of classification scores; classifying an object using a multimodal classifier corresponding to a plurality of modalities corresponding to the plurality of input data based on the plurality of classification scores and the plurality of feature amounts; 10. A method for identifying a target object, comprising:

15. On the computer, inputting each of a plurality of input data having different modalities into a corresponding single-modal classifier to obtain a plurality of classification scores; acquiring a plurality of feature amounts calculated by different single-modal classifiers obtained in the process of calculating the plurality of classification scores; classifying an object using a multimodal classifier corresponding to a plurality of modalities corresponding to the plurality of input data based on the plurality of classification scores and the plurality of feature amounts; A program characterized by executing the following.

Images