Method, information processing device and program

Abstract

To provide a method for generating explanatory information for any range of discrimination object data.SOLUTION: A method for generating explanatory information regarding class classification includes the steps of: inputting input data into a machine learning model again and acquiring a set of M×N-number of first feature vectors corresponding to a size of a first feature map from L-number of first feature maps, the output of a specific layer; generating a known feature vector group consisting of the set of first feature vectors for every class; receiving specification of a range in discrimination object data; inputting the discrimination object data into the machine learning model and acquiring a set of M×N-number of second feature vectors corresponding to a size of the second feature map from a second feature map, the output of the specific layer; and calculating a similarity between the specified range in the discrimination object data and at least one class using the set of second feature vectors and the known feature vector group of at least one class.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a method, an information processing device, and a program. [Background technology]

[0002] Non-Patent Document 1 discloses a technology related to deep neural networks (DNNs) that makes the inference process interpretable. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yuki Ukai and 3 others, “Toward Prototypical Part Interpretable Similarity Learning With ProtoMetric”, IEEE Access, June 19, 2023, [online], [Retrieved January 26, 2024], Internet URL: https: / / doi.org / 10.1109 / ACCESS.2023.3287638 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Non-Patent Document 1, in order to explain the results of class discrimination by DNN, explanatory information is generated that indicates which features of a prototype selected as representative data from the training data the small regions into which the input data are divided are similar to, as well as the degree of similarity. However, the technology described in Non-Patent Document 1 only calculates the degree of similarity when the input data contains a small region that is similar to the features of the prototype. There is a problem in that explanatory information cannot be generated for any range specified by the user within the entire range of the input data. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] According to a first aspect of the present disclosure, there is provided a method for generating explanation information for class classification of a machine learning model that classifies classes of input data, wherein the machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.This method includes the steps of: (a) inputting the input data belonging to one of the classes into the machine learning model again, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) executing step (a) using, as input, each of the plurality of input data belonging to one of the classes; (c) executing step (b) for each of the plurality of classes to generate, for each of the classes, a group of known feature vectors consisting of the set of the first feature vectors associated with the classes; and (d) discrimination target data different from the input data, which is used to determine whether the machine learning model can recognize the first feature vectors. (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature quantities included in the second feature maps along a channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in step (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range of the data to be discriminated and at least one of the classes using the set of second feature vectors acquired in step (e) and the group of known feature vectors of at least one of the classes.

[0007] According to a second aspect of the present disclosure, there is provided an information processing device for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, wherein the machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.This information processing device includes: (a) a process of inputting the input data belonging to one of the classes again to the machine learning model, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) a process of executing the process (a) using, as input, each of the plurality of input data belonging to one of the classes; (c) a process of executing the process (b) for each of the plurality of classes, thereby generating, for each of the classes, a known feature vector group consisting of the set of the first feature vectors associated with the classes; and (d) a process of generating discrimination target data different from the input data, the known feature vector group being associated with the class, the known feature vector group being composed of the set of the first feature vectors associated with the classes, the known feature vector group being obtained by vectorizing, along the channel direction, feature amounts included in the L first feature maps, the L first feature maps being outputs of a specific layer of the machine learning model. (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature amounts included in the second feature maps along the channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in the process (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range of the data to be discriminated and at least one of the classes using the set of second feature vectors acquired in the process (e) and the group of known feature vectors of at least one of the classes.

[0008] According to a third aspect of the present disclosure, there is provided a program for causing a computer to execute a process for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, wherein the machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.This program includes: (a) a process of inputting the input data belonging to one of the classes again into the machine learning model, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) a process of executing the process (a) using, as input, each of the plurality of input data belonging to one of the classes; (c) a process of executing the process (b) for each of the plurality of classes, thereby generating, for each of the classes, a group of known feature vectors consisting of the set of the first feature vectors associated with the class; and (d) discrimination target data that is different from the input data and is input to the machine learning model. The computer is caused to execute the following processes: (a) receiving a specification of a range in the data to be discriminated, for which similarity is to be calculated; (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature amounts included in the second feature maps along the channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in process (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range in the data to be discriminated and at least one of the classes, using the set of second feature vectors acquired in process (e) and the group of known feature vectors of at least one of the classes. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an evaluation system according to an embodiment. [Figure 2]FIG. 1 is an explanatory diagram illustrating the configuration of a machine learning model. [Figure 3] FIG. 2 is an explanatory diagram showing the configuration of an intermediate layer and an output layer. [Figure 4] FIG. 10 is an explanatory diagram illustrating an example of a residual block. [Figure 5] 10 is a flowchart showing a process for generating a group of known feature vectors. [Figure 6] FIG. 2 is an explanatory diagram showing the configuration of a group of known feature vectors. [Figure 7] 1 is a flowchart showing the first half of the process for evaluating a machine learning model. [Figure 8] 10 is a flowchart showing the second half of the process for evaluating a machine learning model. [Figure 9] FIG. 10 is an image diagram of a similarity map expressed in the form of a heat map. DETAILED DESCRIPTION OF THE INVENTION

[0010] A. Implementation: FIG. 1 is a block diagram showing an evaluation system 5 according to an embodiment. The evaluation system 5 includes an information processing device 100 and a camera 400. The camera 400 captures an image of an object. The camera 400 may be a camera that captures a color image, or a camera that captures a monochrome image or a spectral image. The captured image captured by the camera 400 is input to a machine learning model.

[0011] The information processing device 100 is a computer including a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display device 150 connected to the interface circuit 130. A camera 400 is also connected to the interface circuit 130.

[0012] The processor 110 executes a program P1 stored in the memory 120, thereby functioning as a learning execution unit 112 and a classification processing unit 114. The learning execution unit 112 executes a learning process for the machine learning model 200 using a training data group TDG. The trained machine learning model 200 determines to which of multiple classes an input image IM is classified. The classification processing unit 114 includes a class discrimination unit 310 and an evaluation unit 330. The class discrimination unit 310 inputs the input image IM to the machine learning model 200 and determines the class to which the input image IM belongs. The evaluation unit 330 uses intermediate data from the machine learning model 200 to generate explanatory information used to evaluate the class discrimination results of the trained machine learning model 200. The processor 110 not only executes the processes described below, but also has the function of displaying data obtained by the processes and data generated during the processes on the display device 150.

[0013] The memory 120 stores a program P1, a machine learning model 200, a training data group TDG, and a known feature vector group KVcG. The training data group TDG is also referred to as a "training data set." The configuration of the machine learning model 200 will be described in detail later.

[0014] The training data group TDG includes multiple training data TD, which are training data. The training data group TDG consists of a set of pairs of an input image IM and a priori labels LB associated with the input image IM. In this embodiment, the priori labels LB are labels indicating the type of object. The input image IM is also referred to as "input data." In this embodiment, a handwritten digit image from MNIST is used as the input image IM. The digits represented by the input image IM become the priori labels LB associated with the input image IM. For example, if the input image IM is a handwritten digit image representing "2," the prior label associated with this input image IM is "2." Note that in this embodiment, "label" and "class" have the same meaning.

[0015] The known feature vector group KVcG is a set of feature vectors obtained when the training data group TDG is input to the trained machine learning model 200. Details of the known feature vector group KVcG will be described later.

[0016] FIG. 2 is an explanatory diagram showing an overview of the configuration of the machine learning model 200. This machine learning model 200 is a convolutional neural network in which an input layer 210, an intermediate layer 290, and an output layer 300 are arranged in this order. More specifically, the machine learning model 200 is a residual neural network (ResNet). The input layer 210, which receives an input image, is the lowest layer. The output layer 300 is the highest layer. As will be described in detail later, the intermediate layer 290 has multiple convolutional layers that extract features of the input image IM. Each layer of the machine learning model 200 is made up of scalar neurons. Hereinafter, the term "node" will be used as a broader concept of neuron.

[0017] The hidden layer 290 includes a first convolutional layer 220, a second convolutional block 240, a third convolutional block 250, and a fourth convolutional block 260. The convolutional block includes multiple convolutional layers. In the following description, the first convolutional layer 220, the second convolutional block 240, the third convolutional block 250, and the fourth convolutional block 260 will be referred to as the "Conv1 layer 220," the "Conv2_x layer 240," the "Conv3_x layer 250," and the "Conv4_x layer 260." The first convolutional layer 220, the second convolutional block 240, the third convolutional block 250, and the fourth convolutional block 260 may also be referred to simply as "layers."

[0018] FIG. 2 shows a first axis x and a second axis y that define the planar coordinates of the node array, and a third axis z that represents depth. The sizes in the x and y directions are called "resolution." The size in the z direction is the number of channels. FIG. 2 shows, for example, that the sizes in the x, y, and z directions of the first convolutional layer 220 are 128, 128, and 64. These three axes x, y, and z are also used as coordinate axes indicating the position of each node in other layers. However, in FIG. 2, these axes x, y, and z are omitted from the illustration in layers other than the Conv1 layer 220.

[0019] The resolution W1 after convolution is calculated using the following formula: Here, W0 represents the resolution before convolution, Wk represents the surface size of the kernel, S represents the stride, and P represents padding. Ceil{X} is a function that performs an operation to round up the decimal point of X. A kernel is a coefficient matrix used to perform a convolution operation. A kernel is sometimes called a filter. In this embodiment, image data is input to the machine learning model 200, so the surface size of the kernel is also two-dimensional. Note that the parameter values ​​of each layer are merely examples and can be changed as desired. Furthermore, although an example in which the kernel shape is square will be described in the embodiment, the kernel shape may also be rectangular. W1=Ceil{(W0+2P-Wk+1) / S} (A1)

[0020] In the description of each layer, the string before the parentheses is the layer name, and the numbers in the parentheses are, respectively, the output size, the number of channels, the kernel size, and the stride. For example, the Conv1 layer 220 is described as Conv1[128*128,64,3,1]. This indicates that the layer name of the Conv1 layer 220 is "Conv1," the output size is 128x128 pixels, the number of channels is 64, the kernel size is 3x3, and the stride is 1. In Figure 2, these parameters are shown below each layer. As will be described in detail later, the Conv2_x layer 240, the Conv3_x layer 250, and the Conv4_x layer 260 have multiple convolutional layers. Because the stride may differ for each convolutional layer, the stride is described as "S."

[0021] 2 shows the resolution of each layer when the size of the input image IM is 128 x 128 pixels. The size of the intermediate data output from each layer is changed appropriately depending on the size of the input image IM.

[0022] An input image IM having a size of 128 x 128 pixels is input to the Conv1 layer 220. The input image IM is a grayscale image. The input image IM has one channel of information. If the input image IM is an RGB image, the input image IM contains three bands of wavelength information. In this case, the input image IM has three-dimensional (three-channel) information.

[0023] As shown in FIG. 3, the Conv1 layer 220 performs convolution with a kernel size of 3×3, a stride of 1, and padding of 1. The Conv1 layer 220 outputs intermediate data with a size of 128×128 pixels and 64 channels. The intermediate data output from the convolution layer is also called a feature map. In this embodiment, the feature map is represented by a quadratic array. The number of channels represents the number of feature maps output from the convolution layer.

[0024] The Conv2_x layer 240, the Conv3_x layer 250, and the Conv4_x layer 260 each have a residual block.

[0025] FIG. 4 is an explanatory diagram showing an example of a residual block. In the example shown in FIG. 4, one residual block has two convolutional layers L1 and L2 connected in series and activation functions ReLU1 and ReLU2. The output of the first convolutional layer L1 is applied with the activation function ReLU1 and input to the convolutional layer L2. The input to the convolutional layer L1 is added to the output of the convolutional layer L2 via a skip connection (residual connection). It is known that using a residual block can address the gradient vanishing problem that accompanies the multi-layer configuration of a machine learning model. While FIG. 4 shows an example of a PlainBlock structure, a residual block with a Bottleneck structure may also be used.

[0026] FIG. 3 is an explanatory diagram showing the configuration of the hidden layer 290 and the output layer 300. The Conv2_x layer 240 is configured as a residual block. The Conv2_x layer 240 includes a convolutional layer 241, a convolutional layer 242, and a convolutional layer 243. The convolutional layers 241 to 243 are referred to as the "Conv2_1 layer 241," the "Conv2_2 layer 242," and the "Conv2_3 layer 243," respectively. In this embodiment, a convolutional layer is also provided for the skip connection. This allows downsampling to be performed along with the skip connection. Furthermore, the Conv2_1 layer 241 performs convolution with a kernel size of 3×3, a stride of 2, and padding of 1. The Conv2_2 layer 242 performs convolution with a kernel size of 3×3, a stride of 1, and padding of 1. The Conv2_3 layer 243 performs convolution with a kernel size of 1×1, a stride of 2, and padding of 0. The Conv2_1 layer 241, the Conv2_2 layer 242, and the Conv2_3 layer 243 each output intermediate data having a size of 64×64 pixels and a number of channels of 64. Therefore, the Conv2_x layer 240 outputs intermediate data having a size of 64×64 pixels and a number of channels of 64. Note that a convolutional layer does not necessarily have to be provided in the skip connection.

[0027] Intermediate data output by the immediately preceding Conv1 layer 220 is input to the Conv2_1 layer 241. The output of the Conv2_1 layer 241 is input to the Conv2_2 layer 242 via an activation function ReLU. The sum of the output of the Conv2_2 layer 242 and the output of the Conv2_3 layer 243 is input to the Conv3_x layer 250 via an activation function ReLU.

[0028] The Conv3_x layer 250 is configured as a residual block. The Conv3_x layer 250 includes a convolutional layer 251, a convolutional layer 252, and a convolutional layer 253. The convolutional layers 251 to 253 are referred to as the "Conv3_1 layer 251," the "Con3_2 layer 252," and the "Conv3_3 layer 253," respectively. The Conv3_1 layer 251 performs convolution with a kernel size of 3×3, a stride of 2, and padding of 1. The Conv3_2 layer 252 performs convolution with a kernel size of 3×3, a stride of 1, and padding of 1. The Conv3_3 layer 253 performs convolution with a kernel size of 1×1, a stride of 2, and padding of 0. The Conv3_1 layer 251, the Conv3_2 layer 252, and the Conv3_3 layer 253 each output intermediate data having a size of 32×32 pixels and a channel count of 128. Therefore, the Conv3_x layer 250 outputs intermediate data having a size of 32×32 pixels and a channel count of 128.

[0029] The intermediate data output by Conv2_x 240 is input to the Conv3_1 layer 251. The output of the Conv3_1 layer 251 is input to the Conv3_2 layer 252 via an activation function ReLU. The sum of the output of the Conv3_2 layer 252 and the output of the Conv3_3 layer 253 is input to the Conv4_x layer 260 via an activation function ReLU.

[0030] The third convolution block 250 outputs intermediate data having a size of 32×32 pixels and 128 channels.

[0031] The Conv4_x layer 260 is configured as a residual block. The configuration of each layer in the Conv4_x layer 260 is the same as that of the Conv3_x layer 250, except for the size and number of channels of the intermediate data output by each layer. The Conv4_x layer 260 outputs intermediate data whose size is 16x16 pixels and whose number of channels is 256.

[0032] In this way, the machine learning model 200 is configured to reduce the size of the feature map and increase the number of channels as it passes through each convolutional layer.

[0033] In FIG. 3, the output layer 300 includes a pooling layer 301 and a fully connected layer 302. The pooling layer 301 is referred to as the "Avg_pool layer 301." The fully connected layer 302 is referred to as the "FC layer 302." The Avg_pool layer 301 is a GAP (Global Average Pooling) layer. The Avg_pool layer 301 uses the output of the immediately preceding convolutional layer to calculate the average of the feature map for each channel and outputs the calculated average as a vector. The feature map output by the Conv4_x layer 260 has a size of 16x16 pixels and 256 channels. In this case, the Avg_pool layer 301 calculates the average of the 16x16 pixel feature map for each channel. As a result, the output of the Avg_pool layer 301 is one-dimensional and has 256 channels.

[0034] The FC layer 302 outputs a classification result for the class into which the input image IM is classified, based on the output of the Avg_pool layer 301. As shown in FIG. 2, the output layer 300 has CL channels. CL is the number of classes classified by the machine learning model 200. Any integer value can be set as CL. In this embodiment, CL is 10. A value obtained by applying a Softmax function to the output of the FC layer 302 can be used as a class classification value. The class classification values ​​Class_0 to Class_9 corresponding to the 10 classes range from 0 to 1. The sum of the class classification values ​​Class_0 to Class_9 is 1. In this way, 10 class classification values ​​Class_0 to Class_9 are obtained as the output of the machine learning model 200. The class classification values ​​Class_0 to Class_9 correspond to the probability of the class predicted for the input data. For example, the class indicated by the class classification value with the largest value may be output as the class into which the input image IM is classified.

[0035] Alternatively, even if the Softmax function is not applied to the output of the FC layer 302, the class discrimination can be performed for each class using the maximum value of the output of the FC layer 302.

[0036] The known feature vector group KVcG consists of a set of feature vectors collected when the training data group TDG is input to the trained machine learning model 200. The feature vectors are obtained by vectorizing the feature amounts in one subregion Rn from the feature maps whose number corresponds to the number of channels output as intermediate data.

[0037] As shown in FIG. 2, a subregion Rn is depicted in the Conv1 layer 220. The subscript "n" in the subregion Rn refers to the reference character for each layer. Note that FIG. 2 illustrates only the subregions of the Conv1 layer 220. The subregion R220 refers to a subregion in the first convolutional layer 220. A "subregion Rn" is a region in each layer, identified by a planar position (x, y) defined by the position of the first axis x and the position of the second axis y, and including multiple channels along the third axis z. The subregion Rn has dimensions of "Width" × "Height" × "Depth" corresponding to the first axis x, the second axis y, and the third axis z. Vectorizing the features in one subregion Rn along the channel direction means acquiring the features in the subregion Rn in each feature map corresponding to each channel and generating an array of the acquired features. In this embodiment, one "subregion Rn" is expressed as "1 × 1 × depth number," i.e., "1 × 1 × channel number." From the feature maps whose number corresponds to the number of channels output by each layer, feature vectors whose lengths correspond to the number of channels can be collected in numbers corresponding to the size of the feature maps. While FIG. 2 illustrates only the subregions in the first convolutional layer 220, the same applies to the Conv2_x layer 240, the Conv3_x layer 250, and the Conv4_x layer 260.

[0038] For example, the size of the feature map output from the Conv2_x layer 240 is 64 × 64 pixels. Also, since the number of channels is 64, 64 feature maps are output. In this case, 64 × 64 feature vectors of length 64 can be collected from the output of the Conv2_x layer 240.

[0039] Also, for example, the size of the feature map output by the Conv3_x layer 250 is 32 × 32 pixels. Since the number of channels is 128, 128 feature maps are output. From the output of the Conv3_x layer 250, 32 × 32 feature vectors of length 128 can be collected. The number of feature vectors that can be collected is expressed as M × N (M and N are integers equal to or greater than 1).

[0040] The feature vector obtained by inputting the training data TD into the machine learning model 200 is referred to as a first feature vector Vc1.

[0041] In this embodiment, the first feature vector Vc1 is obtained from a specific layer of the machine learning model 200. The feature map used to obtain the first feature vector Vc1 and output from the specific layer is also referred to as the "first feature map." In this embodiment, the specific layer is selected to be the layer immediately preceding the convolutional layer where downsampling is performed. The specific layer may include two or more intermediate layers. In the configuration shown in FIG. 3, for example, the sum of the output of the Conv3_2 layer 252 and the output of the Conv3_3 layer 253 is the output from the specific layer.

[0042] FIG. 5 is a flowchart showing the process for generating the known feature vector group KVcG. The process shown in FIG. 5 is started, for example, when a user inputs an instruction via the input device 140. The process shown in FIG. 5 is executed by the processor 110 functioning as the learning execution unit 112. In step S110, the processor 110 creates training data TD by associating the numbers represented by the input image IM, which is an MNIST handwritten digit image, with the input image IM as a priori labels LB. In this embodiment, handwritten digit images representing 0 to 9 are used as the input image IM. The priori labels LB associated with each handwritten digit image are "0" to "9." There are 10 classes. 1,000 pieces of training data TD are prepared for each class. The total number of training data TD is 10,000. The 10,000 pieces of training data TD are stored in the memory 120 as a training data group TDG.

[0043] In step S120, the processor 110 uses the training data group TDG to train the machine learning model 200. Any loss function can be used during training, but cross entropy is used in this embodiment. When training is completed, data representing the trained machine learning model 200 is stored in the memory 120.

[0044] In step S130, the processor 110 generates a known feature vector group KVcG by again inputting the training data TD included in the training data group TDG into the trained machine learning model 200. The processor 110 stores the known feature vector group KVcG in the memory 120.

[0045] 6 is an explanatory diagram showing the configuration of the known feature vector group KVcG. In this example, the known feature vector group KVcG is made up of a set of first feature vectors Vc1 obtained from the output of the Conv2_x layer 240.

[0046] Each record of the known feature vector group KVcG includes a parameter p indicating the order of the subregion Rn in the feature map, a parameter q indicating a data number, and a first feature vector Vc1. The parameter p indicating the order of the subregion Rn takes a value representing a planar position (x, y) in the feature map, which is intermediate data output from a specific layer. For example, since the size of the feature map output by the Conv2_x layer 240 is 64 × 64 pixels, p = 1 to 4096. The data number parameter q indicates a sequential number for identifying the training data TD. q takes a value from 1 to max. For example, max = 1000. FIG. 6 shows an example of the known feature vector group KVcG, and each record of the known feature vector group KVcG does not necessarily include the parameters p and q.

[0047] The known feature vector group KVcG is generated for each class. FIG. 6 shows a set of known feature vector groups KVcG corresponding to classes "0" to "9." For example, the known feature vector group KVcG_Class0 indicates a known feature vector group KVcG generated using training data TD belonging to class "0." The known feature vector group KVcG_Class1 indicates a known feature vector group KVcG generated using training data TD belonging to class "1."

[0048] The plurality of training data TD used in step S130 (see FIG. 5) of the preparation process does not need to be the same as the plurality of training data TD used in step S120. However, if some or all of the plurality of training data used in step S120 is used in step S130 as well, there is an advantage in that there is no need to prepare new training data.

[0049] 7 and 8 are flowcharts showing the processing involved in the evaluation step of the machine learning model 200. The evaluation step can be executed after the preparation step shown in Fig. 5 is completed. The processing shown in Fig. 7 starts, for example, when an operation instruction from the user is received via the input device 140.

[0050] In step S210, the processor 110, functioning as the class discrimination unit 310, generates a discrimination target data ID. In this embodiment, a 128 x 128 pixel character image is created as the discrimination target data ID by capturing an image of handwritten characters using the camera 400. In step S210, the processor 110 performs preprocessing on the discrimination target data ID as necessary. Preprocessing includes resolution adjustment, data normalization (min-max normalization), and the like. Preprocessing is optional. In step S220, the processor 110 accepts specification of a range in the discrimination target data ID for which similarity is to be calculated. In this embodiment, it is assumed that the entire range of the discrimination target data ID is specified. In step S230, the processor 110 reads the trained machine learning model 200 and the known feature vector group KVcG from the memory 120.

[0051] In step S240, processor 110 inputs the discrimination target data ID to machine learning model 200 to obtain class discrimination values ​​Class_0 to Class_9.

[0052] In step S250, the class classification processing unit 114, functioning as the evaluation unit 330, obtains a second feature vector Vc2 using the output of the specific layer. The feature vector obtained by inputting the discrimination target data ID into the machine learning model 200 is called the second feature vector. The feature map used to obtain the second feature vector Vc2 and output from the specific layer is also called the "second feature map."

[0053] As described above, in this embodiment, the specific layer is selected to be the layer immediately before the convolutional layer where downsampling is performed. For example, the output of the specific layer is the sum of the output of the Conv3_2 layer 252 and the output of the Conv3_3 layer 253.

[0054] The size of the feature map output from the Conv3_x layer 250 is 32 × 32 pixels. The number of channels is 128. Therefore, 32 × 32 feature vectors can be obtained from the output of the Conv3_x layer 250. The length of the feature vector is 128. In step S250, the obtained second feature vector Vc2 is stored in memory 120 together with a value representing the planar position (x, y) of the "subregion Rn" in the feature map, which is the output of the specific layer.

[0055] As shown in FIG. 8, in step S260, processor 110 calculates the similarity between one second feature vector Vc2 among the multiple second feature vectors Vc2 obtained in step S250 and each first feature vector Vc1 in the known feature vector group KVcG belonging to one class. Specifically, the similarity is calculated between one second feature vector Vc2 among the multiple second feature vectors Vc2 obtained in step S250 and the first feature vector Vc1 in each record (see FIG. 6) in the known feature vector group KVcG belonging to one class. For example, if the known feature vector group KVcG includes 1,000 first feature vectors Vc1, 1,000 similarities are calculated. Furthermore, for example, cosine similarity is calculated as the similarity. Cosine similarity is calculated by dividing the inner product of two vectors by the norm of each vector. Note that the processing in step S260 is executed within the range specified in step S220.

[0056] In step S270, processor 110 selects the similarity with the largest value from the multiple similarities calculated in step S260. Here, the similarity with the largest value is selected as a representative value from the multiple similarities calculated. The selected similarity is stored in memory 120 together with a value representing the planar position (x, y) in the feature map of partial region Rn corresponding to second feature vector Vc2 and a value representing the class. In step S270, the selected similarity with the largest value indicates the similarity between second feature vector Vc2 located in that partial region Rn and the most similar first feature vector Vc1 from the set of first feature vectors Vc1 belonging to one class.

[0057] In step S280, processor 110 determines whether the process of calculating similarities and selecting maximum values ​​has been completed for all "subregions Rn" in the feature map output from the specific layer.

[0058] The processes from step S260 onwards are repeated until the processes of step S260 and step S270 are completed for all "partial regions Rn" (step S280; NO). When the processes of step S260 and step S270 are completed for all "partial regions Rn" (step S280; YES), the process of step S290 is executed.

[0059] In step S290, processor 110 determines whether the calculation of similarities and the selection of maximum values ​​have been completed for all classes. Until the processing of steps S260 and S270 has been completed for all classes (step S290; NO), the processing from step S260 onwards is repeated. When the processing of steps S260 onwards has been completed for all classes (step S290; YES), the processing of step S300 is executed.

[0060] In step S300, processor 110 outputs a similarity map for each class. The similarity map is obtained by arranging the maximum similarity of each subregion Rn at a planar position in the feature map of that subregion Rn.

[0061] Furthermore, the processor 110 may display a similarity map for each class in a visually recognizable manner on the display device 150. Here, an example in which the similarity map is displayed as a heat map will be described.

[0062] FIG. 9 is an image diagram of a similarity map expressed in the form of a heat map. Cosine similarity takes values ​​ranging from -1 to 1. Furthermore, the closer the cosine similarity is to 1, the higher the similarity, and the closer it is to -1, the lower the similarity. Therefore, for example, for each partial region Rn, the closer the cosine similarity value is to 1, the brighter the color it is displayed in, and the closer the cosine similarity value is to -1, the darker the color it is displayed in. FIG. 9 shows an example of a similarity map created by calculating the similarity between a second feature vector Vc2 acquired using the discrimination target and a first feature vector Vc1 of a known feature vector group KVcG of class "3."

[0063] The upper left section of Figure 9 shows an image of the handwritten character "3," which is the ID of the data to be discriminated. The lower left section of Figure 9 shows (a) a similarity map generated using the image of the handwritten character "3." To facilitate understanding of the technology, the handwritten character "3," which is the ID of the data to be discriminated, is superimposed on the similarity map. The upper center section of Figure 9 shows an image of the handwritten character "2," which is the ID of the data to be discriminated. The lower center section of Figure 9 shows (b) a similarity map generated using the image of the handwritten character "2." The handwritten character "2," which is the ID of the data to be discriminated, is superimposed on the similarity map. The upper right section of Figure 9 shows an image of the handwritten character "5," which is the ID of the data to be discriminated. The lower right section of Figure 9 shows (c) a similarity map generated using the image of the handwritten character "5." The handwritten character "5," which is the ID of the data to be discriminated, is superimposed on the similarity map.

[0064] As shown in FIG. 9, in the case of (a), the shape of the handwritten character "3" in the discrimination target data ID and the shape of the number "3" represented by the class are nearly identical, so there are no darkly colored portions in the similarity map. In the case of (b), the lower half of the handwritten character "2" in the discrimination target data ID has a portion that is significantly different in shape from the number "3" represented by the class. Therefore, the lower half of the similarity map includes a portion that is darkly colored. In the case of (c), the upper half of the handwritten character "5" in the discrimination target data ID has a portion that is significantly different in shape from the number "3" represented by the class. Therefore, the upper half of the similarity map includes a portion that is darkly colored. When such a similarity map is generated, it is considered that the class classification results of the machine learning model 200 are close to how a person would recognize handwritten characters.

[0065] For example, in case (a), if the similarity map contains areas colored in dark colors, and in cases (b) and (c), if the similarity map is colored brightly overall, it is expected that the processing accuracy of the specific layer of the generated machine learning model 200 or the layer before the specific layer is low.

[0066] Furthermore, although the above embodiment has been described with reference to an example in which there is one specific layer, similarity maps can be generated for multiple specific layers by selecting two or more layers as the specific layer. The closer the specific layer is to the output layer 300, the wider the portion corresponding to the input data for each subregion, and the similarity map for that specific layer can be said to represent the similarity of global structural features. Furthermore, the closer the specific layer is to the input layer 210, the narrower the portion corresponding to the input data for each subregion, and the similarity map for that specific layer can be said to represent the similarity of finer features. In this way, the generated similarity map can be used as explanatory information for class classification.

[0067] In addition to the similarity, the similarity map also includes information on the position within the feature map of the partial region Rn from which the second feature vector Vc2 of the discrimination target data ID was acquired. In the above embodiment, an example was described in which it was specified to calculate the similarity for the entire (entire range) of the discrimination target data ID. However, the user can specify any range, rather than the entire range, of the discrimination target data ID. When the user specifies any range of the discrimination target data ID, the information processing device 100 can provide explanatory information about the specified range.

[0068] B. Other Embodiments: (B1) In the above embodiment, an example has been described in which the output of the layer immediately preceding the convolutional layer where downsampling is performed is selected as the output of the specific layer. Alternatively, the output of the Avg_pool layer 301 may be selected as the output of the specific layer. Alternatively, the output of the Conv4_x layer 260, which is the convolutional layer immediately preceding the Avg_pool layer 301, may be used as the output of the specific layer. Furthermore, the number of specific layers does not have to be limited to one.

[0069] In a convolutional neural network, the features of input data tend to be more condensed by convolution closer to the output layer 300. For example, by selecting multiple different layers as specific layers, it is possible to explain the behavior of the machine learning model 200 at the position of the selected specific layer.

[0070] (B2) In the above embodiment, an example has been described in which image data, which is two-dimensional data, is input to machine learning model 200. However, the data input to machine learning model 200 may be one-dimensional data or time-series data expressed as three-dimensional data. In this case, instead of camera 400, an input data acquisition device appropriate for the type of data is used.

[0071] (B3) When displaying the similarity map as a heat map, the information processing device 100 may also display on the display device 150 the class judgment values ​​for the classes corresponding to the similarity map, among the class judgment values ​​Class_0 to Class_9 output by the machine learning model 200.

[0072] (B4) In the above embodiment, an example was described in which a similarity map was generated for all classes. However, it is not necessary to generate a similarity map for all classes. For example, when examining the difference between the handwritten character "2" and the handwritten character "3," a similarity map may be generated using only the known feature vector group KVcG for class "3."

[0073] (B5) Furthermore, the information processing device 100 may obtain a representative value that quantitatively represents the trend of similarity in the similarity map for each class, and display the representative value together with the similarity map on the display device 150. For example, an average value may be used as the representative value. Alternatively, the information processing device 100 may display only the representative value on the display device 150. The similarity map and the representative value may be stored in the memory 120 of the information processing device 100.

[0074] Alternatively, the information processing device 100 may output only the class into which the discrimination target data ID is classified, without outputting the similarity map.

[0075] (B6) In the above embodiment, an example was described in which a parameter p indicating the order of a subregion Rn in a feature map is associated with a first feature vector Vc1 included in the known feature vector group KVcG (see FIG. 6). The parameter p indicating the order of a subregion Rn represents a planar position (x, y) in a feature map that is intermediate data output from a specific layer. In the above embodiment, an example was described in which the similarity between the first feature vector Vc1 and the second feature vector Vc2 is calculated regardless of the subregion Rn in the feature map for the first feature vector Vc1 (see step S260 in FIG. 8). Alternatively, when calculating the similarity, only the first feature vector Vc1 associated with the region in the feature map corresponding to the subregion Rn of the second feature vector Vc2 may be used.

[0076] Furthermore, the parameter p indicating the order of the partial regions Rn in the feature map does not necessarily have to be associated with the first feature vector Vc1 included in the known feature vector group KVcG.

[0077] (B7) Furthermore, since the training data TD can be identified from the parameter q, the training data TD that can obtain the maximum similarity for each subregion may be displayed. For example, if the specific layer is the Avg_pool layer 301 (GAP layer), the most similar training data TD among the training data group TDG can be displayed.

[0078] (B8) In the machine learning model 200 of the above embodiment, a pooling layer may be provided between the first convolutional layer 220 and the second convolutional block 240. This pooling layer may be configured as a maximum pooling layer.

[0079] (B9) In the above embodiment, the number of channels was not increased for the first convolutional layer 220. However, the number of channels for the first convolutional layer 220 may also be increased.

[0080] (B10) In the above embodiment, an example was described in which cosine similarity was calculated as the similarity between one second feature vector Vc2 among multiple second feature vectors Vc2 and each first feature vector Vc1 in the known feature vector group KVcG belonging to one class. Alternatively, similarity can be expressed using L2 distance. L2 distance is also called Euclidean distance. In the case of cosine similarity, a larger value indicates higher similarity, but when L2 distance is used, a smaller value indicates higher similarity.

[0081] The means for realizing the functions of the information processing device 600 is not limited to software, and some or all of the functions may be realized by dedicated hardware. For example, the dedicated hardware may be a circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0082] C. Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

[0083] (1) According to a first aspect of the present disclosure, there is provided a method for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, wherein the machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.The method includes the steps of: (a) inputting the input data belonging to one of the classes into the machine learning model again, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) executing step (a) using, as input, each of the plurality of input data belonging to one of the classes; (c) executing step (b) for each of the plurality of classes to generate, for each of the classes, a group of known feature vectors consisting of the set of the first feature vectors associated with the classes; and (d) discrimination target data different from the input data, which is used to determine whether the machine learning model can recognize the first feature vectors. (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature quantities included in the second feature maps along a channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in step (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range of the data to be discriminated and at least one of the classes using the set of second feature vectors acquired in step (e) and the group of known feature vectors of at least one of the classes. According to the above aspect, it is possible to provide the similarity regarding the class as explanatory information for the specified range. (2) The method of the above aspect may further include a step of (h) outputting the class into which the discrimination target data indicated by the output of the machine learning model is classified. (3) In the method of the above form, at least one of the classes in step (g) includes the class into which the discrimination target data indicated by the output of the machine learning model is classified, and the method may further include a step of (i) outputting at least one of the class into which the discrimination target data indicated by the output of the machine learning model is classified and the similarity for the class. (4) In the method of the above aspect, the step (g) may include: (g1) calculating the similarity between one of the second feature vectors and each of the first feature vectors included in the group of known feature vectors of one of the classes; (g2) obtaining the maximum value of the multiple similarities calculated in the step (g1); (g3) performing the steps (g1) and (g2) for M×N of the second feature vectors; and (g4) obtaining a similarity map representing the similarity for one of the classes using the maximum value of each of the M×N second feature vectors obtained by performing the step (g3). (5) The method of the above aspect may further include (j) displaying the similarity map for the class in the form of a heat map as the similarity for the class. (6) In the method of the above aspect, the step (g) may include (g5) a step of executing the steps (g1), (g2), (g3), and (g4) for two or more of the plurality of classes. (7) The method of the above aspect may further include a step (k) of outputting a representative value of the similarity map for each of the classes as information quantitatively indicating the similarity between the data to be discriminated and the classes. (8) In the method of the above aspect, the specific layer may be a layer immediately before a convolutional layer that performs downsampling by executing a convolutional process with a stride of 2 or more, a Global Average Pooling layer, or a convolutional layer immediately before the Global Average Pooling layer. (9) In the method of the above aspect, the step (a) may include the step of: (a1) re-inputting the input data belonging to one of the classes into the machine learning model to obtain a set of first feature vectors, and then associating each of the first feature vectors included in the set of first feature vectors with information indicating its position in the first feature map. (10) In the method of the above aspect, the step (a) may include the step of (a2) re-inputting the input data belonging to one of the classes into the machine learning model to obtain a set of first feature vectors, and then associating information identifying the input data with each of the first feature vectors included in the set of first feature vectors. (11) According to a second aspect of the present disclosure, there is provided an information processing device for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, wherein the machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.The information processing device includes: (a) a process of inputting the input data belonging to one of the classes again to the machine learning model, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) a process of executing the process (a) using, as input, each of a plurality of the input data belonging to one of the classes; (c) a process of executing the process (b) for each of the plurality of classes, thereby generating, for each of the classes, a known feature vector group consisting of the set of the first feature vectors associated with the classes; and (d) a process of generating discrimination target data different from the input data, the known feature vector group being associated with the class, the known feature vector group being composed of the set of the first feature vectors associated with the classes, the known feature vector group being obtained by vectorizing, along the channel direction, feature amounts included in the L first feature maps, the L first feature maps being outputs of a specific layer of the machine learning model. (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature amounts included in the second feature maps along the channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in the process (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range of the data to be discriminated and at least one of the classes using the set of second feature vectors acquired in the process (e) and the group of known feature vectors of at least one of the classes. According to the above aspect, it is possible to provide the similarity regarding the class as explanatory information for the specified range. (12) According to a third aspect of the present disclosure, there is provided a program for causing a computer to execute a process for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data. The machine learning model is a convolutional neural network including a plurality of residual blocks and a convolutional layer, and is generated by machine learning using a training dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating one of a plurality of classes to which the input data belongs.The program includes: (a) a process of inputting the input data belonging to one of the classes again to the machine learning model, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers equal to or greater than 1) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes; (b) a process of executing the process (a) using, as input, each of the plurality of input data belonging to one of the classes; (c) a process of executing the process (b) for each of the plurality of classes, thereby generating, for each class, a known feature vector group consisting of the set of the first feature vectors associated with the class; and (d) discrimination target data that is different from the input data and is input to the machine learning model. The computer is caused to execute the following processes: (a) receiving a specification of a range in the data to be discriminated, for which similarity is to be calculated; (e) inputting the data to be discriminated into the machine learning model and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to the size of the second feature map, wherein the second feature vectors are obtained by vectorizing L feature amounts included in the second feature maps along the channel direction; (f) associating each of the second feature vectors included in the set of second feature vectors acquired in process (e) with information indicating its position in the second feature map that is an output of the specific layer; and (g) calculating the similarity between the specified range in the data to be discriminated and at least one of the classes, using the set of second feature vectors acquired in process (e) and the group of known feature vectors of at least one of the classes. According to the above aspect, it is possible to provide the similarity regarding the class as explanatory information for the specified range. [Explanation of symbols]

[0084] 5...Evaluation system, 100...Information processing device, 110...Processor, 112...Learning execution unit, 113...Classification unit, 114...Classification processing unit, 120...Memory, 130...Interface circuit, 140...Input device, 150...Display device, 200...Machine learning model, 210...Input layer, 220...First convolution layer, 240...Second convolution block, 250...Third convolution block, 260...Fourth convolution block, 290...Hidden layer, 30 0...output layer, 301...pooling layer, 302...fully connected layer, 310...classification unit, 330...evaluation unit, 400...camera, ID...data to be classified, IM...input image, KVcG...group of known feature vectors, LB...pre-label, P1...program, ReLU...activation function, Rn...subregion, TD...training data, TDG...training data group, Vc1...first feature vector, Vc2...second feature vector, p, q...parameters, x...first axis, y...second axis, z...third axis

Claims

1. A method for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, comprising: The machine learning model is A convolutional neural network including a plurality of residual blocks and a convolutional layer, The data set is generated by machine learning using a learning dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating a class to which the input data belongs among a plurality of classes; The method comprises: (a) inputting the input data belonging to one of the classes into the machine learning model again, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers of 1 or greater) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes, the first feature vector is obtained by vectorizing feature quantities included in the L number of first feature maps along a channel direction. The process and (b) executing the step (a) using each of the plurality of input data belonging to one of the classes as an input; (c) performing the step (b) for each of the plurality of classes to generate, for each class, a group of known feature vectors, each of which is made up of a set of the first feature vectors associated with the class; (d) receiving a designation of a range for calculating similarity in discrimination target data that is different from the input data and is input to the machine learning model; (e) inputting the discrimination target data into the machine learning model, and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to a size of the second feature map, the second feature vector is obtained by vectorizing feature quantities included in the L number of second feature maps along a channel direction. The process and (f) associating each of the second feature vectors included in the set of second feature vectors acquired in (e) with information indicating a position in the second feature map that is an output of the specific layer; (g) calculating the similarity between the range designated in the discrimination target data and at least one of the classes using the set of second feature vectors acquired in the step (e) and the group of known feature vectors of at least one of the classes; A method comprising:

2. 10. The method of claim 1, (h) outputting the class into which the discrimination target data indicated by the output of the machine learning model is classified; The method further comprises:

3. 10. The method of claim 1, At least one of the classes in step (g) includes the class into which the discrimination target data indicated by the output of the machine learning model is classified; (i) outputting at least one of the class into which the discrimination target data indicated by the output of the machine learning model is classified and the similarity for the class; The method further comprises:

4. 10. The method of claim 1, The step (g) (g1) calculating the similarity between one of the second feature vectors and each of the first feature vectors included in the group of known feature vectors of one of the classes; (g2) obtaining the maximum value of the plurality of similarities calculated in the step (g1); (g3) performing the steps (g1) and (g2) on the M×N second feature vectors; (g4) obtaining a similarity map representing the similarity for one of the classes using the maximum value of each of the M×N second feature vectors obtained by executing the step (g3); A method comprising:

5. 5. The method of claim 4, (j) displaying the similarity map for the class in the form of a heat map as the similarity for the class; The method further comprises:

6. 6. The method of claim 5, The step (g) (g5) performing the steps (g1), (g2), (g3), and (g4) for two or more of the classes; A method comprising:

7. 7. The method of claim 6, (k) outputting a representative value of the similarity map for each of the classes as information quantitatively indicating the similarity between the discrimination target data and the classes; The method further comprises:

8. 8. The method according to any one of claims 1 to 7, The specific layer is The layer is a layer immediately before a convolutional layer that performs downsampling by performing convolution processing with a stride of 2 or more, a Global Average Pooling layer, or a convolutional layer immediately before the Global Average Pooling layer. method.

9. 9. The method of claim 8, The step (a) (a1) inputting the input data belonging to one of the classes into the machine learning model again to obtain the set of first feature vectors, a step of associating information indicating a position in the first feature map with each of the first feature vectors included in the set of first feature vectors; having method.

10. 10. The method of claim 9, The step (a) (a2) inputting the input data belonging to one of the classes into the machine learning model again to obtain the set of first feature vectors, a step of associating information for identifying the input data with each of the first feature vectors included in the set of first feature vectors; having method.

11. An information processing device that generates explanatory information regarding class classification of a machine learning model that classifies classes of input data, The machine learning model is A convolutional neural network including a plurality of residual blocks and a convolutional layer, The data set is generated by machine learning using a learning dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating a class to which the input data belongs among a plurality of classes; The information processing device includes: (a) inputting the input data belonging to one of the classes into the machine learning model again, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers of 1 or greater) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes, the first feature vector is obtained by vectorizing feature quantities included in the L number of first feature maps along a channel direction. Processing and (b) a process of executing the process (a) using each of the plurality of input data belonging to one of the classes as an input; (c) performing the process (b) for each of the plurality of classes to generate, for each class, a group of known feature vectors, each of which is made up of a set of the first feature vectors associated with the class; (d) receiving a designation of a range for calculating similarity in discrimination target data that is different from the input data and is input to the machine learning model; (e) inputting the discrimination target data into the machine learning model, and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to a size of the second feature map, the second feature vector is obtained by vectorizing feature quantities included in the L number of second feature maps along a channel direction. Processing and (f) a process of associating information indicating a position in the second feature map, which is an output of the specific layer, with each of the second feature vectors included in the set of second feature vectors acquired in the process (e); (g) calculating the similarity between the range designated in the discrimination target data and at least one of the classes using the set of second feature vectors acquired in the process (e) and the group of known feature vectors of at least one of the classes; An information processing device that executes the above.

12. A program that causes a computer to execute a process for generating explanatory information regarding class classification of a machine learning model that classifies classes of input data, The machine learning model is A convolutional neural network including a plurality of residual blocks and a convolutional layer, The data set is generated by machine learning using a learning dataset consisting of a set of pairs of a plurality of input data and a prior label associated with the input data, the prior label indicating a class to which the input data belongs among a plurality of classes; The program (a) inputting the input data belonging to one of the classes into the machine learning model again, and acquiring, from L first feature maps (L is the number of channels) that are outputs of a specific layer of the machine learning model, a set of M×N first feature vectors (M and N are integers of 1 or greater) corresponding to the size of the first feature maps, the first feature vectors being associated with one of the classes, the first feature vector is obtained by vectorizing feature quantities included in the L number of first feature maps along a channel direction. Processing and (b) a process of executing the process (a) using each of the plurality of input data belonging to one of the classes as an input; (c) performing the process (b) for each of the plurality of classes to generate, for each class, a group of known feature vectors, each of which is made up of a set of the first feature vectors associated with the class; (d) receiving a designation of a range for calculating similarity in discrimination target data that is different from the input data and is input to the machine learning model; (e) inputting the discrimination target data into the machine learning model, and acquiring, from a second feature map that is an output of the specific layer of the machine learning model, a set of M×N second feature vectors corresponding to a size of the second feature map, the second feature vector is obtained by vectorizing feature quantities included in the L number of second feature maps along a channel direction. Processing and (f) a process of associating information indicating a position in the second feature map, which is an output of the specific layer, with each of the second feature vectors included in the set of second feature vectors acquired in the process (e); (g) calculating the similarity between the range designated in the discrimination target data and at least one of the classes using the set of second feature vectors acquired in the process (e) and the group of known feature vectors of at least one of the classes; A program that causes the computer to execute the above.

Images