Learning device, method, program, and inference system

Abstract

To improve inference accuracy with reduced costs.SOLUTION: An information processing apparatus relating to the present embodiment includes a dividing unit, a first predicting unit, a first calculating unit, a second predicting unit, a second calculating unit, and an updating unit. The dividing unit divides object data to a plurality of pieces of partial data. The first predicting unit inputs the plurality of pieces of partial data to a first model to output a first prediction result. The first calculating unit calculates a first certainty factor showing a degree of contribution to the first prediction result with respect to each of the plurality of pieces of partial data. The second predicting unit inputs the object data to a second model to output a second prediction result. The second calculating unit calculates a second certainty factor showing a degree of contribution to the second prediction result with respect to an area corresponding to the partial data out of the object data. The updating unit updates a parameter of the first model on the basis of the first prediction result, the second prediction result, the first certainty factor, and the second certainty factor.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a learning device, method, program, and inference system.

Background Art

[0002] In recent years, distributed inference processing has been proposed in which inference processing of a deep neural network (DNN) is distributed among a plurality of edge devices. In distributed inference processing, since the resources of a plurality of edge devices can be adaptively utilized, not only can the processing load be distributed, but also stable processing can be performed, and it is difficult to stop even in the event of a failure. However, in distributed inference, in order to maintain inference accuracy, it is necessary to communicate intermediate data between devices. Therefore, if the size of the intermediate data is large, the communication volume increases and the processing speed decreases. There is also a method of processing a plurality of patch images, which are partial images of an image, on different edge devices in order to reduce the communication volume. However, since the amount of information in the patch image is small, it is difficult to maintain the performance of inference by DNN.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure has been made to solve the above-described problems, and an object thereof is to provide a learning device, method, program, and inference system that improve inference accuracy while suppressing costs.

Means for Solving the Problems

[0005] The learning device according to the present embodiment includes a division unit, a first prediction unit, a first calculation unit, a second prediction unit, a second calculation unit, and an update unit. The division unit divides the target data into a plurality of sub-data. The first prediction unit outputs a first prediction result by inputting the plurality of sub-data into a first network model. The first calculation unit calculates a first confidence level indicating the degree of contribution to the first prediction result for each of the plurality of sub-data. The second prediction unit outputs a second prediction result by inputting the target data into a second network model. The second calculation unit calculates a second confidence level indicating the degree of contribution to the second prediction result for a region corresponding to the sub-data in the target data. The update unit updates the parameters of the first network model based on the first prediction result, the second prediction result, the first confidence level, and the second confidence level.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0007] Hereinafter, the learning device, method, program, and inference system according to the present embodiment will be described in detail with reference to the drawings. In the following embodiments, parts denoted by the same reference numerals perform the same operations, and redundant descriptions will be omitted as appropriate.

[0008] (First Embodiment) The learning device according to the first embodiment will be described with reference to the block diagram of FIG. 1. The learning device 10 according to the first embodiment includes an acquisition unit 101, a division unit 102, a first prediction unit 103, a first confidence calculation unit 104, a second prediction unit 105, a second confidence calculation unit 106, an update unit 107, and a storage unit 108.

[0009] The acquisition unit 101 acquires target data, which is data for training a network model, from the storage unit 108 described later or from the outside. The division unit 102 divides the target data into a plurality of sub-data. The first prediction unit 103 outputs a first prediction result by inputting a plurality of sub-data into a first network model. The first confidence calculation unit 104 calculates a first confidence indicating the degree of contribution to the first prediction result for each of the plurality of sub-data.

[0010] The second prediction unit 105 outputs a second prediction result by inputting target data into the second network model. The second network model may have a model structure different from that of the first network model, or may be a model with different parameters but the same model structure. The second confidence calculation unit 106 calculates a second confidence indicating the degree of contribution to the second prediction result for a region corresponding to partial data in the target data. The update unit 107 updates the parameters of the first network model based on the difference between the first prediction result and the second prediction result and the difference between the first confidence and the second confidence. When the second network model is not a trained model, the update unit 107 also updates the parameters of the second network model accordingly. By completing the training for the first network model and the second network model, respective trained models are generated. The storage unit 108 stores the target data, the first network model, the second network model, parameter values related to the network model, and the trained model, etc.

[0011] Also, the first prediction unit 103 includes an aggregation unit 1031. The aggregation unit 1031 generates intermediate data related to feature extraction of a plurality of partial data from the first network model, weights the intermediate data based on the confidence, and executes ensemble processing to output the first prediction result.

[0012] Next, a training example of the learning device 10 according to the first embodiment will be described with reference to the flowchart of FIG. 2. In this embodiment, a classification task will be described as an example of the inference task, but other tasks such as segmentation, object detection, and regression may also be applicable.

[0013] In step S201, the acquisition unit 101 acquires target data. Hereinafter, an image is assumed as the target data, but it is not limited thereto, and it may be multi-dimensional data of two or more dimensions, or one-dimensional time-series data such as audio data and sensor values acquired from sensors, and the same processing can be performed. In step S202, the splitting unit 102 splits the target data into a plurality of sub-data. Here, the splitting unit 102 splits an image into a plurality of sub-images (hereinafter referred to as patch images). For the convenience of explanation here, the image obtained in step S201 before being split into patch images is referred to as the whole image.

[0014] In step S203, the first prediction unit 103 extracts first feature amounts for each patch image using a first network model. The first network model is a network model for extracting features of data, and a deep neural network model including a convolutional neural network (CNN) such as ResNet is assumed. Note that any model can be applied as long as it is a network model used for feature extraction or dimensional compression, not limited to ResNet. In step S204, the first confidence calculation unit 104 calculates a first confidence for each of the extracted first feature amounts. The first confidence may be calculated from information on the region of interest obtained from intermediate data of the first network model such as saliency or attention, for example. The first confidence is a value between 0 and 1, for example.

[0015] In step S205, the aggregation unit 1031 aggregates the first feature amounts based on the first confidence and outputs a first prediction result. Here, for example, the first feature amounts are aggregated by an ensemble process using a weighted average of the first feature amounts according to the first confidence. Specifically, for q (q is an integer of 2 or more) patch images, the feature amounts output from the first network model for each are denoted as l i (i is an integer where 1 ≤ i ≤ q), and the first confidence is denoted as c i Then, the aggregated feature amount l p can be expressed by equation (1).

[0016]

Equation

[0017] Note that the aggregated feature amount l p and the first confidence ci feature quantity l that is maximum i may be adopted. Aggregated feature quantity l p If the logit of is x, the parameter of the classifier of the first network model to be learned is the weight coefficient W, and the bias is b, the first prediction result y p can be expressed, for example, by the following equation (2).

[0018]

Equation

[0019] Here, W in equation (2) is a matrix, x, b, and y p are vectors respectively, and "softmax" is for each element a of the input vector i about z i = exp(a i ) / Σ j exp(a j ) is the softmax function that outputs In step S206, the second prediction unit 105 calculates a second feature quantity from the entire image using the second network model and outputs a second prediction result. The second network model may be a model such as a CNN that can extract feature quantities from the entire image, similar to the first network model. Note that the second prediction result is a classification result for the entire image. In step S207, the second confidence calculation unit 106 calculates the second confidence of the extracted second feature quantity. The second confidence is the same as the first confidence, but is calculated for each position corresponding to the patch image in the entire image.

[0020] In step S208, the update unit 107 calculates a loss function. Here, a loss function L that measures the difference between the probability distribution of the classification of the first prediction result and the probability distribution of the classification of the second prediction result, and the difference between the first confidence and the second confidence is calculated. For example, the loss function L1 representing the difference in probability distribution can be expressed as in equation (3).

[0021]

Equation

[0022] Here, α ∈ [0, 1] is a hyperparameter. M is the resolution degree of the patch image. If there is only one type of resolution for the patch image, then M = 1. As will be described later, two or more types of resolution degrees of the patch image may be set. t n is a one-hot vector representing the correct class, L f ( ) is a loss function for the entire image, and the cross-entropy function C( ) is used. θ f is the parameter (weight coefficient, bias, etc.) of the second network model, θ p is the parameter (weight coefficient, bias, etc.) of the first network model.

[0023] Also, y n f (θ p ) is the second prediction result, ŷ n f is the second prediction result based on the softmax function with temperature parameter. y n,m p (θ p ) is the first prediction result for the m-th resolution when the resolution of the patch image changes for the n-th image, ŷ n,m p (θ p ) is the first prediction result for the m-th resolution when the resolution of the patch image changes for the n-th image, based on the softmax function with temperature parameter. y n f (θ f ), ŷ n f , y n,m p (θ p ) and ŷ n,m p (θ p ) are calculated as in equation (4).

[0024]

Equation

[0025] T is the temperature parameter, and l n f is the logit for the entire image, and l n,m p is the logit for the m-th resolution when the resolution of the patch image changes for the n-th image. Here, L in equation (3) p is the loss function for the patch image and is defined as in equation (5).

[0026]

Equation

[0027] KL represents the Kullback-Leibler divergence. β satisfies β ∈ [0, 1] and is a hyperparameter that adjusts the balance between the loss with the correct answer (hard target) and the loss by knowledge distillation (soft target). On the other hand, the loss function L2 that measures the difference in confidence is based on the first confidence c i corresponding to each patch image and the second confidence d i corresponding to the region of the patch image, and can be expressed as in equation (6) using the sum of squared errors (SSE). Note that other methods such as the mean squared error (MSE) and the Kullback-Leibler divergence KL(d||c) may also be used.

[0028]

Equation

[0029] The final loss function L calculated in step S208 can be expressed as L = L1 + γL2. Note that γ is an arbitrarily set hyperparameter. In step S209, the update unit 107 trains so that the value of the loss function L is minimized, and determines whether the training of the first network model and the second network model has ended. To determine whether the training has ended, for example, it may be determined that the training has ended when the loss value of the loss function L is equal to or less than a threshold value. Alternatively, it may be determined that the training has ended when the decrease width of the loss value has converged. Furthermore, it may be determined that the training has ended when the training for a predetermined number of epochs has ended. If the training has ended, the process ends, and if the training has not ended, the process proceeds to step S210.

[0030] In step S210, the update unit 107 updates the parameter θ of the first network model p and the parameter θ of the second network model. f Specifically, the update unit 107 updates, for example, the weight coefficients and biases of the first network model and the second network model by the gradient descent method and / or the error backpropagation method so that the loss value is minimized. After updating the parameters θ p and θ f , the process returns to step S303, and the training of the first network model and the second network model is continued.

[0031] Note that the second network model that outputs the prediction result of the entire image is assumed to be learned simultaneously with the first network model in the example of FIG. 2, but is not limited thereto, and the second network model may use a pre-trained model whose learning has been completed. In this case, in step S210, since the parameter θ of the second network model f has already been learned and is fixed, the parameter θ of the first network model p may be updated. In addition, regarding the calculation process of the first confidence level shown in step S204 and the calculation process of the second confidence level shown in step S207, examples are shown in which they are executed immediately after the process of step S203 for extracting the first feature amount and the process of step S206 for extracting the second feature amount, respectively, but it is not limited to this. For example, the first confidence level calculation unit 104 may execute the process of calculating the first confidence level from the patch image in parallel with step S203. Similarly, the second confidence level calculation unit 106 may execute the process of calculating the second confidence level from the entire image in parallel with step S206.

[0032] Next, an example of dividing the patch image will be described with reference to FIG. 3. The example of FIG. 3 is a diagram assuming a case where the entire image 30 is divided into four parts to generate four patch images. It can be divided as the upper left patch image 31-1, the lower left patch image 31-2, and so on.

[0033] In addition, the method of dividing the entire image into patch images is not limited to dividing the patch images so that the regions divided at a predetermined patch size do not overlap as shown in FIG. 3, for example. The patch images may be divided by overlapping a part of the regions. Also, a region randomly selected from the entire image may be used as a patch image, or a region related to the prediction target included in the entire image may be used as a patch image. The prediction target included in the entire image may be, for example, a rectangular region by object detection, or if the label of the object is assigned to each pixel by semantic segmentation processing, the region of the set of pixels to which the label of the object is assigned may be used as a patch image.

[0034] Furthermore, the patch images may be divided so that their sizes are different. For example, they may be divided into patch images of a plurality of different sizes, such as combining a patch image of a quarter size of the entire image and a patch image of an eighth size of the entire image. When the sizes of the patch images are different, it is sufficient that position information with respect to the entire image is defined for each corresponding size. In addition, patch images with the same size but different image resolutions may be generated. For example, by selecting patch images from the entire image and selecting patch images from the entire image with the resolution changed by reducing the entire image, patch images with different resolutions may be combined and used. When the resolutions of the images are different, for each entire image with a different resolution, position information corresponding to the patch image is assigned, and a plurality of entire images with different resolutions are input to the second network model to calculate the corresponding second confidence level. Alternatively, in one entire image, the position information of regions corresponding to a plurality of patch images with different resolutions may be defined, and the corresponding second confidence level may be calculated.

[0035] Note that the position in the whole image to which the divided patch image corresponds may be additionally used for identification. For example, values obtained by normalizing the vertical and horizontal coordinates of the entire image to 0-1 respectively (for example, if there are 256 pixels, the coordinates from 0-255 are divided by 255) are added to each pixel value of the entire image. Alternatively, the normalized values may be used as input data for another channel. For example, if the entire image is an RGB image, in addition to the three channels of the R image, G image, and B image, the normalized values may be used as the fourth and fifth channels. In this way, by dividing the entire image with the added position information, the information on where in the whole the patch image was located is also retained, so that the inference performance can be improved. The normalization process is performed to absorb differences in resolution. In general, when the size of the convolutional kernel is 2 or more, padding processing for adding new pixels to the edge of the image is required. Usually, a fixed value such as zero is substituted regardless of the position, but the position information can be embedded by changing this value according to the position of the patch. An example of embedding position information using different padding methods for each position of the patch image is shown in FIG. 4. Figure 4 shows the region of the overall image 40, the adjacent pixel region 41 that is one pixel outside the overall image, and the region of the patch image 42. For example, in the case of four-way division as shown in Figure 4, zero-padding can be used for the outside of the patch image, and replicate (repetition of pixel values) padding can be used for the inside. For example, the pixel value of the adjacent pixel region 41 adjacent to the upper-right pixel value "4" of the upper-right patch image 42 is set to "0", and the pixel value of the adjacent pixel directly below the lower-right pixel value "8" is set to "8". In this way, if the patch image with added position information is used for inference, information on where in the whole the patch image was will be retained in the patch image, so the performance of the inference can be improved.

[0036] Note that position information corresponding to the patch image may be pre-learned by self-supervised learning. For example, a patch image is input, and the first network model is trained using the position of the patch image with respect to the overall image based on the position information obtained by the above method as the teacher label. Note that in self-supervised learning, in the first network model, a layer that outputs the position of the patch image (for example, the ID for each divided region) from the first feature amount as a class classification result may be added and trained.

[0037] Next, a first structural example of the first network model and the second network model will be described with reference to Figure 5. The first network model shown in Figure 5 includes a plurality of convolutional layers, two fully connected layers (FC layers), and an output layer. Here, the case where there are two FC layers is shown, but it may be one layer, three or more layers, or there may be only an output layer without an FC layer.

[0038] Each convolutional layer in Figure 5 may be a single convolutional layer or a block unit including a plurality of convolutional layers like the residual block in ResNet. In the first-stage convolutional layer of the first network model, q (q is a natural number of 2 or more) patch images 51-1 to 51-N are input, feature amounts are extracted from the patch images, and the extracted feature amounts become the input of the subsequent convolutional layer as intermediate data.

[0039] In the convolutional layers after the second stage, feature quantities are extracted in the same manner as in the convolutional layer of the first stage, and the extracted feature quantities serve as intermediate data for input to the subsequent stage. In the convolutional layer immediately before the FC layer, the corresponding first confidence level is extracted together with the first feature quantity. In the example of FIG. 5, confidence level 1 is calculated for the feature quantity of patch image 51-1, and confidence level N is calculated for the feature quantity of patch image 51-N. The feature quantities and the corresponding confidence levels for the N patch images are aggregated, for example, by the process shown in step S205 of FIG. 2, input to the two-stage FC layer, and logits are output. In the output layer, for example, a softmax function is applied to the logits from the FC layer, and a probability distribution for multi-class classification is output as the first prediction result 52.

[0040] On the other hand, the second network model, like the first network model, includes a plurality of convolutional layers, a two-stage fully connected layer (FC layer), and an output layer. In the plurality of convolutional layers, the entire image 50 is input, and second feature quantities are extracted for the entire image 50. Also, in the final convolutional layer, a second confidence level corresponding to the second feature quantity is calculated. At this time, based on the position information given to the patch image 51 as shown in FIG. 3, the second confidence level for the corresponding region is calculated from the entire image 50. Specifically, patch image 51-1 is the upper left region of the entire image 50, and the confidence level in the upper left region corresponding to patch image 51-1 in the entire image 40 is calculated as the second confidence level.

[0041] The feature quantities for the entire image are input to the two-stage FC layer, and logits are output. In the output layer, for example, a softmax function is applied to the logits output from the FC layer, and a probability distribution for classification of multiple classes is output as the second prediction result 53. Based on the loss function related to the first prediction result 52 and the second prediction result 53, and the loss function related to the first confidence level and the second confidence level, the parameters of the first network model and the second network model are repeatedly updated so that the loss value is minimized. As a result, the first network model and the second network model are trained, and the learned models of the first network model and the second network model are generated respectively. In addition, when the second network model has been pre-trained, only the first network model is trained.

[0042] Note that the first confidence level and the second confidence level are shown as examples calculated based on the feature amounts at the final stage of the convolutional layer, but they may be calculated based on the feature amounts extracted by any convolutional layer.

[0043] Generally, since the patch image represents only a part of the entire image, it is considered that the classification accuracy of the second prediction result is higher than that of the first prediction result. Therefore, the prediction result of the probability distribution of classification by the entire image is knowledge-distilled for the prediction of the probability distribution of classification by the patch image, and further, the knowledge of the second confidence level when viewed with the entire image is reflected in the first confidence level in the patch image. As a result, for the processing of the patch image alone, the knowledge of the entire image regarding which divided region of the entire image contributes to the prediction result can be reflected in the training of the first network model.

[0044] In addition, in distributed inference, in the processing node which is an edge device, as a feature extractor 55 for inference processing, for example, a partial network of a plurality of convolutional layers of the first network model is deployed, and in the central node, a predictor 56 which is a partial network including a FC layer and an output layer is held.

[0045] Here, a first example of an inference system that executes the distributed inference according to the present embodiment will be described with reference to FIGS. 5 and 6. The inference system shown in FIG. 6 shows a situation where a plurality of processing nodes 1-1 and 1-2 and one central node 6 are connected by a network NW. In the example of FIG. 6, two processing nodes 1-1 and 1-2 are shown, but three or more may be used. Also, when not particularly distinguished, it is simply described as the processing node 1.

[0046] Each processing node 1 includes a communication unit 11 and an execution unit 12. The execution unit 12 includes a feature extractor 55 which is a network model related to the extraction of feature amounts and is included in the learned first network model shown in FIG. 5. The communication unit 11 receives a patch image of the entire image to be processed for inference from the central node 6. The execution unit 12 inputs the patch image to the feature extractor 55 and extracts a feature amount and a confidence level. The communication unit 11 transmits the extracted feature amount and confidence level to the central node 6.

[0047] Note that each processing node 1 may receive the entire image, divide the patch image from the entire image at its own processing node, and execute processing on the divided patch image. In this case, each processing node 1 may grasp in advance the area of the patch image to be processed at its own node, that is, the position information of the area to be divided from the entire image.

[0048] The central node 6 includes a communication unit 61 and an execution unit 62. The execution unit 62 includes a predictor 56 shown in FIG. 5. The communication unit 61 receives a feature amount and a confidence level from each of the plurality of processing nodes 1. The execution unit 62 aggregates the received multiple feature amounts by performing ensemble processing based on the confidence levels. Also, the communication unit 61 may receive only the feature amounts from the plurality of processing nodes 1. In this case, the execution unit 62 may calculate the confidence levels from the multiple feature amounts received and perform ensemble processing. For the confidence level calculation, an FC layer and a softmax layer may be separately used. The execution unit 62 generates an inference result by inputting the aggregated feature amounts to the predictor 56. In this way, the load of processing can be dispersed by aggregating the feature amounts of the patch images processed by each processing node 1 at the central node 6.

[0049] Note that the layers up to before the aggregation of the network model may be arranged in the processing nodes, and the arrangement method is not limited to the example of FIG. 5. A second structural example of the first network model and the second network model will be described with reference to FIG. 7. The second structural example shown in FIG. 7 is different from the first structural example in that the aggregation process is after the output layer of each first network model 71. Also, in each first network model 71, the feature amounts extracted by the convolutional layer are input to the pooling layer, and the prediction result and the confidence level are output from the output layer. The pooling layer performs, for example, Global Average Pooling processing. The prediction results from each first network model 71 are aggregated by the aggregation unit 72 to generate an inference result 73. On the other hand, the second network model includes a plurality of convolutional layers, a pooling layer, and an output layer, similar to the first network model. The entire image 50 is input, and the second prediction result 74 and the confidence level are output from the output layer. The method for calculating the confidence level and the method for training the network model are the same as those in the case of the first structural example above.

[0050] Next, a second example of the inference system according to the second structural example will be described with reference to FIG. 8. In the inference system according to the second structural example, as in the inference system shown in FIG. 6, a situation is shown in which a plurality of processing nodes 1-1 and 1-2 and one central node 6 are connected by a network NW. In the processing node 1, a learned model of the first network model 71 is deployed, and the central node 6 includes an aggregation unit 72. In each processing node 1, the execution unit 12 inputs the patch image into the learned model of the first network model 71 and generates a prediction result and a confidence level. Then, the communication unit 11 transmits the prediction result and the confidence level to the central node 6. In the central node 6, the communication unit 61 receives the prediction result and the confidence level from the plurality of processing nodes 1 respectively. The execution unit 62 aggregates the received prediction results by performing ensemble processing based on the confidence levels and generates an inference result. In addition, in this embodiment, an example of minimizing using a loss function for measuring differences has been described, but it may be a problem of maximizing a function such as cosine similarity. That is, the parameters may be updated so as to optimize the objective functions of the first network model and the second network model respectively.

[0051] According to the first embodiment shown above, in training the first network model that processes partial data that is part of the target data, the first prediction result of the partial data and the degree of contribution to the inference of the first prediction result are shown. The first confidence level is calculated. Further, a second prediction result regarding the entire target data and a second confidence level indicating the degree of contribution to the inference of the second prediction result obtained from the intermediate data of the second network model that processes the target data are calculated. By training the first network model using the difference between the first prediction result and the second prediction result and the difference between the first confidence level and the second confidence level as a loss function, the inference result of the target data can be knowledge-distilled into the inference of the partial data. As a result, in distributed inference processing, it is possible to improve the inference accuracy of the learned model that processes partial data while suppressing the communication cost.

[0052] (Second Embodiment) In the first embodiment, a case where parameters are different between the first network model and the second network model is assumed. In the second embodiment, however, the difference lies in sharing parameters in each network model. Since the configuration of the learning device 10 according to the second embodiment is the same as that of the first embodiment, the description thereof will be omitted.

[0053] The training example of the learning device 10 according to the second embodiment will be described with reference to the flowchart of FIG. 9. In the second embodiment, the first network model and the second network model have the same network model structure and the same parameters. Note that, not limited to the example of FIG. 9, only a part of the model structure may share parameters, and different structures and parameters may be used for the rest. Further, when the network model has a Batch Normalization layer, the learnable weights and biases may be shared, and the mean and variance parameters may be separately held in the first network model and the second network model. Steps S201 to S210 are the same as those in the first embodiment. In step S208 according to the second embodiment, the loss function L1 representing the difference in probability distributions may be calculated based on equation (7) using common parameters.

[0054]

Equation

[0055] In step S901, the update unit 107 shares the values of the parameters updated in step S210 between the first network model and the second network model. That is, the update unit 107 sets the parameters to have the same value.

[0056] According to the second embodiment described above, knowledge distillation can be performed by sharing parameters between the first network model and the second network model during the learning of the network model. That is, by learning the whole image and the patch image with the same model, parameters that can be used for inference of both the whole image and the patch image are used. Since information necessary for recognizing the whole image can also be used from the patch image, the performance of the model can be improved. That is, similar to the first embodiment, in the distributed inference process, it is possible to improve the inference accuracy of the learned model that processes partial data while suppressing the communication cost.

[0057] (Third Embodiment) In the third embodiment, it is different from the above-described embodiment in that parameters are shared without calculating the confidence.

[0058] The learning device according to the third embodiment will be described with reference to the block diagram of FIG. 10. The learning device 20 according to the third embodiment includes an acquisition unit 101, a division unit 102, a first prediction unit 103, a second prediction unit 105, an update unit 107, and a storage unit 108. The update unit 107 causes the first network model and the second network model to share parameters, similarly to the second embodiment.

[0059] Next, the training example of the learning device 20 according to the third embodiment will be described with reference to the flowchart of FIG. 11. Steps S201 to S203, step S206, steps S208 to S210, and step S901 are the same as those in the second embodiment. In step S1101, the aggregation unit 1031 aggregates the feature amounts extracted for each patch image. For example, the aggregated feature amount may be calculated by a simple average with the first confidence c i in the above-described formula (1) being a uniform value. As the loss function in step S208, a loss function L1 related only to the difference in probability distributions may be used. For example, the above-described formula (3) may be used.

[0060] According to the third embodiment described above, knowledge distillation can be performed by sharing parameters between the first network model and the second network model during the learning of the network model, and the performance of the model can be improved. As a result, similar to the first embodiment, in the distributed inference process, it is possible to improve the inference accuracy of the learned model that processes partial data while suppressing the communication cost.

[0061] Next, an example of the hardware configuration of the learning device 10 and the learning device 20 according to the above-described embodiment is shown in the block diagram of FIG. 12. The learning device 10 and the learning device 20 include a CPU (Central Processing Unit) 1201, a RAM (Random Access Memory) 1202, a ROM (Read Only Memory) 1203, a storage 1204, a display device 1205, an input device 1206, and a communication device 1207, and are each connected by a bus.

[0062] The CPU 1201 is a processor that executes arithmetic processing, control processing, and the like according to a program. The CPU 1201 uses a predetermined area of the RAM 1202 as a work area and executes the processing of each part of the learning device 20 described above in cooperation with programs stored in the ROM 1203, the storage 1204, and the like.

[0063] The RAM 1202 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory). The RAM 1202 functions as a work area for the CPU 1201. The ROM 1203 is a memory that stores programs and various information in a non-rewritable manner.

[0064] Storage 1204 is a device that writes and reads data to and from a magnetic recording medium such as a HDD (Hard Disc Drive), a semiconductor-based storage medium such as a flash memory, or a magnetically recordable storage medium such as a HDD, an optically recordable storage medium, etc. Storage 1204 writes and reads data to and from the storage medium according to the control from CPU 1201.

[0065] The display device 1205 is a display device such as an LCD (Liquid Crystal Display). The display device 1205 displays various information based on the display signal from CPU 1201. The input device 1206 is an input device such as a mouse and a keyboard. The input device 1206 receives the information input by the user's operation as an instruction signal and outputs the instruction signal to CPU 1201. The communication device 1207 communicates with an external device via a network according to the control from CPU 1201.

[0066] The instructions shown in the processing procedures presented in the above-described embodiments can be executed based on a program that is software. A general-purpose computer system can store this program in advance and, by loading this program, can also obtain effects similar to those achieved by the control operations of the information processing apparatus described above. The instructions described in the above-described embodiments can be recorded as a program executable by a computer on a magnetic disk (such as a floppy disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray (registered trademark) Disc, etc.), a semiconductor memory, or a recording medium similar thereto. As long as it is a recording medium readable by a computer or an embedded system, the storage format can be in any form. If a computer reads a program from this recording medium and causes a CPU to execute the instructions described in this program, it can realize operations similar to the control of the information processing apparatus in the above-described embodiments. Of course, when a computer acquires or reads a program, it may be acquired or read through a network.

[0067] Also, based on the instructions of a program installed from a recording medium into a computer or an embedded system, the OS (operating system), database management software, MW (middleware) such as a network, etc. that are operating on the computer may execute a part of each process for realizing this embodiment.

[0068] Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, and also includes a recording medium that has downloaded and stored or temporarily stored a program transmitted via a LAN, the Internet, etc. Also, the recording medium is not limited to one. Even when the processing in this embodiment is executed from a plurality of media, it is included in the recording medium in this embodiment, and the configuration of the media can be in any configuration.

[0069] Note that the computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored in a recording medium, and may have any configuration such as a device consisting of one of a personal computer, a microcomputer, etc., or a system in which a plurality of devices are network-connected. In addition, the computer in this embodiment is not limited to a personal computer, but also includes an arithmetic processing unit, a microcomputer, etc. included in an information processing device, and generically refers to devices and apparatuses capable of realizing the functions in this embodiment by a program.

[0070] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0071] 1... Processing node, 6... Central node, 10... Learning device, 11, 61... Communication unit, 12, 62... Execution unit, 20... Learning device, 30, 40, 50... Whole image, 41... Adjacent pixel region, 31-1, 31-2, 42, 51-1 to 51-N... Patch image, 52... First prediction result, 53, 74... Second prediction result, 55... Feature extractor, 66... Predictor, 71... First network model, 73... Inference result, 1201... CPU, 1202... RAM, 1203... ROM, 1204... Storage, 1205... Display device, 1206... Input device, 1207... Communication device, 101... Acquisition unit, 102... Division unit, 103... First prediction unit, 104... First confidence calculation unit, 105... Second prediction unit, 106... Second confidence calculation unit, 107... Update unit, 108... Storage unit, 72, 1031... Aggregation unit

Claims

1. A splitting unit that splits target data into a plurality of partial data; A first prediction unit that outputs a first prediction result by inputting the plurality of partial data into a first network model; A first calculation unit that calculates a first confidence level indicating the degree of contribution to the first prediction result for each of the plurality of partial data; A second prediction unit that outputs a second prediction result by inputting the target data into a second network model; A second calculation unit that calculates a second confidence level indicating the degree of contribution to the second prediction result for a region corresponding to the partial data among the target data; An update unit that updates the parameters of the first network model so that the weighted sum of a first loss function related to the difference between the first prediction result and the second prediction result and a second loss function related to the difference between the first confidence level and the second confidence level is minimized; A learning device comprising the above.

2. The learning device according to claim 1, wherein part or all of the model structures of the first network model and the second network model are the same and part or all of the parameters are shared.

3. The update unit calculates an objective function based on the first prediction result and the second prediction result, and the first confidence level and the second confidence level, and updates the parameters so as to optimize the value of the objective function. The learning device according to claim 1 or claim 2.

4. The learning device further comprises an aggregation unit that generates intermediate data related to feature extraction of the plurality of partial data from the first network model, weights the intermediate data based on the first confidence level, and performs ensemble processing. The first prediction unit outputs the first prediction result based on the intermediate data after the ensemble processing. The learning device according to any one of claims 1 to 3.

5. The first confidence level and the second confidence level are calculated based on the saliency or attention of the intermediate data of the first network model and the second network model, respectively. The learning device according to any one of claims 1 to 4.

6. A splitting unit that splits target data into a plurality of partial data; A first prediction unit that outputs a first prediction result by inputting the plurality of partial data into a first network model; A second prediction unit that outputs a second prediction result by inputting the target data into a second network model that has part or all of the same model structure as the first network model and shares part or all of the parameters; An update unit that updates the parameters so that a loss function regarding the difference between the first prediction result and the second prediction result is minimized; A learning device comprising the above.

7. The learning device according to any one of claims 1 to 6, wherein the plurality of partial data corresponds to at least one of an overlapping region of a part of the target data, a non-overlapping region of the target data, a region randomly selected from the target data, and a region related to a prediction target included in the target data.

8. A dividing means divides target data into a plurality of partial data, A first prediction means outputs a first prediction result by inputting the plurality of partial data into a first network model, A first calculation means calculates a first confidence level indicating the degree of contribution to the first prediction result for each of the plurality of partial data, A second prediction means outputs a second prediction result by inputting the target data into a second network model, A second calculation means calculates a second confidence level indicating the degree of contribution to the second prediction result for a region corresponding to the partial data in the target data, An update means updates the parameters of the first network model so that a weighted sum of a first loss function regarding the difference between the first prediction result and the second prediction result and a second loss function regarding the difference between the first confidence level and the second confidence level is minimized. Learning method.

9. A computer, A dividing means for dividing target data into a plurality of partial data, A first prediction means that outputs a first prediction result by inputting the plurality of partial data into a first network model, A first calculation means for calculating a first confidence level indicating the degree of contribution to the first prediction result for each of the plurality of partial data, A second prediction means that outputs a second prediction result by inputting the target data into a second network model, A second calculation means for calculating a second confidence level indicating the degree of contribution to the second prediction result for a region corresponding to the partial data in the target data. Updating means for updating the parameters of the first network model such that the weighted sum of a first loss function regarding the difference between the first prediction result and the second prediction result and a second loss function regarding the difference between the first confidence level and the second confidence level is minimized. A learning program for causing it to function.

10. An inference system including a plurality of processing nodes and a central node, wherein each of the plurality of processing nodes is a feature extractor which is a network model regarding extraction of feature quantities included in a first network model learned by the learning device according to any one of Claims 1 to 7, a first execution unit for inputting partial data of target data to the feature extractor to extract feature quantities, and a communication unit for transmitting the feature quantities to the central node, wherein the central node is a communication unit for receiving feature quantities from the plurality of processing nodes respectively, is a predictor which is a network model included in the learned first network model and for executing processing according to a task on the feature quantities, and a second execution unit for performing ensemble processing on the plurality of feature quantities transmitted from the plurality of processing nodes, inputting the processed feature quantities to the predictor, and generating an inference result. An inference system.

11. An inference system including a plurality of processing nodes and a central node, wherein each of the plurality of processing nodes is a first network model learned by the learning device according to any one of Claims 1 to 7, a first execution unit for inputting partial data of target data to the first network model and generating a prediction result, and a communication unit for transmitting the prediction result to the central node, wherein the central node is a communication unit for receiving prediction results from the plurality of processing nodes respectively, and a second execution unit for performing ensemble processing on the plurality of prediction results transmitted from the plurality of processing nodes and generating an inference result. An inference system.

Images