Information processing method

Abstract

To enable privacy protection in learning of a learning model.SOLUTION: A learning model consists of a plurality of first models and a second model different from the first models. An information processing method comprises: dividing a first image used for learning of the learning model into a plurality of first patches; inputting the plurality of first patches to the plurality of first models respectively at random without overlapping, and then inputs a plurality of calculation results, output from the plurality of first models, to a second model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to the technical field of information processing methods. [Background technology]

[0002] As an example of this type of method, Patent Document 1 describes a method in which a neural network is divided into multiple parts and parameters are specified based on input / output characteristics. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6784162 [Non-patent literature]

[0004] [Non-Patent Document 1] Federated Learning: Strategies for Improving Communication Efficiency, NIPS Workshop on Private Multi-Party Machine Learning (2016), URL: “https: / / arxiv.org / pdf / 1610.05492.pdf” [Non-Patent Document 2] Split learning for health: Distributed deep learning without sharing raw patient data, December 2018, URL: “https: / / arxiv.org / pdf / 1812.00564.pdf” Summary of the Invention [Problem to be solved by the invention]

[0005] In the above prior art, by dividing a neural network, each neural network is given a learning result up to a certain point, which allows consideration to privacy protection, but there are also methods that restore the original data from data in the middle of calculation. Therefore, it is expected that there are cases where privacy protection is not sufficient just by dividing a neural network, and there is room for improvement.

[0006] An objective of the present disclosure is to provide an information processing method that enables privacy protection during learning of a learning model. [Means for solving the problem]

[0007] In an information processing method according to one embodiment of the present disclosure, a learning model is composed of a plurality of first models and a second model different from the first models, a first image used for training the learning model is divided into a plurality of first patches, the plurality of first patches are input randomly and without overlapping to each of the plurality of first models, and a plurality of calculation results output from each of the plurality of first models are input to the second model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram showing the hardware configuration of a user terminal and each server as a computer. [Diagram 3] FIG. 2 is a diagram illustrating an example of the data flow of an information processing system and a configuration example of a learning model. [Figure 4] 13 is a flowchart showing an information processing method during learning. [Diagram 5] FIG. 13 is a diagram showing the flow of data during inference. [Figure 6] 13 is a flowchart showing an information processing method during inference. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment of an information processing method will be described with reference to Figs. 1 to 6. First, an information processing method related to deep learning of image recognition will be described. In this embodiment, an image used for learning is divided into a plurality of patches. In this embodiment, a patch split neural network is used as a configuration of a learning model. The learning model may be, for example, a model in which a CNN (Convolutional Neural Network) is divided into two models, an Upper model and a Lower model.

[0010] In FIG. 1, the information processing system 10 includes a user terminal 102 and a plurality of patch servers 110 (110 1~N ) and a plurality of Upper servers 112 (112) each storing a plurality of Upper models. 1~N ) and a Lower server 114 that stores the Lower model. The user terminal 102, the multiple patch servers 110, the multiple Upper servers 112, and the Lower server 114 are connected via a network N.

[0011] The user terminal 102 is a terminal used for inputting an image to be used for learning. The patch server 110 is a storage server for storing a plurality of patches generated by dividing an image inputted using the user terminal 102. The upper server 112 and the lower server 114 are servers for storing an upper model and a lower model, respectively, as models for learning.

[0012] The user terminal 102 and each server (110, 112, 114) may be realized by, for example, a computer CM shown in Fig. 2. In Fig. 2, the computer CM has a calculation unit 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface 17. The calculation unit 11, the ROM 12, the RAM 13, the storage 14, the input unit 15, the display unit 16, and the communication interface 17 are connected to each other so as to be able to communicate with each other via a bus 18. The calculation unit 11 may include, for example, at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0013] The calculation unit 11 executes various programs and controls each component of the computer CM. That is, the calculation unit 11 reads the program from at least one of the ROM 12 and the storage 14, and executes the program using the RAM 13 as a working area. The calculation unit 11 controls each component of the computer CM and performs at least one of various arithmetic processing according to the program. At least one of the ROM 12 and the storage 14 stores an information processing program for performing the information processing method according to this embodiment. Note that a general hardware configuration of a computer can be applied to the computer CM, so other explanations regarding the hardware configuration will be omitted.

[0014] Next, the data flow of the information processing system 10 during learning of the learning model will be described with reference to FIG. 3. In (1) of FIG. 3, an input image input by a user via a user terminal 102 is divided into a plurality of patches. In FIG. 3, the input image is divided into nine patches, but the number of divisions is not limited to nine. The division of the input image may be performed by the user terminal 102. The division of the input image may also be performed by one patch server 110. In (2) of FIG. 3, a plurality of patches generated by dividing one input image are stored in different patch servers 110. The plurality of patches may be stored in a predetermined patch server 110. One of the plurality of patch servers 110 may store only a patch corresponding to a predetermined position of the input image (for example, patch P1 corresponding to the upper left corner of the input image).

[0015] In (3) of FIG. 3, combinations of multiple patches generated by dividing one input image and multiple Upper models to which the multiple patches are respectively input are determined randomly (in other words, randomly). At this time, the combinations of multiple patches and multiple Upper models are determined so that two or more patches are not input to one Upper model. In (4) of FIG. 3, multiple patches are input to multiple Upper models according to the combinations determined in (3) above. As a result, learning is performed by the Upper models. Here, each Upper model starts and operates on an individual independent Upper server 112.

[0016] In (5) of FIG. 3, the calculation results of the upper models are integrated in the lower server 114. For example, the calculation results of the upper models are integrated spatially based on the combination determined in (3) above and the positions of the multiple patches in one input image. For example, for multiple patches P1 to P9, the calculation results of the upper models are integrated so as to reproduce the positional relationship of the multiple patches P1 to P9, which corresponds to the calculation results of the upper models being integrated spatially. The integrated calculation result of the upper model is input to the lower model. In the lower model, a calculation is performed based on the calculation result of the integrated upper model. As a result, the recognition result of the learning model is obtained. For example, the learning model may be evaluated by calculating the loss related to the calculation result of the lower model. In this manner, the information processing system 10 generates and outputs a trained model.

[0017] In (6) of FIG. 3, necessary patches may be collected according to the learning status of the learned learning model. An input image may be restored from the collected patches. The restored input image may be analyzed, and the patches may be appropriately labeled with correct answer data. Note that this process is optional and may not be performed.

[0018] The division of the input image may be a simple division or an overlap division. In the simple division, the input image may be divided without overlapping, for example, based on the size of the input image. For example, in the case of an input image of 32x32 size, if the patch size is 16x16, the input image may be divided into 4 patches. For example, in the case of an input image of 32x32 size, if the patch size is 8x8, the input image may be divided into 16 patches. In the overlap division, the input image may be divided by overlapping with a fixed length, for example. For example, in the case of an input image of 32x32 size, if the fixed length is 8 and the patch size is 16x16, the input image may be divided into 9 patches. For example, in the case of an input image of 32x32 size, if the fixed length is 4 and the patch size is 8x8, the input image may be divided into 49 patches. Note that one of the simple division and the overlap division may be selected according to the complexity of the input image.

[0019] The operation of the information processing system 10 during learning of the learning model will be described with reference to the flowchart in Fig. 4. In Fig. 4, the user terminal 102 divides an input image into a plurality of patches (step S101). The user terminal 102 transmits the plurality of patches to a plurality of predetermined patch servers 110, respectively.

[0020] Then, combinations of the multiple patches generated by dividing one input image and the multiple Upper models to which the multiple patches are input are randomly determined (step S102). Note that the process of step S102 may be performed by, for example, the user terminal 102, one patch server 110, or one Upper server 112. Note that information indicating the combinations of the multiple patches and the multiple Upper servers is transmitted to the Lower server 114.

[0021] According to the combination determined in the process of step S102, a plurality of patches generated by dividing one input image are input to the corresponding upper models of the plurality of upper servers 112 (step S103). As a result, a calculation result is output from each upper model. Each of the plurality of upper servers 112 transmits the calculation result of the upper model to the lower server 114.

[0022] The Lower server 114 spatially integrates the calculation results of each Upper model based on information indicating a combination of a plurality of patches and a plurality of Upper servers and the positions of each of the plurality of patches in one input image (step S104). The Lower server 114 inputs the calculation results of the integrated Upper model to the Lower model (step S105). In the Lower model, calculation is performed based on the calculation results of the integrated Upper model. As a result, the recognition result of the learning model is obtained.

[0023] The Lower server 114, for example, calculates a loss related to the calculation result of the Lower model (step S106). In the information processing system 10, for example, the weight parameters of each of the multiple Upper models and Lower models may be adjusted based on the calculated loss.

[0024] An information processing method for image recognition using a trained model generated by the above-mentioned information processing method for deep learning of image recognition will be described. The trained model has a plurality of trained upper models and a trained lower model.

[0025] In the information processing system 10, one trained Upper model may be selected from a plurality of trained Upper models. For example, an image for evaluation (so-called test data) may be input to each of the plurality of trained Upper models. The plurality of trained Upper models may be evaluated based on the calculation results of each of the plurality of trained Upper models. For example, the trained Upper model with the highest evaluation may be selected as the one trained Upper model.

[0026] In the information processing system 10, image recognition (for example, inference) may be performed using the selected one trained Upper model and the Lower model. Here, the data flow of the information processing system 10 during inference of an unknown image (i.e., an image not used in training the training model) will be described with reference to FIG.

[0027] In FIG. 5, an input image input by a user via a user terminal 102 is divided into a plurality of patches. In FIG. 5, the input image is divided into nine patches, but the number of divisions is not limited to nine. The plurality of patches are input to a selected trained Upper model. For example, the plurality of patches are input to an Upper server 112 that stores a selected trained Upper model. As a result, a plurality of calculation results corresponding to the plurality of patches are output from the selected trained Upper model. The Lower server 114 spatially integrates the plurality of calculation results. The Lower server 114 inputs the integrated calculation result to a trained Lower model. As a result, a recognition result of the input image by the trained model is obtained.

[0028] The operation of the information processing system 10 during inference of an unknown image will be described with reference to the flowchart of Fig. 6. It is assumed that one trained Upper model is selected from a plurality of trained Upper models before the operation shown in the flowchart of Fig. 6 is performed.

[0029] 6, the user terminal 102 divides an input image into a plurality of patches (step S201). The user terminal 102 transmits the plurality of patches to the Upper server 112 that stores a selected trained Upper model. The Upper server 112 inputs the plurality of patches to the trained Upper model (step S202). As a result, a plurality of calculation results corresponding to the plurality of patches are output from the trained Upper model. The Upper server 112 transmits the plurality of calculation results to the Lower server 114.

[0030] The Lower server 114 spatially integrates the multiple calculation results (step S203). The Lower server 114 inputs the integrated calculation results to the Lower model (step S204). As a result, a recognition result of the input image by the trained model is obtained.

[0031] In addition, the selected trained Upper model and the trained Lower model may be stored in the same device.

[0032] (Technical effect) For example, data containing personal information such as facial images may be used to train learning models for image recognition. In addition, privacy protection laws such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) have been enacted in various countries, and privacy protection has become important in data collection and model training.

[0033] For example, the methods described in Non-Patent Document 1 and Non-Patent Document 2 achieve privacy protection by not collecting data centrally, but have the problem that data cannot be adjusted when improving the performance of the model. In addition, a method of masking parts containing personal information such as faces can be considered, but this is likely to affect the recognition performance of the model.

[0034] When training a neural network in deep learning for image recognition, a large amount of images is required as training data. In addition, in the case of supervised learning, after collecting images, it is necessary to label the images with correct answer data. The collected images may contain privacy-related information such as faces and vehicle license plates. When storing collected images in one place, care must be taken when handling them, even if security is ensured.

[0035] In the information processing system 10, when learning a learning model, an input image is divided into a plurality of patches. Since the size of the patch is small compared to the size of the input image, even if the input image contains privacy information, it is extremely difficult to identify the privacy information from the patch. In other words, by dividing the input image into a plurality of patches, each patch can be made into non-privacy information.

[0036] In the information processing system 10, a plurality of patches generated by dividing one input image are stored in separate patch servers 110. Therefore, privacy information will not be restored unless a plurality of patches related to one input image are extracted from each of the plurality of patch servers 110. Note that some users may be permitted to extract a plurality of patches related to one input image from the plurality of patch servers 110. With this configuration, it is possible to verify a learning model using one input image restored from a plurality of patches.

[0037] In this way, the information processing system 10 makes it possible to protect privacy during the training of a training model.

[0038] In the information processing system 10, when learning a learning model, if a patch corresponding to a specific position in an input image is always input to one Upper model, the one trained Upper model tends to have high accuracy of calculation results related to the patch corresponding to the specific position, while the accuracy of calculation results related to patches corresponding to positions other than the specific position in the input image tends to be low. Therefore, when inferring an unknown image, it is necessary to use all of the multiple trained Upper models included in the trained model in order to obtain a recognition result with the expected accuracy.

[0039] In contrast, in the above-described embodiment, when the learning model is trained, a combination of a plurality of patches and a plurality of Upper models to which the plurality of patches are respectively input is randomly determined. When a patch corresponding to an arbitrary position in the input image is input to one Upper model during training, the accuracy of the calculation result of the one trained Upper model is less affected by the input patch. Therefore, when inferring an unknown image, even if only one trained Upper model out of the multiple trained Upper models is used, a recognition result with expected accuracy can be obtained.

[0040] In the above-described embodiment, an unknown image is inferred using one selected trained Upper model and a trained Lower model. Therefore, resource consumption can be reduced compared to when an unknown image is inferred using multiple trained Upper models and trained Lower models. Furthermore, the inventor's research has revealed that the accuracy of the recognition result when an unknown image is inferred using one selected trained Upper model and a trained Lower model is equal to or higher than the accuracy of the recognition result when an unknown image is inferred using multiple trained Upper models and trained Lower models.

[0041] Aspects of the invention derived from the above-described embodiment will be described below.

[0042] In an information processing method according to one aspect of the invention, a learning model is composed of a plurality of first models and a second model different from the first models, a first image used for learning the learning model is divided into a plurality of first patches, the plurality of first patches are input randomly and without overlapping to the plurality of first models, respectively, and a plurality of calculation results output from the plurality of first models are input to the second model. In the above embodiment, the "Upper model" corresponds to an example of the "first model", and the "Lower model" corresponds to an example of the "second model".

[0043] In the information processing method, a second image different from the first image may be divided into a plurality of second patches, the plurality of second patches may be input into a trained first model among the plurality of trained first models constituting the trained learning model, and a calculation result output from the trained first model may be input into the trained second model constituting the trained learning model.

[0044] In the information processing method, the second model may integrate the multiple calculation results based on information indicating a correspondence between the positions of each of the multiple first patches in the first image and a first model into which each of the multiple first patches is input.

[0045] In the information processing method, the one trained first model and the trained second model may operate on different devices. Alternatively, in the information processing method, the one trained first model and the trained second model may operate on the same device.

[0046] The present disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope that does not contradict the gist or concept of the invention that can be read from the claims and the entire specification, and information processing methods involving such modifications are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0047] 10: information processing system, 102: user terminal, 110: patch server, 112: upper server, 114: lower server

Claims

1. The learning model is composed of a plurality of first models and a second model different from the first models, Dividing a first image used for training the learning model into a plurality of first patches; inputting the first patches randomly and without overlap into the first models, respectively; A plurality of calculation results outputted from the plurality of first models are inputted into the second model. Information processing methods.

2. Dividing a second image, different from the first image, into a plurality of second patches; The plurality of second patches are input to one trained first model among the plurality of trained first models constituting the trained learning model; A calculation result output from the one trained first model is input to the trained second model constituting the trained learning model. The information processing method according to claim 1 .

3. The second model integrates the plurality of calculation results based on information indicating a correspondence relationship between the positions of the plurality of first patches in the first image and a first model into which the plurality of first patches are input. The information processing method according to claim 1 .

4. The trained first model and the trained second model operate on different devices. The information processing method according to claim 2 .

5. The trained first model and the trained second model operate on the same device. The learning processing method according to claim 2 .

Images