Information processing method

Abstract

To enable privacy protection during training of a learning model.SOLUTION: A learning model is configured of a plurality of first models and a second model different from the plurality of first models. An information processing method includes a division step of dividing an image used for training the learning model into a plurality of patches, a first input step of inputting the plurality of patches into the plurality of first models, respectively, without overlap, an adding step of adding noise to each of a plurality of calculation results output from the plurality of first models, and a second input step of inputting the plurality of calculation results to which noise has been added into the second model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the technical field of information processing methods. [Background technology]

[0002] As one such method, for example, a method has been proposed in which a neural network constituting a learning model is divided into multiple parts to enable privacy protection during learning of the learning model (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-030614 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 leaves room for improvement.

[0005] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide an information processing method that enables privacy protection in the training of a training model. [Means for solving the problem]

[0006] An information processing method according to one embodiment of the present disclosure is such that a learning model is composed of a plurality of first models and a second model different from the plurality of first models, and includes a division process for dividing an image used for training the learning model into a plurality of patches, a first input process for inputting the plurality of patches into the plurality of first models respectively without overlapping, an addition process for adding noise to each of a plurality of calculation results output respectively from the plurality of first models, and a second input process for inputting the plurality of calculation results with added noise into the second model. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 2] FIG. 1 is a diagram illustrating an example of the data flow of an information processing system and the configuration of a learning model. [Figure 3] FIG. 2 is a diagram showing the main part of the data flow in the information processing system. [Figure 4] FIG. 10 is a diagram for explaining a specific example of a noise calculation method. [Figure 5] 10 is a flowchart illustrating an example of an information processing method. [Figure 6] FIG. 2 is a diagram showing the main part of the data flow in the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment A first embodiment of an information processing method will be described with reference to FIGS. 1 to 5. First, an information processing method related to deep learning for image recognition will be described. In this embodiment, an image used for learning is divided into a plurality of patches. In this embodiment, a patch split neural network is used as the configuration of the learning model. The learning model may be, for example, a model in which a CNN (Convolutional Neural Network) is divided into two models, an upper model and a lower model.

[0009] In FIG. 1, the information processing system 10 includes a user terminal 102 and a plurality of patch servers 110 (110 1~N ) and a plurality of Upper servers 112 (112) each storing a plurality of Upper models. 1~N ) and a Lower server 114 that stores the Lower model. The user terminal 102, the multiple patch servers 110, the multiple Upper servers 112, and the Lower server 114 are connected via a network NW.

[0010] The user terminal 102 is a terminal used to input an image to be used for learning. The patch server 110 is a storage server for storing multiple patches generated by dividing an image input using the user terminal 102. The upper server 112 and the lower server 114 are servers for storing an upper model and a lower model, respectively, as models for learning.

[0011] The user terminal 102 may be realized by a computer such as a personal computer. The user terminal 102 may include a computing device, a storage device, and a communication interface. An example of the computing device is at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). An example of the storage device is at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, and an SSD (Solid State Drive). The user terminal 102 may include at least one of an input device (e.g., a keyboard, a mouse, a touchpad, etc.) and an output device (e.g., a display, a speaker, etc.).

[0012] (study) Next, the data flow of the information processing system 10 during training of a training model will be described with reference to Figures 2 and 3(a). In Figure 2 (1), an input image input by a user via a user terminal 102 is divided into a plurality of patches. In Figure 2, the input image is divided into nine patches, but the number of divisions is not limited to nine. The division of the input image may be performed by the user terminal 102. Note that the division of the input image may also be performed by a single patch server 110.

[0013] The division of the input image may be simple division or overlapping division. In simple division, the input image may be divided without overlapping, for example, based on the size of the input image. For example, for a 32x32 input image, if the patch size is 16x16, the input image may be divided into four patches. For example, for a 32x32 input image, if the patch size is 8x8, the input image may be divided into 16 patches. In overlapping division, the input image may be divided with a fixed length of overlap. For example, for a 32x32 input image, if the fixed length is 8 and the patch size is 16x16, the input image may be divided into nine patches. For example, for a 32x32 input image, if the fixed length is 4 and the patch size is 8x8, the input image may be divided into 49 patches. Note that either simple division or overlapping division may be selected depending on the complexity of the input image.

[0014] In (2) of Fig. 2, multiple patches generated by dividing one input image are stored in separate patch servers 110. The multiple patches may each be stored in a predetermined patch server 110. One of the multiple patch servers 110 may store only a patch corresponding to a predetermined position of the input image (for example, patch P1 corresponding to the upper left corner of the input image).

[0015] In (3) of FIG. 2, each of the multiple patches is input to a predetermined upper model. At this time, the multiple patches are input to each of the multiple upper models without overlapping (in other words, so that two or more patches are not input to one upper model). For example, as shown in FIG. 3(a), patch P1 corresponding to the upper left corner of the input image may be input to "Upper #01" as the upper model. For example, patch P2 corresponding to the upper center of the input image may be input to "Upper #02" as the upper model. In this way, the upper model to which the patch is input may be predetermined based on the position of the patch in the input image. As a result of the patch being input to the upper model, learning is performed by the upper model. Here, each upper model is started and operated by an independent upper server 112.

[0016] 3(a), after learning by each Upper model, noise is added to the calculation result of each Upper model. The process of adding noise to the calculation result of the Upper model may be performed by each Upper server 112. An example of the calculation result of the Upper model is a feature amount related to a patch.

[0017] An example of noise added to the calculation result of the Upper model will be described with reference to FIG. 4. In FIG. 4, "F" represents the calculation result of the Upper model. In FIG. 4(b), "Conv2d," "BatchNorm2d," and "Tanh" represent a "two-dimensional convolutional layer," a "regularization layer," and an "activation function," respectively. The activation function is not limited to "Tanh," and may be, for example, a sigmoid function or a ReLU (Rectified Linear Unit). In FIG. 4(b), "Conv2d," "BatchNorm2d," and "Tanh" represent one neuron in the neural network. In FIG. 4(c), "Activation" represents the "activation function." In FIG. 4(c), "Block_1," "Block_2," ..., "Block_n" represent one neuron in the neural network.

[0018] For example, in the method shown in FIG. 4(a), the product N_new (i.e., α*N) of Gaussian noise N and coefficient α may be added to the calculation result F as the noise. Note that Gaussian noise may also be referred to as white noise. For example, in the method shown in FIG. 4(b), the calculation result F may be input to a neural network. The sum W_new (i.e., F+W) of the output W of the neural network and the calculation result F may be calculated. The product N_new (i.e., α*W_new / |W_new|*N) of Gaussian noise N and coefficient “α*W_new / |W_new|” may be added to the calculation result F as the noise.

[0019] For example, in the method shown in FIG. 4(c), the calculation result F may be input to the neural network. The product N_new (i.e., α*W / W_mean*N) of Gaussian noise N and a coefficient “α*W / W_mean” including the output W of the neural network may be added to the calculation result F as the noise. Each neuron (see “Block” in FIG. 4(c)) of the neural network may have a two-dimensional convolutional layer (e.g., Conv2d), a regularization layer (e.g., BatchNorm2d), and an activation function. The input x_in of each neuron may be input to the two-dimensional convolutional layer. The sum of the output of the regularization layer and the input x_in may be input to the activation function. The activation function may output an output x_out. Note that in the case of Block_1, the input x_in is the calculation result F. In the case of Block_n, the output x_out is the output W of the neural network. For example, the activation function of Block_1 to Block_(n-1) may be a Mish function, and the activation function of Block_n may be a sigmoid function. Note that the activation functions are not limited to the Mish function and the sigmoid function, and other activation functions may also be applied. Note that the number of neurons in the neural network shown in Figure 4(c) may be arbitrary.

[0020] The noise added to the calculation result of the Upper model is not limited to the noise calculated by the method described with reference to FIG. 4, and may be noise calculated by other methods. The noise added to the calculation result of the Upper model is not limited to Gaussian noise, and any random noise can be applied. The configuration of each neuron of the neural network is not limited to the configuration shown in FIGS. 4(b) and 4(c). For example, a neuron may have a pooling layer between a two-dimensional convolution layer and a regularization layer. For example, a neuron may have two or more two-dimensional convolution layers. For example, a neuron may have two or more pairs of a two-dimensional convolution layer and a pooling layer.

[0021] Returning to FIG. 2, after the calculation results of each Upper model to which noise has been added are integrated (see FIG. 3(a)), the integrated calculation result of the Upper model is input to the Lower model. The process of integrating the calculation results to which noise has been added may be performed by the Lower server 114. In the Lower model, calculations are performed based on the calculation results of the integrated Upper model. As a result, the recognition result of the training model is obtained. For example, the training model may be evaluated by calculating the loss related to the calculation results of the Lower model. In this way, the information processing system 10 generates and outputs a trained model.

[0022] In (5) of FIG. 2, necessary patches may be collected according to the learning status of the trained learning model. An input image may be restored from the collected patches. The restored input image may be analyzed, and the patches may be appropriately labeled with correct answer data. Note that this process is optional and may not be performed.

[0023] The operation of the information processing system 10 during training of a training model will be further described with reference to the flowchart of FIG. 5(a). In FIG. 5(a), the user terminal 102 divides an input image into a plurality of patches (step S111). The user terminal 102 transmits the plurality of patches to a plurality of predetermined patch servers 110, respectively. Thereafter, the plurality of patches generated by dividing one input image are input to a predetermined upper model (step S112). As a result, a calculation result is output from each upper model. For example, each of the plurality of upper servers 112 adds noise to the calculation result of the upper model (step S113). Each of the plurality of upper servers 112 transmits the calculation result with the added noise to the lower server 114.

[0024] The Lower server 114 may spatially integrate the calculation results of the Upper models (step S114). The Lower server 114 inputs the integrated calculation results of the Upper models to the Lower model (step S115). In the Lower model, calculations are performed based on the calculation results of the integrated Upper models. As a result, a recognition result of the training model is obtained. The Lower server 114, for example, calculates a loss related to the calculation results of the Lower model (step S116). In the information processing system 10, for example, weight parameters of each of the multiple Upper models and Lower models may be adjusted based on the calculated loss.

[0025] (inference) An information processing method for image recognition using a trained model generated by the above-described information processing method for deep learning of image recognition will be described. The trained model has a plurality of trained upper models and a trained lower model.

[0026] In the information processing system 10, image recognition (e.g., inference) may be performed using a plurality of trained upper models and lower models. Here, the flow of data in the information processing system 10 during inference of an unknown image (i.e., an image not used in training the training model) will be described with reference to Fig. 3(b).

[0027] In FIG. 3(b), an input image input by a user via the user terminal 102 is divided into a plurality of patches. In FIG. 3(b), the input image is divided into nine patches, but the number of divisions is not limited to nine. The plurality of patches are input to a predetermined upper model. For example, during learning, patch P1 corresponding to the upper left corner of the input image may be input to "Upper #01" as an upper model. In this case, patch P1 corresponding to the upper left corner of an unknown image may be input to "Upper #01" as a trained upper model trained using patch P1.

[0028] As a result, a plurality of calculation results corresponding to the plurality of patches are output from the plurality of trained upper models. After the calculation results of each trained upper model are integrated, the integrated calculation result is input to the lower model. The lower server 114 inputs the integrated calculation result to the trained lower model. As a result, a recognition result of the input image by the trained model is obtained. Note that, during inference, noise is not added to the calculation results of each trained upper model.

[0029] The operation of the information processing system 10 during inference of an unknown image will be further described with reference to the flowchart of FIG. 5(b). In FIG. 5(b), the user terminal 102 divides an input image into a plurality of patches (step S121). The user terminal 102 inputs the plurality of patches into a predetermined trained upper model (step S122). As a result, a plurality of calculation results corresponding to the plurality of patches are output from the plurality of trained upper models. The upper server 112 transmits the plurality of calculation results to the lower server 114. The lower server 114 may spatially integrate the plurality of calculation results (step S123). The lower server 114 inputs the integrated calculation result into the lower model (step S124). As a result, a recognition result of the input image using the trained model is obtained.

[0030] (Technical Effects) Data containing personal information, such as facial images, is sometimes used to train models for image recognition. Meanwhile, privacy protection laws such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) have been enacted in various countries, making privacy protection increasingly important in data collection and model training.

[0031] For example, federated learning protects privacy by not collecting data centrally. However, there is a problem in that it is not possible to adjust the data when trying to improve the model's performance. Another method to protect privacy is to mask parts of the image that contain personal information, such as faces, but this is likely to affect the model's recognition performance.

[0032] In contrast, in the information processing system 10, an input image is divided into multiple patches when learning a learning model. Because the size of the patches is small compared to the size of the input image, even if the input image contains private information, it is extremely difficult to identify the private information from the patches. In other words, by dividing the input image into multiple patches, each patch can be made into non-private information.

[0033] Furthermore, when training a neural network in deep learning for image recognition, a large amount of images is required as training data. In addition, in the case of supervised learning, after collecting images, it is necessary to label the images with correct data. The collected images may contain privacy-related information such as faces or vehicle license plates. When storing collected images in one location, even if security is ensured, care must be taken in how they are handled.

[0034] In contrast, in the information processing system 10, multiple patches generated by dividing one input image are stored in separate patch servers 110. Therefore, privacy information will not be restored unless multiple patches related to one input image are retrieved from each of the multiple patch servers 110. Note that some users may be permitted to retrieve multiple patches related to one input image from multiple patch servers 110. With this configuration, it is possible to verify a learning model using one input image reconstructed from multiple patches.

[0035] The information processing system 10 uses a method called a partitioned neural network, which partitions a neural network. For example, the calculation results of the upper model (e.g., feature quantities related to the patch) are input to the lower model, but the patch is not input. In this way, the partitioned neural network can take privacy protection into consideration. However, research is being conducted on a method for restoring original data from data in the middle of calculation (e.g., the calculation results of the upper model input to the lower model).

[0036] In contrast, in the information processing system 10, noise is added to the calculation results of the Upper model when the learning model is trained. Therefore, the information processing system 10 can make it difficult to restore the original data from the data input to the Lower model (i.e., the calculation results to which noise has been added). In addition, research by the inventors of the present application has revealed that adding noise to the calculation results of the Upper model improves the generalization performance of the trained model. In other words, the information processing system 10 can improve the performance of the trained model.

[0037] As described above, the information processing system 10 enables privacy protection during training of a training model.

[0038] Second Embodiment A second embodiment of the information processing method will be described with reference to Fig. 6 in addition to Fig. 1 and Fig. 2. The second embodiment is similar to the first embodiment described above, except that the information processing method is partially different. Therefore, for the second embodiment, explanations that overlap with the explanation for the first embodiment will be omitted as appropriate.

[0039] (study) As shown in FIG. 6(a), before each of the multiple patches is input to the upper model, combinations of the multiple patches generated by dividing an input image and the multiple upper models to which the multiple patches are respectively input may be determined randomly (in other words, randomly). At this time, the combinations of the multiple patches and the multiple upper models are determined so that two or more patches are not input to one upper model. In FIG. 2(3), the multiple patches are input to the multiple upper models according to the combinations determined as described above. As a result, learning is performed using the upper models.

[0040] As shown in FIG. 6(a), after learning is performed using each upper model, noise is added to the calculation results of each upper model. Then, as shown in FIG. 6(a), the calculation results of each upper model with the noise added are integrated. At this time, the calculation results of each upper model are spatially integrated based on the combination determined as described above and the positions of each of the multiple patches in one input image. For example, for multiple patches P1 to P9, the calculation results of each upper model are integrated so as to reproduce the positional relationship between the multiple patches P1 to P9, which corresponds to spatially integrating the calculation results of each upper model. Then, the integrated calculation results of the upper model are input to the lower model.

[0041] The operation of the information processing system 10 according to the second embodiment will now be described. In the information processing system 10 according to the second embodiment, after the processing of step S111 in FIG. 5(a) and before the processing of step S112, combinations of multiple patches generated by dividing one input image and multiple upper models to which the multiple patches are respectively input are randomly determined. This processing may be performed by, for example, the user terminal 102, one patch server 110, or one upper server 112. Information indicating the combinations of multiple patches and multiple upper servers is transmitted to the lower server 114.

[0042] 5(a), in accordance with the combinations determined as described above, a plurality of patches generated by dividing one input image are input to the corresponding upper models of the plurality of upper servers 112. As a result, calculation results are output from each upper model.

[0043] In the process of step S114 in Fig. 5(a), the Lower server 114 spatially integrates the calculation results of each Upper model based on information indicating the combination of multiple patches and multiple Upper servers and the positions of each of the multiple patches in one input image.The Lower server 114 then inputs the calculation results of the integrated Upper model to the Lower model (step S115).The Lower model performs calculations based on the calculation results of the integrated Upper model.As a result, the recognition result of the learning model is obtained.

[0044] (inference) An information processing method for image recognition using a trained model generated by the above-described information processing method for deep learning of image recognition will be described.

[0045] In the second embodiment, one trained upper model may be selected from multiple trained upper models. For example, an evaluation image (so-called test data) may be input to each of the multiple trained upper models. The multiple trained upper models may be evaluated based on the calculation results of each of the multiple trained upper models. For example, the trained upper model with the highest evaluation may be selected as the single trained upper model.

[0046] In the second embodiment, as shown in FIG. 6(b), image recognition (e.g., inference) may be performed using the selected one trained upper model (see "Upper #x" in FIG. 6(b)) and a lower model. In the second embodiment, as shown in FIG. 6(b), a plurality of patches are input to the selected one trained upper model. In this case, a plurality of calculation results corresponding to the plurality of patches are output from the one trained upper model.

[0047] (Technical Effects) According to the information processing system 10 of the second embodiment, privacy protection during training of a training model is possible, similar to the first embodiment described above.

[0048] In the first embodiment described above, when a training model is trained, a patch corresponding to a specific position in an input image is always input to one upper model. In this case, the trained upper model tends to have high accuracy in calculation results related to patches corresponding to the specific position, while the accuracy of calculation results related to patches corresponding to positions other than the specific position in the input image tends to be low. For this reason, when inferring an unknown image, it is difficult to obtain a recognition result with the expected accuracy unless all of the multiple trained upper models included in the trained model are used.

[0049] In contrast, in the second embodiment, when a learning model is trained, a combination of multiple patches and multiple upper models to which the multiple patches are respectively input is randomly determined. When a patch corresponding to an arbitrary position in an input image is input to one upper model during training, the accuracy of the calculation results of the trained one upper model is less affected by the input patch. Therefore, when inferring an unknown image, even if only one trained upper model out of multiple trained upper models is used, it is possible to obtain recognition results with the expected accuracy.

[0050] In the second embodiment, an unknown image is inferred using a selected trained upper model and a trained lower model. This reduces resource consumption compared to when an unknown image is inferred using multiple trained upper models and trained lower models. Furthermore, research by the present inventors has revealed that the accuracy of the recognition result when an unknown image is inferred using a selected trained upper model and a trained lower model is equal to or higher than the accuracy of the recognition result when an unknown image is inferred using multiple trained upper models and trained lower models.

[0051] Aspects of the invention derived from the above-described embodiments will be described below.

[0052] In one aspect of the invention, an information processing method includes a learning model consisting of a plurality of first models and a second model different from the plurality of first models, and includes a division step of dividing an image used for learning the learning model into a plurality of patches, a first input step of inputting the plurality of patches into the plurality of first models without overlapping, an addition step of adding noise to each of a plurality of calculation results output from the plurality of first models, and a second input step of inputting the plurality of calculation results with added noise into the second model. In the above embodiment, the "Upper model" corresponds to an example of the "first model," and the "Lower model" corresponds to an example of the "second model."

[0053] In the information processing method, in the first input step, the multiple patches may be input randomly and without overlapping to the multiple first models, respectively. In this aspect, in the second input step, the multiple calculation results to which noise has been added may be integrated based on information indicating a correspondence between the positions of the multiple patches in the image and the first models to which the multiple patches have been input, and then the multiple calculation results to which noise has been added may be input to the second model. In the information processing method, the noise may be Gaussian noise.

[0054] The present disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing methods involving such modifications are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0055] 10...information processing system, 102...user terminal, 110...patch server, 112...upper server, 114...lower server

Claims

1. the learning model is configured by a plurality of first models and a second model different from the plurality of first models, a division step of dividing an image used for training the learning model into a plurality of patches; a first input step of inputting the plurality of patches into the plurality of first models, respectively, without overlapping; an adding step of adding noise to each of a plurality of calculation results output from the plurality of first models; a second input step of inputting the plurality of calculation results to which noise has been added into the second model; An information processing method including:

2. In the first input step, the plurality of patches are input randomly and without overlapping to the plurality of first models, respectively. The information processing method according to claim 1 .

3. In the second input step, the plurality of calculation results to which noise has been added are integrated based on information indicating a correspondence relationship between the position of each of the plurality of patches in the image and a first model to which each of the plurality of patches has been input, and then the plurality of calculation results to which noise has been added are input to the second model. The information processing method according to claim 2 .

4. The noise is Gaussian noise The information processing method according to claim 1 .

Images