Information processing method

Abstract

The learning model is configured from a plurality of first models and a second model different from the plurality of first models. The information processing method includes: a segmentation process of segmenting an image for learning of a learning model into a plurality of blocks; a first input process of respectively inputting the plurality of blocks to the plurality of first models in a non-repetitive manner; an addition process for adding noise to each of a plurality of calculation results output from each of the plurality of first models; and a second input process of inputting the plurality of calculation results to which the noise has been added to the second model.

Description

Technical Field

[0001] This disclosure relates to the technical field of information processing methods. Background Technology

[0002] As an example of this method, the following approach has been proposed: the neural network constituting the learning model is divided into multiple parts, thereby enabling privacy protection during the learning of the learning model (see Japanese Patent Application Publication No. 2024-030614).

[0003] There is room for improvement in the technology described in Japanese Patent Application Publication No. 2024-030614. Summary of the Invention

[0004] The present invention was made in view of the above circumstances, and its object is to provide an information processing method that can realize privacy protection in the learning of learning models.

[0005] One information processing method disclosed herein includes: a segmentation process, dividing an image used for learning a learning model into multiple blocks, wherein the learning model consists of multiple first models and a second model different from the multiple first models; a first input process, inputting the multiple blocks into the multiple first models respectively in a non-repeating manner; an appending process, appending noise to multiple calculation results output from the multiple first models respectively; and a second input process, inputting the multiple calculation results with the appended noise into the second model. Attached Figure Description

[0006] Hereinafter, with reference to the accompanying drawings, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described, wherein the same reference numerals denote the same elements, wherein:

[0007] Figure 1 It is a diagram showing the structure of an information processing system.

[0008] Figure 2 This is a diagram illustrating the data flow and the structure of the learning model in an information processing system.

[0009] Figure 3A It is a diagram representing the main part of the data flow of an information processing system during learning.

[0010] Figure 3B This is a diagram representing the main part of the data flow in the information processing system when making the conclusion.

[0011] Figure 4A This is a diagram illustrating a specific example of a method for calculating noise.

[0012] Figure 4B This is a diagram illustrating a specific example of a method for calculating noise.

[0013] Figure 4C This is a diagram illustrating a specific example of a method for calculating noise.

[0014] Figure 5A This is a flowchart illustrating an example of an information processing method during learning.

[0015] Figure 5B This is a flowchart illustrating an example of an information processing method used when drawing conclusions.

[0016] Figure 6A It is a diagram representing the main part of the data flow of an information processing system during learning.

[0017] Figure 6B This is a diagram representing the main part of the data flow in the information processing system when making the conclusion. Detailed Implementation

[0018] First Implementation Method

[0019] Reference Figures 1 to 5B A first embodiment of the information processing method will be described. First, an information processing method related to deep learning for image recognition will be described. In this embodiment, the image used for learning is segmented into multiple patches. In this embodiment, a patch-splitting neural network is used as the structure of the learning model. The learning model can be, for example, a CNN (Convolutional Neural Network) segmented into an upper layer model and a lower layer model.

[0020] exist Figure 1 In this system, the information processing system 10 includes a user terminal 102, multiple patch servers 110 (110-1 to 110-N), multiple upper-layer servers 112 (112-1 to 112-N) that store multiple upper-layer models respectively, and lower-layer servers 114 that store lower-layer models. The user terminal 102, the multiple patch servers 110, the multiple upper-layer servers 112, and the lower-layer servers 114 are connected via a network NW.

[0021] User terminal 102 is a terminal used for inputting images for learning. Block server 110 is a storage server for storing multiple blocks generated by segmenting the image input using user terminal 102. Upper-layer server 112 and lower-layer server 114 are servers that respectively store upper-layer and lower-layer models used as learning models.

[0022] User terminal 102 can be implemented, for example, by a computer such as a personal computer. User terminal 102 may include a computing device, a storage device, and a communication interface. Examples of computing devices include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Examples of storage devices include at least one of RAM (Random Access Memory), ROM (Read Only Memory), a hard disk drive, and an SSD (Solid State Drive). User terminal 102 may also include at least one of an input device (e.g., a keyboard, mouse, touchpad, etc.) and an output device (e.g., a display, speakers, etc.).

[0023] study

[0024] Next, refer to Figure 2 , Figure 3A , Figure 3B The data flow of the information processing system 10 during the learning process of the learning model is described. Figure 2 In (1), the input image input by the user via user terminal 102 is segmented into multiple blocks. Figure 2 In this process, the input image is segmented into nine blocks, but the number of segments is not limited to nine. The segmentation of the input image can be performed via user terminal 102. It should be noted that the segmentation of the input image can also be performed via a block server 110.

[0025] Input image segmentation can be either simple segmentation or overlapping segmentation. In simple segmentation, the input image can be segmented non-repeatingly based on its size. For example, a 32×32 input image can be segmented into four blocks when the block size is set to 16×16. Similarly, a 32×32 input image can be segmented into sixteen blocks when the block size is set to 8×8. In overlapping segmentation, the input image can be segmented by overlapping blocks of a fixed length. For example, a 32×32 input image can be segmented into nine blocks when the fixed length is set to 8 and the block size to 16×16. Similarly, a 32×32 input image can be segmented into forty-nine blocks when the fixed length is set to 4 and the block size to 8×8. It should be noted that the choice between simple and overlapping segmentation can also be made based on the complexity of the input image.

[0026] exist Figure 2 In (2), multiple blocks generated by segmenting an input image are stored in different block servers 110. The multiple blocks can be stored in designated block servers 110 respectively. Alternatively, one of the multiple block servers 110 may store only the block corresponding to a designated position of the input image (e.g., block P1 corresponding to the upper left corner of the input image).

[0027] exist Figure 2 In step (3), multiple blocks are input to predetermined upper-level models. At this time, the multiple blocks are input to multiple upper-level models in a non-repeating manner (in other words, no more than two blocks are input to one upper-level model). For example, as... Figure 3A As shown, block P1 corresponding to the upper left corner of the input image can be input to "upper layer #01" as the upper layer model. For example, block P2 corresponding to the upper center of the input image can be input to "upper layer #02" as the upper layer model. Thus, the upper layer model for block input can be predetermined based on the position of the blocks in the input image. The result of the blocks being input to the upper layer model is that learning based on the upper layer model is performed. Here, each upper layer model is started and operates in a separate upper layer server 112.

[0028] like Figure 3A As shown, after learning based on each upper-layer model, noise is added to the calculation results of each upper-layer model. It should be noted that the process of adding noise to the calculation results of the upper-layer models can be performed by each upper-layer server 112. It should also be noted that, as an example of the calculation results of the upper-layer models, block-related feature quantities are given.

[0029] Here, refer to Figures 4A to 4C An example of noise added to the computational results of the upper-level model is given. It should be noted that... Figures 4A to 4C In this context, "F" represents the calculation result of the upper-level model. It should be noted that... Figure 4B In this context, "Conv2d", "BatchNorm2d", and "Tanh" represent "two-dimensional convolutional layer", "regularization layer", and "activation function", respectively. It should be noted that the activation function is not limited to "Tanh"; for example, it can also be the sigmoid function, ReLU (Rectified Linear Unit), etc. Figure 4B In this context, "Conv2d", "BatchNorm2d", and "Tanh" represent a neuron in a neural network. Figure 4C In this context, "Activation" refers to the "activation function". Figure 4CIn this context, "Block_1", "Block_2", ..., "Block_n" represent a neuron in a neural network.

[0030] For example, in Figure 4A In the method shown, the product of Gaussian noise N and the coefficient α, N_new (i.e., α*N), can be added to the calculation result F as the aforementioned noise. It should be noted that Gaussian noise can also be called white noise. For example, in... Figure 4B In the method shown, the calculation result F can be input into a neural network. The sum W_new (i.e., F+W) of the output W of the neural network and the calculation result F can be calculated. The product N_new (i.e., α*W_new / |W_new|*N) of Gaussian noise N and the coefficient "α*W_new / |W_new|" can be added to the calculation result F as the aforementioned noise.

[0031] For example, in Figure 4C In the method shown, the calculation result F can be input into the neural network. The product N_new (i.e., α*W / W_mean*N) of Gaussian noise N and the coefficients “α*W / W_mean” including the output W of the neural network can be added to the calculation result F as the aforementioned noise. The neurons of the neural network (refer to...) Figure 4C A "Block" can have a two-dimensional convolutional layer (e.g., Conv2d), a regularization layer (e.g., BatchNorm2d), and an activation function. The input x_in of each neuron can be fed into the two-dimensional convolutional layer. The sum of the output of the regularization layer and the input x_in can be fed into the activation function. The activation function outputs x_out. It should be noted that in the case of Block_1, the input x_in is the computational result F. In the case of Block_n, the output x_out is the output W of the neural network. For example, the activation function for Block_1 to Block_(n-1) can be the Mish function, and the activation function for Block_n can be the Sigmoid function. It should be noted that the activation function is not limited to the Mish and Sigmoid functions; other activation functions can also be applied. Figure 4C The number of neurons in the neural network shown can be arbitrary.

[0032] It should be noted that the noise added to the calculation results of the upper-level model is not limited to noise from reference. Figures 4A to 4C The noise calculated using the methods described herein can also be noise calculated using other methods. It should be noted that Gaussian noise is not limited to the noise added to the calculation results of the upper-level model; any random noise can be used. It should also be noted that the composition of each neuron in the neural network is not limited to... Figure 4B and Figure 4C The configuration is shown. For example, a neuron may also have a pooling layer between a two-dimensional convolutional layer and a regularization layer. For example, a neuron may also have more than two two-dimensional convolutional layers. For example, a neuron may also have more than two groups formed by two-dimensional convolutional layers and pooling layers.

[0033] Back Figure 2 After the calculation results of each upper-level model with added noise are integrated (refer to...) Figure 3A The integrated calculation results of the upper-layer model are then input into the lower-layer model. It should be noted that the integration of calculation results with added noise can be performed by the lower-layer server 114. In the lower-layer model, calculations are performed based on the integrated calculation results of the upper-layer model. The result is the recognition result of the learned model. For example, the learned model can be evaluated by calculating the loss associated with the calculation results of the lower-layer model. In this way, the information processing system 10 generates and outputs the learned model.

[0034] exist Figure 2 In (5), the required blocks can be collected based on the learning status of the learned learning model. The input image can be reconstructed based on the collected blocks. The blocks can be appropriately labeled with correct answer data by analyzing the reconstructed input image. It should be noted that this process is arbitrary and can be omitted.

[0035] Reference Figure 5A The flowchart illustrates the actions of the information processing system 10 during the learning process of the learning model. Figure 5A In this process, user terminal 102 segments the input image into multiple blocks (S111). User terminal 102 sends the multiple blocks to designated block servers 110 respectively. Then, the multiple blocks generated by segmenting an input image are input to a predetermined upper-level model (S112). As a result, calculation results are output from each upper-level model. For example, the multiple upper-level servers 112 add noise to the calculation results of the upper-level models (S113). The multiple upper-level servers 112 send the calculation results with added noise to the lower-level server 114 respectively.

[0036] The lower-level server 114 can spatially integrate the computation results of each upper-level model (S114). The lower-level server 114 inputs the integrated computation results of the upper-level models into the lower-level model (S115). In the lower-level model, calculations are performed based on the integrated computation results of the upper-level models. The result is the recognition result of the learning model. The lower-level server 114, for example, calculates the loss related to the computation results of the lower-level model (S116). In the information processing system 10, for example, the weight parameters of the lower-level model and the multiple upper-level models can be adjusted based on the calculated loss.

[0037] Conclusion Presumption

[0038] An information processing method related to image recognition is described, wherein the information processing method for image recognition uses a learned model generated by the aforementioned information processing method related to deep learning for image recognition. The learned model has a learned lower-level model and multiple learned upper-level models.

[0039] In the information processing system 10, image recognition (e.g., conclusion inference) can be performed using a learned lower-level model and multiple learned upper-level models. Here, refer to... Figure 3B The data flow of the information processing system 10 is described when making conclusions about unknown images (i.e., images not used for learning the model).

[0040] exist Figure 3B In this process, the input image input by the user via user terminal 102 is segmented into multiple blocks. Figure 3B In this model, the input image is segmented into nine blocks, but the number of segments is not limited to nine. Multiple blocks are input to a pre-defined upper-layer model. For example, during learning, block P1 corresponding to the top-left corner of the input image can be input to "upper-layer #01" which serves as the upper-layer model. In this case, block P1 corresponding to the top-left corner of the unknown image can be input to "upper-layer #01," which has been learned using block P1 and serves as the learned upper-layer model.

[0041] The result is that multiple learned upper-layer models output multiple computational results corresponding to multiple blocks. After the computational results of each learned upper-layer model are integrated, the integrated computational result is input to the lower-layer model. The lower-layer server 114 inputs the integrated computational result to the learned lower-layer model. The result is the recognition result of the input image obtained through the learned model. It should be noted that no noise is added to the computational results of each learned upper-layer model when extrapolating the conclusion.

[0042] Reference Figure 5BThe flowchart illustrates the actions of the information processing system 10 when drawing conclusions about unknown images. Figure 5B In the process, user terminal 102 segments the input image into multiple blocks (S121). User terminal 102 inputs the multiple blocks into a pre-determined, pre-learned upper-layer model (S122). As a result, multiple calculation results corresponding to the multiple blocks are output from the multiple learned upper-layer models. Upper-layer server 112 sends the multiple calculation results to lower-layer server 114. Lower-layer server 114 can spatially integrate the multiple calculation results (S123). Lower-layer server 114 inputs the integrated calculation results into the lower-layer model (S124). As a result, the recognition result of the input image obtained through the learned model is obtained.

[0043] Technical effect

[0044] In the learning of models related to image recognition, data containing personal information, such as facial images, is sometimes used. On the other hand, various countries have enacted privacy-related laws such as the GDPR (General Data Protection Regulation) and the CCPA (California Consumer Privacy Act). Privacy protection is becoming increasingly important in data collection and model learning.

[0045] For example, privacy is protected in federated learning by not centrally collecting data. However, this presents a problem: data cannot be adjusted when improvements to model performance are desired. Furthermore, methods such as masking personal information like faces are considered for privacy protection, but these methods can potentially negatively impact the model's recognition performance.

[0046] In contrast, in information processing system 10, during the learning of the learning model, the input image is segmented into multiple blocks. The size of each block is smaller than the size of the input image, making it extremely difficult to determine the privacy information based on the blocks, even if the input image contains privacy information. In other words, by segmenting the input image into multiple blocks, each block becomes non-privacy information.

[0047] Furthermore, deep learning for image recognition requires a large number of images as training data when training neural networks. In addition, supervised learning necessitates assigning labels to the images after collection. The collected images may sometimes contain privacy-related information such as faces and vehicle license plates. Even with security measures in place, the processing of these collected images still needs to be carefully considered.

[0048] In contrast, in information processing system 10, multiple blocks generated by segmenting an input image are stored on different block servers 110. Therefore, privacy information will not be restored as long as multiple blocks related to an input image are not retrieved separately from multiple block servers 110. It should be noted that some users may also be permitted to retrieve multiple blocks related to an input image from multiple block servers 110. If configured in this way, the verification of a learning model using an input image reconstructed from multiple blocks can be performed.

[0049] Furthermore, in the information processing system 10, a method such as segmenting a neural network to divide the neural network is used. For example, the computation results of the upper-layer model (e.g., block-related features) are input into the lower-layer model instead of the blocks themselves. In this way, privacy protection considerations can be made in the segmented neural network. However, a method is being investigated to reconstruct the original data based on data obtained during the computation (e.g., the computation results of the upper-layer model input into the lower-layer model).

[0050] In contrast, in information processing system 10, noise is added to the computation results of the upper-level model during the learning process. Therefore, according to information processing system 10, it is possible to reconstruct the original data from the data input to the lower-level model (i.e., the computation results with added noise). Furthermore, according to the inventors' research, it has been demonstrated that by adding noise to the computation results of the upper-level model, the generalization performance of the learned model is improved. In other words, information processing system 10 can improve the performance of the learned model.

[0051] As described above, the information processing system 10 enables privacy protection during the learning process of the learning model.

[0052] Second Implementation Method

[0053] In addition to reference Figure 1 and Figure 2 In addition, refer to Figure 6A and Figure 6B A second embodiment of the information processing method will be described. In the second embodiment, except for a portion of the information processing method, it is the same as the first embodiment described above. Therefore, regarding the second embodiment, descriptions that overlap with those of the first embodiment will be appropriately omitted.

[0054] study

[0055] like Figure 6AAs shown, before multiple blocks are input into the upper-level models, the combination of multiple blocks generated by segmenting an input image with the multiple upper-level models to which these blocks are input can be randomly determined. In this case, the combination of multiple blocks with multiple upper-level models is determined in a way that does not input more than two blocks into a single upper-level model. Figure 2 In (3), multiple blocks are input into multiple upper-level models in combinations determined as described above. As a result, learning is performed based on the upper-level models.

[0056] like Figure 6A As shown, after learning based on each upper-level model, noise is added to the calculation results of each upper-level model. Then, as... Figure 6A As shown, the computational results of each upper-layer model, with added noise, are integrated. At this point, the computational results of each upper-layer model are spatially integrated based on the combinations determined as described above and the positions of each block within an input image. For example, regarding multiple blocks P1 to P9, the computational results of each upper-layer model are integrated in a way that reproduces the positional relationships of the multiple blocks P1 to P9, which is equivalent to spatially integrating the computational results of each upper-layer model. Afterward, the integrated computational results of the upper-layer models are input to the lower-layer model.

[0057] The operation of the information processing system 10 according to the second embodiment will be described. In the information processing system 10 of the second embodiment, in... Figure 5A After the processing in S111 and before the processing in S112, a combination of multiple blocks generated by segmenting an input image and multiple upper-level models for inputting those blocks is randomly determined. It should be noted that this processing can be performed, for example, by a user terminal 102, a block server 110, or an upper-level server 112. It should also be noted that information indicating the combination of multiple blocks with multiple upper-level servers is sent to the lower-level server 114.

[0058] exist Figure 5A In the S112 processing, multiple blocks generated by segmenting an input image are input to the upper-layer models of the corresponding upper-layer servers 112 according to the combination determined as described above. As a result, the calculation results are output from each upper-layer model.

[0059] exist Figure 5AIn the processing of S114, the lower-layer server 114 spatially integrates the computation results of each upper-layer model based on information representing the combination of multiple blocks and multiple upper-layer servers, and the position of each block in an input image. Then, the lower-layer server 114 inputs the integrated computation results of the upper-layer models into the lower-layer model (S115). In the lower-layer model, calculations are performed based on the integrated computation results of the upper-layer models. The result is the recognition result of the learned model.

[0060] Conclusion Presumption

[0061] An information processing method related to image recognition is described, wherein the information processing method related to image recognition uses a learned model generated by the information processing method related to deep learning of image recognition described above.

[0062] In the second embodiment, a learned upper-level model can be selected from multiple learned upper-level models. For example, evaluation images (so-called test data) can be input into each of the multiple learned upper-level models. The multiple learned upper-level models can be evaluated based on their respective calculation results. For example, the learned upper-level model with the highest evaluation can be selected as the aforementioned learned upper-level model.

[0063] In the second embodiment, such as Figure 6B As shown, a lower-level model and one of the selected, fully learned upper-level models can be used (see reference). Figure 6B Image recognition (e.g., conclusion estimation) is performed using the "upper layer #x". In the second embodiment, as... Figure 6B As shown, multiple blocks are input into a selected, fully learned upper-level model. In this case, the fully learned upper-level model outputs multiple computational results corresponding to the multiple blocks.

[0064] Technical effect

[0065] The information processing system 10 according to the second embodiment, like the first embodiment described above, can achieve privacy protection in the learning of the learning model.

[0066] In the first embodiment described above, during the learning of the learning model, blocks corresponding to specific locations in the input image are always input into a single upper-level model. In this case, a learned upper-level model tends to have higher accuracy for calculations of blocks corresponding to the specific locations, but lower accuracy for calculations of blocks corresponding to locations in the input image other than the specific locations. Therefore, when making conclusions about unknown images, it is difficult to obtain the desired accuracy of the recognition result without using all the multiple learned upper-level models included in the learned model.

[0067] In contrast, in the second embodiment, during the learning of the learning model, multiple combinations of blocks and multiple upper-level models for each block are randomly determined. During learning, when a block corresponding to any position in the input image is input into an upper-level model, the accuracy of the calculation result of the learned upper-level model is less affected by the input block. Therefore, when making conclusions about unknown images, even using only one of the multiple learned upper-level models, the desired accuracy of the recognition result can be obtained.

[0068] In the second embodiment, a learned lower-level model and a selected learned upper-level model are used to make a conclusion inference for an unknown image. Therefore, compared to using a learned lower-level model and multiple learned upper-level models to make a conclusion inference for an unknown image, resource consumption is reduced. Furthermore, through the inventors' research, it has been clarified that the accuracy of the recognition result when using a learned lower-level model and a selected learned upper-level model to make a conclusion inference for an unknown image is equal to or greater than the accuracy of the recognition result when using a learned lower-level model and multiple learned upper-level models to make a conclusion inference for an unknown image.

[0069] The following describes the inventive solution derived from the embodiments described above.

[0070] In one aspect of the information processing method of the invention, the learning model consists of multiple first models and a second model different from the multiple first models. The information processing method includes a segmentation process, a first input process, an appending process, and a second input process. In the segmentation process, the image used for learning the learning model is segmented into multiple blocks. In the first input process, the multiple blocks are input to the multiple first models in a non-repeating manner. In the appending process, noise is appended to multiple calculation results output from the multiple first models. In the second input process, the multiple calculation results with the appended noise are input to the second model. In the above embodiment, "upper-layer model" corresponds to an example of "first model," and "lower-layer model" corresponds to an example of "second model."

[0071] In this information processing method, during the first input process, the plurality of blocks may be randomly and non-repeatingly input into the plurality of first models. In this scheme, during the second input process, after integrating the plurality of calculation results with added noise based on information representing the correspondence between the respective positions of the plurality of blocks in the image and the first models to which each of the plurality of blocks has been input, the integrated result of the plurality of calculation results with added noise is input into the second model. In this information processing method, the noise may be Gaussian noise.

[0072] This disclosure is not limited to the embodiments described above. Appropriate modifications may be made without departing from the spirit or idea of ​​the invention as read from the claims and the entire specification. Information processing methods with such modifications are also included within the technical scope of this disclosure.

Claims

1. An information processing method, comprising: The segmentation process divides the image used for learning the model into multiple blocks, wherein the learning model consists of multiple first models and a second model that is different from the multiple first models; In the first input process, the multiple blocks are input into the multiple first models in a non-repeating manner; The appending process involves adding noise to the multiple calculation results output from the multiple first models, respectively; and The second input process involves inputting the multiple calculation results, with noise added, into the second model.

2. The information processing method according to claim 1, wherein, During the first input process, the plurality of blocks are randomly and non-repeatingly input into the plurality of first models.

3. The information processing method according to claim 2, wherein, During the second input process, based on information representing the correspondence between the positions of the plurality of blocks in the image and the first model that has been respectively input to each of the plurality of blocks, the plurality of calculation results with added noise are integrated, and the integrated result of the plurality of calculation results with added noise is input to the second model.

4. The information processing method according to claim 1, wherein, The noise is Gaussian noise.

Images