Learning method, apparatus, and program

Abstract

To provide a method, an apparatus, and a program for training an inference machine for applying an inference machine trained using encrypted image data to non-encrypted image data.SOLUTION: A learning method trains an inference machine using a neural network by using a group of images and training data. The inference machine is composed of a front layer and a rear layer that include a plurality of layers. The learning method comprises: fixing the rear layer of the first inference machine trained using a first group of images and first training data; disposing the fixed rear layer in the rear layer of a second inference machine that inputs a second group of images; and inputting the second group of images, and feeding back the error of the inference result of the second inference machine with respect to second training data to the front layer of the second inference machine for learning.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a method, device, and program for training an inferencer, for applying an inferencer based on a neural network trained using encrypted images to unencrypted images. [Background technology]

[0002] In recent years, when camera users send captured image data to the cloud, it is common for the image data to be encrypted in order to protect privacy. There is also a need to input encrypted image data into an inference device using a neural network to perform inference such as identification, classification, and area determination.

[0003] One technology that meets this need is the learning method described in Patent Document 1. This learning method is composed of an image conversion unit using a neural network, and an inference unit that inputs image data converted and encrypted by the image conversion unit. Then, parameters for the image conversion unit are learned using an evaluation value calculated from the difference between the image data before and after conversion, and the error between the output of the inference unit and the teacher data. This makes it possible to obtain a trained image conversion unit that outputs encrypted image data while ensuring the inference accuracy of the inference unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-152725 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional techniques disclosed in the above-mentioned patent documents have the following problems. That is, the inference unit ensures inference accuracy only when encrypted image data is input, and inference accuracy cannot be ensured for input of images before encryption. Therefore, an inference machine trained using encrypted images cannot be used as input for unencrypted images. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention provides a learning method for training an inference device based on a neural network using a group of images and training data, the inference device being composed of an upstream layer including multiple layers and a downstream layer, the downstream layer of a first inference device trained using a first group of images and first training data being fixed, the fixed downstream layer being placed in a downstream layer of a second inference device which inputs a second group of images, and the error in the second training data resulting from the inference made by the second inference device by inputting the second group of images is fed back to the upstream layer of the second inference device for training. Effect of the Invention

[0007] A highly accurate inference machine created using a large collection of encrypted training images for training can also be used with unencrypted images as input. [Brief description of the drawings]

[0008] [Figure 1] Overall configuration diagram of the learning system according to the present invention [Diagram 2] Hardware configuration diagram of the learning system according to the present invention [Diagram 3] A configuration diagram of a learning system according to a first embodiment of the present invention. [Figure 4] A configuration diagram of a learning unit in the first embodiment of the present invention. [Diagram 5] Conceptual diagram of training image data in the first embodiment of the present invention. [Figure 6] A learning flow chart according to the first embodiment of the present invention. [Figure 7] A learning flow chart according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Example 1 Hereinafter, a learning method for an inference device according to a first embodiment of the present invention will be described with reference to FIGS.

[0010] Fig. 1 is a diagram showing an example of the overall configuration of a learning system according to the present invention. As shown in Fig. 1, the learning system according to the present invention is composed of a learning server 101, a data collection server 102, a file server 103, and a camera 104, which communicate with each other via the Internet 100.

[0011] A camera user first stores image data captured by the camera 104 in the image storage unit 320 of the file server 103. Next, the camera user selects image data to be used for training the inference device, and transmits the image data to the data collection server 102, thereby providing the image data to the inference device creator.

[0012] Next, the inference machine creator uses the image data collected in the data collection server 102 to train the inference machine in the learning server 101 .

[0013] In this embodiment, an image captured by the camera 104 will be described, but the capturing device is not limited to a camera and may be a mobile phone, a tablet terminal, or the like.

[0014] 2 is a diagram showing the configuration of hardware resources of each device constituting the system of FIG 1, with the hardware configuration of the learning server 101 and file server 103 shown as server 200. The hardware configuration of the camera 104 is shown as client 210.

[0015] In the server 200, a CPU (Central Processing Unit) 202, a memory 203, a neural network processing unit 204, a communication unit 205, a data storage unit 206, and an input unit 207 are connected to a system bus 201. These units exchange data with each other.

[0016] The memory 203 is, for example, a RAM (Random Access Memory), and is used as a work memory for the CPU 202 and the neural network processing unit 204 .

[0017] The data storage unit 206 is, for example, a hard disk drive (HDD) or a solid state drive (SSD). Alternatively, it is a read only memory (ROM) or an magneto-optical (MO) disk, etc. The data storage unit 206 stores various programs for the operation of the CPU 202.

[0018] The neural network processing unit 204 is configured with one or more GPUs, DSPs, FPGAs, or boards equipped with HW for inference calculation processing, etc. It performs learning of an inference device using a neural network, or calculation processing for inference.

[0019] GPU is an abbreviation of Graphics Processing Unit, and is a computing device that can perform efficient calculations by processing data in parallel. Therefore, when performing learning multiple times using a learning model such as deep learning, it is effective to use a GPU for processing. It is also effective to use a GPU to perform calculations for inference using a trained inference device.

[0020] In this embodiment, for the calculation processing in learning, a neural network processing unit 204 is used in addition to the CPU 202. Specifically, when a learning program including an inference device using a neural network is executed, the CPU 202 and the neural network processing unit 204 cooperate to perform calculations to perform learning. Note that the calculations in learning may be performed only by the CPU 202 or the neural network processing unit 204.

[0021] A communication unit 205 enables data communication between each hardware resource connected to the system bus 201 and devices connected to the Internet 100 via, for example, Ethernet or WiFi, under the control of the CPU 202.

[0022] The input unit 207 receives a user operation, generates a control signal according to the operation, and supplies the control signal to the CPU 202. For example, the input unit 207 includes a keyboard, a mouse, and the like as input devices that receive the user operation.

[0023] The CPU 202 controls the communication unit 205, the memory 203, and the neural network processing unit 204 in accordance with a program loaded from the data storage unit 206 to the memory 203, based on a control signal supplied in response to a user operation performed from the input unit 207. This allows the learning server 101 to cause the inference device to perform learning. Also, inference can be performed in the inference unit 321 of the file server 103.

[0024] The client 210 represents the configuration of the hardware resources of the camera 104 .

[0025] In the client 210, a CPU 212, a memory 213, a communication unit 214, a neural network processing unit 215, an input unit 216, a display unit 217, an imaging unit 218, and a data storage unit 219 are connected to a system bus 211, and exchange data among them.

[0026] The memory 213 is, for example, a RAM, and is used as a work memory for the CPU 212 and the neural network processing unit 215 .

[0027] The data storage unit 219 stores various programs for the operation of the CPU 212. The storage location of the various programs for the operation of the CPU 212 is not limited to the data storage unit 219, and may be, for example, a ROM or the like.

[0028] The CPU 212 controls each unit in accordance with a program loaded from the data storage unit 219 to the memory 213, based on a control signal supplied in response to a user operation performed from the input unit 216. This allows the display unit 217, the imaging unit 218, the communication unit 214, the memory 213, and the neural network processing unit 215 to perform operations in response to the user operation.

[0029] A communication unit 214 enables data communication between each hardware resource connected to the system bus 211 and devices connected to the Internet 100 via, for example, Ethernet or WiFi, under the control of the CPU 212.

[0030] The neural network processing unit 215 performs inference calculations using an inference device based on deep learning.

[0031] The input unit 216 receives a user operation, generates a control signal according to the operation, and supplies it to the CPU 212. For example, the input unit 217 has a physical operation button, a touch panel, or the like as an input device that receives a user operation. Note that the touch panel is an input device that outputs coordinate information according to a position touched on an input unit configured in a planar manner, for example.

[0032] The display unit 217 is, for example, a display, and includes a mechanism for outputting a display signal for displaying an image on the display. When a touch panel is used as the input unit 216, the input unit 216 and the display can be configured as one unit. For example, the touch panel is configured so that the light transmittance does not interfere with the display on the display, and is attached to the upper layer of the display surface of the display. Then, the input coordinates on the touch panel can be associated with the display coordinates on the display.

[0033] The imaging unit 218 is a mechanism for carrying out a series of photographing processes, including an imaging unit consisting of a lens, a shutter with an aperture function, a CCD or CMOS element for converting an optical image into an electric signal, and an image processing unit for carrying out various image processes such as exposure control and distance measurement control based on the signal from the imaging unit. Under the control of the CPU 212, photographing can be performed in response to user operations of the input unit 216.

[0034] FIG. 3 is a diagram showing a software configuration realized by using the hardware resources and programs shown in the hardware configuration diagram of FIG. 2. Boundary 330 and boundary 335 are shown as examples. The upper side of boundary 330 is software managed by an inference device creator who creates an inference device by training the inference device. The lower side of boundary 330 is software managed by the camera user. Furthermore, among the software managed by the camera user, the upper side of boundary 335 belongs to the cloud, and the lower side belongs to the camera 104 which is an edge device. The camera user encrypts the image he / she has taken from the image storage unit 320 and transmits it to the data collection unit 313 managed by the inference device creator. Then, the camera user downloads the inference device that the inference device creator has trained in the training unit 312 to the inference unit 321 of the file server 103 and the inference unit 323 of the camera 104 for use.

[0035] That is, in order to protect privacy, the camera user encrypts an image captured by the imaging unit 324 of the camera 104 in the image conversion unit 325, for example, by homomorphic encryption. Homomorphic encryption is an encryption method in which, for example, an encrypted result of a binary operation is equivalent to the encrypted result of the binary operation before encryption. The operations are addition and multiplication. It is known that this property makes it possible to perform statistical processing and convolution operations on encrypted data.

[0036] Next, the encrypted image data is stored in the image storage unit 320 of the file server 103 on the cloud. Then, the data collection server 102 collects the encrypted image data from the image storage unit 320 under the control of the camera user. Also, the data collection server 313 collects non-encrypted image data captured by the imaging unit 310 of the camera 350 of the inference device creator under the control of the inference device creator. Furthermore, the collected encrypted image data or non-encrypted image data is stored in the data storage unit 314.

[0037] In response to a request from the learning unit 312, the data providing unit 315 transmits encrypted image data or non-encrypted image data stored in the data storage unit 314 to the learning data generating unit 311. The learning unit 312 uses the training encrypted image data and teacher data generated by the training data generating unit 311 to train an inference device that inputs the encrypted image data.

[0038] Furthermore, in response to a request from the learning unit 312, the data providing unit 315 transmits the non-encrypted image data stored in the data storage unit 314 to the learning data generating unit 311. The learning unit 312 uses the non-learning encrypted image data and teacher data generated by the learning data generating unit 311 to train an inference device that receives the non-encrypted image data as input.

[0039] The camera user downloads the non-encrypted image inference device, which inputs the non-encrypted image that has completed learning in the learning unit 312, to the inference unit 323 of the camera 104, performs inference on the captured image, and displays the inference result on the display unit 322. In addition, the encrypted image inference device, which inputs the encrypted image data that has completed learning in the learning unit 312, is downloaded to the inference unit 321 of the file server 103, and the result of inferring the encrypted image data stored in the image storage unit 320 is displayed on the display unit 322.

[0040] Fig. 4 is a diagram showing an example of the flow of data when training an inference machine in the training unit 312. Fig. 4(a) is a diagram related to training an inference machine for encrypted images, and Fig. 4(b) is a diagram related to training an inference machine for non-encrypted images.

[0041] 4(a), the front-stage layer 410 is composed of an input layer 411, a convolutional layer 412, and a pooling layer 413. The convolutional layer 412 and the pooling layer 413 may each be composed of a plurality of layers. The back-stage layer 420 is composed of fully connected layers 421 and 422, and an output layer 423. The fully connected layers 421 and 422 may each be a single layer, or three or more layers.

[0042] To start learning, the learning unit 312 obtains the encrypted image data 400 and the teacher data T (402) from the learning data generating unit 311.

[0043] Next, the learning unit 312 inputs the encrypted image data 400 to an encrypted image inference device that is the learning target and is composed of a front-stage layer 410 and a back-stage layer 420 .

[0044] The inference result value Y1 (401) output by the encrypted image inference device and the teacher data T (402) are input to an error function 404 to obtain an error L1 (403). The error L1 (403) is fed back 405 to all layers constituting the encrypted image inference device using the error backpropagation method, and the parameter values ​​of each layer are adjusted so that the value of error L1 (403) is reduced for each epoch from the output layer 423 to the input layer 411. Here, an epoch is one learning step in which learning data consisting of one or more pieces of data is input to the inference device once, and the parameters are adjusted once from the obtained error. Learning is completed when this step is repeated multiple times and the value of the error becomes equal to or less than a predetermined value.

[0045] 4(b), the front-stage layer 440 is composed of an input layer 441, a convolution layer 442, and a pooling layer 443. Note that the convolution layer 442 and the pooling layer 443 may each be composed of a plurality of layers.

[0046] The latter layer 450 is obtained by fixing the parameters and layer structure of the latter layer 420 of the trained encrypted image inference device. Therefore, the parameter values ​​and layer structure of the latter layer do not change in the training using the non-encrypted image data 430 in Fig. 4(b).

[0047] The learning data generation unit 311 inputs non-encrypted image data 430 corresponding to teacher data T (432) having the same value as teacher data T (402) to the learning unit.

[0048] Unencrypted image data 430 is input to an unencrypted image inference device, which is the learning target, and which is composed of a front-stage layer 440 and a back-stage layer 450 .

[0049] The inference result value Y2 (431) output by the non-encrypted image inference device and the training data T (432) are input to an error function 434 to obtain an error L2 (433). The error L2 (433) is fed back 435 to the previous layer of the non-encrypted image inference device using the error backpropagation method, and the parameter values ​​of each layer are adjusted for each epoch from the first layer 451 to the input layer 441 in the subsequent layers so as to reduce the value of the error L2 (433).

[0050] FIG. 5 shows examples of the encrypted image data 400, the non-encrypted image data 430, the training data T (402) for the encrypted image data 400, and the training data T (432) for the non-encrypted image data 430 in FIG.

[0051] In FIG. 5(a), an unencrypted image 501 of a cat taken by a camera user is linked to a label 502 whose label name is "cat". The camera user encrypts the unencrypted image 501 in the image conversion unit 325 to obtain an encrypted image 503. The camera user then transmits the encrypted image 503 to the data collection server 102, and stores encrypted image data 505 with a "cat" label in the data storage unit 314. The "cat" label may be linked to the image by a label linking function of the camera 104 (not shown) or by another device. The "cat" label may also be encrypted using a homomorphic encryption method.

[0052] 5(b), an unencrypted image 506 of a cat taken by the inference machine creator using the imaging unit 310 is associated with a label 507 whose label name is "cat". The inference machine creator transmits the unencrypted image 506 to the data collection server 102, and stores unencrypted image data 508 with the label "cat" in the data storage unit 314.

[0053] The learning data generation unit 311 first receives encrypted image data 505 labeled with a "cat" from the data provision unit 315, and sets the teacher data T (402) to "cat". Furthermore, the encrypted image data 505 labeled with a "cat" is used to train the inference device. Next, the learning data generation unit 311 receives non-encrypted image data 508 labeled with a "cat" from the data provision unit 315, and sets the teacher data T (432) to "cat". Further, the inference device is trained using non-encrypted image data 508 labeled with a "cat".

[0054] FIG. 6 is a flowchart showing the flow of learning.

[0055] In FIG. 6(a), learning 600 using the unencrypted image data 430 is shown in steps 601 to 604.

[0056] In FIG. 6(b), learning 601 of the front and rear layers using encrypted image data 400, which is one of the steps of learning 600 using non-encrypted image data 430, is shown by steps 611 to 613.

[0057] In FIG. 6(c), learning 612 using learning image data, which is one step of learning 600 using encrypted image data 400 and learning 601 in the front and back layers using non-encrypted image data 430, is shown as steps 621 to 625.

[0058] The flow of learning 600 using the non-encrypted image data 430 shown in FIG. 6(a) will be described.

[0059] In step 601, the CPU 202 trains an encrypted image reasoner using encrypted image data, and as a result, obtains a trained subsequent layer 614 of the encrypted image reasoner.

[0060] Next, in step 602, the CPU 202 arranges the latter layer 614 of the trained encrypted image reasoner as the latter layer 450 of the unencrypted image reasoner that trains by inputting the unencrypted image data 430.

[0061] Next, in step 603, the CPU 202 requests the unencrypted image data 320 from the data collection server 102. Then, the process proceeds to step 604 under the condition that the parameters of the front-stage layer 440 of the reasoner are not fixed, and the parameters of the back-stage layer 450 are fixed.

[0062] In step 604, the CPU 202 performs learning using the non-encrypted image data 430. At this time, in step 622, the CPU 202 sets parameters belonging to the previous layer 440. In step 623, the CPU 202 inputs the non-encrypted image data 430. In step 624, the output of the inference device is Y2 (431), the teacher data is T (432), and the loss value is L2 (433). By completing learning using the non-encrypted image data in step 604, a trained non-encrypted image inference device is obtained.

[0063] The flow of learning 601 of the front and rear layers using encrypted image data shown in FIG. 6(b) will be described.

[0064] In step 611, the CPU 202 requests the encrypted image data 400 from the data collection server 102, and proceeds to step 612 on the condition that the parameters of the inference device are not fixed.

[0065] In step 612, the CPU 202 trains the inference machine using the encrypted image data 400. At this time, the CPU 202 sets parameters for the previous layer 410 and the next layer 420.

[0066] In step 623, the CPU 202 inputs the encrypted image data 400. Then, in step 624, the output of the inference unit is Y1 (401), the training data is T (402), and the loss value is L1 (403).

[0067] When the learning of the encrypted image inference machine is completed in step 612, the process proceeds to step 613.

[0068] In step 613, the CPU 202 fixes the latter layer 420 of the trained encrypted image reasoner, and outputs the latter layers 450 and 614 of the encrypted image reasoner.

[0069] The flow of learning 612 using learning image data shown in Fig. 6(c) will be described. When this flow starts, the position of the layer where the parameter value is fixed in learning is specified.

[0070] The CPU 202 determines whether or not learning data consisting of learning image data (400, 430) and teacher data (T (402), T (432)) has been received in step 621. When the learning data has been received, the process proceeds to step 622.

[0071] In step 622, CPU 202 sets values ​​of parameters belonging to the unfixed layer. When transitioning from step 621 to step 622, CPU 202 sets initial values ​​to the parameters belonging to the unfixed layer. When transitioning from step 625 to step 622, CPU 202 sets values ​​adjusted based on the backpropagation algorithm to the parameters belonging to the unfixed layer, and proceeds to step 623.

[0072] In step 623, the CPU 202 inputs the learning image data (400, 430) to the inference unit, obtains the output (Y1 (401), Y2 (431)) of the inference unit, and proceeds to step 624.

[0073] In step 624, the CPU 202 inputs the output of the inference unit (Y1 (401), Y2 (431)) and the training data (T (402), T (432)) to the loss function (404, 434) to obtain loss values ​​(L1 (403), L2 (433)).

[0074] Next, in step 625, the CPU 202 determines whether the loss values ​​(L1 (403), L2 (433)) are less than a predetermined value. If the loss values ​​are equal to or greater than the predetermined value, the CPU 202 transitions to step 622 and performs learning again. When the loss values ​​become less than the predetermined value, the learning of the inference unit is completed.

[0075] As described above, in the first embodiment, a trained encrypted image inference machine is fixedly arranged at the rear of a non-encrypted image inference machine, and the non-encrypted image inference machine is trained to obtain a highly accurate non-encrypted image inference machine. Furthermore, it is considered that the psychological barrier for camera users to encrypt and send their captured photos to the cloud is lower than that for sending non-encrypted images. Therefore, it is possible to collect a large amount of encrypted images from camera users on the cloud. If more encrypted images than non-encrypted images can be used in training the inference machine, it is possible to create an inference machine with higher accuracy than when training only with non-encrypted images. Then, by training the trained encrypted image inference machine to correspond to the input of non-encrypted images, a similarly highly accurate non-encrypted image inference machine can be obtained.

[0076] Example 2 The learning method for an inference device according to the second embodiment of the present invention will now be described.

[0077] This embodiment is an example of learning for updating an inference device in the present invention.

[0078] The overall configuration of the learning system in this embodiment is the same as that in the first embodiment of the present invention, and is shown in FIG.

[0079] The hardware configuration of the learning system in this embodiment is the same as that in the first embodiment of the present invention, and is shown in FIG.

[0080] The configuration of the learning system in this embodiment is the same as that in the first embodiment of the present invention, and is shown in FIG.

[0081] The configuration of the learning unit of the learning system in this embodiment is the same as that in the first embodiment of the present invention, and is shown in FIG.

[0082] The concept of the training image data for the learning system in this embodiment is the same as that in the first embodiment of the present invention, and is shown in FIG.

[0083] The learning flow in this embodiment will be described with reference to FIG.

[0084] FIG. 7(a) shows a learning flow for simultaneously improving the accuracy of an encrypted image reasoner that inputs encrypted image data and performs inference, and an unencrypted image reasoner that inputs unencrypted image data and performs inference.

[0085] FIG. 7B is a diagram for explaining the flow of step 704 in FIG. 7A, where learning of the previous layer is performed using non-encrypted image data.

[0086] FIG. 7C is a diagram for explaining the flow of step 705 in FIG. 7A, in which the subsequent layer is trained using non-encrypted image data.

[0087] FIG. 7D is a diagram for explaining the flow of step 706 in FIG. 7A, in which the previous layer is trained using encrypted image data.

[0088] FIG. 7(e) is a diagram for explaining the flow of step 707 in FIG. 7(a) for learning the subsequent layer using encrypted image data.

[0089] First, a learning flow for simultaneously improving the accuracy of an inference device for encrypted images and an inference device for non-encrypted images shown in FIG. 7(a) will be described.

[0090] In step 701, the CPU 202 defines the structure of the encrypted image inference device and the non-encrypted image inference device. Specifically, the number of inputs and the number of outputs of the neural network constituting the inference device are defined. Next, a function having a weight parameter and a bias, for example, a perceptron, which is a model of a neural network, is defined. Next, the number of perceptrons constituting the layers is determined according to the above definition. Next, the type and number of layers constituting the neural network are defined and arranged according to the above definition. For example, from the second layer, which is the second layer counting from the input layer, convolution layers and pooling layers are alternately stacked three times, and the first to seventh layers are set as front layers. Then, the layer structure is defined such that the eighth to tenth layers are fully connected layers, the eleventh layer is an output layer, and the eighth to eleventh layers are set as rear layers. From the layer structure defined in step 701, the CPU 202 obtains a front layer 715 of the encrypted image inference device, and a front layer 716 and a rear layer 717 of the non-encrypted image inference device.

[0091] In step 702, the CPU 202 repeatedly performs one-round learning. The one-round learning is a single process that is the target of repetition in step 702. This process is a process in which learning 704 of the front layer of the inference device for non-encrypted images, learning 705 of the rear layer of the inference device for non-encrypted images, learning 706 of the front layer of the inference device for encrypted images, learning 707 of the rear layer of the inference device for encrypted images, and error determination 708 are performed once each in sequence. Learning 727 of the inference device for non-encrypted images is performed in steps 704 to 705, and learning 728 of the inference device for encrypted images is performed in steps 706 to 707. The termination condition 727 of the repeated execution of the one-round learning in step 702 can be, for example, "the loss value of each learning that constitutes the one-round learning 704 to 708 becomes equal to or less than a predetermined value" and "the upper limit of the number of repeated executions of the one-round learning". When the end condition for the repeated execution of the one-cycle learning is met, the learning of the inference unit is completed, and the CPU 202 outputs a trained encrypted image inference unit 725 and a trained encrypted image inference unit 726 in step 709. When the end condition for the repeated execution of the one-cycle learning is not met, the one-cycle learning is repeatedly executed as long as the number of repeated executions is equal to or less than the "upper limit value of the number of repeated executions of one-cycle learning", and when it is exceeded, an error is output and the process is terminated.

[0092] The iterative process of step 702 shown in FIG. 7(a) will be described in detail. The targets of the iterative learning in step 702 are the front layer 718 of the encrypted image inferencing device, the front layer 719 of the non-encrypted image inferencing device, and the back layer 720. When the process transitions from step 701 to step 702, the front layer 715 of the encrypted image inferencing device is substituted for the front layer 718 of the encrypted image inferencing device. Also, the front layer 716 of the non-encrypted image inferencing device is substituted for the front layer 719 of the non-encrypted image inferencing device, and the back layer 717 is substituted for the back layer 720. When the iterative process is in step 702, the front layer 723 of the encrypted image inferencing device is substituted for the front layer 718 of the encrypted image inferencing device. Also, the front layer 721 of the non-encrypted image inferencing device is substituted for the front layer 719 of the non-encrypted image inferencing device, and the back layer 724 is substituted for the back layer 720.

[0093] In step 703, the CPU 202 sets a one-round learning condition. The one-round learning condition is each of the learning end conditions in steps 704 to 707 and the allowable number of epochs until each learning end condition is satisfied. The learning end condition may include, for example, "when the loss value does not decrease for three consecutive epochs within 10 epochs from the start of learning in step 704." "When the loss value does not decrease for three consecutive epochs" is the "learning end condition in step 704," and "10 epochs" is the "allowable number of epochs in step 704." In steps 704 to 707, if the learning end condition is not satisfied even if the number of epochs reaches the allowable number of epochs set in step 703, the CPU 202 issues an error. At this time, the error determination result in step 708 is YES, and the process transitions to step 701 for setting the layer structure.

[0094] Next, in step 704, the CPU 202 inputs the fixed rear layer 720 and the fixed front layer 719 of the non-encrypted image reasoner, and trains the front layer using non-encrypted image data. It also fixes and outputs the front layer 721 of the non-encrypted image reasoner that has completed training.

[0095] Next, in step 705, the CPU 202 fixes the previous layer for the inference device that has completed the previous layer learning 704 using non-encrypted image data, and performs learning on the subsequent layer. Also, the CPU 202 fixes and outputs the subsequent layer 722 of the inference device for non-encrypted image that has completed learning.

[0096] Next, in step 706, the CPU 202 inputs the fixed rear layer 722 and the fixed front layer 718 of the encrypted image reasoner, and trains the front layer using the encrypted image data. It also fixes and outputs the front layer 723 of the encrypted image reasoner that has completed training.

[0097] Next, in step 707, the CPU 202 fixes the previous layer for the inference device that has completed learning 706 of the previous layer using encrypted image data, and performs learning of the subsequent layer. Also, the CPU 202 fixes and outputs the subsequent layer 724 of the encrypted image inference device that has completed learning.

[0098] The flow of learning 704 of the front-stage layer using non-encrypted image data shown in Fig. 7(b) will be described below. In this figure, the learning target is the front-stage layer of the non-encrypted image inference device.

[0099] In step 740, the CPU 202 places the input front-stage layer 719 of the reasoner for unencrypted images in the front-stage layer of the reasoner.

[0100] Next, in step 741, the weight parameters and biases in the front layer of the reasoner are unfixed so that they can be updated during learning.

[0101] Next, in step 742, the input back layer 720 is placed in the back layer of a reasoner for unencrypted images.

[0102] In step 743, the CPU 202 requests the unencrypted image data from the data collection server.

[0103] In step 612, under the condition that the previous layer is not fixed and the subsequent layer is fixed, learning of the previous layer of the non-encrypted image reasoner is performed using the non-encrypted image data as learning image data.

[0104] In step 744, the front layer of the non-encrypted image reasoner that has already been trained in step 612 is fixed and output. The output data is the front layer 721 of the non-encrypted image reasoner.

[0105] The flow of learning 705 of the subsequent layer using non-encrypted image data, shown in Fig. 7(c), will be described below. In this figure, the learning target is the subsequent layer of the non-encrypted image reasoner.

[0106] In step 750, the CPU 202 unlocks the rear layers of the non-encrypted image reasoner inherited from the previous step 704, allowing it to be updated during learning.

[0107] Next, in step 751, the front layer of the reasoner for unencrypted images is fixed so that it cannot be updated during training.

[0108] The CPU 202, in step 752, requests the unencrypted image data from the data collection server.

[0109] In step 612, under the condition that the previous layer is fixed and the subsequent layer is not fixed, learning of the subsequent layer of the non-encrypted image reasoner is performed using the non-encrypted image data as learning image data.

[0110] In step 753, the latter layer of the non-encrypted image reasoner that has already been trained in step 612 is fixed and output. The output data is the latter layer 722 of the non-encrypted image reasoner.

[0111] The flow of learning 706 of the front-stage layer using non-encrypted image data, shown in Fig. 7(d), will be described below. In this figure, the learning target is the front-stage layer of the inference device for encrypted images.

[0112] In step 760, CPU 202 places the input encrypted image reasoner front-stage layer 718 in the reasoner front-stage layer.

[0113] Next, in step 761, the weight parameters and biases in the front layer of the reasoner are unfixed so that they can be updated during learning.

[0114] Next, in step 762, the input back layer 720 is placed in the back layer of the encrypted image reasoner.

[0115] In step 763, the CPU 202 requests the encrypted image data from the data collection server.

[0116] In step 612, the front layer of the encrypted image reasoner is trained using the encrypted image data as training image data under the condition that the front layer is not fixed and the rear layer is fixed.

[0117] In step 764, the front layer of the encrypted image reasoner that has already been trained in step 612 is fixed and output. The output data is the front layer 723 of the encrypted image reasoner.

[0118] The flow of learning 707 of the subsequent layer using encrypted image data, shown in Fig. 7(e), will be described below. In this figure, the learning target is the subsequent layer of the encrypted image inference device.

[0119] In step 770, the CPU 202 releases the fixation of the weight parameters and biases of the subsequent layers of the encrypted image reasoner inherited from the previous step 706, so that they can be updated during learning.

[0120] Next, in step 771, the front layer of the encrypted image reasoner is fixed so that it cannot be updated during training.

[0121] The CPU 202, in step 772, requests the encrypted image data from the data collection server.

[0122] In step 612, the encrypted image data is used as training image data to train the subsequent layer of the encrypted image reasoner, with the previous layer being fixed and the weight parameters and biases of the subsequent layer not being fixed.

[0123] In step 773, the latter layer of the encrypted image reasoner that has already been trained in step 612 is fixed and output. The output data is the latter layer 724 of the encrypted image reasoner.

[0124] As described above, in the second embodiment, by repeatedly learning using encrypted image data and non-encrypted image data, a front layer of an inference device for encrypted images, a front layer of an inference device for non-encrypted images, and a common back layer are obtained. By connecting the front layer of the inference device for encrypted images to the common back layer, an inference device for encrypted images is obtained, and by connecting the front layer of the inference device for non-encrypted images to the common back layer, an inference device for non-encrypted images is obtained.

[0125] A condition for starting the flow 700 for improving the accuracy of the inference machine may be, for example, "a predetermined number of encrypted image data or non-encrypted image data has been newly accumulated in the data storage unit 314." In other words, by appropriately performing the learning described in this embodiment when the amount of learning data increases, the accuracy of the encrypted image inference machine and the non-encrypted image inference machine can be improved.

[0126] The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0127] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims

[Claim 1] A learning method for training an inference device using a neural network by using images and training data, comprising the steps of: The inference device is composed of a front layer including a plurality of layers and a rear layer, Fixing a subsequent layer of the first inference device trained using the first image and the first teacher data; The fixed subsequent layer is disposed in a subsequent layer of a second inference device that inputs a second image; A learning method, characterized in that an error resulting from the inference made by the second inference device when the second image is input is fed back to a previous layer of the second inference device for learning.

Images