Information processing device, information processing method

Abstract

To provide a learning technique to bring a feature vector output from an arithmetic section operating as a student model and a feature vector output from an arithmetic section operating as a teacher model closer to substantially the same even in highly difficult distillation.SOLUTION: An information processing apparatus determines a first difference vector between a first feature vector and a second feature vector extracted with a smaller amount of calculation. The information processing apparatus updates a parameter of a second arithmetic section based on a first changed difference vector obtained by changing a value of an element in the first difference vector. An amount of change from the value of an element in the first difference vector corresponding to a first element exceeding a threshold in the first feature vector to a value of an element of the first changed difference vector corresponding to the first element, is larger than an amount of change in a value from a value of an element in the first difference vector corresponding to a second element not exceeding the threshold in the first feature vector to a value of an element of the first changed difference vector corresponding to the second element.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to learning technology.

Background Art

[0002] [[ID=II]] In recent years, many feature extraction techniques have been proposed to highly process the images of objects shown in photographed images and extract useful information. Among them, in particular, feature extraction techniques for extracting the feature vectors of objects in images using a multi-layer neural network called a deep net (also referred to as a deep neural net, deep learning, or deep learning) have been actively studied.

[0003] Regarding the prosperity of the feature extraction technique using a deep net, as is well known, in recent years, a deep net learning method called distillation as disclosed in Patent Document 1 has attracted attention. Distillation is a method of learning a deep net (referred to as a student model) having a different network architecture using a learned deep net model (referred to as a teacher model). Generally, learning by distillation is often performed for the purpose of reducing the weight of the teacher model. Therefore, a network architecture lighter than the teacher model is often prepared for the student model. In distillation, the student model is learned using the feature vector output by the teacher model instead of the correct label. Therefore, in learning by distillation, a large amount of labeled learning images required for normal learning are not necessary. It is known that the knowledge of the teacher model can be propagated to the student model by such a distillation technique.

[0004] The student model learned by the distillation technique can output a feature vector substantially the same as that of the teacher model. Therefore, although the network architecture and the parameters associated therewith are different between the student model and the teacher model, when the same image is input to both models, substantially the same feature vector is output from both models.

[0005] Furthermore, Non-Patent Document 1 discloses techniques to improve the ease of learning by distillation, such as training the intermediate feature vectors generated by the teacher model during the process of calculating feature vectors to be approximately identical to the intermediate feature vectors generated by the student model during the process of calculating feature vectors, and research and development in this area is actively underway. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] US10289962 [Non-patent literature]

[0007] [Non-Patent Document 1] FITNETS: HINTS FOR THIN DEEP NETS Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta & Yoshua Bengio [Overview of the project] [Problems that the invention aims to solve]

[0008] However, when the difficulty of distillation increases, such as when distilling a student model with a significantly reduced number of neural network parameters (e.g., number of layers or neurons) compared to the teacher model, conventional methods may result in the feature vectors of the student model and the teacher model not being nearly identical.

[0009] This invention provides a learning technique that makes the feature vector output from the computation unit acting as the student model and the feature vector output from the computation unit acting as the teacher model nearly identical, even in highly difficult distillation processes. [Means for solving the problem]

[0010] One aspect of the present invention involves learning based on a first modified difference vector obtained by changing the element values ​​of the first difference vector between a first feature vector extracted in a teacher neural network model based on input data and a second feature vector extracted in a student neural network model based on the same input data with less computational effort than the teacher neural network model. inside A holding means for holding the parameters of the student neural network model, An update means for updating the parameters of the student neural network model based on the first change difference vector, The parameters are updated by the update means. Output means for outputting feature vectors for the image input to the aforementioned student neural network model. and Equipped with, The aforementioned first change difference vector is generated by multiplying the elements of the aforementioned first difference vector by weight coefficients, The values ​​of the elements of the first difference vector corresponding to the first element in the first feature vector that exceeds the threshold. Weight coefficient multiplied by This is the value of the element of the first difference vector that corresponds to the second element in the first feature vector that does not exceed the threshold. Weight coefficient multiplied by Larger than It is characterized by the following: [Effects of the Invention]

[0011] According to the present invention, even in highly complex distillation processes, the feature vector output from the computation unit operating as the student model and the feature vector output from the computation unit operating as the teacher model can be made nearly identical. [Brief explanation of the drawing]

[0012] [Figure 1] A block diagram showing an example of the hardware configuration of the information processing device 100. [Figure 2] A block diagram showing an example of the functional configuration of the information processing device 100. [Figure 3] A flowchart illustrating the operation of the information processing device 100. [Figure 4] A block diagram showing an example of the functional configuration of extraction unit 202 and extraction unit 203. [Figure 5]A diagram showing an example of an intermediate feature vector and a difference vector. [Figure 6] A block diagram showing an example of the functional configuration of the information processing apparatus 100. **Embodiments for Carrying Out the Invention**

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] [First Embodiment] In the present embodiment, an example of an information processing apparatus will be described in which a first difference vector between a first feature vector extracted by a first arithmetic unit based on input data and a second feature vector extracted by a second arithmetic unit based on the input data with a smaller arithmetic amount than the first arithmetic unit is obtained, a first modified difference vector in which the values of the elements in the first difference vector are changed is generated, and the parameters of the second arithmetic unit are updated based on the first modified difference vector. Here, the amount of change from the value of the element of the first difference vector corresponding to the first element exceeding the threshold value in the first feature vector to the value of the element of the first modified difference vector corresponding to the first element is greater than the amount of change from the value of the element of the first difference vector corresponding to the second element not exceeding the threshold value in the first feature vector to the value of the element of the first modified difference vector corresponding to the second element.

[0015] First, an example of the hardware configuration of the information processing apparatus 100 according to the present embodiment will be described using the block diagram of FIG. 1. The CPU 101 executes various processes using computer programs and data stored in the RAM 103 and the ROM 102. Thereby, the CPU 101 controls the operation of the entire information processing apparatus 100 and executes or controls various processes described as being performed by the information processing apparatus 100.

[0016] ROM 102 stores configuration data for the information processing device 100, computer programs and data related to the startup of the information processing device 100, computer programs and data related to the basic operation of the information processing device 100, and so on.

[0017] The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 and external storage device 104, and a work area used by the CPU 101 when executing various processes. In this way, the RAM 103 can provide various areas as appropriate.

[0018] The external storage device 104 is a large-capacity information storage device such as a hard disk drive. The external storage device 104 stores the OS (operating system), computer programs and data that cause the CPU 101 to execute or control various processes described as being performed by the information processing device 100. The computer programs and data stored in the external storage device 104 are loaded into the RAM 103 as appropriate according to the control of the CPU 101 and become the target of processing by the CPU 101.

[0019] The external storage device 104 may include optical discs such as flexible disks (FD) or Compact Discs (CDs) that can be attached to and detached from the information processing device 100, as well as magnetic or optical cards, IC cards, memory cards, etc.

[0020] The CPU 101, ROM 102, RAM 103, and external storage device 104 are all connected to the system bus 108. Furthermore, the system bus 108 is also connected to an input interface 105 and an output interface 106.

[0021] Input I / F 105 is connected to input unit 109. Input unit 109 is a user interface such as a keyboard, mouse, or touch panel screen, and allows the user to input various instructions to CPU 101 by operating it.

[0022] A monitor 110 is connected to the output I / F 106. The monitor 110 has an LCD screen or a touch panel screen and can display the processing results from the CPU 101 as images, text, etc. The monitor 110 may also be a projection device such as a projector that projects images and text.

[0023] Such information processing device 100 can be a computer device such as a PC (personal computer), WS (workstation), smartphone, or tablet terminal. The hardware configuration applicable to the information processing device 100 is not limited to the configuration shown in Figure 1, and can be modified or changed as appropriate.

[0024] Next, an example of the functional configuration of the information processing device 100 will be explained using the block diagram in Figure 2. In the following explanation, the functional unit in Figure 2 will be described as the main processing unit, but in reality, the functions of the functional unit are realized by the CPU 101 executing a computer program that causes the CPU 101 to execute or control the functions of the functional unit. Such a computer program is stored in the external storage device 104, loaded into the RAM 103 as appropriate according to the control of the CPU 101, and executed by the CPU 101. Note that one or more of the functional units in Figure 2 may be implemented in hardware.

[0025] The data acquisition unit 201 loads (acquires) the image 301 stored in the external storage device 104 into the RAM 103. Note that the image 301 is just one example of input data; text data, audio data, etc., may also be used as input data.

[0026] The extraction unit 202 inputs the image 301 into a deep network such as a Convolutional Neutral Network (CNN), which is a training model, and obtains a first feature vector 302 extracted from the image 301 in the intermediate layer of the deep network. In this embodiment, the first feature vector 302 is assumed to be a 512-dimensional vector. The 512-dimensional first feature vector 302 is intended for use, for example, in image classification systems and facial recognition systems.

[0027] The parameters (such as weight coefficients) of the deep network used by the extraction unit 202 are parameters that have already been acquired through training, and these parameters are not changed during the training process of this embodiment, which will be described below.

[0028] It should be noted that DeepNet is an example of a hierarchical neural network, and the extraction unit 202 in this embodiment may also use other types of hierarchical neural networks to obtain the first feature vector 302 from the image 301.

[0029] The extraction unit 203 inputs the image 301 into a deep network (referred to as Deep Network B) that requires less computation than the deep network (referred to as Deep Network A) used by the extraction unit 202, and obtains a second feature vector 303 (512 dimensions) extracted from the image 301 in the intermediate layer of Deep Network B.

[0030] Deepnet B is a student model, for example, a deepnet with fewer parameters than deepnet A (for example, a deepnet with fewer hidden layers than deepnet A, or a deepnet with fewer neurons than deepnet A).

[0031] In this embodiment, the activation function in the deep network used by the extraction unit 202 and the deep network used by the extraction unit 203 is assumed to be ReLU (Rectified Linear Unit), but the following explanation is not limited to this. Furthermore, a 1024-dimensional vector corresponding to image 301 is assumed to be output from the output layer of the deep network used by the extraction unit 202 and the deep network used by the extraction unit 203.

[0032] The difference acquisition unit 204 calculates the difference value 306 based on the difference vector between the first feature vector 302 acquired by the extraction unit 202 and the second feature vector 303 acquired by the extraction unit 203.

[0033] The acquisition unit 204a generates a difference vector 304 between the first feature vector 302 acquired by the extraction unit 202 and the second feature vector 303 acquired by the extraction unit 203. For example, if the acquisition unit 204a takes the i-th element (1≦i≦512) of the first feature vector 302 as Ai and the i-th element of the second feature vector 303 as Bi, then the value of the i-th element Ci of the difference vector 304 is (Ai-Bi). 2 This is how to find the value of element Z. In this embodiment, the value of element Z may also be referred to as Z. If the non-negative difference between Ai and Bi is to be found as the i-th element Ci of the difference vector 304, then other methods may be used to find the i-th element Ci of the difference vector 304.

[0034] The function application unit 204b identifies elements Aj (1 ≤ j ≤ 5¹²) in the first feature vector 302 that have a value exceeding the threshold TH1, and generates a difference vector 305 by increasing the value of element Cj in the difference vector 304 corresponding to element Aj. Hereafter, the set of elements Cj in the difference vector 304 will be referred to as the "function application region". In this embodiment, the threshold TH1 is set to 0.

[0035] For example, the function application unit 204b, if Dj is the j-th element in the difference vector 305, applies the function f to the element Cj of the difference vector 304 as shown in (Equation 1) below to find Dj.

[0036] Dj=f(Cj)=αxCj … (Formula 1) Here, α is a weight value with a real value of 1 or greater, and function f is a function for finding Dj by increasing the value of element Cj according to the weight value α. Note that the function applied to element Cj by function application unit 204b is not limited to function f shown in (Equation 1), but may be any other linear function or nonlinear function as long as it is a function for finding Dj by increasing the value of element Cj. Furthermore, it is not limited to using a function as long as the same objective can be achieved.

[0037] In other words, any method for generating the difference vector 305 is acceptable as long as the following condition is met: "The change in value from the element of the difference vector 304 corresponding to the first element that exceeds the threshold TH1 in the first feature vector 302 to the element of the difference vector 305 corresponding to the first element is greater than the change in value from the element of the difference vector 304 corresponding to the second element that does not exceed the threshold TH1 in the first feature vector 302 to the element of the difference vector 305 corresponding to the second element."

[0038] For example, the function application unit 204b may identify an element A'k (1≦k≦512) in the first feature vector 302 that does not exceed the threshold TH1, and generate a difference vector 305 by decreasing the value of element C'k of the difference vector 304 corresponding to element A'k. For example, the function application unit 204b obtains the k-th element Dk in the difference vector 305 by applying the function f' shown in (Equation 1-1) below to element C'k of the difference vector 304.

[0039] Dk=f'(C'k)=βxC'k … (Formula 1-1) Here, β is a weight value that has a real value satisfying 0 < β < 1. In this case, the function application unit 204b may or may not apply (Equation 1). The following table shows examples of the first feature vector 302, the second feature vector 303, the difference vector 304, and the difference vector 305.

[0040] [Table 1]

[0041] Here, the weight value α = 64 is used. For example, among the elements of the first feature vector 302 (from the 1st element (the element of the 1st dimension) to the 512th element (the element of the 512th dimension)) as specified in the table, the elements whose values ​​exceed the threshold TH1 = 0 are the elements of the 2nd dimension, the 3rd dimension, and the 510th dimension. Therefore, the elements belonging to the function application domain are the elements of the 2nd dimension, the 3rd dimension, and the 510th dimension of the difference vector 304, and the vector obtained by multiplying the values ​​of these elements by the weight value α = 64 is the difference vector 305.

[0042] The calculation unit 204c calculates the difference value 306 by dividing the sum of the values ​​of all elements in the difference vector 305 by 1024, which is the number of dimensions of the feature vector output from the output layer of the deep network described above. Note that the method for calculating the difference value 306 from the element values ​​in the difference vector 305 is not limited to a specific method. For example, the calculation unit 204c may calculate the difference value 306 by summing the values ​​of all elements in the difference vector 305, or it may calculate the difference value 306 by summing the values ​​of elements in the difference vector 305 that are above a threshold, or by dividing that sum by 1024.

[0043] The update unit 205 calculates the "deep network parameters 307 used by the extraction unit 203" using backpropagation so that the difference value 306 becomes smaller, and updates the currently set "deep network parameters used by the extraction unit 203" to parameters 307. As a result, the extraction unit 203 inputs the next input image into the deep network B reconstructed according to parameters 307, and obtains a second feature vector 303 (512 dimensions) extracted from the image in the intermediate layer of the deep network B. The deep network B reconstructed according to parameters 307 has changed weight coefficients and other properties compared to the deep network B before reconstruction.

[0044] The augmentation unit 206 increases the amount by which it increases the value of element Cj in the difference vector 304 that corresponds to element Aj in the first feature vector 302 that has a value exceeding the threshold TH1. In the example above, this would be done by making the weight value α larger or the weight value β smaller. Generally, the augmentation unit 206 updates its settings so that the amount of increase in element values ​​from the difference vector 304 to the difference vector 305 in the function application domain increases. Alternatively, the augmentation unit 206 updates its settings so that the amount of decrease in element values ​​from the difference vector 304 to the difference vector 305 in the non-function application domain decreases.

[0045] The increase in weight value α by the augmentation unit 206 is performed when the number of elements belonging to the function application domain no longer decreases even as the number of training iterations increases. When ReLU is used as the activation function for a deep network, all outputs corresponding to inputs with values ​​less than or equal to 0 are 0. Therefore, if the learning rate during parameter updates by backpropagation is small, the output will remain 0, making it easy to fall into a local minimum. On the other hand, if the learning rate is too large, the fluctuations of the parameters contributing to elements of the second feature vector 303 that are not 0 will also increase, making proper learning impossible. For this reason, the initial value of weight value α is set to 1 at the start of training, and the parameters of the extraction unit 203 that output elements of the first feature vector 302, which can be reproduced relatively easily, are obtained first.

[0046] By increasing the weight value α at the aforementioned timing, the difference value corresponding to the function application domain is increased. When the difference value increases due to the weight value α, the learning process promotes the learning of parameters that contribute to the function application domain. By sequentially increasing the weight value α in this procedure, it becomes possible to ultimately obtain appropriate parameters for the extraction unit 203.

[0047] The timing for updating the settings so that the increase amount by the increase unit 206 increases is not limited to the timing described above; for example, it may be determined by the increase or decrease in the difference value 306 (the same applies to the timing for updating the settings so that the decrease amount by the increase unit 206 decreases). In other words, the timing for updating may be set as when the change in the difference value 306 obtained this time from the previous time is less than a threshold. Alternatively, the settings may be updated periodically or irregularly depending on the number of learning iterations (number of trials) or the elapsed time since the start of learning.

[0048] By repeating the above processes (data acquisition unit 201, extraction unit 202, extraction unit 203, acquisition unit 204a, function application unit 204b, calculation unit 204c, update unit 205, and increase unit 206), the "deep network used by the extraction unit 203" is trained.

[0049] The determination unit 207 determines whether the learning termination conditions have been met. The learning termination conditions are not limited to specific conditions. For example, the determination unit 207 determines that the learning termination conditions have been met when the user inputs a learning termination instruction by operating the input unit 109. Also, for example, the determination unit 207 determines that the learning termination conditions have been met when the number of learning repetitions exceeds a specified number, or when the elapsed time from the start of learning exceeds a specified time.

[0050] Next, the operation of the information processing device 100 described above will be explained according to the flowchart in Figure 3. Since the details of the processing in each step are as described above, a brief explanation will be given here.

[0051] In step S401, the data acquisition unit 201 loads (acquires) the image 301 stored in the external storage device 104 into the RAM 103. In step S402, the extraction unit 202 inputs the image 301 acquired in step S401 into the deep network A, and the intermediate layer of the deep network A acquires the first feature vector 302 extracted from the image 301.

[0052] In step S403, the extraction unit 203 inputs the image 301 acquired in step S401 into the deep network B, and obtains a second feature vector 303 extracted from the image 301 in the intermediate layer of the deep network B.

[0053] In step S404, the acquisition unit 204a generates a difference vector 304 between the first feature vector 302 acquired in step S402 and the second feature vector 303 acquired in step S403.

[0054] In step S405, the function application unit 204b identifies an element Aj in the first feature vector 302 that has a value exceeding the threshold TH1, and generates a difference vector 305 by increasing the value of element Cj in the difference vector 304 corresponding to element Aj.

[0055] In step S406, the calculation unit 204c obtains the difference value 306 by dividing the sum of the values ​​of all elements of the difference vector 305 generated in step S405 by 1024.

[0056] In step S407, the update unit 205 calculates the "deep net parameters 307 used by the extraction unit 203" using the backpropagation method such that the difference value 306 obtained in step S406 becomes smaller. The update unit 205 then updates the currently set "deep net parameters used by the extraction unit 203" to parameters 307.

[0057] In step S408, the incrementing unit 206 determines whether it is time to update the weight value α. If it is time to update the weight value α, the process proceeds to step S409; otherwise, the process proceeds to step S410.

[0058] For example, if the condition is met that the number of elements belonging to the function application domain does not change continuously throughout 100 training iterations, it is determined that it is time to update the weight value α. If this condition is not met, it is determined that it is not time to update the weight value α.

[0059] In step S409, the increment unit 206 increases the weight value α. Here, the initial value of the weight value α is set to 1, and the increment of the weight value α to be increased in one step is set to 64. In step S410, the determination unit 207 determines whether or not the learning termination condition has been met. If the learning termination condition is met as a result of this determination, the process according to the flowchart in Figure 3 is terminated; otherwise, the process proceeds to step S401.

[0060] The CPU 101 may save the "deep network parameters 307 used by the extraction unit 203" obtained through the learning process according to the flowchart in Figure 3 to the external storage device 104, or it may transmit them to an external device via the network. Furthermore, the CPU 101 may display the "deep network parameters 307 used by the extraction unit 203" on the monitor 110 as an image or text. Thus, the output destination and output format of the "deep network parameters 307 used by the extraction unit 203" are not limited to a specific destination or output format.

[0061] Thus, according to this embodiment, weight values ​​are assigned to the difference in the function application domain during distillation, and the weight values ​​are increased according to the progress of learning. As a result, even in distillation of high difficulty, which was difficult to achieve with conventional methods, it becomes possible to make the feature vectors of the student model and the teacher model nearly identical.

[0062] [Second Embodiment] The differences from the first embodiment will be described below, and unless otherwise specified, it will be assumed that it is the same as the first embodiment. An example of the functional configuration of the extraction unit 202 and the extraction unit 203 according to this embodiment will be explained using the block diagram in Figure 4.

[0063] The extraction unit 202 according to this embodiment includes a first half extraction unit 501 and a first second half extraction unit 502. The first half extraction unit 501 inputs the input image 301 to the deep network A and obtains a first intermediate feature vector 505 extracted from the image 301 at the intermediate layer A of the deep network A. The intermediate layer A is an intermediate layer in the deep network A between the input layer and the "intermediate layer that outputs the first feature vector 302". The first second half extraction unit 502 generates a first feature vector 302 by performing calculations in each layer from the intermediate layer A onward using the first intermediate feature vector 505 as input.

[0064] The extraction unit 203 according to this embodiment includes a second first-half extraction unit 503 and a second second-half extraction unit 504. The second first-half extraction unit 503 inputs the input image 301 to the deep network B and obtains a second intermediate feature vector 506 extracted from the image 301 in the intermediate layer B of the deep network B. The intermediate layer B is an intermediate layer in the deep network B between the input layer and the "intermediate layer that outputs the second feature vector 303". The second second-half extraction unit 504 generates a second feature vector 303 by performing calculations in each layer from the intermediate layer B onward using the second intermediate feature vector 506 as input.

[0065] The first feature vector 302 is obtained from a trained deep network, and therefore its values ​​are close to those of the ground truth labels. Consequently, if the values ​​of most elements in the ground truth label vector are zero, then the first feature vector 302 will also have many elements with zero values. In this embodiment, if the threshold TH1 is set to 0, there may be cases where almost no elements belong to the function application region.

[0066] On the other hand, the first intermediate feature vector 505 is a vector obtained by acquiring various features from image 301 in order to obtain the first feature vector 302 so that it has a value close to the correct label, and many of the values ​​of the elements of the vector are non-zero. Therefore, many of the elements of the first intermediate feature vector 505 belong to the function application domain, and the above learning can be performed more effectively by performing distillation using the intermediate feature vector.

[0067] Figure 5 will be used to explain intermediate feature vectors and difference vectors. Figure 5 shows an example of a 3D intermediate feature vector with nine 2D data points. The black-filled areas indicate a value of 0, and the white-filled areas indicate a value that is not zero.

[0068] Figure 5(a) shows an example of the first intermediate feature vector 505, where the intermediate feature vector from the trained deep network (extractor 202) has acquired various features of image 301, and therefore many of them are non-zero.

[0069] Figure 5(b) shows an example of the second intermediate feature vector 506 from the deep network (extraction unit 203) during training, where all three 2D data points have a value of zero, and the other 2D data points have a value equivalent to the first intermediate feature vector 505.

[0070] Figure 5(c) shows an example of the difference vector between the first intermediate feature vector 505 and the second intermediate feature vector 506, where differences occur across three 2D data points. Since all regions where differences occur are function application regions, increasing the weight value α can accelerate the learning of parameters contributing to the three 2D data points.

[0071] An example of the functional configuration of the information processing device 100 according to this embodiment will be explained using the block diagram in Figure 6. The extraction unit 202 obtains a first feature vector 302 and a first intermediate feature vector 505 from the image 301. The extraction unit 203 obtains a second feature vector 303 and a second intermediate feature vector 506 from the image 301.

[0072] The difference acquisition unit 204 acquires a difference value 701 from the first feature vector 302 and the second feature vector 303 in the same manner as in the first embodiment, and acquires a difference value 702 from the first intermediate feature vector 505 and the second intermediate feature vector 506 in the same manner as in the first embodiment. There are various processes for calculating the difference value from two vectors, as explained in the first embodiment. Therefore, the process for calculating the difference value 701 and the process for calculating the difference value 702 may be the same process, or they may be different processes, or even if they are the same process, the threshold and weight value α may be changed.

[0073] The integration unit 703 calculates the sum of the difference value 701 and the difference value 702 as the difference value 306. Thereafter, the same process as in the first embodiment is performed to obtain the parameter 307, and the parameter of the extraction unit 203 is updated with the obtained parameter 307.

[0074] Furthermore, the timing of updating the threshold and weight values ​​α used to calculate the difference value 701, and the timing of updating the threshold and weight values ​​α used to calculate the difference value 702, may be the same or different. Also, the amount of increase in weight values ​​α used to calculate the difference value 701, and the amount of increase in weight values ​​α used to calculate the difference value 702, may be the same or different.

[0075] In this embodiment, as shown in the flowchart of Figure 3, in step S402, the extraction unit 202 obtains a first feature vector 302 and a first intermediate feature vector 505 from the image 301. Then, in step S403, the extraction unit 203 obtains a second feature vector 303 and a second intermediate feature vector 506 from the image 301.

[0076] Then, in steps S404 to S406, the difference acquisition unit 204 acquires a difference value 701 from the first feature vector 302 and the second feature vector 303, similar to the first embodiment. The difference acquisition unit 204 also acquires a difference value 702 from the first intermediate feature vector 505 and the second intermediate feature vector 506, using the same processing as in steps S404 to S406. Then, in step S406, the integration unit 703 calculates the sum of the difference value 701 and the difference value 702 as the difference value 306. The processing in the other steps is the same as in the first embodiment.

[0077] Thus, according to this embodiment, even for highly complex distillation that was difficult to achieve with conventional methods, distillation using intermediate feature vectors makes it possible to make the feature vectors of the student model and the teacher model nearly identical with higher accuracy.

[0078] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0079] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0080] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0081] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0082] 201: Data acquisition unit 202: Extraction unit 203: Extraction unit 204: Difference acquisition unit 204a: Acquisition unit 204b: Function application unit 204c: Calculation unit 205: Update unit 206: Increase unit 207: Judgment unit

Claims

1. A holding means for holding parameters of the student neural network model under training based on a first modified difference vector obtained by changing the values ​​of elements in the first difference vector between a first feature vector extracted based on input data in the teacher neural network model and a second feature vector extracted based on the input data in the student neural network model with less computational effort than the teacher neural network model, An update means for updating the parameters of the student neural network model based on the first change difference vector, An output means that outputs a feature vector for the image input to the student neural network model whose parameters have been updated by the update means. Equipped with, The first change difference vector is generated by multiplying the elements of the first difference vector by weight coefficients, The weight coefficient multiplied by the value of the element in the first difference vector corresponding to the first element in the first feature vector that exceeds the threshold is greater than the weight coefficient multiplied by the value of the element in the first difference vector corresponding to the second element in the first feature vector that does not exceed the threshold. An information processing device characterized by the following:

2. The first change difference vector is, The information processing apparatus according to claim 1, characterized in that it is generated by increasing the value of the element of the first difference vector corresponding to the first element in the first feature vector that exceeds a threshold.

3. The first change difference vector is, The information processing apparatus according to claim 1 or 2, characterized in that it is generated by reducing the value of the element of the first difference vector corresponding to the second element in the first feature vector that does not exceed a threshold.

4. moreover, The information processing apparatus according to claim 2, further comprising an increasing means for increasing the amount by which the value of the element of the first difference vector corresponding to the first element is increased.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the first feature vector is a feature vector extracted from the input data in the intermediate layer of the training neural network model after inputting the input data into the training neural network model.

6. The information processing apparatus according to claim 5, characterized in that the activation function of the aforementioned teacher neural network model is ReLU (Rectified Linear Unit).

7. The information processing apparatus according to any one of claims 1 to 6, characterized in that the second feature vector is a feature vector extracted from the input data in the intermediate layer of the student neural network model, which has fewer parameters than the teacher neural network model, by inputting the input data into the student neural network model.

8. The information processing apparatus according to claim 7, characterized in that the activation function of the student neural network model is ReLU (Rectified Linear Unit).

9. The storage means further stores the parameters of the student neural network model learned based on the first modified difference vector, obtained by changing the values ​​of the elements in the second difference vector of the first intermediate feature vector extracted in the teacher neural network model based on the input data and the second intermediate feature vector extracted in the student neural network model based on the input data with less computational effort than the teacher neural network model, and the first modified difference vector. The change in value from the element of the second difference vector corresponding to the third element that exceeds the threshold in the first intermediate feature vector to the value of the element of the second modification difference vector corresponding to the third element is greater than the change in value from the element of the second difference vector corresponding to the fourth element that does not exceed the threshold in the first intermediate feature vector to the value of the element of the second modification difference vector corresponding to the fourth element. The information processing apparatus according to any one of claims 1 to 4.

10. An information processing method performed by an information processing device, An update step in which the parameters of the student neural network model are updated based on a first modified difference vector obtained by changing the values ​​of the elements in the first difference vector between a first feature vector extracted in the teacher neural network model based on the input data and a second feature vector extracted in the student neural network model based on the input data with less computational effort than the teacher neural network model, An output step which outputs a feature vector for the image input to the student neural network model whose parameters were updated in the update step. Equipped with, The first change difference vector is generated by multiplying the elements of the first difference vector by weight coefficients, The weight coefficient multiplied by the value of the element in the first difference vector corresponding to the first element in the first feature vector that exceeds the threshold is greater than the weight coefficient multiplied by the value of the element in the first difference vector corresponding to the second element in the first feature vector that does not exceed the threshold. An information processing method characterized by the following:

11. A computer program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 9.

Images