Information processing apparatus and information processing method

Abstract

To provide a learning technique to bring a feature vector output from an arithmetic section operating as a student model and a feature vector output from an arithmetic section operating as a teacher model closer to substantially the same even in highly difficult distillation.SOLUTION: An information processing apparatus determines a first difference vector between a first feature vector and a second feature vector extracted with a smaller amount of calculation. The information processing apparatus updates a parameter of a second arithmetic section based on a first changed difference vector obtained by changing a value of an element in the first difference vector. An amount of change from the value of an element in the first difference vector corresponding to a first element exceeding a threshold in the first feature vector to a value of an element of the first changed difference vector corresponding to the first element, is larger than an amount of change in a value from a value of an element in the first difference vector corresponding to a second element not exceeding the threshold in the first feature vector to a value of an element of the first changed difference vector corresponding to the second element.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to learning techniques. [Background technology]

[0002] In recent years, many feature extraction techniques have been proposed to extract useful information from images of objects captured in captured images through advanced processing. In particular, research has been actively conducted on feature extraction techniques that use multi-layered neural networks called deep nets (also called deep neural networks, deep learning, and deep learning) to extract feature vectors of objects in images.

[0003] It is well known that feature extraction techniques using deep nets are on the rise, but in recent years, a deep net learning technique called distillation, as described in Patent Document 1, has been attracting attention. Distillation is a technique in which a deep net (called a student model) having a different network architecture is trained using a trained deep net model (called a teacher model). In general, learning by distillation is often performed with the aim of reducing the weight of the teacher model, and therefore a network architecture that is lighter than that of the teacher model is often prepared for the student model. In distillation, the student model is trained using the feature vector output by the teacher model instead of the correct answer label. Therefore, learning by distillation does not require a large number of labeled learning images that are required for normal learning. It is known that such distillation techniques can propagate the knowledge of the teacher model to the student model.

[0004] A student model trained by distillation technology can output a feature vector that is approximately the same as that of a teacher model. Therefore, although the network architecture and associated parameters of the student model and teacher model are different, when the same image is input to both models, the two models output approximately the same feature vector.

[0005] In addition, Non-Patent Document 1 discloses a technique for improving the ease of learning by distillation by learning so that the intermediate feature vectors generated in the process of calculating the feature vectors by the teacher model and the intermediate feature vectors generated in the process of calculating the feature vectors by the student model are approximately identical, and research and development of this technique is being actively carried out. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US10289962 [Non-patent literature]

[0007] [Non-Patent Document 1] FITNETS: HINTS FOR THIN DEEP NETS Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta & Yoshua Bengio Summary of the Invention [Problem to be solved by the invention]

[0008] However, when the difficulty of distillation increases, for example when distilling a student model in which the number of neural network parameters (such as the number of layers or neurons) is significantly reduced from that of the teacher model, the feature vectors of the student model and the teacher model may not be approximately identical using conventional methods.

[0009] The present invention provides a learning technique for making the feature vector output from a calculation unit operating as a student model and the feature vector output from a calculation unit operating as a teacher model approximately identical, even in highly difficult distillation. [Means for solving the problem]

[0010] One aspect of the present invention comprises an acquisition means for calculating a first difference vector between a first feature vector extracted by a first calculation unit based on input data and a second feature vector extracted by a second calculation unit based on the input data with less calculation effort than the first calculation unit, and an update means for generating a first modified difference vector by changing values ​​of elements in the first difference vector and updating parameters of the second calculation unit based on the first modified difference vector, wherein an amount of change from a value of an element of the first difference vector corresponding to a first element in the first feature vector that exceeds a threshold to a value of an element of the first modified difference vector corresponding to the first element is greater than an amount of change from a value of an element of the first difference vector corresponding to a second element in the first feature vector that does not exceed the threshold to a value of an element of the first modified difference vector corresponding to the second element. Effect of the Invention

[0011] According to the present invention, even in highly difficult distillation, it is possible to make the feature vector output from the calculation unit operating as a student model and the feature vector output from the calculation unit operating as a teacher model approximately identical. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device 100. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device 100. [Diagram 3] 4 is a flowchart showing the operation of the information processing device 100. [Figure 4] FIG. 2 is a block diagram showing an example of the functional configuration of an extraction unit 202 and an extraction unit 203. [Diagram 5] FIG. 13 is a diagram showing an example of an intermediate feature vector and a difference vector. [Figure 6] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device 100. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] [First embodiment] In this embodiment, an example of an information processing device is described that obtains a first difference vector between a first feature vector extracted by a first calculation unit based on input data and a second feature vector extracted by a second calculation unit based on the input data with a calculation amount less than that of the first calculation unit, generates a first modified difference vector by modifying values ​​of elements in the first difference vector, and updates parameters of the second calculation unit based on the first modified difference vector. Here, the amount of change from the value of an element of the first difference vector corresponding to a first element that exceeds a threshold in the first feature vector to the value of an element of the first modified difference vector corresponding to the first element is greater than the amount of change from the value of an element of the first difference vector corresponding to a second element that does not exceed the threshold in the first feature vector to the value of an element of the first modified difference vector corresponding to the second element.

[0015] First, a hardware configuration example of an information processing device 100 according to this embodiment will be described with reference to the block diagram of Fig. 1. A CPU 101 executes various processes using computer programs and data stored in a RAM 103 and a ROM 102. As a result, the CPU 101 controls the operation of the entire information processing device 100, and executes or controls various processes that will be described as being performed by the information processing device 100.

[0016] The ROM 102 stores setting data for the information processing device 100, computer programs and data related to the startup of the information processing device 100, computer programs and data related to the basic operations of the information processing device 100, and the like.

[0017] The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 or the external storage device 104, and a work area used when the CPU 101 executes various processes. In this way, the RAM 103 can provide various areas as needed.

[0018] The external storage device 104 is a large-capacity information storage device such as a hard disk drive device. The external storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as being performed by the information processing device 100, and the like. The computer programs and data stored in the external storage device 104 are loaded into the RAM 103 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101.

[0019] The external storage device 104 may include an optical disk such as a flexible disk (FD) or a compact disk (CD) that is detachable from the information processing device 100, a magnetic or optical card, an IC card, a memory card, or the like.

[0020] The CPU 101, the ROM 102, the RAM 103, and the external storage device 104 are all connected to a system bus 108. In addition, an input I / F 105 and an output I / F 106 are further connected to the system bus 108.

[0021] An input unit 109 is connected to the input I / F 105. The input unit 109 is a user interface such as a keyboard, a mouse, a touch panel screen, etc., and can input various instructions to the CPU 101 by a user operating the input unit 109.

[0022] A monitor 110 is connected to the output I / F 106. The monitor 110 has a liquid crystal screen or a touch panel screen, and can display the processing results by the CPU 101 as images, characters, etc. The monitor 110 may be a projection device such as a projector that projects images and characters.

[0023] Computer devices such as a PC (personal computer), a WS (workstation), a smartphone, a tablet terminal device, etc. can be applied to such information processing device 100. Note that the hardware configuration applicable to the information processing device 100 is not limited to the configuration shown in FIG. 1 and can be modified / changed as appropriate.

[0024] Next, an example of the functional configuration of the information processing device 100 will be described with reference to the block diagram of Fig. 2. In the following, the functional units of Fig. 2 will be described as the main processing units, but in reality, the functions of the functional units are realized by the CPU 101 executing a computer program for causing the CPU 101 to execute or control the functions of the functional units. Such computer programs are stored in the external storage device 104, and are appropriately loaded into the RAM 103 under the control of the CPU 101 and executed by the CPU 101. Note that one or more of the functional units of Fig. 2 may be implemented by hardware.

[0025] The data acquisition unit 201 loads (acquires) an image 301 stored in the external storage device 104 into the RAM 103. Note that the image 301 is an example of input data, and text data, audio data, etc. may also be used as the input data.

[0026] The extraction unit 202 inputs an image 301 to a deep net such as a convolutional neural network (CNN) that is a teacher model, and acquires a first feature vector 302 extracted from the image 301 in an intermediate layer of the deep net. In this embodiment, the first feature vector 302 is a 512-dimensional vector. The 512-dimensional first feature vector 302 is intended to be used in, for example, an image classification system or a face recognition system.

[0027] The deep net parameters (weighting coefficients, etc.) used by the extraction unit 202 are parameters that have already been acquired by learning, and these parameters are not changed during the learning process of this embodiment described below.

[0028] It should be noted that the deep net is an example of a hierarchical neural network, and the extraction unit 202 according to this embodiment may acquire the first feature vector 302 from the image 301 using another type of hierarchical neural network.

[0029] The extraction unit 203 inputs the image 301 to a deep net (called deep net B) that requires less computational effort than the deep net (called deep net A) used by the extraction unit 202, and obtains a second feature vector 303 (512 dimensions) extracted from the image 301 in the intermediate layer of deep net B.

[0030] Deep net B is a student model, for example, a deep net with fewer parameters than deep net A (e.g., a deep net with fewer hidden layers than deep net A, or a deep net with fewer neurons than deep net A).

[0031] In this embodiment, the activation function in the deep net used by the extraction unit 202 and the deep net used by the extraction unit 203 is ReLU (Rectified Linear Unit), but the following description is not limited to this. Also, it is assumed that a 1024-dimensional vector corresponding to the image 301 is output from the output layer of the deep net used by the extraction unit 202 and the deep net used by the extraction unit 203.

[0032] The difference acquisition unit 204 obtains a difference value 306 based on a difference vector between the first feature vector 302 acquired by the extraction unit 202 and the second feature vector 303 acquired by the extraction unit 203 .

[0033] The acquisition unit 204a generates a difference vector 304 between the first feature vector 302 acquired by the extraction unit 202 and the second feature vector 303 acquired by the extraction unit 203. For example, assuming that the i-th (1≦i≦512) element in the first feature vector 302 is Ai and the i-th element in the second feature vector 303 is Bi, the acquisition unit 204a calculates (Ai-Bi) as the value of the i-th element Ci of the difference vector 304.2 In this embodiment, the value of element Z may also be referred to as Z. Note that, as long as a non-negative difference between Ai and Bi is obtained as the i-th element Ci of difference vector 304, the i-th element Ci of difference vector 304 may be obtained using another method.

[0034] The function application unit 204b identifies an element Aj (1≦j≦512) having a value exceeding a threshold TH1 among the elements of the first feature vector 302, and generates a difference vector 305 by increasing the value of an element Cj of the difference vector 304 corresponding to the element Aj. Hereinafter, the set of elements Cj in the difference vector 304 is referred to as a "function application domain." In this embodiment, the threshold TH1 is set to 0.

[0035] For example, if the j-th element in the difference vector 305 is Dj, the function application unit 204b obtains Dj by applying a function f to an element Cj of the difference vector 304 as shown in the following (Equation 1).

[0036] Dj=f(Cj)=αxCj … (Formula 1) Here, α is a weighting value having a real value of 1 or more, and the function f is a function for obtaining Dj by increasing the value of the element Cj according to the weighting value α. Note that the function applied to the element Cj by the function application unit 204b is not limited to the function f shown in (Equation 1) as long as it is a function for obtaining Dj by increasing the value of the element Cj, but other linear functions or nonlinear functions may be used. Also, as long as the same purpose can be achieved, it is not limited to using a function.

[0037] In other words, any method of generating the difference vector 305 may be applied as long as the following condition is satisfied: "the amount of change from the value of the element of the difference vector 304 corresponding to a first element in the first feature vector 302 that exceeds the threshold TH1 to the value of the element of the difference vector 305 corresponding to the first element is greater than the amount of change from the value of the element of the difference vector 304 corresponding to a second element in the first feature vector 302 that does not exceed the threshold TH1 to the value of the element of the difference vector 305 corresponding to the second element."

[0038] For example, the function application unit 204b may identify an element A'k (1≦k≦512) having a value not exceeding a threshold TH1 among the elements of the first feature vector 302, and generate a difference vector 305 by reducing the value of an element C'k of the difference vector 304 corresponding to the element A'k. For example, the function application unit 204b applies a function f' to the element C'k of the difference vector 304 as shown in the following (Equation 1-1) to obtain the k-th element Dk in the difference vector 305.

[0039] Dk=f'(C'k)=βxC'k … (Formula 1-1) Here, β is a weighting value having a real value satisfying 0<β<1. In this case, the function application unit 204b may or may not further apply (Equation 1). The following table shows examples of a first feature vector 302, a second feature vector 303, a difference vector 304, and a difference vector 305.

[0040] [Table 1]

[0041] Here, the weight value α=64. For example, among the elements of first feature vector 302 specified in the table (the first element (first-dimension element) to the 512th element (512th-dimension element)), the elements whose element values ​​exceed the threshold TH1=0 are the second-dimension element, the third-dimension element, and the 510th-dimension element. Therefore, the elements belonging to the function application domain are the second-dimension element, the third-dimension element, and the 510th-dimension element of difference vector 304, and the vector obtained by multiplying the values ​​of these elements by weight value α=64 becomes difference vector 305.

[0042] The calculation unit 204c calculates the difference value 306 by dividing the sum of all element values ​​of the difference vector 305 by 1024, which is the number of dimensions of the feature vector output from the output layer of the deep net. The method for calculating the difference value 306 from the element values ​​in the difference vector 305 is not limited to a specific method. For example, the calculation unit 204c may calculate the difference value 306 by the sum of all element values ​​of the difference vector 305, or may calculate the difference value 306 by dividing the sum of element values ​​equal to or greater than a threshold in the difference vector 305 or by dividing the sum by 1024.

[0043] The update unit 205 calculates "parameters 307 of the deep net used by the extraction unit 203" by the backpropagation method so that the difference value 306 becomes smaller, and updates the currently set "parameters of the deep net used by the extraction unit 203" to the parameters 307. As a result, the extraction unit 203 inputs the next input image to the deep net B reconstructed according to the parameters 307, and obtains a second feature vector 303 (512 dimensions) extracted from the image in the intermediate layer of the deep net B. The deep net B reconstructed according to the parameters 307 has changed weighting coefficients and the like in the deep net B before reconstruction.

[0044] The increasing unit 206 increases the amount by which the value of element Cj of the difference vector 304 corresponding to element Aj having a value exceeding the threshold value TH1 among the elements of the first feature vector 302 is increased (increase amount). In the above example, the weight value α is increased and the weight value β is decreased. In general, the increasing unit 206 updates the setting so that the amount of increase in the element value from the difference vector 304 to the difference vector 305 in the function application region is increased. Alternatively, the increasing unit 206 updates the setting so that the amount of decrease in the element value from the difference vector 304 to the difference vector 305 in the non-function application region is decreased.

[0045] The weight value α is increased by the increase unit 206 when the number of elements belonging to the function application domain no longer decreases even if the number of learning times increases. When ReLU is used as the activation function of the deep net, all outputs corresponding to inputs with values ​​of 0 or less are 0, so if the learning rate at the time of updating the parameters by error backpropagation is small, the output remains 0 and it is easy to fall into a local solution. On the other hand, if the learning rate is made large, the fluctuation of the parameters contributing to the elements of the second feature vector 303 whose values ​​are not 0 also increases at the same time, making it impossible to perform appropriate learning. Therefore, the value (initial value) of the weight value α at the start of learning is set to 1, and the parameters of the extraction unit 203 that outputs the elements of the first feature vector 302 that can be reproduced relatively easily are first obtained.

[0046] By increasing the weight value α at the aforementioned timing, the difference value corresponding to the function application region is increased. When the difference value is increased by the weight value α, the learning promotes the learning of the parameters that contribute to the function application region. By sequentially increasing the weight value α in this manner, it is possible to finally obtain appropriate parameters for the extraction unit 203.

[0047] The timing for updating the setting so that the amount of increase by the increase unit 206 increases is not limited to the above timing, and may be determined, for example, based on the increase or decrease in the difference value 306 (the same applies to the timing for updating the setting so that the amount of decrease by the increase unit 206 decreases). In other words, the timing for updating may be determined when the amount of change in the difference value 306 calculated this time from the previous time is less than a threshold value. Also, the timing may be periodically or irregularly updated depending on the number of learning repetitions (number of trials) or the time elapsed since the start of learning.

[0048] By repeating the above-mentioned processing (processing of the data acquisition unit 201, the extraction unit 202, the extraction unit 203, the acquisition unit 204a, the function application unit 204b, the calculation unit 204c, the update unit 205, and the increase unit 206), the "deep net used by the extraction unit 203" is learned.

[0049] The determination unit 207 determines whether or not a learning end condition has been satisfied. The learning end condition is not limited to a specific condition. For example, the determination unit 207 determines that the learning end condition has been satisfied when the user operates the input unit 109 to input an instruction to end learning. In addition, for example, the determination unit 207 determines that the learning end condition has been satisfied when the number of times the learning has been repeated exceeds a specified number of times or when the time elapsed since the start of learning exceeds a specified time.

[0050] Next, the operation of the information processing device 100 will be described with reference to the flowchart in Fig. 3. The details of the processing in each step have been described above, so only a brief description will be given here.

[0051] In step S401, the data acquisition unit 201 loads (acquires) the image 301 stored in the external storage device 104 into the RAM 103. In step S402, the extraction unit 202 inputs the image 301 acquired in step S401 to the deep net A, and acquires a first feature vector 302 extracted from the image 301 in an intermediate layer of the deep net A.

[0052] In step S403, the extraction unit 203 inputs the image 301 acquired in step S401 to the deep net B, and acquires a second feature vector 303 extracted from the image 301 in the intermediate layer of the deep net B.

[0053] In step S404, the acquisition unit 204a generates a difference vector 304 between the first feature vector 302 acquired in step S402 and the second feature vector 303 acquired in step S403.

[0054] In step S405, the function application unit 204b identifies an element Aj among the elements of the first feature vector 302 that has a value exceeding a threshold TH1, and generates a difference vector 305 by increasing the value of element Cj of the difference vector 304 corresponding to the element Aj.

[0055] In step S406, the calculation unit 204c obtains the difference value 306 by dividing the sum of all the element values ​​of the difference vector 305 generated in step S405 by 1024.

[0056] In step S407, the update unit 205 calculates, by the backpropagation algorithm, "parameters 307 of the deep net used by the extraction unit 203" that make the difference value 306 calculated in step S406 smaller. Then, the update unit 205 updates the currently set "parameters of the deep net used by the extraction unit 203" to the parameters 307.

[0057] In step S408, the increase unit 206 determines whether it is time to update the weight value α. If it is time to update the weight value α as a result of this determination, the process proceeds to step S409, and if it is not time to update the weight value α, the process proceeds to step S410.

[0058] For example, if the condition that the number of elements belonging to the function application domain does not change continuously through 100 learning cycles is met, it is determined that it is time to update the weight value α, and if the condition is not met, it is determined that it is not time to update the weight value α.

[0059] In step S409, the increase unit 206 increases the weight value α. Here, the initial value of the weight value α is 1, and the increment of the weight value α at one time is 64. In step S410, the determination unit 207 determines whether or not the learning end condition is satisfied. If the result of this determination is that the learning end condition is satisfied, the process according to the flowchart in FIG. 3 ends, and if the learning end condition is not satisfied, the process proceeds to step S401.

[0060] 3 in the external storage device 104, or may transmit the "deep net parameters 307 used by the extraction unit 203" to an external device via a network. Also, the CPU 101 may display the "deep net parameters 307 used by the extraction unit 203" on the monitor 110 using images or text. In this way, the output destination and output form of the "deep net parameters 307 used by the extraction unit 203" are not limited to a specific output destination or a specific output form.

[0061] In this way, according to the present embodiment, a weight is assigned to the difference in the function application domain during distillation, and the weight is increased according to the progress of learning. This makes it possible to make the feature vectors of the student model and the teacher model approximately identical, even in highly difficult distillation that was difficult to achieve with conventional methods.

[0062] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified, it is assumed that the present embodiment is the same as the first embodiment. An example of the functional configuration of the extraction unit 202 and the extraction unit 203 according to this embodiment will be described with reference to the block diagram of FIG.

[0063] The extraction unit 202 according to this embodiment has a first first half extraction unit 501 and a first second half extraction unit 502. The first first half extraction unit 501 inputs an input image 301 to a deep net A, and acquires a first intermediate feature vector 505 extracted from the image 301 in an intermediate layer A of the deep net A. The intermediate layer A is an intermediate layer between an input layer in the deep net A and an "intermediate layer that outputs the first feature vector 302". The first second half extraction unit 502 generates the first feature vector 302 by performing calculations on each layer after the intermediate layer A using the first intermediate feature vector 505 as an input.

[0064] The extraction unit 203 according to this embodiment has a second first half extraction unit 503 and a second second half extraction unit 504. The second first half extraction unit 503 inputs an input image 301 to a deep net B, and acquires a second intermediate feature vector 506 extracted from the image 301 in an intermediate layer B of the deep net B. The intermediate layer B is an intermediate layer between the input layer and the "intermediate layer that outputs the second feature vector 303" in the deep net B. The second second half extraction unit 504 generates the second feature vector 303 by performing calculations on each layer after the intermediate layer B using the second intermediate feature vector 506 as an input.

[0065] The first feature vector 302 is obtained from a trained deep net, and therefore has a value close to the correct label. Therefore, if most of the element values ​​of the vector of the correct label are 0, many of the element values ​​of the first feature vector 302 will also be 0. If the threshold value TH1 is set to 0 as in this embodiment, there may be few elements that belong to the function application region.

[0066] On the other hand, the first intermediate feature vector 505 is a vector in which various features are obtained from the image 301 in order to obtain the first feature vector 302 so that the value is close to the correct label, and many of the values ​​of the vector elements are non-zero. Therefore, the first intermediate feature vector 505 has many elements that belong to the function application region, and the above learning can be performed more effectively by performing distillation using the intermediate feature vector.

[0067] We will explain the intermediate feature vector and the difference vector using Figure 5. Figure 5 shows an example of a three-dimensional intermediate feature vector with nine two-dimensional data. The black filled areas indicate values ​​of 0, and the white filled areas indicate values ​​that are non-zero.

[0068] FIG. 5(a) shows an example of a first intermediate feature vector 505, in which the intermediate feature vector from the trained deep net (extraction unit 202) captures various features of the image 301, and therefore many of the values ​​are non-zero.

[0069] FIG. 5(b) shows an example of a second intermediate feature vector 506 from a deep net (extraction unit 203) in the middle of training, in which all three two-dimensional data have values ​​of zero, and the other two-dimensional data have values ​​equivalent to those of the first intermediate feature vector 505.

[0070] 5(c) shows an example of the difference vector between the first intermediate feature vector 505 and the second intermediate feature vector 506, where differences occur in three pieces of two-dimensional data. Since the regions in which differences occur are all function application regions, increasing the weight value α can promote learning of parameters that contribute to the three pieces of two-dimensional data.

[0071] An example of the functional configuration of the information processing device 100 according to this embodiment will be described with reference to the block diagram of Fig. 6. The extraction unit 202 acquires a first feature vector 302 and a first intermediate feature vector 505 from an image 301. The extraction unit 203 acquires a second feature vector 303 and a second intermediate feature vector 506 from the image 301.

[0072] The difference acquisition unit 204 acquires a difference value 701 from the first feature vector 302 and the second feature vector 303 in the same manner as in the first embodiment, and acquires a difference value 702 from the first intermediate feature vector 505 and the second intermediate feature vector 506 in the same manner as in the first embodiment. There are various processes for obtaining a difference value from two vectors, as described in the first embodiment. Therefore, the process for obtaining the difference value 701 and the process for obtaining the difference value 702 may be the same process or different processes, or may be the same process but with different threshold values ​​or weighting value α.

[0073] The integration unit 703 calculates the sum of the difference value 701 and the difference value 702 as the difference value 306. After that, the same process as in the first embodiment is performed to calculate the parameter 307, and the parameter of the extraction unit 203 is updated with the calculated parameter 307.

[0074] Furthermore, the update timing of the threshold value and weight value α used to calculate the difference value 701 and the update timing of the threshold value and weight value α used to calculate the difference value 702 may be the same or different. Furthermore, the increment of the weight value α used to calculate the difference value 701 and the increment of the weight value α used to calculate the difference value 702 may be the same or different.

[0075] 3, in step S402, the extraction unit 202 acquires a first feature vector 302 and a first intermediate feature vector 505 from the image 301. Then, in step S403, the extraction unit 203 acquires a second feature vector 303 and a second intermediate feature vector 506 from the image 301.

[0076] Then, in the processes of steps S404 to S406, the difference acquisition unit 204 acquires a difference value 701 from the first feature vector 302 and the second feature vector 303 in the same manner as in the first embodiment. The difference acquisition unit 204 also acquires a difference value 702 from the first intermediate feature vector 505 and the second intermediate feature vector 506 in the same manner as in the processes of steps S404 to S406. Then, in step S406, the integration unit 703 obtains the sum of the difference value 701 and the difference value 702 as the difference value 306. The processes in the other steps are the same as in the first embodiment.

[0077] In this way, according to this embodiment, even in the case of highly difficult distillation that was difficult to achieve using conventional methods, distillation using intermediate feature vectors makes it possible to make the feature vectors of a student model and the feature vectors of a teacher model approximately identical with higher accuracy.

[0078] In addition, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0079] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0080] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0081] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0082] 201: Data acquisition unit 202: Extraction unit 203: Extraction unit 204: Difference acquisition unit 204a: Acquisition unit 204b: Function application unit 204c: Calculation unit 205: Update unit 206: Increase unit 207: Determination unit

Claims

1. A holding means for holding parameters of the student neural network model learned based on a first modified difference vector obtained by modifying values ​​of elements in a first difference vector between a first feature vector extracted based on input data in a teacher neural network model and a second feature vector extracted based on the input data in a student neural network model with less computational effort than the teacher neural network model; an output means for outputting a feature vector for an image input to the trained student neural network model; A change amount from a value of an element of the first difference vector corresponding to a first element in the first feature vector that exceeds a threshold to a value of an element of the first modified difference vector corresponding to the first element is greater than a change amount from a value of an element of the first difference vector corresponding to a second element in the first feature vector that does not exceed a threshold to a value of an element of the first modified difference vector corresponding to the second element.

23. An information processing apparatus comprising:

2. The first modified difference vector is 2 . The information processing apparatus according to claim 1 , wherein the first difference vector is generated by increasing a value of an element of the first difference vector that corresponds to a first element in the first feature vector that exceeds a threshold value.

3. The first modified difference vector is 3 . The information processing apparatus according to claim 1 , wherein the first difference vector is generated by decreasing values ​​of elements of the first difference vector corresponding to second elements in the first feature vector that do not exceed a threshold value.

4. The parameters of the trained student neural network model are:

4. The information processing apparatus according to claim 1, wherein the parameter has a smaller value based on the values ​​of elements in the first modification difference vector.

5. moreover, 3. The information processing apparatus according to claim 2, further comprising: an increasing unit that increases an amount by which a value of the element of the first difference vector corresponding to the first element is increased.

6. 6. The information processing device according to claim 5, wherein the increasing means increases an amount by which the value of the element of the first difference vector corresponding to the first element is increased at a timing when the number of elements of the first difference vector corresponding to the first element stops decreasing.

7. 7. The information processing apparatus according to claim 1, wherein the first feature vector is a feature vector extracted from input data in an intermediate layer of a teacher neural network model by inputting the input data into the teacher neural network model.

8. 8. The information processing apparatus according to claim 7, wherein an activation function of the teacher neural network model is ReLU (Rectified Linear Unit).

9. 9. The information processing device according to claim 1, wherein the second feature vector is a feature vector extracted from the input data in an intermediate layer of the student neural network model by inputting input data to the student neural network model having a smaller number of parameters than the teacher neural network model.

10. 10. The information processing apparatus according to claim 9, wherein an activation function of the student neural network model is ReLU (Rectified Linear Unit).

11. the retaining means further retains parameters of the student neural network model trained based on: a second modified difference vector obtained by modifying values ​​of elements in a second difference vector between a first intermediate feature vector extracted based on input data in the teacher neural network model and a second intermediate feature vector extracted based on the input data in the student neural network model with a smaller amount of calculation than the teacher neural network model; and the first modified difference vector; A change amount from a value of an element of the second difference vector corresponding to a third element in the first intermediate feature vector that exceeds a threshold to a value of an element of the second modified difference vector corresponding to the third element is greater than a change amount from a value of an element of the second difference vector corresponding to a fourth element in the first intermediate feature vector that does not exceed a threshold to a value of an element of the second modified difference vector corresponding to the fourth element.

7. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

12. The parameters of the trained student neural network model are:

12. The information processing apparatus according to claim 11, wherein the parameter has a value based on an element value in the first modification difference vector and an element value in the second modification difference vector, the value being smaller.

13. An information processing method performed by an information processing device, an output step of outputting a feature vector for an image input to the trained student neural network model using parameters of the student neural network model trained based on a first changed difference vector obtained by changing values ​​of elements in a first difference vector between a first feature vector extracted based on input data in a teacher neural network model and a second feature vector extracted based on the input data in a student neural network model with a smaller amount of calculation than that of the teacher neural network model, A change amount from a value of an element of the first difference vector corresponding to a first element in the first feature vector that exceeds a threshold to a value of an element of the first modified difference vector corresponding to the first element is greater than a change amount from a value of an element of the first difference vector corresponding to a second element in the first feature vector that does not exceed a threshold to a value of an element of the first modified difference vector corresponding to the second element.

23. An information processing method comprising:

14. A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images