Information processing apparatus, information processing method, and program

Abstract

To secure various parameters while regularizing some of parameters of a machine learning model.SOLUTION: An information processing apparatus calculates an error between an output into which learning data is input and ground truth data for the learning data in a machine learning model, regularizes some of parameters of the machine learning model, and updates parameters of the machine learning model on the basis of the calculated error and the regularized parameters.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, multilayer neural networks have achieved high accuracy in various computer vision tasks, such as image class classification (Non-Patent Document 2) and object tracking (Non-Patent Document 3). However, multilayer neural networks have a large number of parameters, and learning these parameters so that they can be applied to each task requires a lot of computational resources and time. In particular, when there is little data available for learning, it is known that there are cases in which multilayer neural networks overfit, making them difficult to apply to real problems.

[0003] Known methods for suppressing overlearning while efficiently learning include L2 regularization, dropout, and batch regularization. For example, Non-Patent Document 1 discloses a technique for orthogonalizing the parameters of each layer in a neural network and reducing the similarity between the parameters by performing orthogonal regularization. Here, by orthogonalizing the parameters, a variety of feature quantities are acquired, and the efficiency of learning is improved, overlearning is suppressed, and estimation accuracy is improved. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Nitin Bansal et al., “Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?”, NIPS 2018, December 2018, p.4266-4276 [Non-Patent Document 2] Karen Simonyan et al., “Very Deep Convolutional Networks For Large-Scale Image Recognition”, ICLR 2015, [Non-Patent Document 3] Luca Bertinetto et al., “Fully-Convolutional Siamese Networks For Object Tracking”, ECCV 2016, November 2016, p.850-865 Summary of the Invention [Problem to be solved by the invention]

[0005] In Non-Patent Document 1, orthogonal regularization is applied to a convolutional neural network. In Non-Patent Document 1, in the experiment, orthogonal regularization is applied to both the convolutional layer and the fully connected layer of the convolutional neural network. By applying orthogonal regularization to the kernel of the convolutional layer, it is possible to acquire a variety of features, but there is a problem in that if all parameters are orthogonalized, it may become impossible to extract important features.

[0006] The present invention aims to ensure diversity of parameters while regularizing some of the parameters of a machine learning model. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: A calculation means for calculating an error between an output of a machine learning model to which learning data is input and correct answer data for the learning data, a regularization means for regularizing a part of parameters of the machine learning model, and an update means for updating the parameters of the machine learning model based on the calculated error and the regularized parameters. Effect of the Invention

[0008] Regularize some of the parameters of the machine learning model while ensuring diversity of parameters. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 2] 5 is a flowchart showing an example of information processing according to the first embodiment. [Diagram 3] FIG. 4 is a diagram for explaining parameters of a machine learning model according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining a kernel for extracting a primitive feature according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining parameters of a machine learning model according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 7] 10A to 10C are diagrams for explaining search features and reference features according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of a hardware configuration of an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [Embodiment 1] The information processing device according to this embodiment is described below. The information processing device according to this embodiment can be used, for example, when performing an image classification task using a convolutional neural network as shown in Non-Patent Document 2.

[0012] 1 is a block diagram showing an example of the functional configuration of an information processing device according to this embodiment. Note that only an overview of each functional unit will be described here, and details will be described later.

[0013] The information processing device 100 according to the present embodiment is connected to the imaging device 110 and the result output unit 120 so as to be able to communicate with them. The imaging device 110 is composed of an optical system, an imaging element, etc., and captures an image and outputs it to the data acquisition unit 101. The imaging device 110 may be, for example, an imaging device such as a digital camera or a security camera. Note that the information processing device 100 according to the present embodiment will be described as acquiring an image captured by the imaging device 110 and using the image as a processing target, but is not limited to such a configuration as long as it can acquire an image. The information processing device 100 may acquire image data from, for example, a separate device different from the imaging device 110, or may acquire image data via the Internet.

[0014] The information processing device 100 includes a data acquisition unit 101, a learning unit 102, a regularization unit 103, a selection unit 104, a feature acquisition unit 105, and a pattern recognition unit 106. The data acquisition unit 101 receives an input of a pair of an image captured by an imaging device 110 and corresponding supervised answer data. The data acquisition unit 101 according to this embodiment receives as input a captured image as input data, and receives as input a pair of the input data and information indicating a class (category) of an object captured in the image as supervised answer data.

[0015] The feature acquisition unit 105 takes the acquired data (image) as input and extracts features using a machine learning model. The pattern recognition unit 106 calculates a recognition result based on the extracted features (here, it calculates a recognition result of the class of the object in the image). In the following, the parameters of the machine learning model may be expressed as "parameters of the feature acquisition unit 105". The learning unit 102 sums up the difference (task error) between the recognition result and the ground truth data and an orthogonal regularization term described later, and updates the parameters of the feature acquisition unit 105 so that the sum becomes smaller.

[0016] The regularization unit 103 regularizes some of the parameters of the feature acquisition unit 105. The regularization unit 103 according to the present embodiment uses orthogonal regularization as a parameter regularization process, for example, regularizing kernel parameters in a specific convolution layer in a convolutional neural network. In the following, the regularization process performed by the regularization unit 103 will be described as orthogonal regularization, but the regularization process is not particularly limited to this as long as some of the parameters of the feature acquisition unit 105 are used in the regularization process and the result can be used as a regularization term in learning a machine learning model.

[0017] The selection unit 104 selects parameters to be regularized. When the selection unit 104 performs the selection in the convolution layer, the selection unit 104 selects a channel to be regularized. For example, the selection unit 104 may select a channel to be regularized by a calculation process shown in Equation (7) using a mask matrix M for a matrix indicating parameters of a machine learning model. The mask matrix M according to this embodiment is a matrix for selecting parameters to be regularized from among parameters of the machine learning model, and the detailed process will be described later.

[0018] The result output unit 120 displays the recognition result by the pattern recognition unit 106. Here, the result output unit 120 will be described as a display device separate from the information processing device 100, but may be a display unit included in the information processing device 100. The result output unit 120 is, for example, a device equipped with a display, and in this embodiment, displays a category (recognition result of an object in an image) corresponding to the input image.

[0019] Hereinafter, information processing according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of information processing by the information processing device 100 according to this embodiment. Note that only an overview will be described here, and details of the processing in each step will be described later.

[0020] S200 is a process (preparation step) for preparing for the learning process related to S201 to S210. In S200, the data acquisition unit 101 acquires a set of learning data and supervised data to be used for learning. Here, the data acquisition unit 101 captures a sufficient number of images (learning data) in advance, converts the learning data, acquires and assigns supervised data to the learning data, and encodes the supervised data.

[0021] In S201, the feature acquisition unit 105 acquires features from the image acquired in S200. In S202, the pattern recognition unit 106 performs pattern recognition based on the acquired features and outputs the recognition result. In S203, the learning unit 102 compares the recognition result with the correct answer data and calculates the task error L task The task error is lower when the recognition result and the ground truth data match, and higher when they differ.

[0022] In S204, the learning unit 102 calculates a Gram matrix G of the parameters W of the feature acquisition unit 105. In S205, the learning unit 102 generates a mask matrix M for selecting parameters to be regularized, and the selection unit 104 selects parameters for regularization. In S206, the selection unit 104 selects parameters for orthogonal regularization by taking the Hadamard product of G and M. The Hadamard product of G and M is defined as T.

[0023] In S207, the learning unit 102 calculates the Frobenius norm L orth Calculate the calculated L orth is the orthogonal regularization term. The orthogonal regularization term becomes lower as the parameters of the feature extraction unit become more orthogonal, and becomes higher when the parameters are not orthogonal. In S208, the learning unit 102 calculates the task error L task and the orthogonal regularization term L orth and are summed up to calculate the error L. In S209, the learning unit 102 updates the parameters of the machine learning model based on the error L. In S210, the learning unit 102 determines whether or not a learning end condition is satisfied. If the end condition is satisfied, the process ends, and if not, the process returns to S201. Here, it is assumed that the learning end condition is satisfied when the parameter update process in S209 has been executed a predetermined number of times.

[0024] [Preparation Step] The process executed in S200 will be described below. As described above, in S200, the data acquisition unit 101 collects a sufficient number of pairs of training data and supervised data, converts the training data, and encodes the supervised data. Here, the training data is an RGB image of 1000 pixels wide and 1000 pixels high captured by a digital camera. The information processing device 100 according to this embodiment will be described as performing each inference process using a convolutional neural network VGG16 described in Non-Patent Document 2.

[0025] It should be noted that the conversion process of the learning data is assumed to be a process of converting a captured image into a size corresponding to the input of the VGG16 (here, reducing the image to a width of 224 pixels and a height of 224 pixels). In this embodiment, the information processing device 100 executes a classification task using an image as an input, and the super-answer data is the class of an object appearing in the image used in such classification. However, the inference process by the information processing device 100 and the super-answer data used are not particularly limited to this, and super-answer data corresponding to a different inference process may be used.

[0026] The data acquisition unit 101 calculates the task error L using the recognition result and the correct answer data. task To calculate this, the correct answer data is encoded (here, one-hot encoding). Note that the number of dimensions of the encoded correct answer data is equal to the number of categories (e.g., 1000).

[0027] [Feature Acquisition and Pattern Recognition] The process executed in S201 to S202 will be described below. It is known that a multilayer neural network that has sufficiently learned a specific task acquires features useful for the specific task in its intermediate layer. In this embodiment, an image is input to VGG16 to learn a classification task, and the intermediate layer of VGG16 becomes a feature acquisition unit that extracts features useful for classification. In this embodiment, the number of neurons in the final layer of VGG16 is set to 1000, which is the same as the number of categories. The pattern recognition unit 106 according to this embodiment regards the category corresponding to the neuron with the strongest response among the responses of the neurons in the final layer of VGG16 as the pattern recognition result. Note that here, considering ease of learning, a softmax function (the following formula (1)) is applied to the final layer of VGG16.

number

[0028] where y i is the final layer value of VGG16 after applying the softmax function. x i is the response of the i neuron in the final layer of the VGG16, and i represents its index. In the following, y i may be referred to as a predicted value. n indicates the number of categories, and in this embodiment, the description will be given assuming that n=1000.

[0029] [Task error] The calculation process of the task error in S203 will be described below. task is calculated using the cross-entropy error, for example, as shown in equation (2) below.

number

[0030] Here, t i is the correct data, and y i indicates the predicted value. n indicates the number of categories, and in this embodiment, n=1000. i Since it is one-hot encoded, if the correct class is the cth one, then t c Only L is 1 and the others are 0. task is the correct data t i and the predicted value y i is a minimum when they match perfectly, and takes on larger values ​​as they mismatch.

[0031] [How to obtain parameter W of the feature acquisition part] A method for acquiring the parameter W of the feature acquisition unit that performs orthogonal regularization will be described with reference to FIG. 3. FIG. 3(a) is a diagram for explaining a method for acquiring the parameter W of the feature acquisition unit when the feature acquisition unit is a fully connected layer of a multi-layer neural network. Consider a case where the input layer 301 is made up of four nodes and the output layer 303 is made up of three nodes. Since the fully connected layer connects all the nodes of the input layer and the output layer, the fully connected layer shown in FIG. 3(a) has a total of 12 connection weight parameters 302. The parameter W (304) of the feature acquisition unit is a matrix in which the connection weight parameters 302 are arranged. The number of nodes in the input layer is n_in, and the number of nodes in the output layer is n_out. The connection weight parameter from the i-th node of the input layer to the j-th node of the output layer is w (j) i Then, the parameter W of the feature acquisition unit can be expressed as in equation (3). In the example of Fig. 3(a), the parameter W is a 4 × 3 matrix.

number

[0032] FIG. 3(b) is a diagram for explaining a method of acquiring the parameter W of the feature acquisition unit when the feature acquisition unit is a convolution layer of a multi-layer neural network. The number of channels in_ch of the input layer 311 is 4, and the number of channels out_ch of the output layer 313 is 3. The kernel size k of the convolution filter 312 shown in FIG. 3(b) is 2, and there are a total of 12 convolution filters. The parameter W (314) of the feature acquisition unit is a matrix in which one-dimensional vectors obtained by flattening the kernel on the input side are arranged in the number of output channels. The number of rows of the parameter W 314 of the feature acquisition unit is k×k×in_ch, and the number of columns is out_ch, which is a 16×3 matrix in the example of FIG. 3(b).

[0033] FIG. 3(c) is a diagram for explaining a method of acquiring the parameter W of the feature acquisition unit when the feature acquisition unit is a depthwise separable convolution layer of a multilayer neural network. In a depthwise separable convolution layer, the number of channels of the input layer and the output layer must match, and in FIG. 3(c), the number of channels in_ch of the input layer 321 and the number of channels out_ch of the output layer 323 are both 3. The kernel size k of the convolution filter 312 is 2, and each channel has three convolution filters. The parameter W (324) of the feature acquisition unit is a matrix in which one-dimensional vectors with flattened kernels are arranged in the number of input and output channels. The number of rows of the parameter W 324 of the feature acquisition unit is k×k, and the number of columns is in_ch=out_ch. In the example of FIG. 3(c), the parameter 324 is a 4×3 matrix.

[0034] [Gram matrix] The calculation process of the Gram matrix in S204 will be described below. The Gram matrix G can be calculated using the parameter W of the feature acquisition unit as shown in the following formula (4). G=W T W formula (4)

[0035] The i-th row and j-th column component of the Gram matrix G ij is the i-th column vector w of the parameter W of the feature acquisition part (i) and the j-th column vector w (j)w (i) And w (j) If is orthogonal, G ij becomes 0. The diagonal element G ii is the i-th column vector w of the parameter W of the feature acquisition part (i) Represents the norm of .

[0036] [Mask generation for selecting parameters to be regularized] The mask matrix generation process in S205 will be described below. The number of columns of the parameter W of the feature acquisition unit is p, and the number of parameters to be regularized in advance is q. Here, q≦p. The learning unit 102 randomly selects q column indexes without duplication. If the set of selected column indexes is s, the mask matrix M is expressed by equation (5).

number

[0037] Here, ε is a tuning parameter that can be set to any small value that satisfies 0≦ε<1. The larger ε is, the stronger the degree of orthogonal regularization for the unselected column indexes. An example of M for p=5, q=3, and s={2,3,5} is shown in Equation (6).

number

[0038] Orthogonal Regularization The orthogonal regularization process in S206 to S207 will be described below. The learning unit 102 uses the Gram matrix G and the mask matrix M to calculate an orthogonal regularization term L orth can be calculated.

number

[0039] Here, in equation (7), T is the Hadamard product of the Gram matrix G and the mask matrix M, and in equation (8), the Frobenius norm of T is calculated. For simplicity, if ε = 0, the non-zero elements of matrix T are the elements in row i and column j, where i and j satisfy the second condition in equation (5). In other words, i ≠ j (off-diagonal elements), and i and j are included in the selected column index set s. As mentioned above, G ij W (i) And w (j) To express the inner product of, assuming ε=0, L orth When becomes 0, the parameters of the feature acquisition parts corresponding to the selected column index are orthogonal to each other.

[0040] [Error L] The calculation process of the error L in S208 and the parameter update process in S209 will be described below. task and the orthogonal regularization term L orth Using this, the error L can be calculated using equation (9). L=L task +λL orth Formula (9)

[0041] Here, λ is an adjustment parameter such that 0≦λ, and the larger it is, the greater the weight of the error in the orthogonal regularization. In this embodiment, λ is set to 0.1. Based on the calculated error L, the learning unit 102 updates the parameters of the feature acquisition unit realized by the convolutional neural network using the stochastic gradient descent method with a part of the learning data collected in S200 as a batch. Note that the update process of the parameters of the machine learning model based on the error can be performed by a known general machine learning technology, and a detailed description thereof will be omitted here.

[0042] In this embodiment, one column index set is considered, but there may be multiple column index sets. The following describes the case where there are multiple column indexes. Let R be the number of column index sets, and each column index set be a set s r Let us assume that s rThe number of elements of each is the number of parameters to be regularized, where q r The mask matrix M is expressed by modifying equation (5) as shown in the following equation (10).

number

[0043] An example of the mask matrix M when p=5, R=2, q1=2, q2=2, s1={1,3}, and s2={2,4} is shown in equation (11).

number

[0044] [Effects of the First Embodiment] [Effects of the first embodiment; primitive features] The regularization unit 103 according to this embodiment regularizes some of the parameters of the feature acquisition unit 105. To this end, the selection unit 104 according to this embodiment can select the parameters to be regularized. By selecting the parameters to be regularized, some parameters can be excluded from the targets of regularization, and these parameters can be maintained without degradation.

[0045] It is known that shallow layers (intermediate layers close to the input layer) in a convolutional neural network are trained to extract primitive features such as edges and corners of an image. For example, the selection unit 104 can select parameters so as not to regularize parameters determined to extract primitive features such as edges in a specific direction. Here, the selection unit 104 may select parameters so as not to regularize parameters of a predetermined number of layers from the input layer of the convolutional neural network, for example.

[0046] FIG. 4 is a diagram showing an example of parameters for extracting primitive features when the kernel size of the convolution layer is set to 3×3. Parameter 401 is a parameter for extracting horizontal edges. Parameter 402 is a parameter for extracting diagonal edges. Some human photoreceptors react to edges in such specific directions, so these primitive features are considered to be important. However, when parameter 401 and parameter 402 are each made one-dimensional and the inner product is taken, it becomes 1, so these parameters are not orthogonal. When a parameter orthogonal to parameter 1 is obtained using a conventional method, a parameter such as parameter 403 is obtained.

[0047] [Effects of the first embodiment; Features of subtle differences] Orthogonal regularization is an effective method for acquiring various parameters, but depending on the task to be solved, the parameters for extracting important features to be extracted may be similar (not orthogonal). In a task that requires extracting subtle differences in features from an image, such as classifying objects with similar appearances (e.g., leopards and cheetahs), the important features to be extracted are likely to be similar. Below, using FIG. 5, the difference between the conventional method and the processing according to this embodiment when the important features to be extracted are similar will be explained. Here, the space in which the column vectors of the parameter W of the feature acquisition unit exist is called the feature space.

[0048] Figure 5 is a conceptual diagram showing five feature parameters as vectors (feature vectors) in a feature space, assuming that parameter W has two rows and five columns, i.e., five feature parameters exist in a two-dimensional feature space. In Figure 5, the arrows shown with solid lines represent the five feature vectors, and the arrows shown with dashed lines represent important features. In the example of Figure 5, it is assumed that the important features are similar, and the parameters that extract them are not orthogonal.

[0049] Figure 5(a) shows the learned parameters in the feature space when λ (the adjustment parameter for orthogonal regularization) is large in the conventional method, and Figure 5(b) shows the parameters learned with a small λ in the feature space. When the weight of orthogonal regularization is increased as in Figure 5(a), emphasis is placed on widening the angle between the parameters, and similar features cannot be captured accurately. On the other hand, when the weight of orthogonal regularization is decreased as in Figure 5(b), the angle between the parameters becomes narrower, but similar features cannot be captured accurately. Figure 5(c) shows an example when λ is set to 0. When λ is set to 0, similar features can be captured accurately, but the diversity of parameters is lost.

[0050] 5(d) illustrates, in the information processing according to this embodiment, feature vectors in a feature space when the number q of parameters to be regularized is set to 3. Here, vectors 503 and 504, which are parameters corresponding to important features, are not orthogonally regularized, and similar features can be accurately captured. Also, here, vectors 501, 502, and 505 are parameters to be orthogonally regularized, and thus a variety of parameters can be obtained.

[0051] According to this process, by obtaining various parameters by parameter regularization processing and selecting parameters to be regularized, it becomes possible to accurately extract some features. Therefore, it becomes possible to accurately extract primitive features or features with subtle differences. In particular, it becomes possible to effectively utilize redundant (mutually similar) parameters used in the intermediate layers of a multi-layer neural network in the subsequent layers, which has a positive effect on the final result.

[0052] [Embodiment 2] The information processing device according to the first embodiment has been described as performing a class classification task in an image. However, as long as a regularization parameter is selected and learning is performed in a similar manner, the subject to which the information processing device according to the present invention is applied is not particularly limited to such an example. In the second embodiment, a case where object tracking is performed instead of class classification will be described.

[0053] Fig. 6 is a block diagram showing an example of a part of the functional configuration of the information processing device according to this embodiment. The information processing device 100 according to this embodiment has the same configuration as that of the first embodiment except that the feature acquisition unit 105 includes a reference feature acquisition unit 601 and a search feature acquisition unit 602, and the pattern recognition unit 106 includes a correlation calculation unit 603 and a tracking target detection unit 604, so duplicated explanations will be omitted. In Fig. 6, some of the configurations that overlap with those of the first embodiment are omitted and not shown.

[0054] The reference feature acquisition unit 601 acquires features (reference features) of the tracking target in the image. Here, the reference feature acquisition unit 601 receives the image (reference image) acquired by the data acquisition unit 101 and information indicating the position of the tracking target in the image, and acquires the reference features. The search feature acquisition unit 602 outputs features (search features) of the entire search image from an image (search image) different from the reference image.

[0055] The reference feature acquisition unit 601 according to this embodiment acquires the reference feature using a part of the convolutional neural network. The reference image is an RGB image of 1000 pixels wide and 1000 pixels high captured by a digital camera. The reference feature acquisition unit 601 according to this embodiment inputs the reference image to the convolutional neural network and outputs a feature of 100 pixels wide and 100 pixels high, 16 channels. The reference feature is a part of the output of the convolutional neural network, and in this embodiment, it is a feature of 3 pixels wide and 3 pixels high, 16 channels. The reference feature acquisition unit 601 can calculate the reference feature based on the position of the object in the reference image and the ratio of the width and height of the reference image to the width and height of the output of the convolutional neural network. In this embodiment, the ratio of the width and height is both 1 / 10. If the position of the object in the reference image is x=343, y=627, then dividing these by 10 and rounding them off results in x=34 and y=63. In this case, the reference feature acquisition unit 601 can extract a 3×3 region centered at x=34, y=64 in each channel of the output of the convolutional neural network, and use the extracted region as the reference feature.

[0056] The search feature acquisition unit 602 according to this embodiment outputs search features using a convolutional neural network. The search image is an RGB image of 1000 pixels wide and 1000 pixels high captured by a digital camera. The search feature acquisition unit 602 according to this embodiment inputs the search image to the convolutional neural network and outputs a 100 pixel wide, 100 pixel high, 16 channel tensor (search feature).

[0057] The correlation calculation unit 603 evaluates the correlation (similarity) between the reference image and the search image (between the reference feature and the search feature). The correlation calculation unit 603 according to this embodiment evaluates the correlation between the reference feature and the search feature by setting the reference feature as the kernel parameter of the convolution layer and performing a depthwise separable convolution process with the search feature as an input to the convolution layer. Such an evaluation of the correlation may be referred to as a correlation calculation between the reference feature and the search feature hereinafter. Here, the output (evaluation) of the correlation calculation unit 603 is the output of the depthwise separable convolution process, and is a tensor of the same size as the search feature. In this output, the reaction of a portion similar to the reference feature of the corresponding channel for each channel of the tensor is large. Here, in this output, for example, the evaluation of the correlation may be output as a tensor so that the smaller the difference in the values ​​of the components at the corresponding positions is, the larger the value becomes.

[0058] The tracking target detection unit 604 outputs the position and size of the tracking target in the search image. The tracking target detection unit 604 according to this embodiment can output the position and size of the tracking target in the search image by inputting the correlation between the reference feature and the search feature evaluated by the correlation calculation unit 603 to the convolutional neural network. In this embodiment, the output by the convolutional neural network of the tracking target detection unit 604 is a tensor with a width of 100, a height of 100, and three channels. The convolutional neural network is trained to output a likelihood map indicating the likelihood of the presence of the tracking target in the first channel, a width map of the tracking target in the second channel, and a height map of the tracking target in the third channel. The training method will be described later.

[0059] The tracking target detection unit 604 finds the point where the value of the likelihood map output by the convolutional neural network with the above correlation as input is maximum. Here, the point where the likelihood map is maximum is x=32, y=55. Next, the tracking target detection unit 604 calculates the position of the object in the search image using the ratio of the width and height between the search image and the likelihood map. In this embodiment, the ratio of the width and height between the search image and the likelihood map is 1 / 10, and the position of the object in the search image is x=320, y=550. Next, the tracking target detection unit 604 acquires the values ​​in the width map and height map at the point where the likelihood map is maximum. Here, the numerical value corresponding to x=32, y=55 in the width map is 120, and similarly, the numerical value in the height map is 180. Through the above process, the tracking target detection unit 604 outputs the position of the tracking target as x=320, y=550, the width as 120, and the height as 180. If the tracking target does not appear in the search image, the tracking target detection unit 604 outputs a message indicating that there is no match.

[0060] [Task error calculation] The learning unit 102 according to this embodiment acquires the position, width, and height of an object in a search image as correct answer data. Next, the learning unit 102 generates a correct answer likelihood map, a correct answer width map, and a correct answer height map based on the acquired correct answer data. The size of the correct answer likelihood map is 100 in width and 100 in height, similar to the likelihood map.

[0061] Consider the case where x=343, y=528 are given as the position of the object. Based on the ratio of the width and height of the search image and the correct answer likelihood map, the position of the object on the correct answer likelihood map is x=34.3, y=42.8. Here, the learning unit 102 generates a correct answer likelihood map having a value representing the correct answer likelihood for each position by using a Gaussian centered on the position of the object. This correct answer likelihood map has a peak at x=34.3, y=42.8, and takes gradually smaller values ​​in a concentric circle shape as it moves away from the peak. At this time, the spread parameter of the Gaussian can be set arbitrarily according to the conditions, and may be adjustable according to the width and height of the object in the search image. In this example, the minimum value of the correct answer likelihood map is 0, and the maximum value is 1.

[0062] The correct width map is a map having as its components the values ​​of the width of the object in the search image in a range where the correct likelihood map value is equal to or greater than a predetermined value (e.g., 0.5) (such a range may be called a corresponding range). If the width of the object is 128, then in the correct width map, the values ​​in the range of 0.5 or greater in the correct likelihood map are 128, and the other values ​​are 0. The correct height map can also be obtained in a similar manner. For example, the width of the object is 189.

[0063] The learning unit 102 according to this embodiment calculates the task error L task can be calculated, for example, by equation (12). L task =L l +λ w L w +λ h L h Formula (12)

[0064] λ w And λ h are adjustment parameters that are 0 or greater, and represent the importance of the width map and the height map, respectively. For example, λ w = 1000, λ h = 1000. L l is the error for the likelihood map, and can be calculated by the least square error between the likelihood map and the correct likelihood map. w , and L h are the errors for the width map and height map, respectively, and are calculated by the least squares error in the same way as for the likelihood map.

[0065] The regularization unit 103 according to this embodiment performs orthogonal regularization to orthogonalize the reference features of the 16 channels. As shown in Fig. 3(c), the correlation calculation unit 603 according to this embodiment evaluates correlation by depth-unit separable convolution processing, so that the parameter W of the reference feature to be orthogonalized is a matrix of k x k rows and in_ch columns as shown in 324. In this embodiment, the matrix is ​​9 rows and 16 columns.

[0066] [Mask Generation] The selection unit 104 according to this embodiment selects a reference feature corresponding to a channel in which search features are similar to each other. The process performed by the selection unit 104 will be described with reference to FIG. 7. FIG. 7 is a schematic diagram for explaining the process of the correlation calculation unit 603 in a tracking task. A 3-row, 3-column matrix 701 shown at the top of FIG. 7 is the parameters of the reference feature for one channel. In this embodiment, there are matrices 701 in the same number as the number of channels (here, 16). This matrix 701 is rearranged into 1 row and 9 columns to form the column vector w of the parameter W.

[0067] An asterisk 703 indicates a correlation calculation. The correlation calculation unit 603 performs correlation calculation between the reference feature and the search feature for each channel. A schematic diagram of the search feature is shown at the bottom of FIG. 7. The search feature according to this embodiment is a tensor with 100 rows, 100 columns, and 16 channels, and represents the feature of the search image. Here, it is assumed that the search feature 704 in the i-th channel and the search feature 705 in the j-th channel are similar. The parameters of the reference feature corresponding to these search features are w (i) and w (j) and is subject to the orthogonal regularization process.

[0068] An example of the process performed by the selection unit 104 will be described using mathematical expressions. Here, F is a matrix of h×w rows and c columns of search features with h rows, w columns, and c columns, and F is normalized in the column direction. n Let us assume that F n Gram matrix G s is calculated in the same way as in equation (4). G s The i-th row and j-th column component of F n This can be considered as the similarity between the i-th channel and the j-th channel of the search feature. The mask matrix M can be obtained by equation (13).

number

[0069] Here, G s(ij) is G srepresents the i-th row and j-th column component of G. θ is a predetermined threshold (e.g., 0.95). In equation (13), the threshold θ is used to adjust the number of selections, but instead of using the threshold θ to perform the selection, s(ij) The ratio of (e.g. G s(ij) Top 10%) or number (e.g. G s(ij) A maximum of three of the above may be used.

[0070] In addition, the learning unit 102 uses the mask matrix M shown in equation (13) to calculate the orthogonal regularization term L orth Next, the learning unit 102 calculates the orthogonal regularization term L orth and the task error L in Eq. (12). task and calculate the error L by equation (9). The learning unit 102 according to this embodiment can perform parameter update processing based on the calculated error L, similar to S209 in the first embodiment. Here, the learning unit 102 updates the parameters of the convolutional neural network in the feature acquisition unit 105 by using a stochastic gradient descent method with a part of the learning data collected in S200 as a batch. In this embodiment, similar to the first embodiment, as described in S210, it is assumed that the learning termination condition is satisfied when the processing has been executed a predetermined number of times.

[0071] [Application example: Selection based on similarity with features of the tracking target detection unit] Consider a case where the selection unit 104 selects parameters corresponding to features with low similarity between the feature amount acquired by the correlation calculation unit 603 and the feature amount acquired by the tracking target detection unit 604. As described above, the tracking target detection unit 604 according to this embodiment detects the tracking target by a convolutional neural network. Hereinafter, the feature amount acquired by the correlation calculation unit 603 is a tensor with a width of 100, a height of 100, and 16 channels, and the output of the intermediate layer of the tracking target detection unit 604 is a tensor with a width of 100, a height of 100, and 8 channels. When the width and height of the output of the intermediate layer are different from the width and height of the output of the correlation calculation unit 603, the output of the correlation calculation unit 603 may be scaled using, for example, bilinear interpolation. U is a 100×100 row, 16 column matrix with each channel of the output of the correlation calculation unit 603 as a column vector. Similarly, V is a 100×100 row, 8 column matrix with each channel of the output of the intermediate layer of the tracking target detection unit 604 as a column vector. Moreover, these matrices are normalized in the column direction to Un and Vn, respectively. The similarity between the feature amount obtained by the correlation calculation unit 603 and the feature amount obtained by the tracking target detection unit 604 may be calculated by, for example, the following formulas (14) and (15).

number

[0072] Here, out_ch is the number of channels of the search feature, and in this embodiment, in_ch=out_ch=16. i is a row vector, and the number of elements is equal to the number of channels of U. The max function extracts the maximum element value from the vector. Each element of SIM indicates the similarity between the column vectors of V and U, and takes a value close to 1 if there is at least one column vector similar to V in U, and takes a value close to -1 if there is no column vector similar to V in U. A predetermined similarity threshold is SIMth (e.g., 0.5). Let s be the set of indices of elements of SIM that are smaller than SIMth. The set s is applied to equation (5) to calculate the mask matrix M, and orthogonal regularization is applied to the reference feature.

[0073] [Application example; Select when the reference feature is a special feature] When the reference features satisfy a condition that the reference features are to be a predetermined special value, it is considered to orthogonalize them. Here, for example, when the number of components of the reference features that have the same value satisfies a predetermined condition (for example, all components have the same value), the values ​​may be determined to be special values. Also, for example, when the number of components of the reference features that have a value of 0 satisfies a predetermined condition (for example, all components have a value of 0), the values ​​may be determined to be special values. Also, for example, when the number of components of the reference features that have a negative value satisfies a predetermined condition (for example, all components have a negative value), the values ​​may be determined to be special values. Hereinafter, a set s is defined as an index set of column vectors that satisfy the condition of being a special value among the column vectors of the parameter W of the reference feature. The set s is applied to the formula (5) to calculate the mask matrix M, and orthogonal regularization can be applied to the reference feature.

[0074] [Effect of the second embodiment: Effective selection of features to be regularized] The information processing device 100 according to the second embodiment can effectively select parameters to be regularized among the parameters of the machine learning model. Here, an example of applying orthogonal regularization to the reference features has been described, but the present invention is not limited to such processing. Even if the reference features are similar to each other, if the search features are different, the result of the correlation calculation will be different, and a variety of features can be expressed. According to the processing described in this embodiment, it is possible to effectively perform orthogonal regularization by focusing on channels in which the reference features and the search features are similar to each other.

[0075] As an application example, a mode focusing on the similarity between the result of the correlation calculation and the output of the convolutional neural network of the tracking target detection unit has been described. As described above, the feature acquisition unit 105 and the pattern recognition unit 106 according to this embodiment perform each inference process using a convolutional neural network. The features appearing in the intermediate layer of the convolutional neural network indicate important features for solving the target task, and generally, the closer to the input layer, the more general the features are, and the closer to the output layer, the more complex the features (features that combine general features). By orthogonalizing the reference features corresponding to the features not appearing in the subsequent convolutional layers among the features of the correlation calculation result, it is possible to effectively acquire a variety of features.

[0076] Furthermore, as an application example, we explained an example of orthogonalizing reference features when they satisfy special conditions. When each element of the reference feature is the same value, the correlation calculation can be considered as a simple averaging process, and effective feature extraction is not possible. In addition, for a channel where each element of the reference feature is all 0, the features corresponding to that channel will all be 0. If we assume that the search feature is non-negative and use the widely used ReLU as the activation function, a channel where each element of the reference feature is all negative will also have all 0 features, just like when all elements are 0. As such, when the reference feature satisfies special conditions, important features cannot be extracted, so by orthogonalizing the corresponding reference features, we can effectively acquire diverse features. [Embodiment 3] In the above-described embodiments, each processing unit shown in, for example, FIG. 1 is realized by dedicated hardware. Some or all of the processing units included in the information processing device 100 may be realized by a computer. In this embodiment, at least a part of the processing according to each of the above-described embodiments is executed by a computer.

[0077] FIG. 8 is a diagram showing the basic configuration of a computer. In FIG. 8, a processor 801 is, for example, a CPU, and controls the operation of the entire computer. A memory 802 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 803 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. for the long term. In this embodiment, a program that realizes the functions of each unit, which is stored in the storage medium 803, is read into the memory 802. Then, the processor 801 operates according to the program on the memory 802, thereby realizing the functions of each unit.

[0078] 8, an input interface 804 is an interface for acquiring information from an external device. An output interface 805 is an interface for outputting information to an external device. A bus 806 connects the above-mentioned units and enables data exchange.

[0079] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) A calculation means for calculating an error between an output of a machine learning model to which learning data is input and correct answer data for the learning data; A regularization means for regularizing a part of the parameters of the machine learning model; An update means for updating parameters of the machine learning model based on the calculated error and the regularized parameters; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein a portion of the parameters is a predetermined number of parameters randomly selected from the parameters of the machine learning model. (Item 3) 2. The information processing device according to item 1, wherein some of the parameters are a preselected set of parameters of the machine learning model. (Item 4) Item 1. The information processing device according to item 1, further comprising a first selection means for selecting a portion of the parameters by a Hadamard product using a matrix that selects parameters to be regularized for a matrix indicating the parameters of the machine learning model. (Item 5) 2. The information processing device according to item 1, wherein the regularization means performs regularization so that a portion of the parameters is orthogonalized. (Item 6) The tracking device further includes an estimation unit that estimates a position and a size of the tracking target in the search image by inputting a search image for tracking the tracking target and a reference image in which the tracking target is captured, 2. The information processing device according to item 1, wherein the regularization means regularizes a portion of parameters of a machine learning model that estimates the position and size of the tracking target. (Item 7) The method further includes an evaluation unit that evaluates a correlation between features of the search image and the reference image by a depth-unit separable convolution process using the search image and the reference image as inputs, 7. The information processing device according to item 6, wherein the estimation means estimates a position and a size of a tracking target in the search image based on the correlation. (Item 8) The image processing device further includes a determination unit for determining whether the first search image and the second search image are similar to each other, The information processing device described in item 6, characterized in that the regularization means regularizes parameters of the machine learning model that correspond to features of reference images corresponding to the first search image and the second search image, respectively, that are determined to be similar. (Item 9) The information processing device described in item 6, characterized in that the regularization means regularizes a portion of parameters of a machine learning model that estimates the position and size of the tracking target, selected based on a similarity between features of the search image and features of a machine learning model that estimates the position and size of the tracking target when the search image is input. (Item 10) 7. The information processing device according to item 6, wherein the regularization means regularizes parameters of the machine learning model that correspond to the features of the reference image and have a number of components with the same value that satisfy a predetermined condition. (Item 11) 7. The information processing device according to item 6, wherein the regularization means regularizes parameters of the machine learning model that correspond to the features of the reference image and have a number of components that are 0 that satisfy a predetermined condition. (Item 12) 7. The information processing device according to item 6, wherein the regularization means regularizes parameters of the machine learning model that correspond to the features of the reference image and have a number of components that take negative values ​​that satisfy a predetermined condition. (Item 13) 2. The information processing device according to item 1, wherein the machine learning model performs inference processing using a convolutional neural network. (Item 14) A step of calculating an error between an output of a machine learning model to which learning data is input and correct answer data for the learning data; Regularizing a portion of the parameters of the machine learning model; updating parameters of the machine learning model based on the calculated error and the regularized parameters; An information processing method comprising: (Item 15) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 13.

[0080] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0081] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0082] 100: information processing device, 110: imaging device, 120: result output unit

Claims

1. A calculation means for calculating an error between an output of a machine learning model to which learning data is input and correct answer data for the learning data; A regularization means for regularizing a part of the parameters of the machine learning model; An update means for updating parameters of the machine learning model based on the calculated error and the regularized parameters; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the part of the parameters is a predetermined number of parameters randomly selected from the parameters of the machine learning model.

3. The information processing apparatus according to claim 1 , wherein a portion of the parameters is a preselected set of parameters of the machine learning model.

4. The information processing device according to claim 1 , further comprising a first selection means for selecting a portion of the parameters by a Hadamard product using a matrix that selects parameters to be regularized for a matrix indicating parameters of the machine learning model.

5. The information processing apparatus according to claim 1 , wherein the regularization means performs regularization so that a part of the parameters is orthogonalized.

6. The tracking device further includes an estimation unit that estimates a position and a size of the tracking target in the search image by inputting a search image for tracking the tracking target and a reference image in which the tracking target is captured, The information processing device according to claim 1 , wherein the regularization means regularizes a part of parameters of a machine learning model that estimates the position and size of the tracking target.

7. The method further includes an evaluation unit that evaluates a correlation between features of the search image and the reference image by a depth-unit separable convolution process using the search image and the reference image as inputs, The information processing apparatus according to claim 6 , wherein the estimation means estimates a position and a size of the tracking target in the search image based on the correlation.

8. The image processing apparatus further includes a determining unit for determining whether the first search image and the second search image are similar to each other, The information processing device according to claim 6, characterized in that the regularization means regularizes parameters of the machine learning model that correspond to features of reference images corresponding to the first search image and the second search image, respectively, that are determined to be similar.

9. The information processing device according to claim 6, characterized in that the regularization means regularizes a portion of parameters of a machine learning model that estimates the position and size of the tracking target, selected based on a similarity between features of the search image and features of a machine learning model that estimates the position and size of the tracking target when the search image is input.

10. The information processing device according to claim 6 , wherein the regularization means regularizes parameters of the machine learning model that correspond to features of the reference image and have a number of components with the same value that satisfy a predetermined condition.

11. The information processing device according to claim 6 , wherein the regularization means regularizes a parameter of the machine learning model that corresponds to a feature of the reference image and has a number of components that are 0 that satisfies a predetermined condition.

12. The information processing device according to claim 6 , wherein the regularization means regularizes a parameter of the machine learning model that corresponds to a feature of the reference image and has a number of components that take negative values ​​that satisfy a predetermined condition.

13. The information processing device according to claim 1 , wherein the machine learning model performs inference processing using a convolutional neural network.

14. A step of calculating an error between an output of a machine learning model to which learning data is input and correct answer data for the learning data; Regularizing a portion of the parameters of the machine learning model; updating parameters of the machine learning model based on the calculated error and the regularized parameters; An information processing method comprising:

15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 13.

Images