Information processing device, information processing method, and program

Abstract

This enables the training of neural network architectures that achieve sufficient inference accuracy while preventing an increase in processing load. [Solution] An information processing device for learning an architecture to optimize the structure of a neural network, which generates multiple candidate edges for the neural network, inputs training data into the neural network to which weight coefficients are set for each of the candidate edges, and obtains an inference result. Here, the information processing device calculates the loss of the neural network based on a specified number of candidates to be selected from among the multiple candidates and the inference result, and updates the weight coefficients for the multiple candidates based on that loss. Then, the information processing device selects a candidate from among the multiple candidates based on the updated weight coefficients.

Description

[Technical Field]

[0001] This invention relates to an information processing technique for learning the architecture of a neural network. [Background technology]

[0002] In recent years, machine learning technologies, particularly Deep Learning, have seen rapid development in areas such as image recognition, speech recognition, and even machine translation. Most existing neural network architectures are still manually created by experts based on their knowledge and experience. While manually created neural networks can achieve high inference accuracy, exploring these architectures is extremely time-consuming, and difficult for non-experts. Recently, however, there has been a surge in research on Neural Architecture Search (NAS), a framework for automatically exploring neural network architectures. For example, Zoph et al.'s "Neural Architecture Search with Reinforcement Learning" utilizes a reinforcement learning framework to explore architectures. Specifically, it uses a Controller RNN (Convolutional Recurrent Neural Network) to explore the structure of a Child Network and generate the optimal Child Network for the task. The accuracy of the generated Child Network on validation data is then used as a reward, and the Controller RNN is updated based on a policy gradient method. However, this method consumes a large amount of computing resources and requires several days to several weeks for training, resulting in significant implementation costs.

[0003] Furthermore, as a more efficient method for learning the architecture of a neural network, Liu et al. propose a method that makes the space for searching the architecture differentiable by treating it as a continuous space and performs optimization using gradient descent. Liu et al., "DARTS: Differentiable Architecture Search". In this way, by enabling search using gradient descent, it is possible to optimize the architecture of a neural network in a few days to a few days. In addition, Non-Patent Literature 1 (FBNet) proposes an architecture search method that balances accuracy and speed by considering not only accuracy but also latency during network inference when optimizing the architecture. In the method shown in Non-Patent Literature 1, when learning the architecture of a neural network, the weight of selected edge candidates becomes large and the weight of unselected edge candidates becomes small. This is achieved by a temperature-controlled Softmax function called Gumbel-Softmax. In this case, only one edge candidate is selected. Thus, in a NAS based on gradient descent (referred to as gradient-based NAS), weights are assigned to multiple edge candidates existing between nodes, and the edge candidate with the highest weight is selected. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Wu Bichen etal, "FBNet:Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search", The IEEE Conference оn Computer Vision and Pattern Recognition 2019. [Overview of the project] [Problems that the invention aims to solve]

[0005] In the case of the technique of automatically searching for a neural network architecture in the gradient-based NAS described above, although a certain level of inference accuracy can be achieved, the processing amount during inference tends to increase. Therefore, it is desired to be able to achieve sufficient inference accuracy while preventing an increase in the processing amount during inference.

[0006] Therefore, an object of the present invention is to enable learning of a neural network architecture that realizes sufficient inference accuracy while preventing an increase in the processing amount.

Means for Solving the Problems

[0007] The present invention is an information processing apparatus that learns an architecture for optimizing the structure of a neural network, and includes candidate generation means for generating a plurality of candidates for edges of the neural network, inference means for inputting learning data into the neural network in which weight coefficients are set for each candidate of the edge to obtain an inference result, loss calculation means for calculating the loss of the neural network based on a designated candidate number as the number of candidates to be selected from the plurality of candidates and the inference result, update means for updating the weight coefficients for the plurality of candidates based on the loss, and selection means for selecting candidates from the plurality of candidates based on the updated weight coefficients.

Effects of the Invention

[0008] According to the present invention, it is possible to learn a neural network architecture that realizes sufficient inference accuracy while preventing an increase in the processing amount.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram showing a schematic hardware configuration example of an information processing apparatus. [Figure 2] It is a functional block diagram showing a functional configuration example of an information processing apparatus. [Figure 3]It is a flowchart showing the flow of information processing according to the first embodiment. [Figure 4] It is a diagram showing an example of a network architecture according to the first embodiment. [Figure 5] It is a diagram showing the configuration of a Layer according to the first embodiment. [Figure 6] It is a diagram showing an example of four types of networks acquired by learning. [Figure 7] It is a diagram showing examples of an input image, a GT map, and an inferred map. [Figure 8] It is a diagram used to explain the weight coefficient and the change by learning. [Figure 9] It is a flowchart showing the flow of information processing according to the second embodiment. [Figure 10] It is a diagram showing an example of a network architecture according to the second embodiment. [Figure 11] It is a diagram showing the configuration of a Layer according to the second embodiment. [Figure 12] It is a diagram used to explain a template image, a search target image, a tracking target, etc. [Figure 13] It is a diagram used to explain pruning (branch pruning).

Mode for Carrying Out the Invention

[0010] <000012​​​​In this embodiment, as an example, we will describe an information processing device that implements the function of a neural network architecture search device that learns an architecture to optimize the structure of a neural network. Before describing the detailed configuration and operation of the information processing device, we will describe an overview of the neural network architecture assumed in this embodiment. In this embodiment, we will give an example of learning the architecture of a neural network using a gradient-based NAS (NAS) method based on gradient descent.

[0012] In gradient-based NAS, it is important that the difference between the weights of edges to be selected and those not to be selected increases as training progresses. In other words, if the difference in weights between candidate edges to be selected and candidate edges not to be selected is sufficiently large, the change in the architecture after edge selection will be small, and the impact on the inference accuracy of the neural network will be small. On the other hand, if the difference in weights between candidate edges to be selected and candidate edges not to be selected is small (they are similar values), the neural network architecture may change significantly when edge selection is performed. As a result, the inference accuracy of the neural network after training may decrease.

[0013] In this embodiment, when training the neural network architecture using the gradient-based NAS method, multiple edge candidates are selected, not just one. One possible method for selecting multiple edges (candidates) is to select them in descending order of weight. However, in this case, if the difference between the weights of each selected edge (candidate) and the weights of the unselected edges (candidates) is small, the network architecture may change significantly, as described above, potentially leading to a decrease in inference accuracy.

[0014] Therefore, in this embodiment, when training the neural network architecture using the gradient-based NAS method, even when selecting multiple candidates, the difference in weights between selected and unselected candidates is increased to suppress the decrease in accuracy during inference. Furthermore, the fewer edges a neural network has, the faster and more memory-efficient it becomes. For this reason, in this embodiment, speed and memory efficiency are achieved by selecting a specified number of candidates from all edges (candidates) to reduce the number of candidates.

[0015] <First Embodiment> In the first embodiment, we will describe a learning method for a neural network architecture in an object detection task that detects the position and size of objects from an image. Figure 1 shows a schematic hardware configuration example of the information processing device 1 of this embodiment, which learns the architecture of a neural network. In the configuration shown in Figure 1, the CPU 101 controls the entire device by executing a control program stored in the ROM 102. The ROM 102 also stores an information processing program according to this embodiment, and the CPU 101 executes this program to perform the process of learning the neural network architecture, which will be described later. The RAM 103 temporarily stores various data from each component. The RAM 103 also unpacks the program stored in the ROM 102, making it executable by the CPU 101. The storage unit 104 stores the data to be processed in this embodiment and stores the data used for learning, which will be described later. The storage unit 104 can use an HDD, flash memory, various optical media, etc. Note that the information processing program according to this embodiment may also be stored in the storage unit 104.

[0016] Figure 2 is a functional block diagram showing an example of the functional configuration of the information processing device 1. Each functional part of the information processing device 1 shown in Figure 2 is formed, for example, by executing the information processing program of this embodiment in the CPU 101. Note that some or all of each functional part may be realized by hardware configurations such as circuits. Figure 3 is a flowchart showing the flow of information processing in the information processing device 1 according to the first embodiment, that is, the flow of the learning process of the neural network architecture. In the first embodiment, the information processing device 1 learns the architecture of a neural network for an object detection task that detects objects from an image.

[0017] The image acquisition unit 201 acquires images stored in the memory unit 208. The images contain objects to be detected, such as people or cars. In the example shown in Figure 2, the memory unit 208 is located outside the information processing device 1, but it may also be located inside the information processing device 1. The image data acquired by the image acquisition unit 201 is used to train the neural network architecture. The Ground Truth (GT) acquisition unit 202 retrieves data on the position and size of objects in the image acquired by the image acquisition unit 201 from the memory unit 208. The position and size data of objects acquired by the GT acquisition unit 202 is used when training the neural network architecture. In other words, the information processing device 1 includes an image acquisition unit 201 and a GT acquisition unit 202, which are functional units that acquire image data and data on the position and size of objects within that image as training data used when learning the architecture of a neural network.

[0018] The candidate generation unit 203 generates edges (candidates) for the neural network. Figure 4 shows an example of the neural network architecture generated by the candidate generation unit 203. As shown in Figure 4, the network is composed of Layer 1 (401), Layer 2 (402), Layer 3 (403), Layer 4 (404), and Layer 5 (411). The outputs of each of Layers 1 to 4 are weighted 405 to 408, then combined in the channel direction in the joining process 409 and input to Layer 5 (411). In the example in Figure 4, each output of Layers 1 to 4 is an edge (candidate), and the information processing device 1 of this embodiment searches for the optimal combination of edges from these edges (candidates).

[0019] Here, in weightings 405, 406, 407, and 408, the corresponding outputs of Layer 1 to Layer 4 are weighted using weight coefficients. The weight coefficient is a value that represents the importance of a Layer, and the higher the importance of a Layer, the larger the weight coefficient. The weight coefficient is a parameter that determines the architecture of the neural network, and the architecture of the neural network is determined by the magnitude of the weight coefficient. The weight coefficient is acquired through the training of the neural network, which will be described later. In other words, the training of the neural network generates weight coefficients that indicate the importance of each edge (each candidate).

[0020] Each layer in Figure 4 consists of a convolution 501, a batch-normalization 502, and a rectified linear unit 503, as shown in Figure 5. Hereafter, the rectified linear unit will be referred to as ReLU. The configuration example in Figure 5 is merely an example; Leaky ReLU or the Sigmoid function may be used instead of ReLU, or a combination of MaxPooling and AveragePooling may be used. This embodiment is not limited to these.

[0021] The selection number specification unit 204 specifies the number of candidates (edges) to be selected from all edges (candidates) in the neural network. In this embodiment, the number of candidates specified by the selection number specification unit 204 is called the "specified number of candidates". The specified number of candidates by the selection number specification unit 204 is predetermined based on the speed requirements for the neural network and the memory requirements available to the neural network. Generally, the fewer candidates there are, the faster and more memory-efficient the neural network becomes. Therefore, in this embodiment, by selecting a specified number of candidates from all edges (candidates) and reducing the number of candidates, it is possible to achieve higher speed and lower memory usage.

[0022] In the example shown in Figure 4, the specified number of candidates corresponds to the number of layers from Layer 1 to 4 whose outputs are input to Layer 5. For example, if the number of candidate selections is specified as 3, the outputs of any three layers from Layer 1 to 4 will be input to Layer 5. That is, if the number of candidate selections is specified as 3 for the four layers from Layer 1 to 4, the network acquired through the learning of the neural network architecture will be one of the four types of networks shown in Figures 6(a) to 6(d). Note that weights 405, 406, 407, and 408 are omitted in Figures 6(a) to 6(d). For example, in Figure 6(a), the outputs of three layers from Layer 1, Layer 2, and Layer 3 are combined in the merging process 409 and then input to Layer 5. Similarly, in Figure 6(b), the outputs of Layer 1, Layer 3, and Layer 4 are combined and input to Layer 5. In Figure 6(c), the outputs of Layer 1, Layer 2, and Layer 4 are combined, and in Figure 6(d), the outputs of Layer 2, Layer 3, and Layer 4 are combined and input to Layer 5. Which of these networks in Figures 6(a) to 6(d) is formed is determined by the magnitude of the weight coefficients in weights 405 to 408, and the outputs of the three layers with the largest weight coefficients from Layer 1 to Layer 4 are combined and input to Layer 5.

[0023] The inference unit 205 inputs the image acquired by the image acquisition unit 201 as training data into a neural network as shown in Figure 4, and obtains the output of the inference result from that neural network. For example, if the image 701 shown in Figure 7(a) is input as training data, the inference unit 205 may obtain an output of the inference result as a map 703, such as the one shown in Figure 7(c). That is, if the person in the image 701 in Figure 7(a) is the detection target 702, the output of the inference result from the inference unit 205 will be an inference map 703 as shown in Figure 7(c). In the inference map 703 exemplified in Figure 7(c), the value of the centroid position 704 corresponding to the detection target 702 in Figure 7(a) is 1, and all other values ​​are 0. Note that the detection target (person) is also drawn in the inference map 703 in Figure 7(c) to make the positional relationship with the detection target 702 in Figure 7(a) easier to understand.

[0024] The loss calculation unit 206 calculates the loss (loss function) based on the output of the inference result by the neural network in the inference unit 205 and the Ground Truth (GT) obtained from the GT acquisition unit 202. If the image acquired by the image acquisition unit 201 is an image 701 as shown in Figure 7(a), the Ground Truth will be a map 706 as shown in Figure 7(b). That is, in the GT map 706 of Figure 7(b), the value of the position corresponding to the centroid position of the detected target 702 (centroid position of the person) in the image 701 of Figure 7(a) is set to 1, and all other values ​​are set to 0. Note that the detected target (person) is also drawn in the GT map 706 of Figure 7(b) to make the positional relationship with the detected target 702 of Figure 7(a) easier to understand. In this embodiment, since the neural network architecture is learned based on the specified number of candidates, the loss calculation unit 206 calculates the loss using the output of the inference result and GT, as well as the information on the specified number of candidates. The details of this will be described later.

[0025] The update unit 207 updates the neural network parameters based on the loss calculated by the loss calculation unit 206, and stores the updated parameters in the storage unit 208. Here, the parameters of a neural network can be divided into two types: parameters related to the architecture of the neural network and the weights of the elements that make up the neural network, such as convolutions. In the case of Figure 4 mentioned above, the parameters related to the architecture of the neural network are the weight coefficients of 405 to 408.

[0026] The selection unit 209 selects each candidate (edge) based on the weight coefficient of the edge (candidate). In this embodiment, taking the example of Figure 4 mentioned above, the selection unit 209 selects each candidate corresponding to the specified number of top candidates when the outputs of Layer 1 to Layer 4 are sorted in descending order based on the weight coefficient values. That is, it selects each candidate up to the top several candidates corresponding to the specified number of candidates. In the information processing device 1 of this embodiment, the selected candidates from the selection unit 209 are adopted to form the architecture of the neural network.

[0027] The learning process flow of the neural network architecture, which is the information processing of this embodiment, will be explained in detail below, with reference to the flowchart in Figure 3. In the following flowcharts, the notation for each step will be omitted by prefixing each step with an "S". Note that the information processing device 1 does not necessarily have to perform all the steps described in the following flowcharts.

[0028] In S301, the candidate generation unit 203 generates a neural network architecture. Figure 4, mentioned above, shows an example of an architecture generated by the candidate generation unit 203. As mentioned above, in the architecture shown in Figure 4, weights 405 to 408 are applied to the outputs of Layer 1, Layer 2, Layer 3, and Layer 4. The magnitude of the weight coefficients at this time determines the network architecture. Here, the output of the i-th layer (i=1 to 4) is o i The weight coefficient for the i-th Layer is α i Therefore, the weighted output of the i-th Layer o i' can be expressed as in equation (1).

[0029] o i (x)=α i 'o i (x) Equation (1)

[0030] α in equation (1) i ' is a coefficient expressed using the Softmax function based on the weight coefficients of the candidate Layers 1 to 4, as shown in equation (2) below. The reason for using the Softmax function in this way is to set the range of the weight coefficients to [0,1].

[0031]

number

[0032] Next, in S302, the selection number specification unit 204 sets the number of candidates to be selected from among the candidates (edges) of the neural network (specified candidate number). In the case of Figure 4 illustrated in this embodiment, the candidates are the outputs of Layer 1, Layer 2, Layer 3, and Layer 4, and the selection number specification unit 204 specifies the number of Layers from the outputs of Layer 1 to Layer 4 that should be input to Layer 5. For example, if the specified candidate number is set to 3, there are 4C3 possible structures for the neural network obtained as a result of training, as shown in Figure 6.

[0033] Next, in S303, the image acquisition unit 201 acquires the image stored in the storage unit 208. In this embodiment, the image 701 in Figure 7(a) mentioned above is given as an example of the image acquired at this time. The image 701 contains the object designated as the detection target 702 (in this example, a person).

[0034] Next, in S304, the GT acquisition unit 202 acquires the GroundTruth (GT) stored in the storage unit 208. In this embodiment, when the image acquisition unit 201 acquires an image 701 as shown in Figure 7(a), the GroundTruth is the GT map 706 as shown in Figure 7(b). That is, in the GT map 706, the value of the centroid position 707 (centroid position of the person) of the detected target 702 in the image 701 of Figure 7(a) is set to 1, and all other values ​​are set to 0.

[0035] Next, in S305, the inference unit 205 inputs the image acquired in S303 into the neural network shown in Figure 4 to obtain the inference result. As mentioned above, if the image 701 in Figure 7(a) is acquired in S303, the inference result by the neural network will be an inference map 703 like the one in Figure 7(c) that corresponds to the image 701 in Figure 7(a). As learning progresses, the value of the centroid position of the object to be detected will be inferred to be 1, and all other values ​​will be 0.

[0036] Next, in S306, the loss calculation unit 206 calculates the loss (loss function) based on the inference result obtained in S305, the GroundTruth (GT) obtained in S305, and the number of candidates specified in S302. Here, the loss calculation unit 206 calculates the following two types of losses. Loss of neural network inference results Loss related to the architecture of neural networks

[0037] First, let's explain the loss on the inference results of a neural network. This embodiment provides an example of training a neural network for an object detection task. Therefore, the training must enable the neural network to correctly detect the position of the object to be detected. For example, in the example shown in Figure 7 above, the neural network is trained so that the inference map 703 in Figure 7(c), which is the output of the neural network, approaches the GroundTruth GT map 706 shown in Figure 7(b). Here, let the output of the neural network output from Layer5(411) be C inf , and let the GT map be C gt . Then, the loss Loss C with respect to the inference result of the neural network is calculated by the sum of squared errors for each pixel of the map as shown in Equation (3). Note that N in Equation (3) is the total number of pixels of the GT map C gt .

[0038]

Equation

[0039] By calculating the sum of squared errors in this way, when the value of the output C inf of the neural network deviates from the GT map C gt , the loss value increases. On the other hand, when C inf and C gt approach each other, the loss value decreases. Since learning proceeds in the direction of decreasing the loss, as learning progresses, the inference map, which is the output of the neural network, approaches the GT map of the GroundTruth.

[0040] For example, in the GT map 706 of Figure 7(b), the value at position 707 is a high value. Also, assume that in the inference map 703 of Figure 7(c), the value of the inference result at position 704 is a high value close to the value at position 707 of the GT map 706. Then, as shown in Figures 7(c) and 7(b), if the position 704 of the inference map 703 and the position 707 of the GT map 706 are, for example, at the same position, the loss value obtained from the calculation of Equation (3) will be small.

[0041] On the other hand, in the inference map 703 of Figure 7(c), for example, at position 705, the value of the inference result shows a high value, while at the corresponding position 708 of the GT map 706 in Figure 7(b), it is a low value. In this case, since there is a large difference in value between position 705 of the inference map 703 and the same position 705 of the GT map 706, the loss obtained from the calculation of Equation (3) will be large. In this embodiment, an example of calculating the sum of squared errors was given, but it is not limited to the sum of squared errors; for example, a loss function such as Cross-Entropy may also be calculated.

[0042] Next, we will explain the calculation of the loss related to the neural network architecture. In this embodiment, we search for the neural network architecture that is optimal for the object detection task. In this embodiment, the neural network architecture is determined by selecting which layer's output is selected from the outputs of Layer 1 to 4 exemplified in Figure 4, depending on the specified number of candidates. For example, if the specified number of candidates is 3, then 3 layers will be selected from the outputs of Layer 1 to 4, resulting in one of the four types of networks shown in Figures 6(a) to 6(d) above.

[0043] One possible method for selecting an output from Layers 1-4 based on a specified number of candidates is to prioritize selecting the output of Layers with larger weight coefficients in weightings 405-408. Figures 8(a) to 8(c) show examples of weight coefficients α1 to α4 for weights 405 to 408 corresponding to each output of Layers 1 to 4. Let α1 be the weight coefficient 405 corresponding to Layer 1, α2 be the weight coefficient 406 corresponding to Layer 2, α3 be the weight coefficient 407 corresponding to Layer 3, and α4 be the weight coefficient 408 corresponding to Layer 4.

[0044] In the initial state before training, as shown in Figure 8(a), the weight coefficients α1 to α4 for Layers 1 to 4 are all the same value. As training progresses, the weight coefficients of Layers that contribute to the accuracy of object detection become larger, while the weight coefficients of Layers that do not contribute become smaller. Figure 8(b) shows an example of how the weight coefficients α1 to α4 have changed as training has progressed. In the case where the weight coefficients α1 to α4 have changed as shown in Figure 8(b), if we were to select a Layer output based on the top three weight coefficients α4, α3, and α1, for example, the outputs of Layer 4, Layer 3, and Layer 1 would be selected.

[0045] In the example shown in Figure 8(b), the weight coefficients α4, α3, and α1 of the selected Layers 4, 3, and 1 are not significantly different from the weight coefficient α2 of the unselected Layer 2. If we were to exclude the output of Layer 2 and select only the outputs of Layers 4, 3, and 1 in this case, the architecture of the neural network could change significantly. In other words, if there is not much difference between the weight coefficients of the selected layers and the weight coefficients of the unselected layers, there is a high probability that the selected and unselected layers will be swapped during the learning process. Whether or not such a swap occurs, the architecture of the neural network generated through learning could change significantly. And if the architecture of the neural network changes significantly, the inference accuracy of object detection by the neural network will also fluctuate greatly and decrease, which is undesirable.

[0046] Therefore, when selecting multiple Layer outputs according to the specified number of candidates, it is desirable that the weight coefficients of the multiple Layer outputs corresponding to each candidate selected according to the specified number become sufficiently large as training progresses. On the other hand, it is desirable that the weight coefficients of the Layer outputs corresponding to each candidate that is not selected (other than the specified number of candidates) become sufficiently small. In other words, it is desirable that the difference between the weight coefficients of the multiple Layer outputs that are selected and the weight coefficients of the Layer outputs that are not selected becomes sufficiently large as training progresses. Specifically, as shown in Figure 8(c), it is desirable that the weight coefficients α4, α3, and α1 of the selected Layers 4, 3, and 1 become sufficiently large as training progresses, while the weight coefficient α2 of the unselected Layer 2 becomes sufficiently small compared to the weight coefficients α4, α3, and α1.

[0047] Therefore, in the information processing device 1 of this embodiment, when calculating the loss related to the neural network architecture, the loss calculation is performed in such a way that the weight coefficients of the Layers selected according to the specified number of candidates are increased, and the weight coefficients of the Layers that are not selected are decreased. Specifically, the loss calculation unit 206 first calculates the weight coefficient α of each layer. i Sort them in descending order. Next, the loss calculation unit 206 calculates the loss related to the neural network architecture, with the specified number of candidates being K. A This is calculated using equation (4).

[0048] Loss A =exp(-(α K -α K+1 ) 2 ) Formula (4)

[0049] Equation (4) is given by a weight coefficient α where K is the number of specified candidates. i The top K-th weight coefficient α when sorted in descending order. K and the next K+1th weight coefficient α K+1 The loss function is designed to decrease as the difference with the specified number of candidates increases, and conversely, to increase as the difference decreases. In other words, the loss calculation unit 206 calculates the loss in such a way that the loss value increases as the difference between the weight coefficient of each candidate with the top K weight coefficients (determined as the specified number of candidates) and the weight coefficients of the other candidates decreases. In other words, the loss calculation unit 206 calculates the loss in such a way that the loss value increases when the difference between the weight coefficient of each candidate with the top specified number of candidates and the weight coefficients of the other candidates becomes smaller than, for example, a predetermined threshold. That is, when the candidates are sorted in descending order according to their weights, the loss calculation unit 206 calculates the loss in such a way that the loss value increases when the difference between the weight of the K-th candidate from the top and the weight of the next K+1-th candidate becomes smaller than a threshold. On the other hand, since learning proceeds in the direction that the loss function decreases, as learning progresses, the K-th weight coefficient α K and the K+1th weight coefficient α K+1 The difference will become larger.

[0050] Furthermore, for example, if the selection unit 209 prioritizes the selection of candidates with larger weight coefficients, a difference may arise between the number of candidates that are likely to be selected by the selection unit 209 due to the large weight coefficients and the number of specified candidates. According to equation (4), when a difference arises between the number of specified candidates and the number of candidates selected by the subsequent selection unit 209, the loss is calculated in such a way that the loss value is high. Therefore, from these points, as learning progresses, the weight coefficients can be brought closer to the state shown in Figure 8(c).

[0051] The loss calculation unit 206 calculates two types of losses as described above: LossC, the loss on the neural network's inference results, and LossA, the loss related to the neural network's architecture. Then, the loss calculation unit 206 calculates the loss on the neural network's inference results using equation (5). C And, the loss related to the architecture of the neural network A By integrating these, the loss of the neural network is calculated. Here, λ in equation (5) is a weight with a value range of [0,1], and increasing λ increases the loss of the neural network's inference result. C The system converges faster, and by reducing λ, the architecture-related loss is reduced. A This converges more quickly. Note that λ is determined empirically.

[0052] Loss = λLoss C +(1-λ)Loss A Formula (5)

[0053] Let's return to the explanation of the flowchart in Figure 3. In S307, the update unit 207 updates the neural network parameters based on the loss calculated in S306. Here, there are two types of parameters: weights for elements such as convolutions in the neural network, and weights related to the architecture of the neural network, and the update unit 207 updates both. The parameter updates are performed using backpropagation, such as Momentum SGD. In the example above, the output of the loss function for a single image was described, but in actual training, the loss value of equation (5) is calculated for multiple different images. Therefore, the update unit 207 updates the neural network parameters so that the loss values ​​for multiple images are all smaller than a predetermined threshold.

[0054] Next, in S308, the update unit 207 saves the updated neural network parameters to the storage unit 208. Then, in S309, the update unit 207 determines whether to terminate the learning process. In determining whether to terminate the learning process, the process may be terminated when the loss value obtained by equation (5) becomes smaller than a predetermined threshold and the learning process converges, or when a predetermined number of learning sessions have been performed. If it is determined in S309 to terminate the learning process, the information processing device 1 proceeds to S310; otherwise, it returns to S303.

[0055] When the process proceeds to S310, the selection unit 209 selects the top K outputs from Layer1 to Layer4 based on their weight coefficient values, forming the architecture of the neural network. For example, in the configuration shown in Figure 4, if the number of specified candidates K is 3, and the weight coefficient values ​​are arranged in descending order, then Layer4, Layer1, Layer3, and Layer2 in descending order of weight coefficient values, the selection unit 209 selects the outputs of the top three layers. In this case, the selection unit 209 selects the outputs of Layer4, Layer1, and Layer3.

[0056] As described above, in the information processing device 1 of the first embodiment, candidates are selected based on a specified number of candidates, and the parameters of the neural network are updated while calculating the loss function shown in equation (4). As a result, the difference between the weight coefficients of the selected candidates and the weight coefficients of the unselected candidates becomes sufficiently large according to the number of candidates selected. In other words, according to this embodiment, a high-speed and memory-efficient neural network can be realized, and even if the output of unselected candidates is excluded at the end of training, a neural network architecture capable of high-accuracy object detection can be learned.

[0057] <Second Embodiment> In the second embodiment, a learning method for the architecture of a neural network in an object tracking task that detects a specific target from an image will be described. In this embodiment, an example of learning the tracking task according to the method of Bertinetto, et al., "Fully-Convolutional Siamese Networks for Object Tracking" will be given. In the second embodiment, the functional configuration of the information processing device 1 is the same as in Figure 2 described above, so its illustration and description will be omitted.

[0058] Figure 9 is a flowchart showing the information processing flow in the information processing device 1 according to the second embodiment, that is, the learning process flow of the neural network architecture.

[0059] In S901, the candidate generation unit 203 in the second embodiment generates a neural network architecture as shown in Figure 10. The network illustrated in Figure 10 consists of Layer 1 (1001), Layer 2 (1002), Layer 3 (1003), and Layer 4 (1004).

[0060] Each of these Layers consists of Convolutions with different kernel sizes, as shown in Figure 11. For example, in Convolutional ReLU1101, a nonlinear transformation such as ReLU is performed after a Convolution with a kernel size of 1x1. Similarly, in Convolutional ReLU1102, a nonlinear transformation is performed after a Convolution with a kernel size of 3x3, and in Convolutional ReLU1103, a nonlinear transformation is performed after a Convolution with a kernel size of 5x5. The outputs of these Convolutional ReLU1101 to 1103 are edges (candidates), and in this embodiment, the optimal combination of edges (candidates) is obtained by learning the neural network architecture.

[0061] Weighting 1104 applies weighting to the output of convolution ReLU1101. Similarly, weighting 1105 applies weighting to the output of convolution ReLU1102, and weighting 1106 applies weighting to the output of convolution ReLU1103. The method for weighting the candidates is the same as in equation (1) described above. Then, in addition 1107, the outputs of weights 1104 to 1106 are added together, and the output of this addition is input to the next layer.

[0062] In the configuration shown in Figure 11, by learning the weight coefficients 1104 to 1106, it becomes possible to select one or more candidates that contribute to the accuracy of the tracking task from the outputs of the candidate convolutional ReLU 1101, 1102, and 1103, respectively. Note that the candidate types of convolutions do not have to be those with different kernel sizes as listed here. For example, the number of convolutional groups can be changed to create candidates. Furthermore, the candidates are not limited to convolutions; for example, the type of activation function (ReLU, Leaky ReLU, ELU, etc.) or the type of pooling (MaxPooling, AveragePooling, etc.) can also be used.

[0063] Let's return to the explanation of the flowchart in Figure 9. In S902, the selection number specification unit 204 sets the candidate selection number, which is the number of candidates to be selected. In this embodiment, the selection number specification unit 204 specifies the number of candidates to be selected (candidate selection number) from the three candidates (outputs of convolutional ReLU 1101, 1102, and 1103) present in each of Layer 1 to Layer 4 shown in Figures 10 and 11. For example, if the candidate selection number is set to 2, when training is complete, two candidates will be selected, such as the outputs of convolutional ReLU 1101 and 1102, or the outputs of convolutional ReLU 1101 and 1103.

[0064] After S902, the information processing device 1 performs the processes from S903 to S905 and the processes from S906 to S908. When the process proceeds to S903, the image acquisition unit 201 acquires an image containing the tracking target as a template image. At the same time, the GT acquisition unit 202 acquires Ground Truth (GT), such as the position and size of the tracking target present in the template image.

[0065] Figure 12(a) shows an example of a template image 1201. The template image 1201 is acquired by the image acquisition unit 201, and the tracking target 1203 is present within the image. A bounding box (BB) 1204 indicating the position and size of the tracking target 1203 is also set for this template image 1201.

[0066] Furthermore, in 904, the image acquisition unit 201 extracts and resizes the image around the tracked target within the template image based on the position and size of the tracked target obtained by the GT acquisition unit 202. As a method for extracting the image around the tracked target, a method can be used in which the image is extracted with the position of the tracked target as the center and in a constant multiple of the size of the tracked target. The region 1202 shown in Figure 12(a) is an example of a region from which the image around the tracked target has been extracted.

[0067] Next, in S905, the inference unit 205 inputs the image extracted from the template image in S904 into the neural network to obtain features of the target to be tracked. After S905, the information processing device 1 proceeds to S909.

[0068] Next, in S906, the image acquisition unit 201 acquires a search target image to be used to search for the tracking target. For example, the image acquisition unit 201 acquires an image from a different time point in the same sequence as the template image acquired in S903 as the search target image for searching for the tracking target. Figure 12(b) shows an example of a search target image 1205. In the search target image 1205 exemplified in Figure 12(b), the tracking target 1207 is present. A bounding box (BB) 1208 indicating the position and size of the tracking target 1207 is also set for the search target image 1205.

[0069] Next, in S907, the image acquisition unit 201 crops the image of the area around the tracked target from the search target image as a search range, based on the position and size of the tracked target obtained by the GT acquisition unit 202, and resizes it. As a method for cropping the area around the tracked target as a search range, a method can be used in which the area is cropped at a constant multiple of the size of the tracked target, with the position of the tracked target as the center. The search target image 1205 shown in Figure 12(b) shows an example of the tracked target 1207, BB1208 representing the position and size of the tracked target 1207, and the search range 1206 cropped around the tracked target.

[0070] Then, in S908, the inference unit 205 inputs the image of the search area, which was extracted from the image to be searched in S907, into the neural network to obtain the features of the target to be tracked. After S908, the processing of the information processing device 1 proceeds to S909.

[0071] Proceeding to S909, the inference unit 205 calculates the cross-correlation between the features of the target to be tracked obtained from the template image in S905 and the features of the target to be tracked obtained from the search target image in S906, and infers the position of the target to be tracked in the search target image. Here, the output of the cross-correlation calculation is an inference map, for example, as explained in Figure 7(c) above, and if the value of the position where the target to be tracked (in this example, a person) exists is 1 and all other values ​​are 0, then tracking can be performed correctly.

[0072] Next, in S910, the loss calculation unit 206 calculates the loss of the neural network. Here, as described in the first embodiment above, two types of losses are calculated: "loss for the inference result of the neural network" and "loss related to the architecture of the neural network". However, since the second embodiment is an example of learning the task of object tracking, the process of calculating these two types of losses is slightly different from that of the first embodiment above.

[0073] First, let's explain the loss on the neural network's inference results in the second embodiment. In the second embodiment, we use the example of learning the task of tracking an object. Therefore, through learning, the neural network needs to be able to correctly detect the position of the object to be tracked.

[0074] Therefore, in S910, the loss calculation unit 206 performs a loss calculation for learning such that the inference map in Figure 7(c), which is an example of the output of the neural network, approaches the GT map in Figure 7(b). For example, the output of the neural network output from Layer 4 (1004) in Figure 10 is C inf , GT map C gt Therefore, the loss on the neural network's inference result is C This is calculated by the sum of squared errors for each pixel of the map, as shown in equation (3) above.

[0075] Next, we will explain the calculation of loss related to the neural network architecture. In the second embodiment, we provide an example of searching for the optimal neural network architecture for an object tracking task. In this case, the network architecture is determined by which of several types of convolution (convolutional ReLU 1101, 1102, and 1103, as exemplified in Figure 11) is selected for each layer.

[0076] One method for selecting convolution candidates is to prioritize the output of convolutions with large weight coefficients in the weightings 1104-1106 shown in Figure 11. In the second embodiment, as with the first embodiment described above, it is desirable that there be a sufficient difference between the weight coefficients of the selected candidates and the weight coefficients of the unselected candidates after the network architecture has been trained. Therefore, in calculating the loss related to the neural network architecture, the loss calculation is performed with the aim of increasing the weight coefficients of the selected convolutions and decreasing the weight coefficients of the unselected convolutions.

[0077] Specifically, the loss calculation unit 206 first calculates the weight coefficient α of each layer. i Sort them in descending order. Next, the loss calculation unit 206 calculates the loss related to the neural network architecture, with the number of candidate selections being K. A Calculate it as shown in equation (6).

[0078]

number

[0079] Equation (6) is given by a weight coefficient α where K is the number of specified candidates. iWhen sorted in descending order, if the weight coefficients of the top 1 to K elements approach a sufficiently large value A, and the values ​​of the weight coefficients from the K+1th element onward approach 0, then the loss will be small. Since learning proceeds in the direction of minimizing loss, after the network architecture has finished learning, the values ​​of the selected weight coefficients will approach a sufficiently large value, and the values ​​of the unselected weight coefficients will approach 0. As a result, even if the unselected convolution candidates are eliminated, it becomes possible to learn a neural network architecture that can achieve highly accurate object tracking.

[0080] Subsequently, the loss calculation unit 206 calculates the loss for the neural network inference result using the aforementioned equation (5). C And, the loss related to the architecture of the neural network A The results are then combined to calculate the loss of the neural network.

[0081] Let's return to the explanation of the flowchart in Figure 9. After S910, proceeding to S911, the update unit 207 updates the neural network parameters based on the loss calculated in S910, similar to the process in S307 in the first embodiment. Furthermore, in S912, the update unit 207 stores the updated neural network parameters in the memory unit 208.

[0082] Then, in S913, the update unit 207 determines whether to terminate the learning process. In determining whether to terminate the learning process, the process may be terminated if the loss value obtained by equation (5) falls below a predetermined threshold, or if a predetermined number of learning sessions have been performed. If it is determined in S913 to terminate the learning process, the information processing device 1 proceeds to S914; otherwise, it returns to S903 and S904.

[0083] Then, proceeding to S914, the selection unit 209 selects candidates based on the learned weight coefficients.

[0084] According to the second embodiment, by updating the neural network parameters while calculating the loss function shown in equation (6), the difference between the weight coefficients of selected candidates and the weight coefficients of unselected candidates becomes sufficiently large depending on the number of candidates selected. In other words, according to this embodiment, a high-speed and memory-efficient neural network can be realized, and even if the output of unselected candidates is excluded at the end of training, a neural network architecture capable of high-precision object tracking can be learned.

[0085] <Third Embodiment> In the third embodiment, a method for pruning a neural network is described in an example of an object tracking task similar to that of the second embodiment. In the third embodiment, the loss is calculated based on the number of candidates to be selected, which is the maximum number of candidates to be selected. In the third embodiment, if the number of candidates whose weight coefficients exceed a predetermined threshold exceeds the maximum number of specified candidates, the loss is calculated in such a way that the loss value increases.

[0086] The functional configuration of the information processing device 1 in the third embodiment is the same as that shown in Figure 2 above, so its illustration and explanation are omitted. Furthermore, the information processing flow in the information processing device 1 of the third embodiment is generally the same as the flowchart in Figure 9 above. In the third embodiment, the network architecture generated by the candidate generation unit 203 is assumed to consist of Layer 1 (1001) to Layer 4 (1004) shown in Figure 10, which are composed of Convolution, Batch-Normalization, and ReLU as shown in Figure 5.

[0087] In the third embodiment, the channels of the convolution are pruned. In this embodiment, the connections between the input and output of each channel of the convolution are edges (candidates), and the optimal combination of edges is found by learning the neural network architecture.

[0088] Specifically, the pruning of the convolution is performed based on the number of candidate selections determined in S902, that is, the maximum number of candidates to be selected. For example, as shown in Figure 13, the output from the i-th input channel to the j-th output channel 1302 of the convolution's input channels 1301 is calculated as shown in equations (7) and (8) below.

[0089]

number

[0090] Here, x in equations (7) and (8) i is the input to the i-th channel, w ij α is the weight coefficient of the convolution from the i-th channel to the j-th channel. ij This is a weighting coefficient that determines the network architecture.

[0091] And when the number of candidates to be selected, which is the maximum number of candidates to be selected, is K, the loss related to the network architecture is A This can be calculated as shown in equation (9) below.

[0092]

number

[0093] In equation (9), th is a threshold for the weight coefficients and is determined empirically. A is the set of weight coefficients that exceed the threshold th. Also, λ a is the weight coefficient for the first term of equation (9), and is determined empirically. According to the first term of equation (9), a loss occurs when the number of candidate weight coefficients that exceed the threshold th exceeds K. This loss, which occurs when the number of weight coefficients that exceed the threshold th exceeds K, has the effect of keeping the number of selected weight coefficients below K. The second term of equation (9) is the L1 regularization term for the weight coefficients, and aims to obtain sparse weight coefficients.

[0094] Thus, in the third embodiment, if the number of candidates whose weight coefficients exceed a predetermined threshold exceeds the maximum value of the specified number of candidates, the loss value increases. In other words, according to the third embodiment, by updating the parameters of the neural network while calculating the loss function shown in equation (9), it becomes possible to prune the convolution so that the number of candidates selected in the selection unit 209 is in the range of 1 to K. As a result, according to the third embodiment, the network can be made lighter while suppressing a decrease in accuracy.

[0095] This embodiment includes the following configurations and methods. (Composition 1) An information processing device that learns an architecture to optimize the structure of a neural network, Candidate generation means for generating multiple candidate edges of a neural network, An inference means that inputs training data into the neural network, in which weight coefficients are set for each candidate edge, and obtains an inference result, A loss calculation means that calculates the loss of the neural network based on the specified number of candidates to be selected from the aforementioned plurality of candidates and the inference result, An update means for updating the weight coefficients for the plurality of candidates based on the loss, A selection means for selecting a candidate from among the aforementioned multiple candidates based on the updated weight coefficient, An information processing device characterized by having the following features. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the loss calculation means performs the loss calculation such that the loss value increases when there is a difference between the number of designated candidates and the number of candidates selected by the selection means. (Composition 3) The information processing apparatus according to Configuration 1, characterized in that the loss calculation means calculates the loss such that the loss value increases when the difference between the weight of each of the specified number of candidates with higher weights among the plurality of candidates and the weights of the other candidates becomes smaller than a predetermined threshold. (Composition 4) The loss calculation means calculates the loss such that when the plurality of candidates are sorted in descending order according to their weights, the difference between the weight of the K-th candidate on the top side and the weight of the K+1-th candidate following the K-th candidate becomes smaller than a predetermined threshold, and K is the number of specified candidates, as described in configuration 1 or 3 of the information processing apparatus. (Composition 5) The information processing apparatus according to configuration 1, characterized in that the loss calculation means calculates the loss based on the maximum value of the specified number of candidates. (Composition 6) The information processing apparatus according to configuration 5, characterized in that the loss calculation means calculates the loss such that the value of the loss increases when the number of candidates with weights exceeding a predetermined threshold exceeds the maximum value of the specified number of candidates. (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, wherein the loss calculation means calculates a loss for the inference result of the neural network and a loss related to the architecture of the neural network, and in calculating the loss related to the architecture of the neural network, it calculates the loss based on the specified number of candidates and the inference result. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the loss calculation means acquires a loss obtained by integrating the loss for the inference result of the neural network and the loss related to the architecture of the neural network as the loss of the neural network. (Composition 9) The information processing device according to any one of configurations 1 to 8, characterized in that the weight of the candidate is a weighting coefficient indicating importance. (Composition 10) The information processing device according to configuration 1, characterized in that the neural network is a neural network for detecting a target from an image or for tracking a target. (Method 1) An information processing method executed by an information processing device that learns an architecture to optimize the structure of a neural network, A candidate generation process that generates multiple candidate edges for a neural network, An inference process in which training data is input to the neural network, which has weight coefficients set for each candidate edge, in order to obtain an inference result, A loss calculation step that calculates the loss of the neural network based on the specified number of candidates to be selected from the aforementioned multiple candidates and the inference result, An update step to update the weight coefficients for the plurality of candidates based on the loss, A selection step of selecting a candidate from among the aforementioned multiple candidates based on the updated weight coefficient, An information processing method characterized by having the following features. (Program 1) A program for causing a computer to function as an information processing device described in any one of configurations 1 through 9.

[0096] In the embodiments described above, the human body was used as an example of the target of detection, but the target of detection is not limited to the human body; it may also be a car, bicycle, motorcycle, animal, etc. The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of symbols]

[0097] 1: Information processing unit, 201: Image acquisition unit, 202: GT acquisition unit, 203: Candidate generation unit, 204: Selection number specification unit, 205: Inference unit, 206: Loss calculation unit, 207: Update unit

Claims

1. An information processing device that learns an architecture to optimize the structure of a neural network, Candidate generation means for generating multiple candidate edges of a neural network, An inference means that inputs training data into the neural network, in which weight coefficients are set for each candidate edge, and obtains an inference result, A loss calculation means that calculates the loss of the neural network based on the specified number of candidates to be selected from the aforementioned plurality of candidates and the inference result, An update means for updating the weight coefficients for the plurality of candidates based on the loss, A selection means for selecting a candidate from among the aforementioned multiple candidates based on the updated weight coefficient, An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the loss calculation means performs the loss calculation such that the value of the loss increases if there is a difference between the number of designated candidates and the number of candidates selected by the selection means.

3. The information processing apparatus according to claim 1, characterized in that the loss calculation means calculates the loss such that the loss value increases when the difference between the weight of each of the specified number of candidates with higher weights among the plurality of candidates and the weights of the other candidates becomes smaller than a predetermined threshold.

4. The loss calculation means calculates the loss such that when the plurality of candidates are sorted in descending order according to their weights, the difference between the weight of the K-th candidate on the top side and the weight of the K+1-th candidate following the K-th candidate becomes smaller than a predetermined threshold, and K is the number of specified candidates, as described in claim 1.

5. The information processing apparatus according to claim 1, characterized in that the loss calculation means calculates the loss based on the maximum value of the specified number of candidates.

6. The information processing apparatus according to claim 5, characterized in that the loss calculation means calculates the loss such that the value of the loss increases when the number of candidates with weights exceeding a predetermined threshold exceeds the maximum value of the specified number of candidates.

7. The information processing apparatus according to claim 1, wherein the loss calculation means calculates a loss for the inference result of the neural network and a loss related to the architecture of the neural network, and in calculating the loss related to the architecture of the neural network, it calculates the loss based on the specified number of candidates and the inference result.

8. The information processing apparatus according to claim 7, characterized in that the loss calculation means acquires a loss obtained by integrating the loss for the inference result of the neural network and the loss related to the architecture of the neural network as the loss of the neural network.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that the weight of the candidate is a weighting coefficient indicating importance.

10. The information processing apparatus according to claim 1, characterized in that the neural network is a neural network for detecting a target from an image or for tracking a target.

11. An information processing method executed by an information processing device that learns an architecture to optimize the structure of a neural network, A candidate generation process that generates multiple candidate edges for a neural network, An inference process in which training data is input to the neural network, which has weight coefficients set for each candidate edge, in order to obtain an inference result, A loss calculation step that calculates the loss of the neural network based on the specified number of candidates to be selected from the aforementioned multiple candidates and the inference result, An update step to update the weight coefficients for the plurality of candidates based on the loss, A selection step of selecting a candidate from among the aforementioned multiple candidates based on the updated weight coefficient, An information processing method characterized by having the following features.

12. A program for causing a computer to function as an information processing device as described in claim 1.

Images