Information processing device and control method of the same

Abstract

To update parameters of a neural network more efficiently.SOLUTION: An information processing device comprises: tracking means which tracks a tracking target included in a moving image using a neural network (NN) including a plurality of processing layers; selection means which selects one or more processing layers based on respective output feature maps of the plurality of processing layers when a plurality of photographed images included in the moving image are input to the NN; and update means which updates parameters of the one or more processing layers selected by the selection means when the tracking means performs inference processing using the NN.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to training neural networks. [Background technology]

[0002] In recent years, tracking technology (online tracking method) that uses online learning of deep neural networks (DNNs) has attracted attention as a technique for tracking specific subjects with high accuracy. Online tracking methods use images of the tracked object obtained during tracking to adapt the parameters of an object detection DNN to the task of detecting the tracked object. For example, Non-Patent Documents 1 and 2 propose a technique for fine-tuning the parameters of an object detection DNN using a gradient method with reference images. This allows information about the tracked object to be incorporated into the object detection DNN, enabling it to detect the tracked object with higher accuracy. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Bhat et al., "Learning Discriminative Model Prediction for Tracking", ICCV2019, 2019 [Non-patent document 2] Wang et al., "Tracking by Instance Detection: A Meta-Learning Approach", CVPR2020, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in existing online tracking methods, the parameters to be updated online are manually determined, so the parameters selected may not necessarily be suitable for online updating. Also, in tracking processing, it is sometimes desirable to use a model that is overfitted to the tracked target rather than a balanced model that can detect various objects.

[0005] The present invention has been made in view of the above problems, and aims to provide a technique that enables more efficient parameter updating of a neural network. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an information processing device according to the present invention has the following arrangement. a tracking means for tracking a target included in a video image using a neural network (NN) including a plurality of processing layers; a selection means for selecting one or more processing layers based on an output feature map of each of the plurality of processing layers when a plurality of captured images included in a video are input to the NN; an update means for updating parameters of the one or more processing layers selected by the selection means when the tracking means performs inference processing using the NN; Equipped with. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique for more efficiently updating parameters of a neural network. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Figure 3] 10 is a flowchart for selecting an update layer. [Figure 4]FIG. 10 is a diagram illustrating processing in the NN that constitutes the tracking unit. [Figure 5] FIG. 10 is a diagram showing an example of an input image and a GT. [Figure 6] FIG. 10 is a diagram showing an example of an input image and correct answer data given to a NN, as well as a feature map and inference results output by the NN. [Figure 7] 10 is a flowchart of online tracking (inference). [Figure 8] 10 is a flowchart for selecting an update layer (second embodiment); DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of an information processing device according to the present invention, an information processing device for online updating of a tracking neural network (tracking NN) will be described below as an example. More specifically, a method for selecting a layer to be updated online from multiple layers included in the tracking NN and a method for online updating of the selected layer will be described.

[0011] <Device configuration> FIG. 1 is a diagram showing the hardware configuration of an information processing device that trains a tracking NN. The information processing device may be configured, for example, by a computer. A CPU 101 controls the entire device by executing a control program stored in a read-only memory (ROM) 102. A random access memory (RAM) 103 temporarily stores various data from each component. The RAM 103 is also used as a work area where the CPU 101 loads and uses the program.

[0012] The storage unit 104 stores data to be processed in the first embodiment. For example, it stores data used for learning and data to be subjected to inference processing. The storage unit 104 may be a hard disk drive (HDD), flash memory, optical media, or the like.

[0013] Fig. 2 is a diagram showing the functional configuration of an information processing device. Fig. 2(a) shows the functional configuration during pre-learning. During pre-learning, a layer to be updated online during tracking processing (online update layer) is determined, and information about this is stored. During pre-learning, the information processing device 1 has a storage unit 200, a captured image acquisition unit 201, a tracking target setting unit 202, a tracking unit 203, and a selection unit 204.

[0014] The memory unit 200 stores captured images for learning, ground truth data corresponding to each captured image, and information about the online update layer. The captured image acquisition unit 201 acquires captured images to be processed from the memory unit 200 or an imaging device (not shown). The tracking target setting unit 202 accepts designation of an area in the captured image from a user and sets the area as a tracking target object. The tracking unit 203 performs tracking processing on the input time-series captured images (video sequence) using a tracking NN. The selection unit 204 determines the online update layer based on the information obtained by the tracking unit 203. The operation of each functional unit during pre-learning will be described later with reference to FIGS. 2(a) and 3.

[0015] On the other hand, Fig. 2(b) shows the functional configuration during tracking processing (inference). It differs from the pre-learning (Fig. 2(a)) in that it does not have a selection unit 204 but has an online update unit 205. The online update unit 205 acquires information on the online update layer from the storage unit 200 and updates the parameters of the tracking NN. The operation of each functional unit during tracking processing will be described later with reference to Fig. 2(b) and Fig. 7.

[0016] <Configuration of tracking NN> Figure 4 is a diagram explaining the processing in a tracking NN that tracks a target included in an input video image (a time-series image group). Figure 4(a) shows the overall configuration of the tracking NN. As shown in Figure 4(a), the tracking NN includes two NNs (a feature extraction NN and a classification NN). Each of the feature extraction NN and classification NN is composed of multiple processing layers, including multiple convolution layers that perform linear transformations and normalization layers that normalize the output of the linear transformations.

[0017] Figure 4(b) shows the detailed configuration of the feature extraction NN. That is, image 4101 corresponds to 4001, and feature map 4111 corresponds to feature map 4003. Here, the feature extraction NN must include multiple normalization layers. Although not shown in Figure 4(b), it may also include a nonlinear transformation such as ReLU (Rectified Linear Unit).

[0018] In tracking processing using a tracking NN, there is a demand for the tracking NN to quickly adapt to the ever-changing "appearance" of the tracked target. However, the amount of data on the tracked target obtained during inference is less than the data used for pre-training. Also, in tracking, a model that is overfitted to the tracked target is more desirable than a balanced model that can stably detect various objects. Furthermore, it is desirable to keep the processing load low.

[0019] Therefore, in this embodiment, a method for efficiently updating a feature extraction NN included in a tracking NN when performing online updating of the tracking NN (parameter updating during inference processing) will be described. Specifically, a layer with a higher learning effect in online updating is selected during pre-learning. In particular, an example of selecting a normalization layer with a larger variance (=larger fluctuation) of output values ​​will be described.

[0020] <Device Operation> In the following, we first explain the process of selecting an online update layer in pre-learning, which is performed before the tracking process, and then explain the process flow when performing tracking inference using the selected online update layer.

[0021] <Select online update layer> 3 is a flowchart for selecting an online update layer in the first embodiment. However, the information processing device does not necessarily have to perform all of the steps described in this flowchart.

[0022] In S301, the captured image acquisition unit 201 acquires one or more pairs of an input image showing one or more objects and ground truth data (GT (Ground Truth)) from a storage unit. In the following description, it is assumed that GT is bounding box (BB) information indicating the center position and size of an object present in the input image.

[0023] 5 is a diagram showing an example of an input image and a GT. An input image 501 includes an image of a tracking target object 503 (a person in this case). A rectangular frame 504 indicates BB, which is the GT of the tracking target object 503. It is assumed that images of various types of objects that can be tracking targets are stored in advance in the storage unit 200.

[0024] In S302, the tracking target setting unit 201 receives a selection of one object from the objects in the input image from the user. Based on the position and size GT (i.e., BB) of the selected object, the tracking target setting unit 201 cuts out an image of the area around the object in the input image as a template and resizes it. For example, it is sufficient to cut out an image of a constant multiple of the size of the object / BB, with the position of the object / BB as the center. Area 502 shows an example of the area around the tracking target object 503 being cut out.

[0025] In S303, the tracking unit 203 inputs the image obtained in S302 to a feature extraction NN 4002 to obtain a feature map 4003. As shown in FIG. 4(b), the feature extraction NN is configured to include multiple convolutional layers and normalization layers, and is configured to use the feature map output from the previous layer as the input to the next layer. Note that the weights of the feature extraction NN 4002 and the convolutional layers of the classification NN 4004 that constitute the tracking unit are assumed to have been trained in advance using a method such as Non-Patent Document 1.

[0026] In S304, the selection unit 204 acquires feature maps output from each normalization layer (normalization layers 4104, 4108, 4112) included in the feature extraction NN 4002. Then, for each output feature map, the variance of all elements (output values) is calculated as a scalar value. For example, the variance σ of all elements of the feature map 4105 output by the normalization layer 4104 is calculated as follows: t (σ t Here, the size of the feature map f is set to "NxWxHxC", where N is the number of input images. Also, W is the width, H is the height, and C is the number of channels, which are values ​​determined for each NN.

[0027] Then, the processes of S301 to S304 are executed for each of the N input images. In the t-th loop, as shown in Equation (1), the variances σ and σ calculated in the past 1st to t-1th loops are t Alternatively, σ may be updated by taking the moving average of m, where m is a real number between 0 and 1, and is a constant that weights the moving average. σ ← mσ t +(1-m)σ (1)

[0028] In S305, the selection unit 204 determines whether or not to end the loop process of S301 to S304. For example, when the number of input images is large (N is large), there is a problem that the amount of calculation for variance becomes enormous. Therefore, for example, it is preferable to determine that the process should end when the amount of change in σ becomes smaller than a certain value. Alternatively, it is preferable to determine that the process should end when the number of processed input images reaches a predetermined number. The conditions for determining that the process should end are not limited to these.

[0029] In S306, the selection unit 204 selects one or more layers to be updated online from among the normalization layers of the feature extraction NN. Specifically, among the variances of each layer calculated in S304, a layer whose variance is greater than a given threshold is selected as the online update layer. The selection result (information on which layer was selected) is then stored in the storage unit 200. Note that a configuration may be adopted in which a layer with a relatively large variance value is selected from among multiple layers.

[0030] By selecting the online update layer in advance in this way, it becomes possible to efficiently perform online updates to adapt to the tracking target during tracking (inference processing) using less data.

[0031] <Inference processing> During tracking (inference processing), the normalization parameters for the layers (online update layers) preselected as described above are updated online to match the possible distribution of the features of the tracked target.

[0032] FIG. 7 is a flowchart of online tracking (during inference). FIG. 6 is a diagram showing an example of the input image and ground truth data given to the NN, and a feature map (likelihood map) output by the NN. Here, the NN used for online tracking is assumed to have undergone pre-training using a method such as Non-Patent Document 1. In addition, in online updating, the parameters of the discriminative NN and the normalization layer selected by the above-mentioned process are assumed to be updated.

[0033] In S701, the captured image acquisition unit 201 acquires a captured image (FIG. 6(a)) that shows the object to be tracked. In S702, the tracking target setting unit 202 specifies an area in the captured image and sets that area as the object to be tracked (FIG. 6(b)). For example, the user specifies the area by touching a subject displayed on the display unit 105 configured as a touch panel. Methods that may be used include detecting the object with an object detector, or, if tracking is in progress, specifying the tracking target area using a tracking target area obtained as a tracking result for an image captured at a past time. Then, the position and size of a bounding box (BB) that surrounds the area of ​​the object to be tracked is set as the GT of the object to be tracked.

[0034] In S703, the tracking unit 203 inputs the image obtained in S701 to the feature extraction NN 4002 to obtain a feature map 4003. Then, the feature map is further input to the classification NN 4004 to obtain a likelihood map 4005. Here, the width of the feature map is W f , height is H f, the number of channels is C f The width of the likelihood map is W l , height is H l Let's say.

[0035] The likelihood map is a map such as that shown in image 603 in FIG. 6(c), in which a pixel value indicating likelihood (a real value between 0 and 1) is set for each pixel. If the pixel value at the position where the tracking target object (in this case, a person) is present is relatively large compared to the values ​​of other pixels in the map, tracking can be performed correctly. Therefore, the pixel with the largest value in the likelihood map is taken as the position of the tracking target, and the pixel value is considered to be the likelihood of the tracking target. Furthermore, the size of the tracking target object obtained in S702 may be used as is, or may be estimated using a neural network, and the method is not limited thereto. Furthermore, the obtained tracking results are stored in a storage unit.

[0036] In S704, the tracking unit 203 determines whether to end the tracking process. The determination method may be, for example, whether or not a tracking end instruction has been input by the user. If the tracking process is not to be ended, the process proceeds to S705, and the tracking process is continued.

[0037] In S705, the online update unit 205 updates the parameters of the classification NN and the feature extraction NN online based on the tracking result. The parameters of the classification NN may be updated using a method similar to that described in Non-Patent Document 1. For example, the position of the tracked object obtained in S703 is used as the GT, and the parameters are updated using the backpropagation method or the like. Meanwhile, the parameters of the feature extraction NN are updated by updating the normalization layer selected by the above-mentioned method as described below.

[0038] In conventional object detection, the normalization of the features output by the convolutional layer ideally uses the population mean and population variance of the features of the objects that can be input. These are then replaced with the mean and variance of the features in the training data. However, in the case of tracking, the appearance of the tracked target changes dramatically depending on the shooting environment and changes in posture, so the distribution of the image data obtained during training and tracking can differ significantly. As a result, the distribution of the input to each normalization layer differs from that during training, and this difference in distribution from that during training can cause a deterioration in tracking accuracy. Therefore, it is desirable to align the normalization parameters of the normalization layer selected using the method described above to the data obtained during tracking.

[0039] Therefore, in the first embodiment, a process is performed to bring the mean and variance of the features of the training data closer to the population mean and variance of the features of the tracking target object. One method for this is to use the features and mean and variance extracted from the captured image of the tracking target object and the moving average of the mean and variance of the features of the training data. Specifically, the online update unit 205 calculates the mean and variance for each channel for each feature map input to each selected normalization layer during the process in which the feature extraction NN processes the captured image in S703. Then, the online update unit 205 calculates the moving average of the mean and variance, which are normalization parameters for each normalization layer, and reflects information about the tracking target in the normalization parameters. Note that while the mean and variance calculated using only the data being tracked may be used, using the moving average of the mean and variance calculated using the training data can be more robust against outliers in the features of the tracking target obtained during tracking.

[0040] The update timing may be only for a predetermined time period after the start of tracking in the time-series images to be tracked, or may be at regular intervals. Only features obtained when the likelihood of the tracking target obtained as a tracking result is sufficiently high may be used as the update features. It is also considered that updating using features from multiple time periods, or even the most recent features at the time of tracking, can more accurately capture the distribution of possible features of the tracking target. Therefore, a database of update features may be provided to control the time period over which features are used. For example, a lower and upper limit may be set for the size of the database, or old data may be deleted from the database.

[0041] As described above, according to the first embodiment, the information processing device that controls the learning of the NN selects a layer to be updated online in advance during pre-learning. In particular, it selects a normalization layer in which the distribution of features that can be obtained varies greatly (= has large variance) depending on the object to be tracked. Then, during inference (online learning), it updates the normalization parameters in the selected normalization layer.

[0042] This control allows the tracking section to be more adapted to the tracked target than when only the discriminative NN is updated online, which is expected to improve tracking performance.In addition, by selecting the normalization layer to be updated online in advance, it becomes possible to adapt to the tracked target using the limited data obtained during tracking.

[0043] (Variation 1) For the NN of the tracking section, instead of an online tracking method in which the convolution layer is updated online using error backpropagation or the like, the Siam method as typified by Literature A may be used. In other words, the classification NN 4004 in the first embodiment performs correlation calculations and processing of multiple layers of NN.

[0044] Document A: Li et al., "High Performance Visual Tracking with Siamese Region Proposal Network", CVPR2018, 2018 Even in the SIAM method, which does not perform online parameter updates, it is possible to expect improved tracking performance by performing online updates of only the normalization parameters according to the tracking target.

[0045] (Variation 2) In tracking, the appearance of the tracked target changes in a variety of ways depending on the shooting environment, posture changes, etc., so as time passes from the initial stage of tracking, the distribution of the input image and the input to each normalization layer can change significantly. As described in literature B, such changes in the domain of the input to each layer are generally known to cause a deterioration in accuracy.

[0046] Document B: Nado et al., "Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift", arXiv:2006.10963, 2020 Therefore, the normalization parameters are updated according to the data obtained during tracking. However, since the data obtained during tracking may contain noise, it is preferable to update only the minimum number of layers rather than updating all layers, even those with little domain change.

[0047] Therefore, in S705, the online update unit 205 may be configured to control the timing of online update of the normalization layer according to the distribution of the input to the normalization layer. For example, during the tracking process of time-series images, the online update unit 205 periodically calculates the mean and variance of the input features for each normalization layer selected by the method of the first embodiment, regardless of whether online update is performed, and stores them in the storage unit 200. The online update unit 205 may then compare the mean and variance calculated at the previous time, and perform update processing if the difference is greater than a predetermined value. Alternatively, the online update unit 205 may be configured to perform update processing if the likelihood of the tracking target obtained as the tracking result in S703 is low (for example, lower than a threshold value).

[0048] In this way, we focus on normalization layers with large domain changes and update the normalization parameters of those layers online. This is expected to improve tracking performance when the distribution of possible tracking targets changes rapidly over time, compared to online updates using only images of the tracking target at the beginning of tracking.

[0049] (Second embodiment) In the second embodiment, a configuration will be described in which normalization parameters are not calculated using only data obtained during tracking, but normalization parameters calculated from a large amount of data obtained during learning are utilized.

[0050] <Device Operation> <Select online update layer> FIG. 8 is a flowchart for selecting an update layer in the second embodiment. S801, S802, and S803 are the same processes as S301, S302, and S303 in the first embodiment, and therefore their explanations are omitted. As in the first embodiment, the tracking NN is assumed to have been pre-trained using various categories. Then, in S804, the input feature map for each normalization layer is updated to a length W f xH f xC f The multidimensional vector d is stored in the memory. f , H f , C f ) are the width, height, and number of channels of the input feature map. The vector d can also be calculated from the mean or variance of each channel or region depending on the normalization method.

[0051] In S809, the selection unit 204 determines whether or not to end the loop process of S801 to S804. For example, it may be determined that the process should end when the number of processed input images reaches a predetermined number (for example, N times). The conditions for determining the end are not limited to these.

[0052] In S805, the selection unit 204 clusters the vector d calculated for N images for each normalization layer into k clusters, where k is determined in advance. The clustering can be performed using a method such as k-means clustering, but is not limited to this.

[0053] In S806, the selection unit 204 selects a normalization layer in which the intra-cluster variance of the vector d is small and the inter-cluster variance is large, for example, the layer in which the value obtained by dividing the inter-cluster variance by the intra-cluster variance is the largest.

[0054] In S807, the selection unit 204 calculates normalization parameters for each cluster in the selected layer. Because the vector d is a parameter corresponding to one captured image, the average and variance of all feature elements belonging to each cluster are calculated again. Alternatively, the average for all images and all channels in the cluster may be calculated for the average and variance of each feature channel for each image calculated as described above.

[0055] That is, in the second embodiment, in addition to selecting an update layer, a normalization parameter is calculated for each cluster, and the result is stored in the storage unit 200.

[0056] <Inference processing> Inference processing is performed in a manner similar to that of the first embodiment (FIG. 7). However, in S705, the above-mentioned vector d is calculated for the selected normalization layer regardless of whether online updating is performed. Then, distance calculations are performed with respect to the vector d representing each of the k clusters, and the normalization parameters are replaced with those of the cluster with the closest distance (stored in the storage unit 200).

[0057] This replacement process can be considered a type of online update in the sense that it changes parameters to match the tracking target during inference. After this, the same parameters may be used during tracking, or the normalized parameters may be further updated online using the method described in the first embodiment.

[0058] As described above, according to the second embodiment, the information processing device that controls the learning of the NN selects a layer to be updated online in advance during pre-learning. In particular, it selects a layer with small intra-cluster variance and large inter-cluster variance. Then, during inference, it updates the replacement parameters with normalized parameters corresponding to the cluster to be tracked.

[0059] This control allows the tracking NN to better adapt to the target being tracked, improving tracking performance compared to when all training data containing various objects is used, or when only the features of the target being tracked obtained during inference are used.

[0060] (Variation 3) In the second embodiment described above, clustering is performed. However, if the category of the tracking target is known in advance, a normalization parameter may be calculated for each image in the category prepared in advance and used.

[0061] Specifically, in the loop of S301 to S304 for selecting the online update layer in the second embodiment, clustering is not performed, and a normalization layer in which the intra-category variance of the vector d is small and the inter-category variance is large is selected. Then, normalization parameters pre-trained using various categories and moving averages of the means and variances of the input features to the normalization layer calculated using images for each category are calculated, and normalization parameters are prepared in advance for each category.

[0062] During inference, the tracking target image is subjected to category determination, and tracking is started using the normalized parameters of the determined category. After this, the same parameters may be used during tracking, or the normalized parameters may be further updated online in the same manner as in the first embodiment.

[0063] The disclosure of this specification includes the following information processing device, control method, and program. (Item 1) a tracking means for tracking a target included in a video image using a neural network (NN) including a plurality of processing layers; a selection means for selecting one or more processing layers based on an output feature map of each of the plurality of processing layers when a plurality of captured images included in a video are input to the NN; an update means for updating parameters of the one or more processing layers selected by the selection means when the tracking means performs inference processing using the NN; An information processing device comprising: (Item 2) The selection means selects one or more processing layers from the plurality of processing layers in which the variance of the output values ​​included in the output feature map is relatively large. 2. The information processing device according to item 1, (Item 3) the plurality of processing layers include a plurality of convolution layers and a plurality of normalization layers that normalize outputs of the plurality of convolution layers, The selecting means selects, from the plurality of normalization layers, one or more normalization layers having a relatively large variance of output values ​​included in an output feature map as the one or more processing layers. 2. The information processing device according to item 1, (Item 4) The update means controls the timing of updates according to the distribution of inputs to the plurality of normalization layers. 4. The information processing device according to item 3, (Item 5) the selection means selects the one or more processing layers during pre-training of the NN; The update means updates parameters of the one or more processing layers based on a result of the inference process during the inference process using the NN. 5. The information processing device according to any one of items 1 to 4. (Item 6) The selection means clusters inputs to each of the plurality of normalization layers into a plurality of clusters, and selects one or more normalization layers having a relatively small intra-cluster variance and a relatively large inter-cluster variance as the one or more processing layers. 5. The information processing device according to item 3 or 4. (Item 7) The selection means calculates a normalization parameter for each cluster, The update means updates the parameters of the one or more normalization layers using normalization parameters corresponding to clusters determined based on inputs to each normalization layer. 7. The information processing device according to item 6, (Item 8) The update means updates the normalization parameters using a moving average of a distribution of inputs to the plurality of normalization layers when the tracking means performs an inference process using the NN. 8. The information processing device according to item 7, (Item 9) A control method for an information processing device having a tracking means that tracks a tracking target included in a moving image using a neural network (NN) including a plurality of processing layers, a selection step of selecting one or more processing layers based on output feature maps of the plurality of processing layers when a plurality of captured images included in a video are input to the NN; an updating step of updating parameters of the one or more processing layers selected by the selecting step when the tracking means performs inference processing using the NN; A control method comprising: (Item 10) A program for causing a computer to execute the control method described in item 9.

[0064] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0065] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0066] 200 Memory unit; 201 Captured image acquisition unit; 202 Tracking target setting unit; 203 Tracking unit; 204 Selection unit; 205 Online update unit

Claims

1. a tracking means for tracking a target included in a moving image using a neural network (NN) including a plurality of processing layers; a selection means for selecting one or more processing layers based on an output feature map of each of the plurality of processing layers when a plurality of captured images included in a video are input to the neural network; an update means for updating parameters of the one or more processing layers selected by the selection means when the tracking means performs inference processing using the NN; An information processing device comprising:

2. The selection means selects one or more processing layers from the plurality of processing layers, the processing layers having a relatively large variance of output values ​​included in an output feature map.

2. The information processing apparatus according to claim 1, wherein:

3. the plurality of processing layers include a plurality of convolution layers and a plurality of normalization layers that normalize outputs of the plurality of convolution layers, The selecting means selects, from the plurality of normalization layers, one or more normalization layers having a relatively large variance of output values ​​included in an output feature map as the one or more processing layers.

2. The information processing apparatus according to claim 1, wherein:

4. The update means controls the timing of updates according to the distribution of inputs to the plurality of normalization layers.

4. The information processing apparatus according to claim 3,

5. the selection means selects the one or more processing layers during pre-training of the neural network; The update means updates parameters of the one or more processing layers based on a result of the inference process during the inference process using the NN.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. The selection means clusters inputs to each of the plurality of normalization layers into a plurality of clusters, and selects one or more normalization layers having a relatively small intra-cluster variance and a relatively large inter-cluster variance as the one or more processing layers.

4. The information processing apparatus according to claim 3,

7. The selection means calculates a normalization parameter for each cluster, The update means updates the parameters of the one or more normalization layers using normalization parameters corresponding to clusters determined based on inputs to each normalization layer.

7. The information processing apparatus according to claim 6,

8. The update means updates the normalization parameters using a moving average of a distribution of inputs to the plurality of normalization layers when the tracking means performs an inference process using the NN.

8. The information processing apparatus according to claim 7,

9. A control method for an information processing device having a tracking means that tracks a tracking target included in a moving image using a neural network (NN) including a plurality of processing layers, a selection step of selecting one or more processing layers based on output feature maps of the plurality of processing layers when a plurality of captured images included in a video are input to the neural network; an updating step of updating parameters of the one or more processing layers selected by the selecting step when the tracking means performs inference processing using the NN; A control method comprising:

10. A program for causing a computer to execute the control method according to claim 9.

Images