Information processing device and control method thereof

Abstract

To provide an information processing device, a method and a program that further precisely track an object to be tracked.SOLUTION: An information processing device includes: a template region setting unit that sets a first region and a second region for a template image which is contained in a video sequence and which contains the image of an object to be tracked; a template feature extracting unit that extracts, using a feature extracting neural network NN, template features from the first and second regions, respectively; a search range feature extracting unit that extracts, using a feature extracting neural network NN, a search range feature from a search range image which is contained in the video sequence and which is subsequent to the template image; a feature correlation map extracting unit that performs a correlation operation between each of first and second features and the search range feature so as to derive first and second feature correlation maps, respectively; a feature correlation map unifying unit that unifies the first and second feature correlation maps so as to derive a unified feature; and a tracking object detecting unit that detects, based on the unified feature, an object to be tracked from the search range image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for tracking a target object in an image. [Background technology]

[0002] In recent years, technology using Deep Neural Network (DNN) has been attracting attention as a technology for tracking a specific subject in an image with high accuracy. For example, the Siam method, as typified by Non-Patent Document 1, detects the tracking target from the search range image by performing correlation calculation between the DNN features of a reference image, which serves as a template for the tracking target, and the search range image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Li et al., "High Performance Visual Tracking with Siamese Region Proposal Network", CVPR2018 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the reference image generally contains the background other than the tracking target, and the above-mentioned Siam method may mistakenly track the background as a feature of the tracking target. In addition, the tracking target changes in appearance over time, so it is easy to lose sight of the tracking target due to the changes. For these reasons, the Siam method has the problem that the tracking results tend to drift into the background.

[0005] The present invention has been made in consideration of such problems, and has an object to provide a technique that enables tracking of a tracking target with higher accuracy. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an information processing device according to the present invention has the following configuration. That is, an information processing device for tracking a tracking target in a video sequence includes: A region setting means for setting a first region and a second region corresponding to different image regions of the tracking target in a template image including an image of the tracking target that is included in the video sequence; a first feature extraction means for extracting a first region feature and a second region feature, which are template features, from the first region and the second region using a feature extraction neural network (NN); A second feature extraction means for extracting search range features from a search range image included in the video sequence and subsequent to the template image, using the feature extraction neural network; a correlation map deriving means for performing a correlation calculation between the first region feature and the second region feature, and the search range feature, and deriving a first feature correlation map and a second feature correlation map, which are correlation maps; an integration means for integrating the first feature correlation map and the second feature correlation map to derive an integrated feature; a target detection means for detecting the tracking target from the search range image based on the integrated features; Equipped with. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that enables tracking of a tracking target with higher accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device (at inference time). [Diagram 3] FIG. 1 is a diagram showing the overall structure of a neural network. [Figure 4] 13 is a flowchart of an inference process. [Diagram 5]1A and 1B are diagrams illustrating examples of a template image and a search range image. [Figure 6] FIG. 1 is a diagram showing an example of an input image (search range image) and an inference result (likelihood map). [Figure 7] FIG. 2 is a diagram illustrating a functional configuration of an information processing device (during learning). [Figure 8] 13 is a flowchart of a learning process. [Figure 9] 1A and 1B are diagrams illustrating examples of a template image and a search range image. [Figure 10] FIG. 1 is a diagram showing an example of an input image (search range image), correct answer data (GT map), and an inference result (likelihood map). [Figure 11] FIG. 13 is a diagram illustrating an example of a feature map of a search range image. [Figure 12] 13 is a flowchart of a template region setting process. [Figure 13] FIG. 13 is a diagram illustrating a functional configuration of an information processing device (fifth embodiment, during learning). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (First embodiment) As a first embodiment of an information processing device according to the present invention, an information processing device that tracks a target in an image using a neural network will be described below. First, an inference process for tracking a target in an image will be described below, and a learning process will be described after the inference process.

[0011] <Explanation of inference process> <Device configuration> 1 is a diagram showing a hardware configuration of an information processing device 1. The information processing device 1 can be used for inference processing using a neural network. The information processing device 1 can also be used for learning processing of a neural network.

[0012] A CPU 101 controls the entire device by executing a control program stored in a ROM 102. A RAM 103 temporarily stores various data from each component. The RAM 103 is also used as a work memory when the CPU 101 executes a program.

[0013] The memory unit 104 stores data to be processed, and saves data (image data and parameters) used for learning. As a medium for the memory unit 104, a HDD, a flash memory, various optical media, etc. can be used. The input unit 105 is composed of a keyboard, a touch panel, a dial, etc., and receives input from a user, and is used when setting a tracking target, etc. The display unit 106 is composed of a liquid crystal display, etc., and displays the subject and tracking results to the user. The communication unit 107 is a functional unit for communicating with other devices such as a photographing device.

[0014] 2 is a diagram showing the functional configuration of the information processing device during inference processing. The information processing device 1 has an image acquisition unit 201, a tracking target setting unit 202, a template area setting unit 203, a template feature extraction unit 204, and a search range feature extraction unit 205. It also has a feature correlation map calculation unit 206, a weighting coefficient acquisition unit 207, a feature correlation map integration unit 208, a tracking target detection unit 209, and a storage unit 210. Each functional configuration unit will be briefly described.

[0015] The image acquisition unit 201 acquires a template image from a video sequence (a series of frame images), and also acquires a search range image subsequent to the template image (at a later time) from the same video sequence.

[0016] The tracking target setting unit 202 determines a tracking target in the template image acquired by the image acquisition unit 201 according to an instruction specified by the input unit 105, and outputs a bounding box (BB) surrounding the entire tracking target area. Specifically, the tracking target may be determined by touching a subject displayed on the display unit 105, or the tracking target may be automatically detected and set from an image captured by a camera. The method of determining the tracking target is not limited to these.

[0017] Based on the tracking target BB set by the tracking target setting unit 202, the template area setting unit 203 sets two or more areas to be used as the template from within the template image acquired by the image acquisition unit 201, and cuts out images of the areas. Hereinafter, the area to be used as the template will be referred to as the template area, and the image cut out from the area will be referred to as the template area image. The area to be used as the template is selected by selecting an area that overlaps with the tracking target area from a group of areas with high objectness (object-like) detected using an objectness detector. The objectness detector may be a detector using a neural network, as typified by FCOS (Reference 1). Reference 1: Tien et al., "FCOS: Fully Convolutional One-Stage Object Detection", ICCV2019

[0018] The template feature extraction unit 204 inputs each of the multiple template region images obtained from the template region setting unit 203 into a neural network for feature extraction, and extracts template features.

[0019] The search range feature extraction unit 205 inputs the search range image obtained from the image acquisition unit 201 into a neural network for feature extraction, and extracts template features.

[0020] The feature correlation map calculation unit 206 performs correlation calculation between each of the template features extracted by the template region setting unit 203 and the search range features extracted by the search range feature extraction unit 205, and calculates a feature correlation map (derives the correlation map).

[0021] The weighting coefficient acquisition unit 207 acquires the weighting coefficient vector from the storage unit 210 .

[0022] The feature correlation map integration unit 208 integrates the feature correlation maps calculated by the feature correlation map calculation unit 206 to calculate an integrated feature. Specifically, the feature correlation maps calculated by the feature correlation map calculation unit 206 are weighted by the weighting coefficients acquired by the weighting coefficient acquisition unit 207, and then the feature correlation maps are combined.

[0023] The tracked target detection unit 209 inputs the integrated features calculated by the feature correlation map integration unit 208 to a neural network for tracking target detection, and outputs a likelihood map that strongly reacts to the position of the tracked target object and a size estimation map. Then, the tracked target detection unit 209 estimates the position and size of the tracked target object using the likelihood map and the size estimation map.

[0024] It is also assumed that the neural networks for feature extraction used by the template feature extraction unit 204 and the search range feature extraction unit 205 have been trained in advance. It is also assumed that the neural network for tracking target detection used by the tracking target detection unit 209 has been trained in advance. Furthermore, when performing inference, previously trained parameters are obtained from the storage unit 210.

[0025] Figure 3 shows the overall structure of a neural network, where NN is an abbreviation for neural network.

[0026] Feature extraction neural networks (NN) 302 and 305 are neural networks that extract features from images, and correspond to the neural networks used in the template feature extraction unit 204 and the search range feature extraction unit 205, respectively. Both may share parameters, or the parameters of the final layer, etc. may be different.

[0027] Correlation calculation 307 is a process of performing correlation calculation between features extracted by feature extraction NNs 302 and 305, and corresponds to the feature correlation map calculation unit. Feature correlation map combination 310 is a process of combining feature correlation maps obtained by correlation calculation 307, and corresponds to the feature correlation map integration unit 208. Tracking target object detection neural network 312 is a neural network that estimates the position and size of a tracking target object, and corresponds to the neural network used in tracking target detection unit 209. Each neural network is composed of nonlinear transformation such as convolution and ReLU (Rectified Linear Unit), but the type of nonlinear transformation is not limited to these in this embodiment.

[0028] <Device Operation> 4 is a flowchart of the inference process. Note that the information processing device does not necessarily have to perform all of the steps described in this flowchart.

[0029] In S401, the image acquisition unit 201 acquires an image (template image) in which a tracking target object appears.

[0030] 5A and 5B are diagrams showing examples of a template image and a search range image. A template image 501 shown in Fig. 5A is a template image acquired by the image acquisition unit 201. The template image 501 includes a tracking target object 503. A tracking target BB 504 is a BB surrounding the tracking target object 503.

[0031] In S402, the tracking target setting unit 202 determines the tracking target in the template image acquired in S401 according to the instruction specified by the input unit 105. Specifically, the target may be determined by touching the subject displayed on the display unit 105. Furthermore, for the determined tracking target, an image around the region of the tracking target object in the template image is cut out as the template image based on the position and size of the tracking target, and resized. It is sufficient to cut out a constant multiple of the size of the tracking target object with the position of the tracking target object as the center. Region 502 in FIG. 5(a) is an example of the region around the tracking target object being cut out.

[0032] In S403, the template area setting unit 203 sets two or more areas (first area, second area) to be used in the template (template area) based on the tracking target BB set by the tracking target setting unit 202, and cuts out images of the areas. That is, images of two or more areas are cut out from the image cut out from the template image in S402.

[0033] Specifically, first, an image is input to an objectness detector, and multiple BBs of image regions with high detected objectness (relatively high likelihood of object existence) are output. These BBs are set as a group of template region candidate BBs. For example, image 509 in FIG. 5(c) is an image cut out from the template image in S402, region 510 is the tracking target BB, and BBs 511, 512, and 513 are a group of template region candidate BBs. From the group of template region candidate BBs, a region with an IoU (Intersection over Union) with region 510, which is the tracking target BB, that is equal to or exceeds a threshold is selected as the template region. Here, BBs 511 and 513 are selected. After that, a template region is cut out from region 510 and set as a template region image. The objectness detector may be a detector using a neural network, such as FCOS (Reference 1).

[0034] In addition, the number of template region images to be acquired may be fixed, and in the following description, it is assumed that N template region images are acquired. Here, the IoU of two BBs (here, assumed to be BB1 ​​and BB2) is calculated as follows.

[0035]

number

[0036] In S404, the template feature extraction unit 204 inputs the N template region images obtained in S403 to the feature extraction NN 302, and obtains N template features 303 (first region feature, second region feature) that are features corresponding to the region of the tracking target object. Here, the width, height, and number of channels of the template features are W T , H T , C T Let us assume that T , H T , C T is a value determined by the width, height, and number of channels of the template region image and the structure of the neural network.

[0037] In S405, the image acquisition unit 201 acquires an image for searching for a tracking target object. For example, an image at a different time in the same video sequence as the image selected in S401 is acquired as an image for searching for a tracking target object. An image 505 in FIG. 5(b) is an image for searching for a tracking target object. The image 505 includes a tracking target object 507. Also, a tracking target BB 508 is a BB surrounding the tracking target object 507.

[0038] Furthermore, the image acquisition unit 201 cuts out and resizes the image around the tracking target object from the image acquired in S404 based on the position and size of the tracking target object. For example, it is sufficient to cut out a constant multiple of the size of the tracking target object, centered on the position of the tracking target object obtained as the tracking result in the previous frame. Also, in the initial stage of tracking, it is sufficient to cut out a constant multiple of the size of the tracking target object, centered on the position of the tracking target in the template image. Area 506 in FIG. 5(b) is an example of an image cut out to be the search range.

[0039] In S406, the search range feature extraction unit 205 inputs the search range image obtained in S405 to the feature extraction NN 305 to obtain a feature map 306 of the search range image. Here, the width, height, and number of channels (W, H, and C, respectively) of the feature map are determined by the structure of the neural network. Specifically, W and H are determined as follows. Note that C and M are integers that are uniquely determined by the structure of the neural network.

[0040] W = [search range image width / M] H = [search range image height / M] Note that [x] indicates the largest integer not exceeding x.

[0041] In S407, the feature correlation map calculation unit 206 calculates the correlation between each of the two or more template features obtained in S404 and the search range feature obtained in S406, and calculates the feature correlation map 308. The correlation calculation may use depthwise (dw) convolution with the template feature as the kernel. In addition, the width, height, and number of channels of the feature correlation map are all set to be equal regardless of the correlation map. For example, after zero padding of the feature map, dw convolution with the template feature as the kernel is applied with a stride of 1.

[0042] Specifically, zero padding adds [W T / 2] pixels at the top and bottom TFor each [1 / 2] pixel, add a pixel with a value of 0. A stride of 1 means that the template feature is applied at intervals of 1 pixel, and the feature map after zero padding is applied by sliding 1 pixel at a time from the top left to the bottom right.

[0043] In S408, first, the weighting coefficient acquisition unit 207 acquires the weighting coefficient vector 309 of the feature correlation map from the storage unit 210. The weighting coefficient vector 309 is an N-dimensional vector, and the value of each dimension represents the weighting coefficient of each of the N feature correlation maps. The value of the weighting coefficient is determined by learning. The learning method will be described later. The feature correlation map integration unit 208 multiplies each of the feature correlation maps 308 obtained in S407 by the weighting coefficient 309 obtained from the weighting coefficient acquisition unit 207. Thereafter, the feature correlation map integration unit 208 combines the feature correlation maps in the channel direction to obtain an integrated feature 311. For example, when there are two feature correlation maps with widths, heights, and channel numbers W, H, and C, the widths, heights, and channel numbers of the obtained integrated feature 311 are W, H, and 2C.

[0044] In S409, the tracking target detection unit 209 inputs the integrated features 311 obtained in S408 to the tracking target detection neural network 312, and outputs a likelihood map and a size map. Then, based on these maps, the position and size of the tracking target object in the search range image are estimated. After that, the estimation results of the position and size of the tracking target object are stored in the storage unit 210.

[0045] FIG. 6 is a diagram showing an example of an input image (search range image) and an inference result (likelihood map). The likelihood map is a map as shown in likelihood map 603 in FIG. 6(b), and takes real values ​​from 0 to 1. If the pixel value at the position where the tracked object (an elephant in this case) exists is relatively large compared to the pixel values ​​in the other maps, tracking can be performed correctly. The size map is two maps of the same size as the likelihood map, one for estimating width and the other for estimating height. For the width estimation map, it is sufficient that the value of the pixel corresponding to the center position 604 of the tracked object is the width of the tracked object. The same applies to the height estimation map.

[0046] In S410, the tracking target detection unit 205 determines whether to end the inference. The end of the inference may be determined based on an instruction specified by the input unit 105.

[0047] <Explanation of learning process> The learning process (the process of learning the parameters and weighting coefficient vectors of the neural network used in the above-mentioned inference) will be described below. In this embodiment, the learning method of the Siam method will be described.

[0048] <Device configuration> 7 is a diagram showing the functional configuration of the information processing device during learning processing. The information processing device 1 has a learning data acquisition unit 701, a template region setting unit 702, a template feature extraction unit 703, and a search range feature extraction unit 704. It also has a feature correlation map calculation unit 705, a weighting coefficient acquisition unit 706, a feature correlation map integration unit 207, a tracking target detection unit 708, a first error calculation unit 709, a parameter update unit 710, and a storage unit 711.

[0049] Template feature extraction section 703 to tracking target detection section 708 and storage section 711 use the same components as template feature extraction section 204 to tracking target detection section 209 and storage section 210 during inference processing (FIG. 2), respectively.

[0050] The learning data acquisition unit 701 acquires two images showing the same object and correct data on the position and size of the object present in the image from the same video sequence stored in the storage unit 711. Hereinafter, the correct data will be referred to as GT (Ground Truth). The two images will be the template image and the search range image, respectively.

[0051] Based on the GT acquired by the learning data acquisition unit 701, the template region setting unit 702 sets two or more template regions from within the template image acquired by the learning data acquisition unit 701, and cuts out template region images. As with inference, the region to be used as the template is selected from a group of object-like regions detected using an objectness detector, which overlap with the tracking target region. In addition, the number of template region images to be acquired may be fixed, and in the following description, it is assumed that N template region images are acquired.

[0052] The first error calculation unit 709 calculates the loss for the estimation result of the position and size of the tracking target object. Specifically, the first error calculation unit 709 calculates the error between the estimated value of the position and size of the tracking target object obtained by the tracking target detection unit 708 and the GT obtained by the learning data acquisition unit 701.

[0053] The parameter update unit 710 updates the parameters of the neural network based on the loss obtained by the first error calculation unit 709 , and stores the updated parameters in the storage unit 711 .

[0054] <Device Operation> 8 is a flowchart of the learning process. Note that the information processing device does not necessarily have to perform all of the steps described in this flowchart.

[0055] In S801, the learning data acquisition unit 701 acquires one template image in which a tracking target object appears and one search range image from the same video sequence stored in the storage unit 711. The learning data acquisition unit 701 also acquires the GT of the center position and size (width, height) of the tracking target object present in the template image.

[0056] Fig. 9 is a diagram showing an example of a template image and a search range image. A template image 901 shown in Fig. 9(a) is a template image acquired by the learning data acquisition unit 201. The template image 901 includes a tracking target object 903.

[0057] Moreover, the tracking target BB 904 is an illustration of the GT of the position and size of the tracking target object 903 as BB surrounding the tracking target object 903. Furthermore, the learning data acquisition unit 701 cuts out an image of the periphery of the region of the tracking target object in the template image as a template based on the position and size of the tracking target, and resizes it. It is sufficient to cut out a constant multiple of the size of the tracking target object with the position of the tracking target object as the center. FIG. 9(c) is an example of cutting out the periphery of the region of the tracking target object (region 902 in FIG. 9(a)) from the template image.

[0058] In S802, the template region setting unit 702 sets N template regions based on the GT of the position and size of the tracking target object acquired by the learning data acquisition unit 701 in S801, and obtains template region images by cutting out images of the regions. That is, N template region images are obtained from within the image cut out from the template image by the learning data acquisition unit 701 in S801.

[0059] The method of setting the template region is the same as that of the template region setting unit 403, and the region with the large IoU with the tracking target BB is set among the template region candidates detected by objectness detection. In FIG. 9(c), an image 909 is an image cut out from the template image by the learning data acquisition unit 701, and a region 910 is displayed as GT and BB, which are the positions and sizes of the tracking target object. Regions 911, 912, and 913 represent template region candidates. Among the template region candidates, the regions selected as the template region are regions 911 and 913. Region 914 in FIG. 9(d) and region 915 in FIG. 9(e) are examples of template region images cut out from regions 911 and 913, respectively, and the region within the frame (the region displayed dark) corresponds to the template region image. Note that the template region images shown in regions 914 and 915 are illustrated enlarged (resized) for the purpose of explanation, and therefore the size is different from that of regions 911 and 913. However, in actual processing, resizing after cutting out may not be performed.

[0060] In S803, the template feature extraction unit 703 and the learning data acquisition unit 701 input the template region image obtained in S804 to the feature extraction NN 302 in the same procedure as in S404, and obtain the feature map 303 of the template region image.

[0061] Fig. 11 is a diagram showing an example of a feature map of a search range image. A template feature 1102 shown in Fig. 11(b) and a template feature 1103 shown in Fig. 11(c) represent examples of features extracted from template region images 914 and 915.

[0062] In S804, the learning data acquisition unit 701 acquires an image for searching the tracking target object in the same procedure as S405. For example, an image at a different time in the same video sequence as the image selected in S801 is acquired as an image for searching the tracking target object, and a search range image is obtained by cutting out and resizing the image around the tracking target object. Figure 9(f) 916 is an example in which the area 906 in Figure 9(b) is cut out as a search range image.

[0063] In S805, the search range feature extraction unit 704 and the learning data acquisition unit 701 input the search range image obtained in S804 to the feature extraction NN 305 in the same procedure as in S406, and obtain the feature map 306 of the search range image. A feature map 1101 shown in Fig. 11(a) represents an example of a feature map extracted from the search range image 916 shown in Fig. 9(f).

[0064] In S806, the feature correlation map calculation unit 206 calculates the correlation between each of the N template features obtained in S803 and the search range feature obtained in S805 in the same manner as in S407, and calculates the feature correlation map 308.

[0065] In S807, the feature correlation map integration unit 707 combines the feature correlation maps to obtain the integrated feature 311. Specifically, first, the weighting coefficient acquisition unit 706 acquires the weighting coefficient vector 309 of the feature correlation map from the storage unit 711 in the same procedure as in S408. Then, the feature correlation map integration unit 208 multiplies each of the feature correlation maps 308 obtained in S407 by the weighting coefficient vector 309 obtained from the weighting coefficient acquisition unit 207, and combines the feature correlation maps in the channel direction.

[0066] In S808, the tracked target detection unit 708 inputs the integrated features 311 obtained in S707 to the tracked target detection neural network 312 in the same manner as in S409, and outputs a likelihood map and a size map. Then, based on these maps, the position and size of the tracked target object in the search range image are estimated.

[0067] In S809, the first error calculation unit 709 calculates the loss for the inference result of the position and size of the tracking target. The purpose is to proceed with learning so that the tracking target object can be correctly detected. Therefore, the loss Loss for the estimated "position" of the tracking target object is calculated as c and loss vs. size s Calculate.

[0068] Loss cis defined as the following equation (2). In equation (2), the likelihood map of the position of the tracked object obtained in S808 is inf , GT map is C gt Tooki, Map C inf and Map C gt Calculate the sum of squared errors for each pixel of C gt is a map in which the value of a position where a tracked object exists is "1" and the value of other positions is "0".

[0069]

number

[0070] Fig. 10 is a diagram showing an example of an input image (search range image), correct answer data (GT map), and inference results (likelihood map). A search range image 1001 represents the same search range image as 906 in Fig. 9. Also shown are a GT map 1004, which is correct answer data, and a likelihood map 1008, which is the inference result.

[0071] Loss s is defined as the following equation (3). In equation (3), the output map W inf , H inf And the GT map W gt , H gt Calculate the sum of squared errors for each pixel.

[0072]

number

[0073] Here, W gt and H gt The width and height values ​​of the tracked object are embedded in the position of the tracked object. By calculating the loss using Equation (3), W inf , H inf Even in this case, the learning proceeds so that the width and height of the tracked object can be inferred from the position of the tracked object. By combining the above two losses, we obtain Equation (4).

[0074]

number

[0075] Here, the loss is described in the form of Mean Squared Error (MSE), but the loss is not limited to MSE. Smooth L1 loss and the like may be used. There is no limitation on the formula for the loss. Also, the loss function for the position and the loss function for the size may be different.

[0076] In S810, the parameter update unit 209 updates the parameters of the neural network using the backpropagation method based on the loss calculated in S312. The parameters to be updated here refer to the weights of the convolution layers constituting the feature extraction NNs 302 and 305 and the tracked target detection neural network 312 shown in Fig. 3, the weighting coefficient vector 309, etc. In other words, the calculation of the weighting coefficient vector (coefficient calculation) is also performed so that image features effective for detecting the tracked target can be obtained from the search range image.

[0077] In S811, the parameter update unit 710 stores the updated neural network parameters in the storage unit 711. Thereafter, weights are read into the neural network. Specifically, the template feature extraction unit 204 and the search range feature extraction unit 205 read the neural network parameters for feature extraction from the storage unit 711, and the tracking target detection unit 209 reads the neural network parameters for tracking target detection from the storage unit 711. The flow up to this point is regarded as one iteration of learning.

[0078] In S812, the parameter update unit 710 determines whether to end the learning. The learning may be ended when the loss value obtained by the formula (4) becomes smaller than a predetermined threshold value, or may be ended when the learning is performed a predetermined number of times.

[0079] As described above, according to the first embodiment, two or more partial regions including the tracking target region are used as templates, and the neural network is trained to integrate the correlation map. By using the neural network trained in this way, it is possible to perform tracking that effectively uses the features of multiple different regions of the tracking target. In particular, it is possible to perform tracking that is robust against drift into the background region and changes in the appearance of the tracking target.

[0080] (Variation 1-1) As a modified example 1-1, an example in which the result of region division is used for template setting will be described. Specifically, in the template setting in the first embodiment, a region divider is used to set the template region. Here, region division means that the same value is assigned to pixels in image regions in which the same object is likely to appear, and the entire original image is divided into several regions.

[0081] In this modification, as in the first embodiment, when a template region is selected from a group of template region candidates, a region with a large overlap with the tracking target is set as the template region. In the first embodiment, the IoU between BBs is used to calculate this overlap, but in this modification, the result of region division is used instead of the IoU. This makes it difficult for a large background region to be used as a template region, and it is expected that erroneous tracking due to the background region being regarded as a tracking target feature will be reduced. In this modification, the template region setting unit 203 includes a region divider in addition to the objectness detector.

[0082] In S403, template region setting unit 203 uses the result of region division instead of IoU when selecting a template region from the group of template region candidates. First, the template image acquired in S401 is input to a region divider, and then a tracking target region is selected and a mask image of the tracking target region is obtained. The mask image is assumed to be an image of the same size as the input image, with the pixel values ​​of the tracking target region being "1" and the pixel values ​​of other regions being "0". How to create a mask image of the tracking target region will be described later.

[0083] Then, the template region setting unit 203 sets the region with a large overlap between the template region candidate and the tracking target region as the template region. Specifically, the total value of the mask value in the tracking target candidate BB region in the image is calculated, and the ratio of the total value to the area of ​​the BB region is selected to be equal to or greater than a threshold value.

[0084] Next, a method for creating a mask image of the tracking target region will be described in detail. Here, the region divider may be the Felzenszwalb method (Reference 2), but is not limited to this. Reference 2: Felzenszwalb and Huttenlocher, "Efficient Graph-Based Image Segmentation", IJCV 2004

[0085] First, the template region setting unit 203 inputs the template image acquired in S401 to the region divider. The region divider outputs a region division image in which the template image is divided into a plurality of regions. In this image, the same value is assigned to pixels in regions where the same object is likely to exist. After that, the template region setting unit 203 selects a tracking target region from the region division image based on the tracking target BB set by the tracking target setting unit 202 in S402. For example, it selects the region with the largest overlap with the tracking target BB. Then, the template region setting unit 203 creates an image in which the pixel value of the selected tracking target region is "1" and the other pixel values ​​are "0", and sets this as a mask image of the tracking target region.

[0086] According to the modified example 1-1, in the template setting, the result of the region division is used instead of IoU. This makes it difficult for a large background region to be used as a template region, and it is expected that the tracker will be less likely to regard the background region as a tracking target feature and erroneously track it.

[0087] (Variation 1-2) As a modified example 1-2, an example in which the result of specific object detection is used for template setting will be described. Specifically, a detector that performs specific object detection is used for setting the template region, and the performance of tracking the specific object is improved. For example, a detector detects a plurality of different parts of an object (animal, person, etc.). More specifically, in order to track a human face, a detector for "face" or "upper body" is used to detect the face and the upper body, and the face region and the upper body region are set as template regions, respectively. Then, the network is trained to track the face by integrating the feature correlation map obtained by the template of the upper body region and the face region. As a result, even when the face region is difficult to see due to occlusion, etc., it becomes possible to track the face using the upper body region as a clue, and the face tracking performance is improved. Below, the inference method and learning method will be described focusing on the parts different from the first embodiment.

[0088] <Inference processing> An example of the functional configuration during inference processing is the same as that in Fig. 2. However, the template region setting unit 203 includes two or more types of specific object detectors instead of an objectness detector. Here, a face detector and an upper body detector are used.

[0089] In S403, the template area setting unit 203 first inputs an image to a face detector and obtains a BB of the face area of ​​the person to be tracked. Next, the upper body area of ​​the same person is estimated. For example, an image is input to an upper body detector, and multiple BBs of upper body area candidates are output. Then, the BB that has a large overlap with the BB of the face area is set as the BB of the upper body area. Then, the BB of the face area and the BB of the upper body area are each set as the template area.

[0090] In S404, the template feature extraction unit 204 inputs the two template region images obtained in S403 to the feature extraction NN 302, and obtains template features 303 corresponding to the face region and upper body region.

[0091] In S408, the feature correlation map integration unit 208 integrates the feature correlation maps in the channel direction to obtain an integrated feature 311. At this time, the order in which the feature correlation map corresponding to the face region and the feature correlation map corresponding to the upper body region are integrated may be determined in advance.

[0092] <Learning process> An example of the functional configuration during the learning process is similar to that in Fig. 7. However, the operation of each functional unit differs from that of the first embodiment in the following manner.

[0093] The learning data acquisition unit 701 acquires two images showing the same object and GTs of the positions and sizes of the faces and upper bodies present in the images from the same video sequence stored in the storage unit 711. Here, it is assumed that the video sequence shows one or more human bodies, and that the GTs of the faces and upper bodies are assigned to each human body.

[0094] Template region setting unit 702 uses a common unit with template region setting unit 203 and includes two or more specific object detectors. Based on the GT of the face and the GT of the upper body acquired by learning data acquisition unit 701, template region setting unit 702 sets a template region from within the template image acquired by learning data acquisition unit 701, and cuts out a template region image.

[0095] The tracking target detection unit 708 estimates the position and size of the face. The integrated features calculated by the feature correlation map integration unit 707 are input to a neural network for tracking target detection, and a likelihood map that strongly reacts to the position of the tracking target object and a size estimation map are output. Then, the position and size of the face are estimated using the likelihood map and the size estimation map.

[0096] The first error calculation unit 709 calculates the loss for the estimation result of the face position and size. Specifically, it calculates the error between the estimated value of the position and size obtained by the tracking target detection unit 708 and the GT of the face position and size obtained by the learning data acquisition unit 701.

[0097] In S801, the learning data acquisition unit 701 acquires one template image and one search range image showing the same person from the same video sequence stored in the memory unit 711, and acquires a GT of the position and size of the face and upper body of the person present in the image.

[0098] In S802, the template area setting unit 702 obtains a template area image based on the GT of the positions and sizes of the face and upper body of the same person obtained in S801 by the learning data obtaining unit 701. That is, the face area and the upper body area are set as template areas, and the image of each area is cut out to obtain the template area image.

[0099] In S804, the learning data acquisition unit 701 acquires images for searching for a tracking target object in the same procedure as in S405. For example, an image at a different time in the same video sequence as the image selected in S801 is acquired as an image for searching for a face to be tracked, and an area around the upper body of the person having the face is cut out and resized to create a search range image.

[0100] In S806, the feature correlation map calculation unit 206 calculates the correlation between each of the two template features obtained in S803 and the search range feature obtained in S805 in a similar procedure to S407, and calculates the feature correlation map 308.

[0101] In S807, the same procedure as that described in the first embodiment is performed. Here, the feature correlation map integration unit 208 integrates the feature correlation maps in the channel direction to obtain the integrated feature 311. At that time, the order in which the feature correlation maps corresponding to the face region and the feature correlation maps corresponding to the upper body region are integrated may be determined in advance, and the integration may be performed in the same order as during inference.

[0102] In S808, the tracking target detection unit 708 estimates the position and size of the face. The integrated features calculated by the feature correlation map integration unit 707 are input to a neural network for tracking target detection, and a likelihood map that strongly reacts to the position of the face of the tracking target and a size estimation map are output. Then, the position and size of the face that is the tracking target are estimated using the likelihood map and the size estimation map.

[0103] In S809, the first error calculation unit 709 calculates a loss for the estimation result of the position and size of the face of the tracking target. Specifically, the first error calculation unit 709 calculates the error between the estimated value of the position and size obtained by the tracking target detection unit 708 and the GT of the position and size of the face obtained by the learning data acquisition unit 701.

[0104] Second embodiment As the second embodiment, a form will be described in which a region that appears frequently among detection frames obtained by an objectness detector is used as a template. That is, unlike the first embodiment, a partial region of the tracking target that appears most frequently among images at multiple times in the same video sequence is set as a template region.

[0105] In the first embodiment, one image at a certain time is selected from a video sequence, and multiple regions from which objectness is detected are set as template regions. However, objectness detection in the selected image is not necessarily successful, and there is a possibility that a background region that happens to be reflected in the image is used as a clue to track the tracking target. In addition, there is a possibility that a region that happens to have high objectness due to a sudden shadow or the like is set as the template region. Therefore, in the second embodiment, the possibility of the above-mentioned situation occurring is reduced by setting a partial region of the tracking target that appears most frequently as the template region from images at multiple times in the same video sequence. This makes it easier to set a template region that has a typical appearance of the tracking target, and it is expected that more robust tracking can be achieved.

[0106] The flow of the inference process in the second embodiment will be described below with reference to the differences from the first embodiment. In S401 in the first embodiment, the image acquisition unit 201 acquires at least two or more template images showing the same object from at least two or more times in the video sequence. Then, in S403, the template region setting unit sets the template region according to the flow shown in the flowchart of FIG.

[0107] FIG. 12 is a detailed flowchart of the template region setting in the second embodiment.

[0108] In S1201, the template region setting unit performs object detection on all acquired template images using a neural network for objectness detection, such as FCOS (Reference 1).

[0109] In S1202, the template region setting unit selects, from among the regions detected in S1201, a region having a large IoU (a predetermined amount of overlap) with the tracking target BB set in S402 as a template region candidate.

[0110] In S1203, the template region setting unit acquires image features of the template region. The image features may be intermediate features of an objectness detection neural network. In addition, the image features are all set to be the same size in order to be used for clustering in S1204. For example, a method of extracting features corresponding to the template region from the intermediate features of the objectness detection neural network using PrRoIPooling (Reference 3) or the like can be considered. Reference 3: Jiang et al., "Acquisition of Localization Confidence for Accurate Object Detection", ECCV2018

[0111] In S1204, the template region setting unit performs clustering on the image features. Specifically, among the features acquired in S1203, features with high similarity are assigned to the same group. The clustering may be performed using the K-means method, but the method is not limited thereto.

[0112] In S1205, the template region setting unit selects a region belonging to a cluster with a high occurrence count as a template region. The template region may be set to a plurality of regions from the cluster with the highest occurrence count as the template region. Alternatively, regions with a high likelihood of objectness detection may be selected one by one from different clusters.

[0113] In S1206, the template area setting unit cuts out the image of the template area.

[0114] As described above, according to the second embodiment, a partial region of a tracking target that appears across multiple time periods is set as a template region. Also, multiple partial regions are used as the template region. These two ideas are expected to result in more robust tracking compared to the first embodiment, which uses only one template for the entire region of the tracking target from one image showing the tracking target.

[0115] For example, in an image selected as a template image, the posture of the tracking target may look unusual, but if we focus on a partial region, it is highly likely that the target looks the same at any given time. This embodiment enables robust tracking by setting such a partial region as the template region.

[0116] Third embodiment As a third embodiment, a form in which a large template is hollowed out to obtain a small template will be described, which makes the feature extraction process more efficient.

[0117] In the first embodiment, when acquiring multiple template features, feature extraction is performed on each template region image using a neural network. Therefore, if there is overlap between template regions, regions where features are extracted twice are generated, resulting in extra calculation costs. Therefore, in the third embodiment, in order to improve processing efficiency, inclusion region features are extracted from one region (inclusion region) that contains all the template regions, and then region features for each template region are cut out. This allows feature extraction to be completed in one go, making it possible to determine multiple template features more efficiently.

[0118] The flow of the inference process in the third embodiment will be described below, focusing on the differences from the first embodiment. In S403, the template region setting unit 203 selects two or more template regions. In S404, the template feature extraction unit 204 acquires a feature map from the image cut out in S401 using the feature extraction neural network 302.

[0119] In S403, the template feature extraction unit 204 cuts out the feature map based on the set template region to obtain the template feature. A possible method of cutting out the feature corresponding to the template region set in S403 from the feature map acquired from the image cut out in S401 using PrRoIPooling (Reference 3) or the like.

[0120] As described above, according to the third embodiment, by completing feature extraction from the template region image at once, it is possible to reduce calculation costs compared to extracting features for each template region image.

[0121] (Fourth embodiment) As a fourth embodiment, a mode in which two or more regions having an inclusion relationship are set as a template region will be described.

[0122] For example, a large area including the entire tracking target and a small area at the center of the tracking target are set as a large template area and a small template area, respectively. It is considered that the template features obtained from the large template area capture global shape features, and the template features obtained from the small template area capture local features such as color (color features). By using a neural network trained to weight and integrate the correlation features obtained thereby, it is expected that tracking with a balance between shape and color can be performed. Also, by obtaining the template of the small area from the center of the tracking target area, it is expected to reduce the likelihood that the background area is included in the template and that features such as the color of the background area are tracked as clues for the tracking target.

[0123] In S403, the template area setting unit 203 inputs the image into the objectness detector and outputs one BB (BB 1 hereinafter referred to as such) of the region with the highest objectness. It is assumed that BB1 includes the entire tracking target. Next, the template area setting unit 203 obtains a rectangular area BB 2 included in BB1, and sets BB 1 and BB 2 as the template areas. BB 2 may be, for example, a rectangular area having a width and height that are s times (where s satisfies 0 < s < 1) that of BB 1 and whose centers coincide. Alternatively, a mask of the tracking target area may be created in the same manner as described in Modification 1-1, and a BB in which all pixels are the tracking target area may be set as BB 2 .

[0124] In S404, the template feature extraction unit 204 extracts template features, for example, in the same procedure as in the third embodiment. First, a feature map is obtained from the image cut out in S401 using the feature extraction neural network 302. Thereafter, the feature map is cropped based on the template area set in S403 to obtain the template features.

[0125] As described above, according to the fourth embodiment, a neural network is trained to use multiple regions in an inclusive relationship as templates and to weight and integrate the correlation features obtained from the features of these templates. It is expected that using a neural network trained in this way will enable tracking with a balance between shape and color. In addition, by acquiring a small region template from the center of the tracking target region, it is expected that the background region is less likely to be included in the template, and tracking using features such as the color of the background region as a clue to the tracking target will be reduced.

[0126] Fifth embodiment As the fifth embodiment, a form in which the degree of importance of color information is adjusted by changing the way in which style loss is added for each template will be described. Specifically, in the learning process, a constraint is explicitly given to adjust the degree of importance of color information for each template region. This promotes learning that emphasizes different information for each template region.

[0127] More specifically, S809 uses a loss called "style loss" that adjusts the degree of importance of color information in the calculation of the loss used in training the neural network. Style loss is a loss used in CNN-based style transfer methods such as Neural Style Transfer (Reference 2), which is a loss that brings the colors and textures of two images closer together.

[0128] Style transfer is the task of generating a new image from two images, where the style (color and texture information) of one image (style image) is reflected in the other image (content image). This method learns to make the color and texture information of the CNN features extracted from each image similar between the new image and the texture image, and to make the color and texture information of the CNN features extracted from each image similar between the new image and the content image.

[0129] The loss used to calculate the similarity of texture information is called style loss, and can be derived by calculating color and texture information as a Gram matrix of CNN features and calculating the difference between the two Gram matrices. In this embodiment, this style loss is used to calculate the loss, and by changing the lower limit of the style loss for each template region, learning that emphasizes different information for each template region is promoted. As in the first embodiment, the feature extraction neural networks 302 and 305 are CNNs.

[0130] Fig. 13 is a diagram showing the functional configuration of the information processing device 1 during learning processing in the fifth embodiment. A second error calculation unit 1310 is added to Fig. 7 (first embodiment). The learning data acquisition unit 1301 to the first error calculation unit 1309, the parameter update unit 1311, and the storage unit 1312 use the same components as the learning data acquisition unit 701 to the first error calculation unit 709, the parameter update unit 710, and the storage unit 711, respectively.

[0131] The flow of the learning process in the fifth embodiment will be described with reference to Fig. 8. The processes from S801 to S808 and S811 are similar to those in the first embodiment.

[0132] In S809, the first error calculation unit calculates the first error in the same procedure as in the first embodiment. Next, the second error calculation unit 1310 calculates the second error in the procedure described later. First, the second error calculation unit 1310 calculates the N template features F extracted by the template feature extraction unit 1303. 1 ~F N Among them, the n-th template feature F n For each feature, cut out features of the same size from the search range feature and use the search range feature G n Then, N sets of template features F n and the search range feature G n For each of them, the Gram matrix P n , Q n where each Gram matrix P n , Q n The element in the i-th row and j-th column of is calculated as shown in equations (5) and (6), respectively.

[0133]

number

[0134]

number

[0135] Here, F n ik , G n ik is the feature F n , G n It represents the kth pixel counting from the top left among the W x H pixels in the i-th channel.

[0136] Then, N sets of template features F n and the search range feature G n The difference between the Gram matrices for each is calculated as shown in Equation (7). The difference between the Gram matrices is qualitatively determined by the template feature F n and the search range feature G n The more similar the color and texture, the smaller it becomes.

[0137]

number

[0138] Here, R represents the number of elements in the Gram matrix. In addition, when the parameter m is large, it has the effect of suppressing the difference in the values ​​of the elements in the Gram matrix from becoming small. By learning this parameter m, it is possible to change the degree of importance of color information for each template feature.

[0139] Then, the weighted sum is calculated as in equation (8) and is taken as the second error. n w represents the weighting coefficient of the Gram matrix difference calculated by equation (7) for each template feature and search range feature pair. n shall take a value of 0 or greater.

[0140]

number

[0141] In S810, the parameter update unit 1311 calculates the sum of the losses calculated by the first error calculation unit 1309 and the second error calculation unit 1310, and updates the parameters based on the sum of the losses. Here, the parameters of the neural network are updated using the backpropagation method. The parameters to be updated here are the weights of the convolution layers constituting the feature extraction neural networks 302 and 305 and the tracked target detection neural network 312 shown in Fig. 3, and the parameter m in equation (7).

[0142] As described above, according to the fifth embodiment, the importance of color information is adjusted or suppressed by the template, and the background color information is prevented from being taken as a feature of the tracking target. This makes it possible to at least partially solve the problem of tracking using the Siam method.

[0143] The disclosure of this specification includes the following information processing device, control method, and program. (Item 1) An information processing device for tracking a tracking target in a video sequence, comprising: A region setting means for setting a first region and a second region corresponding to different image regions of the tracking target in a template image including an image of the tracking target that is included in the video sequence; a first feature extraction means for extracting a first region feature and a second region feature, which are template features, from the first region and the second region using a feature extraction neural network (NN); A second feature extraction means for extracting search range features from a search range image included in the video sequence and subsequent to the template image, using the feature extraction neural network; a correlation map deriving means for performing a correlation calculation between the first region feature and the second region feature, and the search range feature, and deriving a first feature correlation map and a second feature correlation map, which are correlation maps; an integration means for integrating the first feature correlation map and the second feature correlation map to derive an integrated feature; a target detection means for detecting the tracking target from the search range image based on the integrated features; An information processing device comprising: (Item 2) The integrating means integrates the first feature correlation map and the second feature correlation map according to weighting coefficients corresponding to the first feature correlation map and the second feature correlation map. 2. The information processing device according to item 1, (Item 3) The weighting coefficient calculation means further includes: The coefficient calculation means includes a neural network that calculates a weighting coefficient according to the tracking target using the search range feature. 3. The information processing device according to item 2. (Item 4) further comprising a region dividing means for dividing the image into a plurality of regions by assigning the same value to pixels which are likely to include the same object; The region setting means sets the first region and the second region based on an image region of the tracking target estimated based on the region division of the template image by the region dividing means. 4. The information processing device according to any one of items 1 to 3. (Item 5) The image processing device further includes a detection unit for performing objectness detection to detect an image region in which the likelihood of an object being present is relatively high within the image, The region setting means sets the first region and the second region based on the image region detected by the detection means for the template image. 4. The information processing device according to any one of items 1 to 3. (Item 6) The detection means performs objectness detection on a plurality of template images; The region setting means performs clustering of image features for image regions that have a predetermined or greater overlap with the tracking target among image regions based on objectness detection of the template image by the detection means, and sets the first region and the second region based on image regions corresponding to clusters that appear frequently. 6. The information processing device according to item 5, (Item 7) The image features are features of a neural network included in the detection means. 7. The information processing device according to item 6, (Item 8) the tracking target is an animal, The first region and the second region correspond to different parts of an animal. 8. The information processing device according to any one of items 1 to 7, (Item 9) The first region is a region that includes the second region. 9. The information processing device according to any one of items 1 to 8, (Item 10) The first feature extraction means extracts an inclusion region feature for an inclusion region that includes the first region and the second region using the feature extraction NN, and extracts the first region feature and the second region feature by cutting out region features corresponding to the first region and the second region from the inclusion region feature. 10. The information processing device according to any one of items 1 to 9. (Item 11) The weighting coefficient is derived by the target detection means learning to obtain image features effective for detecting the tracking target from the search range image based on the integrated features. 3. The information processing device according to item 2. (Item 12) The feature extraction NN is trained to emphasize local color features for a relatively small region of the first region and the second region. 12. The information processing device according to any one of items 1 to 11, (Item 13) The feature extraction NN is trained to emphasize global shape features for a relatively large region of the first region and the second region. 13. The information processing device according to any one of items 1 to 12. (Item 14) A method for controlling an information processing device that tracks a tracking target in a video sequence, comprising: A region setting step of setting a first region and a second region corresponding to different image regions of the tracking target in a template image including an image of the tracking target that is included in the video sequence; a first feature extraction step of extracting a first region feature and a second region feature, which are template features, from the first region and the second region using a feature extraction neural network (NN); a second feature extraction step of extracting search range features from a search range image included in the video sequence and subsequent to the template image, using the feature extraction neural network; a correlation map deriving step of performing a correlation calculation between the first region feature and the second region feature, and the search range feature, to derive a first feature correlation map and a second feature correlation map, which are correlation maps; an integration step of integrating the first feature correlation map and the second feature correlation map to derive an integrated feature; a target detection step of detecting the tracking target from the search range image based on the integrated features; A control method comprising: (Item 15) Item 15. A program for causing a computer to execute the control method according to item 14.

[0144] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0145] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0146] 201 image acquisition unit; 202 tracking target setting unit; 203 template region setting unit; 204 template feature extraction unit; 205 search range feature extraction unit; 206 feature correlation map calculation unit; 207 weighting coefficient acquisition unit; 208 feature correlation map integration unit; 209 tracking target detection unit; 210 storage unit

Claims

1. An information processing device for tracking a tracking target in a video sequence, comprising: detection means for performing objectness detection in images in the video sequence; a region setting means for setting a first region and a second region, which correspond to different image regions of the tracking target and are based on a detection result of the objectness detection, in a template image that is included in the video sequence and includes an image of the tracking target; a first feature extraction means for extracting a first region feature and a second region feature from the first region and the second region using a feature extraction neural network (NN); a second feature extraction means for extracting search range features from a search range image included in the video sequence and following the template image using the feature extraction neural network; a correlation map deriving means for calculating correlations between the first region feature and the second region feature and the search range feature, respectively, to derive a first feature correlation map and a second feature correlation map; an integration means for integrating the first feature correlation map and the second feature correlation map to derive an integrated feature; a target detection means for detecting the tracking target from the search range image based on the integrated features; Equipped with The area setting means sets the first area and the second area based on an image area that appears frequently among the image areas based on the objectness detection.

2. The integrating means integrates the first feature correlation map and the second feature correlation map according to weighting coefficients corresponding to the first feature correlation map and the second feature correlation map.

2. The information processing apparatus according to claim 1, wherein:

3. further comprising coefficient calculation means for calculating the weighting coefficients; The coefficient calculation means includes a neural network that calculates a weighting coefficient according to the tracking target using the search range feature.

3. The information processing apparatus according to claim 2, wherein:

4. further comprising area division means for dividing the image into a plurality of areas by assigning the same value to pixels that are likely to show the same object; The region setting means sets the first region and the second region based on an image region of the tracking target estimated based on region division of the template image by the region dividing means.

2. The information processing apparatus according to claim 1, wherein:

5. The objectness detection detects image regions in which the likelihood of an object being present is relatively high.

2. The information processing apparatus according to claim 1, wherein:

6. the detection means performs objectness detection on a plurality of template images; The region setting means performs clustering of image features for image regions that have a predetermined or greater overlap with the tracking target among image regions based on objectness detection of the template image by the detection means, and sets the first region and the second region based on image regions corresponding to clusters that appear frequently.

2. The information processing apparatus according to claim 1, wherein:

7. The image features are features of a neural network included in the detection means.

7. The information processing apparatus according to claim 6,

8. the tracking target is an animal, The first region and the second region correspond to different parts of an animal.

2. The information processing apparatus according to claim 1, wherein:

9. The first region is a region that includes the second region.

2. The information processing apparatus according to claim 1, wherein:

10. The first feature extraction means extracts an inclusion region feature for an inclusion region that includes the first region and the second region using the feature extraction NN, and extracts the first region feature and the second region feature by cutting out region features corresponding to the first region and the second region from the inclusion region feature.

2. The information processing apparatus according to claim 1, wherein:

11. The weighting coefficients are derived by the target detection means learning to obtain image features effective for detecting the tracking target from the search range image based on the integrated features.

3. The information processing apparatus according to claim 2, wherein:

12. The feature extraction NN is trained to emphasize local color features for a relatively small region of the first region and the second region.

2. The information processing apparatus according to claim 1, wherein:

13. The feature extraction NN is trained to emphasize global shape features for a relatively large region of the first region and the second region.

2. The information processing apparatus according to claim 1, wherein:

14. A method for controlling an information processing device that tracks a tracking target in a video sequence, comprising: a detection step for performing objectness detection in images in the video sequence; a region setting step of setting a first region and a second region, which correspond to different image regions of the tracking target and are based on a detection result of the objectness detection, in a template image that is included in the video sequence and includes an image of the tracking target; a first feature extraction step of extracting first region features and second region features from the first region and the second region using a feature extraction neural network (NN); a second feature extraction step of extracting search range features from a search range image included in the video sequence and following the template image using the feature extraction neural network; a correlation map deriving step of calculating correlations between the first region feature and the second region feature and the search range feature, and deriving a first feature correlation map and a second feature correlation map; an integration step of integrating the first feature correlation map and the second feature correlation map to derive an integrated feature; a target detection step of detecting the tracking target from the search range image based on the integrated features; Including, In the region setting step, the first region and the second region are set based on an image region that appears frequently among the image regions based on the objectness detection.

15. A program for causing a computer to execute the control method according to claim 14.

Images