Information processing device, learning device, and tracking method

Abstract

To track an object in a moving image.SOLUTION: An information processing device includes: first acquisition means that obtains a reference image in which a target object to be tracked is captured; determination means that determines a plurality of positions in the reference image for use in tracking processing of the target object; first generation means that generates a plurality of first feature amounts corresponding to the plurality of positions by inputting the reference image to a first processing unit; second acquisition means that obtains a search image for which the tracking processing of the target object is performed; second generation means that generates a second feature amount by inputting the search image to the first processing unit; and identifying means that identifies, a position of the target object included in the search image, on the basis of a result of correlation operation between each of the plurality of first feature amounts and the second feature amount.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for tracking an object in a moving image.

Background Art

[0002] Generally, when tracking a subject in a moving image, image information representing the characteristics of the subject is created before tracking, and tracking is performed by matching with the image information. As such techniques, there are those that use luminance or color information, those that use template matching, those that use a CNN (Convolutional Neural Network), etc. For example, when using template matching, the template image representing the subject becomes the image information for tracking, and when using a CNN, the CNN features of the subject area become the image information for tracking.

[0003] Such image information of the subject required for such tracking cannot be successfully tracked unless it appropriately includes information that is easy to track. For example, in Patent Document 1, by acquiring a plurality of templates for one tracking target, even if one template fails to track, it is possible to track using the remaining templates. At this time, the templates are acquired based on the shape, texture, and color of the object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the multiple template features described in Patent Document 1 were designed manually, and their effectiveness in tracking has not been verified. Furthermore, since the appropriate combination of template features differs for each object, it is necessary to consider different combination patterns of template features depending on the object. Therefore, countless template patterns need to be prepared to apply them to any object.

[0006] This invention has been made in view of these problems and aims to provide a technology that enables more appropriate tracking of objects in moving images. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is A first acquisition means for acquiring a reference image showing the target object to be tracked, The aforementioned reference image is input to the second processing unit to generate a location map having likelihood values ​​that represent the likelihood of a location having features suitable for tracking the target object, and based on the peak position in the location map, A determination means for determining multiple positions in the reference image used for tracking the target object, A first generation means inputs the aforementioned reference image to a first processing unit and generates a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition means for acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation means inputs the search image to the first processing unit to generate a second feature quantity, A means for identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, It has. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology that enables more appropriate tracking of objects within moving images. [Brief explanation of the drawing]

[0009] [Figure 1]This diagram shows the hardware configuration of an information processing device. [Figure 2] This diagram shows the functional configuration of the information processing device according to the first embodiment. [Figure 3] This is a processing flowchart of the information processing device in the first embodiment. [Figure 4] This diagram illustrates the acquisition of a reference image using an object detector. [Figure 5] This diagram illustrates the configuration of the feature location inference unit. [Figure 6] This diagram illustrates the generation of template features. [Figure 7] This is a diagram illustrating the search image. [Figure 8] This diagram illustrates how to obtain likelihood maps and size maps. [Figure 9] This diagram illustrates tracking using likelihood maps and size maps. [Figure 10] This figure shows the functional configuration of the learning device according to the first embodiment. [Figure 11] This is a processing flowchart of the learning device in the first embodiment. [Figure 12] This figure shows the functional configuration of the information processing device according to the second embodiment. [Figure 13] This figure illustrates the generation of template features in the second embodiment. [Figure 14] This figure shows the functional configuration of the learning device according to the second embodiment. [Figure 15] This diagram illustrates the configuration of the feature location inference unit in the third embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) As a first embodiment of the information processing apparatus according to the present invention, an information processing apparatus 100 that performs tracking processing on an object included in an image obtained by an imaging apparatus 200 will be described below as an example. In the following description, it is assumed that the imaging apparatus 200 and the information processing apparatus 100 are integrated digital cameras, but they may be configured as separate apparatuses.

[0012] <Configuration of Information Processing Apparatus> FIG. 1 is a diagram showing the hardware configuration of the information processing apparatus 100. The information processing apparatus 100 can be configured using a general-purpose information processing apparatus, and includes a CPU 101, a memory 102, an input unit 103, a storage unit 104, a display unit 105, a communication unit 106, and the like.

[0013] FIG. 2 is a diagram showing the functional configuration of the information processing apparatus according to the first embodiment. The imaging apparatus 200 can be configured using an optical system, an imaging element, etc., captures an image, and outputs it to the image acquisition unit 110 of the information processing apparatus 100. The information processing apparatus 100 includes an image acquisition unit 110, a feature amount position inference unit 111, a reference image acquisition unit 112, a search image acquisition unit 113, a feature amount generation unit 114, a collation unit 115, a detection unit 116, and a storage unit 104.

[0014] The storage unit 104 stores inference parameters learned by a learning apparatus 1001 described later. Specifically, it includes inference parameters of a model for inferring the position of template feature amounts and inference parameters of a model for tracking an object.

[0015] The image acquisition unit 110 acquires time-series images (frame images that make up the video) sequentially captured from the imaging device 200. The feature location inference unit 111 infers a location suitable for template features used for object tracking from objects in the image. The reference image acquisition unit 112 generates a reference image by cutting out the object to be tracked (target object) from the image. The search image generation unit 113 generates a search image by cutting out the region in which the target object to be tracked is searched from the entire image.

[0016] The feature generation unit 114 generates feature quantities for an image when it is input. The matching unit 115 matches the feature quantities of the search image with those of the reference image obtained from the feature generation unit 114 to generate a map representing the position and size of the target object in the search image. The detection unit 116 detects the target object in the search image based on the map representing the position and size of the target object obtained from the matching unit 115.

[0017] <Operation of the Information Processing Device> Figure 3 is a processing flowchart of the information processing device in the first embodiment. Specifically, it describes the process of specifying a target object to track and tracking the target object from video (multiple frame images in chronological order).

[0018] Figure 3(a) shows the process of specifying the object to be tracked. Specifically, it shows the process of setting the object to be tracked by the user specifying (for example, by touch operation) an object within the image (frame image) displayed on the digital camera's display. Figure 3(b) shows the process of tracking the target object in subsequent frame images. The tracking process in this embodiment will be explained based on the neural network (NN) method described in Non-Patent Literature 1 (Non-Patent Literature 1: Bo Li et al., "High Performance Visual Tracking with Siames Region Proposal Network", 2018).

[0019] In S301, the image acquisition unit 110 acquires the image captured by the imaging device 200. This image is assumed to contain the object to be tracked.

[0020] In S302, the reference image acquisition unit 112 acquires a reference image by cutting out the region of the object to be tracked from the image acquired by the image acquisition unit 110. The reference image acquisition unit 112 also acquires the position of the object to be tracked in the image through user operation (e.g., touch operation).

[0021] Figure 4 illustrates the acquisition of a reference image using an object detector. First, the reference image acquisition unit 112 inputs the image acquired from the image acquisition unit 110 as the input image 401 to the object detector 402. The object detector 402 outputs the center position and size of various objects in the input image as the detection result 403. In Figure 4, two object frames corresponding to two objects (a vehicle and a person) are output. Such an object detector can be realized using an object detection method such as that described in Non-Patent Literature 2 (Non-Patent Literature 2: Xingyi Zhou et al., "Objects as Points", 2019).

[0022] Next, from the detection results 403 (object frames) output by the object detector, the object frame closest to the position specified by operation 404 (user specification of target object to track) is obtained. A partial image corresponding to the obtained object frame is extracted and obtained as a reference image 405 (reference image generation). Alternatively, a method such as extracting a region of a fixed size from the specified position may also be used.

[0023] In S303, the feature location inference unit 111 infers and determines the positions of template features (hereinafter referred to as feature positions) suitable for tracking objects in the reference image acquired from the reference image acquisition unit 112. As a result of the inference, the feature location inference unit 111 generates a feature position map showing the weight distribution of the positions of template features suitable for tracking within the reference image, and outputs the feature positions.

[0024] The feature location inference unit 111 includes an inference unit, which is a second processing unit that, upon inputting an image, infers the positions of template features suitable for tracking objects in the image. Such an inference unit can be implemented using known machine learning methods, such as neural networks (NNs) or support vector machines (SVMs). The memory unit 104 stores the weights of the NN, which have been trained in advance, as inference parameters. During inference, the inference unit 111 reads the inference parameters from the memory unit 104 and performs inference. Here, we will describe the feature location inference unit 111 implemented using a neural network.

[0025] Figure 5 illustrates the configuration of the feature location inference unit 111. The feature location inference unit 111 includes, as an example, a multilayer CNN 500. The multilayer CNN 500 has a network structure used in object detection, such as in Non-Patent Literature 2. In such a network, when an image is input to the Backbone network, convolution and pooling operations are repeatedly performed on the image, and as a result, feature quantities that indicate some features of the image are output. The feature quantities obtained through the Backbone 501 are input to the feature location inference CNN 502. As a result, the feature location inference CNN 502 outputs a feature location map 503 that indicates the locations of template features that are easy to track for objects in the image. The learning method for the feature location inference unit 111 will be described later.

[0026] The feature location map 503 becomes smaller in resolution (size) as the image undergoes repeated convolution and pooling. The feature location map contains likelihood values ​​that represent the likelihood of a location having easily trackable features, corresponding to the positions on the map in the input image.

[0027] In this embodiment, three template features (locations) are inferred for a single object. By using the three template features obtained based on these inference results, tracking performance can be improved. One feature location map is output, and responses showing peaks at three feature locations are obtained on the map. Furthermore, the three inferred feature locations are inferred to be in close proximity to each other.

[0028] The feature location inference unit 111 calculates the likelihood peak location within each acquired feature location map. At this time, it obtains the locations of the three locations with the highest likelihood scores within the map. These are output to the feature generation unit 114 as feature location coordinates.

[0029] In S304, the feature generation unit 114 generates template features based on the reference image obtained from the reference image acquisition unit 112 and the feature location obtained from the feature location inference unit 111. The generated template features are stored in the storage unit 104 and used in the tracking process, which will be explained with reference to Figure 3(b). The generation of template features will be described in more detail below.

[0030] Figure 6 illustrates the generation of template features. Using the tracking method with NN described in Non-Patent Literature 1, when the input image is input to the first processing unit, feature extraction CNN 601, an intermediate feature 602, which is a feature image, is obtained as the network output result. Furthermore, based on the feature locations calculated from the feature location map 503, a portion of the intermediate feature (a small region) is extracted and obtained as a template feature. The peak position of the likelihood calculated from the feature location map 503 obtained in S303 is taken as the feature location coordinate 603.

[0031] Next, the region surrounding the feature coordinate 603 is extracted from the intermediate feature 602 as a template feature. For example, in Figure 6, the feature within a small rectangular region with height 3 and width 3, centered on the feature location, is extracted as a template feature 604. The three extracted template features 604 are stored in the memory unit 104 for use in subsequent tracking processes.

[0032] Once the process shown in Figure 3(a) is completed, the process shown in Figure 3(b), which tracks the target object, begins.

[0033] In S305, the image acquisition unit 110 acquires an image from the imaging device 200. The image is assumed to include the object to be tracked.

[0034] In S306, the search image acquisition unit 113 acquires a search image by extracting a region for searching for the target object from the acquired image. The search image is extracted based on the region of the reference image and the previous tracking results.

[0035] Figure 7 is a diagram illustrating the search image. The search image (the search image region 704 at the current time) is a partial image of the image acquired in S306 (the image 701 at the current time), and is determined based on the detection result or tracking result 703 of the target object at the previous time. The previous time refers to, for example, the time of the preceding frame image. For example, the search image region 704 is determined as a region where the size of the object is enlarged by a predetermined ratio relative to the center position of the target object in the detection result or tracking result 703 of the previous time. If the processing at the previous time is a process in which the user specifies the target object to be tracked (Figure 3(a)), the search image region is determined from the detection result by the object detector 402. On the other hand, if the processing at the previous time is a tracking process, the search image region is determined from the tracking result.

[0036] In S307, the feature generation unit 114 generates features of the search image based on the search image acquired from the search image acquisition unit 113. When a search image is input to the feature generation unit 114, it generates features of that image. Intermediate features obtained by inputting the search image into the neural network of the feature generation unit 114 are acquired as features of the search image.

[0037] In S308, the matching unit 115 retrieves the template features stored in the storage unit 104. In S304, multiple template features are stored for a single target object, and all of these template features are retrieved here.

[0038] In S309, the matching unit 115 uses multiple template features and features from the search image to obtain maps of the position and size of the object to be tracked in the search image.

[0039] Figure 8 illustrates the acquisition of the likelihood map and size map of the position. The matching unit 115 obtains three intermediate features by performing a convolution operation on the three template features 801, 802, and 803 acquired in S308 with the feature 804 of the search image. The three intermediate features are input to the region proposal CNN 805, and likelihood maps 806, 807, and 808 representing the position of the target object in the search image are output. In addition, a size map 809 representing the three sizes corresponding to the three likelihood maps is output. Figure 8 shows an example in which three likelihood maps and one size map are output for three template features. The region proposal CNN 805 can be implemented using a CNN such as the one in Non-Patent Literature 1.

[0040] In S310, the detection unit 116 identifies the position and size of the object to be tracked from the likelihood map and size map acquired in S309 and outputs it as a tracking result. Here, three likelihood maps and one size map are used.

[0041] Figure 9 illustrates tracking (identification of the position of the object being tracked) using likelihood maps and size maps. First, the detection unit 16 calculates an average likelihood map 901 by averaging the likelihood maps 806, 807, and 808 acquired in S309. Next, it calculates the peak position of the likelihood value in the average likelihood map 901 and sets this position as the position of the object being tracked. Subsequently, it obtains the size map value at the same position as the object being tracked and uses this as the size of the object being tracked.

[0042] Through the series of processes described above, the detection unit 116 can output the position and size of the object being tracked as a result of the tracking process. In particular, by outputting multiple likelihood maps using multiple template features and averaging them to obtain an average likelihood map which is used as the position of the object being tracked, it is possible to reduce tracking failures.

[0043] <Configuration of the learning device> The following describes how to obtain the inference parameters of the CNN that constitutes the feature position inference unit 111 (Figure 2) mentioned above. The learning device is a device that learns a CNN that infers the feature positions of the tracking template. Note that the hardware configuration is the same as that of the information processing device 100 (Figure 1), so the explanation is omitted.

[0044] Figure 10 shows the functional configuration of the learning device according to the first embodiment. The learning device 1001 includes a tracking evaluation database unit 1010, a template feature candidate generation unit 1011, a tracking evaluation unit 1012, a training data generation unit 1013, and a feature position learning unit 1014.

[0045] The learning device 1001 works in cooperation with the information processing device 100 to learn the inference parameters that the feature location inference unit 111 uses to infer feature locations. The inference parameters obtained through learning are stored in the memory unit 104.

[0046] The tracking evaluation database unit 1010 stores a set of time-series images in which an object is captured. The tracking evaluation database unit 1010 is a database that holds both images and associated information (metadata) such as the position, size, and ID of the object in the images. The ID is set so that it can be identified as the same object in the time-series images.

[0047] The template feature candidate generation unit 1011 determines candidate positions for extracting template features from the features generated by the feature generation unit 114 from the reference image. The information processing device 100 generates template features based on the candidate positions of the template features. The information processing device 100 also performs tracking processing on a set of time-series images using the template features and outputs the detection results to the tracking evaluation unit 1012.

[0048] The tracking evaluation unit 1012 calculates an evaluation value for the tracking process from the detection results. The training data generation unit 1013 determines the position of the template features to be used as training data from candidate template positions based on the tracking evaluation results. The feature position learning unit 1014 updates the inference parameters of the feature position inference unit 111 based on the inference results of the feature position inference unit 111 and the positions of the template features obtained from the training data generation unit 1013. The updated inference parameters are stored as an inference model in the storage unit 104.

[0049] <Learning Method> Figure 11 is a processing flowchart of the learning device in the first embodiment. Specifically, it describes a method for learning the inference parameters that will be used in the feature location inference unit 111.

[0050] In S1101, the image acquisition unit 110 acquires the first frame image of the time-series image. In S1102, the reference image acquisition unit 112 sets one of the objects in the first frame image as the object to be tracked. Here, based on the position and size information of the object provided with the frame image, a region image of the object to be tracked is extracted from the frame image and output to the feature generation unit 114 as a reference image.

[0051] In S1103, the feature generation unit 114 generates intermediate features of the reference image acquired from the reference image acquisition unit 112. The feature generation unit 114 inputs the reference image to the NN and acquires intermediate features in the same way as the tracking process described above. The acquired intermediate features are output to the template feature candidate generation unit 1011.

[0052] S1104 is the entry point for the loop processing from S1105 to S1108. By having the information processing device 100 and the learning device 1001 repeat this loop processing a predetermined number of times, tracking evaluation results (regarding the set of feature location and time-series image) for the object being tracked will be obtained a predetermined number of times.

[0053] In S1105, the template feature candidate generation unit 1011 determines candidate feature locations (for which the feature generation unit 114 will extract template features from the intermediate features of the reference image). Then, the feature generation unit 114 obtains template features from the candidate feature locations.

[0054] Here, candidate feature locations are obtained as a set of three: the central feature location of the entire reference image, and two locations randomly selected from the eight surrounding neighbors. Subsequently, the feature generation unit 114 extracts a region with a height of 3 and a width of 3, centered on each candidate feature location, as a template feature.

[0055] S1106 is the entry point for the loop processing in S1107. Specifically, the information processing device 100 and the learning device 1001 sequentially acquire the second frame image and subsequent frames in the time-series image set, and obtain the tracking process evaluation results for all frame images included in the time-series image set.

[0056] The method for evaluating the tracking process involves first inputting an image into the information processing device 100 to obtain specific result region information. Next, the Intersection over Union (IOU) is calculated using the region information of the target object obtained from the tracking evaluation database unit 1010. If the IOU is greater than or equal to a predetermined value, it is evaluated as successful. The above evaluation is performed for all frame images of one target object in one time-series image set, and the percentage evaluated as successful is calculated as the final tracking evaluation result.

[0057] In S1107, the information processing device 100 detects the target object to be tracked through a series of processes from S305 to S310 in Figure 3(b). Subsequently, the tracking evaluation unit 1012 evaluates the detection result of the target object to be tracked. As described above, the success or failure of tracking is determined and used as an evaluation based on the IOU of the area of ​​the tracking detection result and the area of ​​the target object to be tracked obtained from the tracking evaluation database unit 1010.

[0058] In S1108, the tracking evaluation unit 1012 aggregates the evaluation results obtained in S1106-S1107 to calculate the tracking evaluation result for the time-series image set. Here, the evaluation results from S1107 are aggregated to calculate the percentage of frames evaluated as successful out of all frames.

[0059] In S1109, the training data generation unit 1013 determines one set of feature position candidates to be used as training data. Here, based on the results of S1108, the training data generation unit 1013 selects the set of feature position candidates that had the highest percentage of frame images evaluated as successful from a predetermined number of sets of feature position candidates as training data.

[0060] In S1110, the training data generation unit 1013 generates a feature location map of the training data using the training feature locations determined in S1109. The training data feature location map is given values ​​by a Gaussian distribution with peaks at the three training feature locations.

[0061] In S1111, the feature location inference unit 111 generates a feature location map using the reference image acquired in S1101 as input.

[0062] In S1112, the feature location learning unit 1014 updates the inference parameters of the feature location inference unit 111 using the feature location map generated in S1111 and the feature location map generated in S1110. The updated inference parameters are stored in the memory unit 104.

[0063] Updating the inference parameters can be achieved, for example, as follows: First, the cross-entropy error is calculated using the feature location map obtained from the feature location inference unit 111 and the training feature location map obtained from the training data generation unit 1013. Next, the inference parameters of the feature location inference unit 111 are updated using backpropagation based on the calculated error value.

[0064] By repeatedly performing the series of processes from S1101 to S1112 using a large number of time-series image datasets, the inference parameters of the feature location inference unit 111 can be learned and acquired.

[0065] As described above, according to the first embodiment, the position of template features that are easy to track is inferred, and template features are obtained. This makes it possible to obtain template features suitable for tracking any object. Furthermore, tracking accuracy can be improved by combining multiple template features.

[0066] In other words, conventionally, it was necessary to manually design combination patterns of template feature locations, but in this embodiment, it is possible to obtain them through learning based on tracking evaluation. Furthermore, for the multiple template features obtained, each template feature is inferred to be in a neighboring position to one another. The likelihood maps obtained by correlation calculations between these multiple template features and the features of the search image are inferred so that the likelihood distributions overlap near the object being tracked. Therefore, even if the likelihood is low near the object being tracked in one likelihood map, if the likelihood is high in other likelihood maps, it is possible to detect the position of the object being tracked using the average likelihood map obtained by averaging.

[0067] In the above explanation, we described a tracking method using a CNN, but it can also be applied to template matching, where a portion of the reference image is extracted and the brightness information of that region is used as a template feature. In this case, the characteristic position inference unit 111 infers the central position of the portion of the reference image to be extracted. Then, the surrounding region is extracted from the inferred central position and obtained as a template feature.

[0068] (Second Embodiment) In the second embodiment, a configuration in which the feature location inference unit infers feature locations from intermediate features will be described. Specifically, in the first embodiment, a reference image was input to the feature location inference unit 111 to obtain feature locations, but in the second embodiment, the intermediate features obtained by inputting the reference image to the feature generation unit 114 are input to the feature location inference unit 111 to output a feature location map.

[0069] <Configuration of the information processing device> Figure 12 shows the functional configuration of the information processing device according to the second embodiment. When the feature location inference unit 1211 receives intermediate features obtained by inputting a reference image to the feature generation unit 114, it outputs a feature location map representing the location of the template features. Based on the output feature location map, the template features are extracted from the intermediate features and stored in the storage unit 104. Subsequently, tracking processing is performed using the template features stored in the storage unit 104, in the same manner as in the first embodiment.

[0070] Figure 13 illustrates the generation of template features in the second embodiment. First, the reference image acquired by the reference image acquisition unit 112 is input to the feature extraction CNN 601 of the feature generation unit 114 as the input image to obtain intermediate features 602. Next, the obtained intermediate features 602 are input to the feature location inference CNN 1301 that constitutes the feature location inference unit 1211. The feature location inference CNN 1301 outputs a feature location map 1302 on the intermediate features that has a high likelihood value at a location suitable for the template features. The feature location inference CNN 1301 is implemented using a CNN with the same configuration as the feature location inference CNN 502 in Figure 5.

[0071] Multiple likelihood peaks (three in Figure 13) are obtained from the feature location map 1302, and these are used as the center positions of the template features. Additionally, a region (3 units vertically and 3 units horizontally in Figure 13) around the center positions of the template features is extracted from the intermediate feature 602 to obtain template feature 1303. Using these multiple template features, tracking processing is then performed in the same manner as in the first embodiment.

[0072] <Configuration of the learning device> Figure 14 shows the functional configuration of the learning device according to the second embodiment. The feature location inference unit 1211, which infers feature location, can be realized by a similar learning method using the learning device 1001 of the first embodiment. However, it differs from the first embodiment in that intermediate features obtained by inputting a reference image to the feature generation unit 114 are input to the feature location inference unit 1211, and the inference parameters are learned using the output feature location map.

[0073] As described above, according to the second embodiment, the feature inference unit learns and infers a position that is easy to track by taking the intermediate features generated by the feature generation unit as input. As a result, compared to the first embodiment which uses an image as input, the likelihood of obtaining features that are easy to track is increased.

[0074] (Third embodiment) In the third embodiment, we will describe a configuration in which the feature location inference unit infers the positions of parts and characteristic colors and patterns of the object being tracked. Specifically, in the first embodiment, the feature location inference unit 111 obtained the position of template features based on the object's center position and its surrounding area, but in the third embodiment, template features of the center position, parts, and colors and patterns of the object being tracked are used.

[0075] <Operation of the Information Processing Device> The configuration of the information processing device in the third embodiment is the same as that of the first embodiment (Figure 2), so its description will be omitted. Furthermore, the operation of the information processing device in the third embodiment is almost the same as that of the first embodiment (Figure 3), but the processing content differs in several steps. The following describes the operation of the steps that differ from those of the first embodiment.

[0076] In S303, the feature location inference unit 111 infers the location of characteristic object parts, colors, and patterns in the reference image.

[0077] Figure 15 illustrates the configuration of the feature location inference unit 111 in the third embodiment. When an input image is input to the multilayer CNN 1501, intermediate features of the image are generated via the Backbone 1502. The generated intermediate features are input to the feature location inference CNN 1503 for parts, which infers the position of parts of an object, and the feature location inference CNN 1504 for color and pattern (texture), which infers the position of characteristic colors and patterns of an object. As inference results, the feature location inference CNNs 1503 and 1504 output feature location maps 1505 and 1506, respectively, which represent the positions of the features.

[0078] In the example in Figure 15, feature location map 1505 indicates that the likelihood peak value is at the location of the car's tires 1507, and feature location map 1506 indicates that the likelihood peak value is at the pattern on the car's body 1508. The method for obtaining the inference parameters of such a feature location inference CNN will be described later.

[0079] In S304, the feature generation unit 114 extracts predetermined regions from the intermediate features of the reference image acquired in S303 to obtain three template features for the object's center position, the object's part positions, and the positions of characteristic colors and patterns. Note that, if a reference image is used from which the object's center and size inferred by the object detector is extracted, as described in S302 of the first embodiment, the center of that reference image becomes the object's center position. Therefore, the template feature for the object's center position is obtained from the center of the feature map.

[0080] In S309, the matching unit 115 matches the three template features with the features of the search image and outputs a likelihood map representing the position of the target object as a likelihood for each template feature, and a size map showing the size of the object.

[0081] In S310, the detection unit 116 uses likelihood maps obtained from each template feature to determine whether tracking was successful or unsuccessful. Then, it uses the size map paired with the likelihood map for which tracking was determined to have been successful to detect the target object to be tracked.

[0082] In this embodiment, a threshold value is set in advance for the likelihood of an object's position, which determines whether the tracking target object is being tracked. The detection unit 116 determines that tracking has been successful and detects the likelihood peak point as the object's position if the likelihood of the peak point in the likelihood map obtained from the matching unit 115 is greater than or equal to the threshold value. Subsequently, the detection unit 116 obtains the size of the object being tracked from the detected object's position on the corresponding size map. On the other hand, if the size is lower than the threshold value, the detection process is performed using the likelihood map and size map obtained from other template features, based on the aforementioned determination of whether tracking was successful or unsuccessful.

[0083] When tracking is determined to be successful using multiple template features, it is advisable to detect the target object using the likelihood map with the highest likelihood of the peak point and its corresponding size map. Alternatively, a priority order can be set for each template feature, and the target object can be detected using the likelihood map and size map of the template feature with the highest priority among those that successfully tracked. The priority order could be, for example, object center, color / pattern, and then parts. This ensures that even if tracking fails using one template feature, tracking can still be attempted using other template features.

[0084] <Learning Method> Next, we will explain how to obtain the inference parameters for the feature-based position inference CNNs 1503 and 1504. The configuration of the learning device in the third embodiment is the same as that of the first embodiment (Figure 10), so we will omit the explanation. In addition, the operation of the learning device in the third embodiment is almost the same as that of the first embodiment (Figure 11), but the processing content of some functional units differs. The operation of the functional units that differ in processing content from the first embodiment will be explained below.

[0085] In this embodiment, the candidate feature locations generated by the template feature candidate generation unit 1011 in S1105 are set to "parts of an object" and "characteristic colors and patterns of an object".

[0086] To use parts of an object as candidate feature locations, the positions of the object parts are detected from a reference image. To detect candidate object part locations, the object detector described in Non-Patent Literature 2 can be used to detect the positions of multiple parts of an object from an image. The template feature generation unit 1011 uses the positions corresponding to each detected part location on the intermediate features generated by the feature generation unit 114 as candidate feature locations.

[0087] To select object color and pattern as candidate feature locations, the following methods can be considered. For object color, first, the luminance value of the object's representative color is obtained by using the average or mode of the luminance values ​​within the object in the image. Next, multiple samples are taken in order of the smallest difference between the luminance value of each position in the image and the luminance value of the representative color, and these positions are designated as the characteristic color locations of the object. For object pattern, texture segmentation, as described in Non-Patent Literature 3, is used to separate the texture region within the object, and the center position of that region is designated as the characteristic pattern location of the object. The template feature generation unit 1011 selects positions corresponding to the detected characteristic color and pattern locations of the object on the intermediate features generated by the feature generation unit 114 as candidate feature locations. (Non-Patent Literature 3: Anil K., Jain et al., "Unsupervised Texture Segmentation Using Gabor Filters", 1991)

[0088] The inference parameters of the feature location inference unit are trained using the candidate feature locations obtained as described above. In the third embodiment, the cross-entropy error is calculated between the output results of feature location inference CNNs 1503 and 1504 and the corresponding feature location maps of the training data, and the sum of these errors is taken as the error. Based on the obtained error values, the inference parameters (CNN weights) of the feature inference unit are updated using backpropagation.

[0089] As explained above, the third embodiment uses template features of the central position, parts, color, and pattern of the object to be tracked. This makes it possible to increase the probability of successful tracking compared to the first embodiment.

[0090] The disclosures herein include the following information processing devices, learning devices, tracking methods, and programs. (Item 1) A first acquisition means for acquiring a reference image showing the target object to be tracked, A determination means for determining multiple positions in the reference image used for tracking the target object, A first generation means inputs the aforementioned reference image to a first processing unit and generates a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition means for acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation means inputs the search image to the first processing unit to generate a second feature quantity, A means for identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, An information processing device characterized by having the following features. (Item 2) The first generation means inputs the reference image to the first processing unit and generates a plurality of small regions corresponding to the plurality of positions in the feature quantity image obtained as the plurality of first feature quantities. The information processing device described in item 1, characterized by the features described herein. (Item 3) The determination means inputs the reference image to a second processing unit to generate a position map having likelihood values ​​that represent the likelihood of a position having features suitable for tracking the target object, and determines the plurality of positions based on the peak positions in the position map. An information processing device according to item 1 or 2, characterized by the above. (Item 4) The determination means inputs the reference image to the first processing unit to generate a feature image, inputs the feature image to the second processing unit to generate a location map having likelihood values ​​that represent the likelihood of a location having features suitable for tracking the target object, and determines the plurality of locations based on the peak positions in the location map. An information processing device according to item 1 or 2, characterized by the above. (Item 5) The determination means generates a first position map having likelihood values ​​representing the likelihood of a position having features suitable for tracking the parts of the target object, and a second position map having likelihood values ​​representing the likelihood of a position having features suitable for tracking the color and pattern of the target object. The plurality of locations include one or more peak locations in the first location map and one or more peak locations in the second location map. The information processing apparatus according to claim 3 or 4. (Item 6) The identification means generates a plurality of likelihood maps having likelihood values ​​representing the likely location of the target object included in the search image, based on the results of a correlation calculation between each of the plurality of first features and the second feature, and identifies the location of the target object included in the search image based on the plurality of likelihood maps. The information processing device according to any one of items 1 to 5. (Item 7) The identification means identifies the location of the target object included in the search image based on an average likelihood map obtained by averaging the plurality of likelihood maps, or identifies the location of the target object included in the search image based on the likelihood map having a higher peak value among the plurality of likelihood maps. The information processing apparatus according to feature 6. (Item 8) The identification means further generates a size map of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, and further identifies the size of the target object based on the size map and the position of the target object. The information processing apparatus according to feature 6. (Item 9) A means for acquiring multiple frame images in a time series, A reference image generation means that generates the reference image, which is a partial image including the target object, from a first frame image included in the plurality of frame images, A search image generation means that generates the search image which is a partial image of a second frame image included in a plurality of frame images that follow the first frame image, It further possesses, The search image is extracted from the second frame image based on the position and / or size of the target object in the frame image preceding the second frame image. The information processing device according to any one of items 1 to 8, characterized by... (Item 10) The first processing unit includes a first CNN (Convolutional Neural Network) for feature extraction. The information processing device according to any one of items 1 to 9, characterized by... (Item 11) The second processing unit includes a backbone network and a second CNN for feature localization inference. The information processing device according to any one of items 3 to 5, characterized by... (Item 12) A learning device for learning the second CNN of the information processing device according to claim 11, A database that stores time-series images and supplementary information regarding the position and size of objects contained in the time-series images, A feature candidate generation means that acquires a feature image generated by the first processing unit for a frame image included in the time series image, and outputs a plurality of feature position candidates for the frame image to the first generation means, A tracking evaluation means calculates an evaluation result for each of the multiple feature position candidates based on the identification result by the identification means for each of the multiple feature position candidates and the associated information contained in the database, A training data generation means that determines the feature positions to be used as training data from the plurality of feature position candidates based on the evaluation results and generates training data, An update means for updating the parameters of the second CNN of the second processing unit based on the training data, A learning device characterized by having the following features. (Item 13) The plurality of feature position candidates for the frame image include the central position of the feature in the frame image and one or more positions in the vicinity of the central position. The learning device according to feature 12. (Item 14) A tracking method in an information processing device that tracks objects contained in a moving image, The first acquisition step involves obtaining a reference image in which the target object to be tracked is captured, A determination step for determining multiple positions in the reference image used for tracking the target object, A first generation step involves inputting the aforementioned reference image into a first processing unit to generate a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition step involves acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation step involves inputting the search image to the first processing unit to generate a second feature quantity, A process of identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, A tracking method characterized by including [a certain element]. (Item 15) A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 11.

[0091] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0092] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0093] 100 Information processing device; 111 Feature location inference unit; 112 Reference image acquisition unit; 113 Search image acquisition unit; 114 Feature generation unit; 115 Matching unit; 116 Detection unit; 1001 Learning device

Claims

1. A first acquisition means for acquiring a reference image showing the target object to be tracked, A determination means inputs the aforementioned reference image to a second processing unit to generate a location map having likelihood values ​​that represent the likelihood of a location having features suitable for tracking the target object, and determines a plurality of locations in the reference image to be used for tracking the target object based on the peak positions in the location map, A first generation means inputs the aforementioned reference image to a first processing unit and generates a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition means for acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation means inputs the search image to the first processing unit to generate a second feature quantity, A means for identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, An information processing device characterized by having the following features.

2. The first generation means inputs the reference image to the first processing unit and generates a plurality of small regions corresponding to the plurality of positions in the feature quantity image as the plurality of first feature quantities. The information processing apparatus according to feature 1.

3. A first acquisition means for acquiring a reference image in which the target object to be tracked is shown, A first generation means inputs the aforementioned reference image to a first processing unit and generates a plurality of first feature quantities corresponding to a plurality of positions in the reference image that are used for tracking the target object, A determination means inputs the feature image generated by the first generation means to a second processing unit to generate a location map having likelihood values ​​that represent the likelihood of a location having features suitable for tracking the target object, and determines the plurality of locations based on the peak positions in the location map. A second acquisition means for acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation means inputs the search image to the first processing unit to generate a second feature quantity, A means for identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, An information processing device characterized by having the following features.

4. The determination means generates a first position map having likelihood values ​​representing the likelihood of a position having features suitable for tracking the parts of the target object, and a second position map having likelihood values ​​representing the likelihood of a position having features suitable for tracking the color and pattern of the target object. The plurality of locations include one or more peak locations in the first location map and one or more peak locations in the second location map. The information processing apparatus according to feature 1.

5. The determination means generates a plurality of likelihood maps having likelihood values ​​representing the likely location of the target object included in the search image, based on the results of a correlation calculation between each of the plurality of first features and the second feature. The identification means identifies the location of the target object included in the search image based on an average likelihood map obtained by averaging the plurality of likelihood maps, or identifies the location of the target object included in the search image based on the likelihood map having a higher peak value among the plurality of likelihood maps. The information processing apparatus according to feature 1.

6. The determination means generates a plurality of likelihood maps having likelihood values ​​representing the likely location of the target object included in the search image, based on the results of a correlation calculation between each of the plurality of first features and the second feature. The identification means identifies the location of the target object included in the search image based on an average likelihood map obtained by averaging the plurality of likelihood maps, or identifies the location of the target object included in the search image based on the likelihood map having a higher peak value among the plurality of likelihood maps. The information processing apparatus according to claim 3.

7. The determination means further generates a size map of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity. The identification means further determines the size of the target object based on the size map and the position of the target object. The information processing apparatus according to feature 5.

8. The determination means further generates a size map of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, The identification means further determines the size of the target object based on the size map and the position of the target object. The information processing apparatus according to feature 6.

9. A means for acquiring multiple frame images in a time series, A reference image generation means that generates a reference image which is a partial image including the target object from a first frame image included in the plurality of frame images, A search image generation means for generating the search image which is a partial image of a second frame image included in a plurality of frame images that follow the first frame image, It further possesses, The search image is extracted from the second frame image based on the position and / or size of the target object in the frame image preceding the second frame image. The information processing apparatus according to feature 7.

10. Acquisition means for acquiring multiple frame images in a time series, A reference image generation means that generates a reference image which is a partial image including the target object from a first frame image included in the plurality of frame images, A search image generation means for generating the search image which is a partial image of a second frame image included in a plurality of frame images that follow the first frame image, It further possesses, The search image is extracted from the second frame image based on the position and / or size of the target object in the frame image preceding the second frame image. The information processing apparatus according to feature 8.

11. The first processing unit includes a first CNN (Convolutional Neural Network) for feature extraction. The information processing apparatus according to feature 1.

12. The second processing unit includes a Backbone network and a second CNN for feature localization inference. The information processing apparatus according to feature 1.

13. A learning device for learning the second CNN of the information processing device according to claim 12, A database that stores time-series images and supplementary information regarding the position and size of objects contained in the time-series images, A feature quantity candidate generation means that acquires a feature quantity image generated by the first processing unit for frame images included in the time series image, and outputs a plurality of feature quantity position candidates for the frame image to the first generation means, A tracking evaluation means calculates an evaluation result for each of the multiple feature position candidates based on the identification result by the identification means for each of the multiple feature position candidates and the associated information contained in the database, A training data generation means that determines the feature positions to be used as training data from the plurality of feature position candidates based on the evaluation results and generates training data, An update means for updating the parameters of the second CNN of the second processing unit based on the aforementioned training data, A learning device characterized by having the following features.

14. The plurality of feature position candidates for the frame image include the central position of the feature in the frame image and one or more positions in the vicinity of the central position. The learning device according to feature 13.

15. A tracking method in an information processing device that tracks objects contained in a moving image, The first acquisition step involves obtaining a reference image in which the target object to be tracked is captured, A determination step of inputting the aforementioned reference image into a second processing unit to generate a position map having likelihood values ​​that represent the likelihood of a position having features suitable for tracking the target object, and determining a plurality of positions in the reference image to be used for tracking the target object based on the peak positions in the position map, A first generation step involves inputting the aforementioned reference image into a first processing unit to generate a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition step involves acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation step involves inputting the search image to the first processing unit to generate a second feature quantity, A process of identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, A tracking method characterized by including [a certain element].

16. A tracking method in an information processing device for tracking an object included in a moving image, The first acquisition step involves obtaining a reference image in which the target object to be tracked is captured, A first generation step involves inputting the reference image into a first processing unit and generating a plurality of first feature quantities corresponding to a plurality of positions in the reference image that are used for tracking the target object. A determination step is to input the feature image generated in the first generation step into a second processing unit to generate a location map having likelihood values ​​that represent the likelihood of a location having features suitable for tracking the target object, and to determine a plurality of locations in the reference image to be used for tracking the target object based on the peak positions in the location map, A first generation step involves inputting the aforementioned reference image into a first processing unit to generate a plurality of first feature quantities corresponding to the plurality of positions, A second acquisition step involves acquiring a search image that is the target of the tracking process for the aforementioned target object, A second generation step involves inputting the search image to the first processing unit to generate a second feature quantity, A process of identifying the position of the target object included in the search image based on the result of a correlation calculation between each of the plurality of first feature quantities and the second feature quantity, A tracking method characterized by including [a certain element].

17. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 12.

Images