Information processing apparatus, information processing method, and program

Abstract

Provided is an information processing apparatus configured to track a specific object in images taken at a plurality of times, the information processing apparatus including: a holding unit configured to hold a feature amount of a tracking target based on a learned model configured to detect a position of a predetermined object in an input image; an acquisition unit configured to acquire a feature amount of an object in a plurality of images based on the learned model; a detection unit configured to detect a candidate object similar to the tracking target based on the feature amount of the tracking target and the feature amount of the object acquired from the plurality of images; and an identification unit configured to identify a correlation between the candidate object detected in a first image among the plurality of images and the candidate object in a second image taken at a different time from the first image.

Description

Technical Field

[0001] This invention relates to a technique for tracking a specific subject in an image. Background Technology

[0002] Examples of techniques for tracking specific subjects in an image include techniques using brightness and color information, as well as template correlation techniques. In recent years, techniques utilizing deep neural networks (hereinafter referred to as DNNs) have attracted increasing attention as high-precision tracking techniques. For example, Non-Patent Document 1 discusses a method for tracking specific subjects in an image. An image including the subject to be tracked and an image to be used as the search range are input into a convolutional neural network (hereinafter abbreviated as CNN) with equal weights. Then, the cross-correlation between the feature quantities obtained from the CNN is calculated to identify the location of the subject to be tracked in the image used as the search range.

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-219531

[0006] Non-patent literature

[0007] Non-patent literature 1: Bertinetto, “Fully-Convolutional Siamese Networks for Object Tracking”, arXiv 2016 Summary of the Invention

[0008] Technical issues

[0009] However, in the technique discussed in Non-Patent Document 1, when the image contains objects similar to the tracked object, the cross-correlation value with the similar objects becomes high, and the error of mistakenly tracking the similar object as the tracked object may occur. In Patent Document 1, the positions of the tracked object and the similar object are predicted when there are objects similar to the tracked object near the tracked object. However, the method discussed in Patent Document 1 only uses the position of the tracked object for prediction. Therefore, the tracked object may be lost when it is located far from the predicted position or when the tracked object and the similar object are close together.

[0010] This invention was designed in view of the above-mentioned problems and is intended for the tracking of specific objects.

[0011] Solution to the problem

[0012] According to one aspect of the present invention, an information processing apparatus is configured to track a specific object in images captured at multiple times, the information processing apparatus comprising: a holding unit configured to hold feature values ​​of the tracked object based on a learned model configured to detect the position of a predetermined object in an input image; an acquisition unit configured to acquire feature values ​​of objects in the multiple images based on the learned model; a detection unit configured to detect candidate objects similar to the tracked object based on the feature values ​​of the tracked object and the feature values ​​of objects acquired from the multiple images; and an identification unit configured to identify a correlation between candidate objects detected in a first image of the multiple images and candidate objects captured at a different time than the first image.

[0013] Beneficial effects of the invention

[0014] According to the present invention, specific objects can be tracked. Attached Figure Description

[0015] [ Figure 1 ] Figure 1 An example of the hardware configuration of an information processing device is shown.

[0016] [ Figure 2 ] Figure 2 This is a block diagram illustrating an example of the functional configuration of an information processing device.

[0017] [ Figure 3 ] Figure 3 It is a flowchart illustrating the processing procedure performed by the information processing device.

[0018] [ Figure 4 ] Figure 4 This is a flowchart illustrating the processing procedure performed by the tracking object determination unit.

[0019] [ Figure 5 ] Figure 5 This is a flowchart illustrating the processing procedure performed by the object detection unit.

[0020] [ Figure 6 ] Figure 6 It is a flowchart illustrating the processing procedure performed by the information processing device.

[0021] [ Figure 7 ] Figure 7 This is a flowchart illustrating the processing procedures performed by the tracking unit.

[0022] [ Figure 8 ] Figure 8 This shows an example where the tracked object is occluded.

[0023] [ Figure 9 ] Figure 9 This shows an example of detecting the location of a tracked object in an image.

[0024] [ Figure 10A ] Figure 10A It is a flowchart illustrating the processing procedure performed by the information processing device.

[0025] [ Figure 10B ] Figure 10B It is a flowchart illustrating the processing procedure performed by the information processing device.

[0026] [ Figure 11 ] Figure 11 An example of occlusion detection is shown.

[0027] [ Figure 12 ] Figure 12 An example image is shown where multiple candidate objects were detected.

[0028] [ Figure 13 ] Figure 13 This is a block diagram illustrating an example of the functional configuration of an information processing device.

[0029] [ Figure 14 ] Figure 14 It is a flowchart illustrating the processing procedure performed by the information processing device.

[0030] [ Figure 15A ] Figure 15A Examples of the obtained template image and search range image are shown.

[0031] [ Figure 15B ] Figure 15B Examples of the obtained template image and search range image are shown.

[0032] [ Figure 16 ] Figure 16 An example graph showing the output of the learned model is displayed.

[0033] [ Figure 17 ] Figure 17 An example of teacher data used in the learning model is shown.

[0034] [ Figure 18 ] Figure 18 This is a block diagram illustrating an example of the functional configuration of an information processing device.

[0035] [ Figure 19 ] Figure 19 This is a block diagram illustrating an example of the functional configuration of an information processing device.

[0036] [ Figure 20 ] Figure 20 It is a flowchart illustrating the processing procedure performed by the information processing device. Detailed Implementation

[0037] <First Exemplary Example>

[0038] <Processing during the execution phase using the learned model>

[0039] An information processing apparatus according to an exemplary embodiment of the present invention will now be described with reference to the accompanying drawings. Components assigned the same reference numerals in the drawings perform the same operations, and therefore repeated descriptions thereof will be omitted. The components described in the exemplary embodiments are merely examples and are not intended to limit the scope of the invention.

[0040] In a first exemplary embodiment, an example will be described whereby a tracking object and objects similar to the tracking object are tracked simultaneously, such that stable tracking continues even when many objects similar to the tracking object are present or when the tracking object is occluded by other objects. More specifically, this exemplary embodiment aims to stably track individual objects even when objects similar to the tracking object are present.

[0041] Figure 1 This illustration shows the hardware configuration of an information processing device 1 for tracking a specific object in images captured at multiple times, according to this exemplary embodiment. A central processing unit (CPU) H101 executes a control program stored in a read-only memory (ROM) H102 to control the entire device. A random access memory (RAM) H103 temporarily stores various types of data from the various components. The program is loaded into the RAM H103 to be in a state executable by the CPU H101.

[0042] Storage unit H104 stores the data to be processed and the tracking object data according to this exemplary embodiment. The storage unit H104 can be a hard disk drive (HDD), flash memory, or various optical media. Input unit H105 includes a keyboard, touch panel, or dial for accepting input from a user and for setting the tracking object. Display unit H106 includes a liquid crystal display and displays the subject and tracking results to the user. Information processing device 1 can communicate with other devices, such as camera equipment, via communication unit H107.

[0043] Figure 2This is a block diagram illustrating an example of the functional configuration of an information processing device 1. The information processing device 1 includes an image acquisition unit 201, a tracking object determination unit 202, a holding unit 203, an object detection unit 204, and a tracking unit 205, and these units are connected to a storage unit 206. The storage unit 206 may be an external device or may be included within the information processing device 1. The various functional units will be briefly described below. The image acquisition unit 201 acquires an image of a predetermined object captured by a camera device. Examples of predetermined objects include people and vehicles, i.e., objects with some individual differences. In the exemplary embodiments described below, tracking of a person will be described as a specific example. The tracking object determination unit 202 determines the object to be tracked (object of interest) among the objects contained in the image. The holding unit 203 retains feature values ​​of the objects to be tracked as candidates from the initial image. The object detection unit 204 detects the position of the object in images captured at multiple times. The tracking unit 205 identifies and tracks the tracking object in images captured at multiple times.

[0044] Figure 3 This is a flowchart illustrating the processing flow according to this exemplary embodiment. In the following description, each process (step) begins with "S" and the term "process (step)" is omitted. However, the information processing device does not necessarily need to perform all the processes shown in the flowchart. The various processes performed by CPU H101 are shown as functional blocks.

[0045] In S301, the image acquisition unit 201 acquires an image (initial image) of the predetermined object. The image acquisition unit 201 may acquire an image captured by a camera device connected to the information processing device, or an image stored in the storage unit H104. The processing in S301 to S303 aims to use the initial image to set the object of interest as the tracking object.

[0046] In S302, the tracking object determination unit 202 determines objects in the image acquired in S301 as tracking objects (objects of interest). One or more tracking objects may exist. In this exemplary embodiment, an example of selecting a tracking object will be described. In this step, the tracking object determination unit 202 acquires the positions of image features indicating the positions of predetermined objects from the image using a learned model for detecting the positions of predetermined objects, and determines a portion of the image containing the object of interest. For example, a model that has pre-learned image features of predetermined objects such as people or vehicles is used as the learned model. The learning method will be described below. When an object is detected in an image, that object becomes a tracking object. For example, when no predetermined object is detected in the image, an image from the next frame can be input. When multiple objects are acquired, the tracking object determination unit 202 outputs tracking object candidates and then determines the tracking object using a pre-specified method. In this case, the tracking object determination unit 202 determines the tracking object (object of interest) in the acquired image according to the instructions specified by the input unit H105.

[0047] Examples of specific methods for determining a tracking object include determining the tracking object by means of a subject displayed on the touch display unit H106. In addition to the specification of the input unit H105, the tracking object determination unit 202 can also determine the tracking object by automatically detecting a main subject in an image. Examples of methods for automatically detecting a main subject in an image include the method discussed in Japanese Patent 6556033. The tracking object determination unit 202 can also determine the main subject based on both the specification of the input unit H105 and the result of detecting an object in the image. Technical examples for detecting objects in an image include "Liu, SSD: Single Shot Multibox Detector, In: ECCV2016".

[0048] Figure 12 The results show the detection of candidate objects for tracking in the image. Figure 12 People 1302, 1305, and 1307 are candidate tracking objects. Frames 1303, 1304, and 1306 are bounding boxes (hereinafter referred to as BBs) indicating the locations of the detected candidates. The user can determine the tracking object by touching any of the candidate BBs displayed on the display unit H106 or by selecting any of the candidate BBs using a dial. As described above, various methods exist for determining tracking objects, and this exemplary embodiment is not intended to limit the methods used to specify tracking objects.

[0049] In S303, the holding unit 203 holds the feature quantity of the tracked object from the image containing the determined tracked object based on the learned model. Figure 4This is a flowchart detailing the feature retention process in S303. The retention unit 203 generates and retains template feature quantities representing the tracked object based on the image acquired by the image acquisition unit 201 and the bounding box (hereinafter referred to as BB) indicating the position of the tracked object acquired by the tracked object determination unit 202.

[0050] In S401, the holding unit 203 acquires information related to the position of the tracked object in the image determined by the tracked object determination unit 202. The acquired information related to the position of the tracked object is referred to as the bounding box (BB) below. Information related to the center position of the tracked object input by the user when determining the tracked object in S302, or the result of detecting a predetermined position (e.g., centroid) of the tracked object using a learning model, is used as information related to the position of the tracked object.

[0051] In S402, the holding unit 203 acquires a template image based on the position of the tracked object in the image, which serves as an indication of the image of the tracked object to be extracted to a predetermined size. More specifically, the holding unit 203 cuts out the periphery of the region acquired in S401 from the initial image as a template image, and then resizes this image to a predetermined size. The predetermined size can be adjusted to the size of the input image of the learned model.

[0052] In S403, the holding unit 203 inputs a template image indicating the tracked object into a learned model used to detect the location of a predetermined object in the input image, thereby acquiring feature quantities of the tracked object. In this case, the holding unit 203 inputs the image resized in S402 into a convolutional neural network (CNN) (the learned model). The CNN has been pre-trained to acquire feature quantities that are easier to distinguish between tracked and non-tracked objects. The learning method will be described below. The CNN includes convolution and non-linear transformations such as the Corrected Linear Unit (hereinafter referred to as ReLU) and Max Pooling. The ReLU and Max Pooling described herein are only considered as examples. Leaky ReLU or the Sigmoid function can be used instead of ReLU. Average pooling can be used instead of Max Pooling. This exemplary embodiment is not limited to these methods. Then, in S404, the holding unit 203 holds the feature quantities of the tracked object acquired in S403 as template feature quantities indicating the tracked object. The above processing is the processing in the tracked object setting stage.

[0053] In S304, the image acquisition unit 201 acquires images captured at multiple times for tracking processing. Subsequent processing describes steps for detecting the tracking object set in the first image from a second image captured at a different time than the first image. The first and second images are captured such that they include as large a portion of the tracking object as possible.

[0054] In S305, the object detection unit 204 detects candidate objects similar to the tracked object based on the feature values ​​of the tracked object and the feature values ​​of the object obtained from multiple images. Figure 5 This is a flowchart illustrating the processing performed by the object detection unit 204 in step S305. The processing in S304 and subsequent steps is the processing of an image captured after the image of the tracked object has been determined, and is the processing for detecting the tracked object in the image.

[0055] In S501, the object detection unit 204 acquires a search range image (partial image) from the current image (second image) to indicate the region where the object to be tracked is to be searched. In this case, the object detection unit 204 acquires the search range image based on the detection position of the previously tracked object or candidate object. More specifically, the object detection unit 204 extracts a partial image of a predetermined size from the region in the second image that corresponds to the vicinity of the candidate object detected in the first image (past image). The size of the search region can be changed according to the speed of the object and the viewing angle of the image. The search region can be the entire search image or the periphery of the position of the previously tracked object. Setting a partial region of the input image, rather than the entire region, as the search range provides the effect of improving processing speed and reducing tracking-related errors.

[0056] In S502, the object detection unit 204 extracts the input image to be input into the learned model from the search range image. The object detection unit 204 crops the search range region from the search range image and then adjusts the size of the search range region. The size of the search range is determined to be a constant multiple of the BB size of the tracked object, etc. When acquiring feature quantities from images of the same size, feature quantities with low noise can be acquired. The object detection unit 204 crops the region based on the determined search region and adjusts the size of the region such that the resizing ratio is equivalent to the ratio in S402.

[0057] In S503, the object detection unit 204 inputs the extracted search range image into a learned model (CNN) for detecting the location of predetermined objects in the input image to obtain feature values ​​for each search range image. More specifically, the object detection unit 204 inputs the image of the clipped region into the CNN. The feature values ​​of each search range image indicate the feature values ​​of the objects present in each search range image. It is assumed that the weights of the CNN in S503 are partially or completely the same as the weights of the CNN in S403. For example, when a particular search range image contains an occluding object used to occlude a person, the CNN is able to obtain feature values ​​indicating the occluding object. When other parts of the image contain an animal instead of a person, feature values ​​indicating the animal are obtained.

[0058] In S504, the object detection unit 204 acquires the cross-correlation between the feature values ​​of the tracked object and the feature values ​​of objects present in the current search range image acquired in S503. Cross-correlation is an indicator of the similarity between detected objects. In this case, objects similar to the tracked object (objects of the same type as the tracked object) are called candidate objects. More specifically, objects with a cross-correlation value greater than a predetermined value are candidate objects. Candidate objects include either the tracked object or non-tracked objects, or both. In a specific example, when the tracked object is a person, the search range image with feature values ​​indicating a person has a high cross-correlation.

[0059] In S505, the object detection unit 204 detects the position of the candidate object in the current image. Since the weights of the CNN in S503 are partially or completely the same as those in S403, the cross-correlation value increases at locations within the search range where the candidate object is likely to exist. This allows the detection of candidate object positions within a search range image with cross-correlation values ​​greater than or equal to a threshold. More specifically, based on the cross-correlation obtained in S504, the object detection unit 204 detects positions with cross-correlation values ​​greater than a predetermined value as candidate object positions. Tracked objects are unlikely to exist at locations with cross-correlation values ​​lower than the predetermined value. In this case, the object detection unit 204 also obtains the bounding box (BB) around the candidate object based on its position. First, the object detection unit 204 determines the position of the BB based on a search range image that already indicates a high cross-correlation response.

[0060] Figure 9An example of the processing results in S305 is shown. Map 901 is obtained based on cross-correlation. The tracked object is person 902, and cell 904 near the center of person 902 indicates a high cross-correlation value. When the correlation value is greater than or equal to a threshold, it can be estimated that person 902 is located at cell 904. Simultaneously, the width and height of the base block (BB) can be pre-learned to allow the CNN to estimate the width and height (described below). Alternatively, the width and height of the BB of the tracked object obtained in S302 can be used as is.

[0061] In S306, the tracking unit 205 identifies the correlation between candidate objects detected in a first image out of multiple images and candidate objects in a second image taken at a different time than the first image. Identifying the correlation between objects detected at multiple times enables the tracking of related objects. Since the feature values ​​and positions of the tracked object are updated based on the images in which the tracked object is detected, more stable tracking is achieved. Figure 7 This is a flowchart illustrating the processing performed by the tracking unit 205.

[0062] In S701, the tracking unit 205 acquires combinations (correlated candidates) of candidate objects detected in images taken at past times and those detected in images taken at the current time, pre-stored in the storage unit 206. In this case, past candidate objects are paired with current candidate objects to generate all possible combinations of past and current candidate objects. Each candidate object detected in the past image is assigned a tracking object / non-tracking object label. When a tracking object exists, each object identified as a tracking object in the past candidate objects can be associated with each object in the current candidate objects.

[0063] In S702, the tracking unit 205 identifies combinations (correlation) with similarity scores higher than or equal to a threshold. High similarity between past and current candidate objects indicates that they are likely the same object. Several methods exist for correlation. For example, there are methods for preferentially correlation of candidate objects with high similarity, and methods using the Hungarian algorithm. The correlation method is not limited here. In this case, the tracking unit 205 identifies the same object based on the similarity between candidate objects in the first image (excluding the tracked object) and candidate objects in the second image. Tracking other objects similar to the tracked object in this way prevents the tracked object from being correlated with other objects. Therefore, stable tracking can be performed. Appropriate correlation in this way enables the identification of past and current tracked objects as the same object.

[0064] For example, the similarity L between past candidate c1 and current candidate c2 is calculated as follows. Here, BB represents a vector comprising four distinct variables (center coordinate x, center coordinate y, width, and height) for each candidate BB, and f represents the feature of each candidate. The feature refers to the feature map extracted from the feature map where each candidate is located, obtained from a CNN. W1 and W2 are coefficients derived empirically, with W1 > 0 and W2 > 0. More specifically, the similarity increases as the feature values ​​become closer, and also increases as the detection locations and detection regions become closer.

[0065] [Equation 1]

[0066] L(c1, c2) = -W1||BB1-BB2||-W2||f1-f2|| (Formula 1-1)

[0067] Then, in S703, the tracking unit 205 identifies the tracking object based on the correlation result. As the correlation result obtained in S702, the tracking unit 205 can identify current candidate objects related to past tracking objects as tracking objects. Information indicating that an object is not a tracking object is assigned to candidate objects other than the tracking object. When there is no current candidate object whose feature quantity similarity to a past tracking object is higher than a predetermined threshold, the tracking object is likely outside the viewpoint or occluded by other objects. In this case, the tracking unit 205 can notify that no tracking object has been identified.

[0068] Finally, in S704, storage unit 206 maintains the feature values ​​of the tracked object in the second image and the feature values ​​of candidate objects in the second image. When a tracked object is identified in the current image, storage unit 206 updates the feature values ​​of the tracked object. When a candidate object is detected in the second image whose feature value is greater than a predetermined threshold similar to that of the tracked object in the first image, storage unit 206 maintains the feature values ​​obtained from the second image as the feature values ​​of the tracked object. When no candidate object is detected in the second image whose feature value is greater than the predetermined threshold similar to that of the tracked object, storage unit 206 maintains the feature values ​​obtained from the first image as the feature values ​​of the tracked object. When no tracked object is detected in the current image, tracking unit 205 maintains the feature values ​​and positions of the tracked objects in past images. Furthermore, storage unit 206 stores the feature values ​​of current candidate objects that have been labeled as tracked objects / non-tracked objects. Storage unit 206 updates the BB (position and size) and features of the tracked object and its candidate objects. Maintaining the feature values ​​and judgment results of candidate objects similar to the tracked object enables more stable tracking.

[0069] In S307, the image acquisition unit 201 determines whether to end the tracking process. If tracking is to continue, the process returns to S304. If tracking is to end, the process concludes. For example, if an end command is received from the user or if the next frame cannot be acquired, the image acquisition unit 201 determines to end the process. When the next frame can be acquired, the process returns to S304. The processing steps in the tracking process execution have been described above. The learning process will be described below.

[0070] <Learning Steps>

[0071] Now, we describe a method for training a learned model (specifically a CNN) to estimate the location of objects in an image. We assume that the learned model used here has already been trained to some extent on an object classification task (e.g., a task to detect people but not animals), enabling the model to learn to distinguish individuals based on the external features of predetermined objects. This allows for the tracking of specific objects.

[0072] Consider the example scenario where person A is wearing red and person B is wearing yellow. Since clothing color is not a necessary feature for a learned model used only for detecting people, the learned model may have already learned to ignore clothing color in a person detection task. However, when detecting (tracking) only person A, the model needs to learn features to distinguish between person A and person B. In this case, clothing color is an important feature needed to identify individuals. In this exemplary embodiment, among objects of the same category, the model learns feature quantities of the object being tracked, distinguishing it from other objects of the same category. Figure 13 An example of the functional configuration of the information processing device 2 during learning is shown. The information processing device 2 includes a ground truth acquisition unit 1400, a template image acquisition unit 1401, a search range image acquisition unit 1402, a tracking object estimation unit 1403, a loss calculation unit 1404, a parameter update unit 1405, a parameter storage unit 1406, and a storage unit 206.

[0073] Storage unit 206 stores images captured at multiple times and ground truth (GT) information indicating the position and size of the tracked object in each image. In this case, storage unit 206 stores GT information related to the center position (or base square) of the tracked object input by the user for each image. The GT information can be generated by methods other than the user's GT assignment. For example, it can also be generated using the results of detecting the position of the tracked object by using other learned models. The GT acquisition unit 1400, template image acquisition unit 1401, and search range image acquisition unit 1402 each acquire the images stored in storage unit 206.

[0074] Ground truth (GT) acquisition unit 1400 acquires GT information to obtain the correct answer positions of the objects being tracked in the template image and the correct answer positions of the tracked objects in the search range image. GT acquisition unit 1400 also acquires the base position (BB) of the tracked objects in the template image acquired by template image acquisition unit 1401 and the BB of the tracked objects in the search range image acquired by search range image acquisition unit 1402. More specifically, refer to... Figure 17 Image 1704 shows that object 1705, which is the object being tracked, is assigned information indicating that object 1705 is the object being tracked, and the remaining areas are assigned information indicating that the area is not the object being tracked. For example, the area of ​​the object being tracked 1705 is assigned a label with the real number 1, while other areas are assigned labels with the real number 0.

[0075] The template image acquisition unit 1401 acquires an image containing the object to be tracked as a template image. The template image may contain multiple objects of the same category. The search range image acquisition unit 1402 acquires an image from which the object to be tracked is to be searched. More specifically, feature values ​​of the specific object to be tracked can be acquired from this image. For example, the template image acquisition unit 1401 selects any frame from a series of sequential images, and the search range image acquisition unit 1402 selects other frames from the sequence of images that were not selected by the template image acquisition unit 1401.

[0076] The tracking object estimation unit 1403 estimates the position of the tracking object in the search range image. The tracking object estimation unit 1403 estimates the position of the tracking object in the search range image based on the template image acquired by the template image acquisition unit 1401 and the search range image acquired by the search range image acquisition unit 1402.

[0077] The loss calculation unit 1404 calculates the loss based on the tracking results obtained by the tracking object estimation unit 1403 and the position of the tracked object in the search range image obtained by the ground truth (GT) acquisition unit 1400. The closer the position is to the estimation result from the teacher data, the smaller the loss. The loss calculation unit 1404 obtains the correct answer for the position of the tracked object in the search range image based on the GT information obtained by the GT acquisition unit.

[0078] The parameter update unit 1405 updates the CNN parameters based on the loss obtained by the loss calculation unit 1404. Here, the parameter update unit 1405 updates the parameters so that the loss values ​​converge. When the sum of the loss values ​​converges or when the loss value becomes less than a predetermined value, the parameter update unit 1405 updates the parameter set and ends the learning process.

[0079] The parameter storage unit 1406 stores the CNN parameters updated by the parameter update unit 1405 as learning parameters in the storage unit 206.

[0080] Reference Figure 14 The flowchart describes the learning process. In S1500, the GT acquisition unit 1400 acquires GT information and, based on the GT information, acquires the correct answer position (BB to be tracked) of the tracked object in the template image and the correct answer position of the tracked object in the search range image. In S1501, the template image acquisition unit 1401 acquires a template image. For example, the template image acquisition unit 1401 acquires... Figure 15A The image shown. In Figure 15A In this context, object 1601 is the tracking object, partial image 1602 indicates the BB of the tracking object acquired by the GT acquisition unit 1400, and partial image 1603 indicates the region to be clipped as a template. In other words, in this case, the template image acquisition unit 1401 acquires partial image 1603 as a template image.

[0081] In S1502, the template image acquisition unit 1401 cuts out the region to be used as a template from the template image, and then adjusts the size of the region to a predetermined size. For example, based on the BB of the tracking object, the size of the region to be cut is determined to be a constant multiple of the size of BB.

[0082] In S1503, the tracking object estimation unit 1403 inputs the template image generated in S1502 into the learning model (CNN) and then obtains the CNN feature quantity of the template.

[0083] In S1504, the search range image acquisition unit 1402 acquires a search range image. Based on the position and size of the tracked object, a portion of the image to be used as the search range is acquired as the portion of the image containing the tracked object. Figure 15B This shows an example of an image to be used as the search scope. Figure 15B In the image, object 1604 indicates the object being tracked, partial image 1605 indicates the target area (BB) of the object being tracked, and partial image 1606 indicates the search area. Search area image 1606 includes objects similar to the object being tracked.

[0084] In S1505, the search range image acquisition unit 1402 cuts out the search range region from the search range image and then adjusts the size of the search range region. The size of the search range is determined to be a constant multiple of the BB size of the tracked object. In S1502, the search range image acquisition unit 1402 adjusts the size of the search range region by a multiple used to adjust the template size (so that the size of the tracked object after adjusting the template size is approximately the same as the size of the tracked object after adjusting the search range size).

[0085] In S1506, the tracking object estimation unit 1403 inputs the search range image generated in S1505 into the learning model (CNN) and then obtains the CNN feature quantity of the search range.

[0086] In S1507, the tracking object estimation unit 1403 estimates the location of the tracking object in the search range image. The tracking object estimation unit 1403 calculates the cross-correlation, which indicates the similarity between the CNN features of the tracking object acquired in S1506 and the CNN features of the search range acquired in S1506, and then outputs the cross-correlation as a map. The tracking object estimation unit 1403 estimates the tracking object by indicating locations with cross-correlation values ​​greater than or equal to a threshold. Figure 16 A map is shown to indicate the estimation results. Map 1701 is obtained based on cross-correlation, where regions 1702 and 1703 indicate locations with large cross-correlation values. When cross-correlation is obtained in this way, it is likely that the locations of objects similar to the tracked object provide high cross-correlation values. On the other hand, Figure 17 In the diagram, 1705 indicates the location of the tracked object acquired by the GT acquisition unit 1400 as the correct answer. More specifically, since 1702 indicates the location of the tracked object, the expected value is estimated. However, although region 1703 is not a tracked object, an unexpected value is estimated because 1703 provides a high cross-correlation value. The learning step aims to update the weights such that the location of the tracked object provides a high cross-correlation value, and the location of the non-tracked object provides a low cross-correlation value.

[0087] In S1508, the loss calculation unit 1404 calculates the loss related to the inferred location of the tracked object and the loss related to the inferred size of the tracked object. Regarding the location-related loss, the loss calculation unit 1404 calculates the loss to advance learning, such that the cross-correlation value at the location of the tracked object indicates a large value. The ground truth (hereinafter referred to as GT) acquisition unit 1400 acquires the BB of the tracked object in the template image acquired by the template image acquisition unit 1401 and the BB of the tracked object in the search range image acquired by the search range image acquisition unit 1402.

[0088] The loss function can be represented by Equation (1-2), where Cin represents graph 1701 obtained in the processing of S1507, and Cgt represents the ground truth graph 1704. Equation (1-2) represents the mean square of the difference between the pixel-based graphs Cin and Cgt. The loss decreases when the tracked object is properly estimated, and increases when a non-tracked object is estimated as a tracked object or when a tracked object is estimated as a non-tracked object.

[0089] [Equation 2]

[0090]

[0091] Similarly, the size-related loss is calculated using formula (1-3).

[0092] [Equation 3]

[0093]

[0094]

[0095] Loss W and Loss H Let W represent the loss related to the estimated width of the tracked object and the loss related to the estimated height of the tracked object, respectively. gt and H gt Let W represent the width and height values ​​of the tracked object embedded at its location, respectively. The learning is advanced by calculating the loss using formulas (1-3) and (1-4), thus improving the learning for W. in and H in The width and height of the tracked object are inferred at its location. By summing all the losses, formula (1-5) is derived.

[0096] Loss = Loss C + Loss W + Loss H Formula (1-5)

[0097] Although the loss is described as mean squared error (hereinafter referred to as MSE) in this case, the loss is not limited to MSE. The loss can be Smooth-L1, etc. The formula for calculating the loss is not restricted. Furthermore, the loss function for location and the loss function for magnitude can be different.

[0098] In S1509, the parameter update unit 1405 (learning unit) updates the CNN parameters based on the loss calculated in S1508. The parameters are updated using backpropagation, such as stochastic gradient descent (SGD) with momentum. The output of the loss function for a single image has been described above. However, in actual learning, the loss value is calculated using equation (1-2) for the scores estimated for multiple different images. The parameter update unit 1405 updates the inter-layer connection weighting factors of the learning model, ensuring that the loss value for each of the multiple images becomes less than a predetermined threshold.

[0099] In S1510, parameter storage unit 1406 stores the updated CNN parameters from S1509 in storage unit 206. In the inference step, inference is performed using the parameters stored in S1510 to enable correct tracking of the target object.

[0100] In S1511, the parameter update unit 1405 determines whether to end the learning process. When the loss value obtained by formula (1-2) becomes less than a predetermined threshold, the parameter update unit 1405 determines that the learning process should end.

[0101] <Effects of the First Exemplary Embodiment>

[0102] This exemplary embodiment is characterized by simultaneously tracking the tracked object and objects similar to the tracked object. (Refer to...) Figure 8 This describes the fact that objects similar to the tracked object are tracked simultaneously to reduce the possibility of mistakenly tracking similar objects. Images 801, 802, and 803 are acquired at times t=0, t=1, and t=2, respectively. People 804 and 805 are in image 801; person 804 is the tracked object, and person 805 is a similar object.

[0103] First, the following describes the case where only object 804 is being tracked. In this case, object 804, which is correctly tracked at time t=0, is occluded by object 808 at time t=1. When occlusion occurs, regarding the feature quantities of object 804, it is likely that feature quantities lacking the likelihood of the object due to occlusion will be detected. For object 808, feature quantities with high object likelihood will be detected. Therefore, at time t=1, the tracking object is likely to be identified as object 808, and then object 808 will begin to be incorrectly tracked as the tracking object.

[0104] The following describes the case where not only object 804 is tracked, but also similar object 805. At time t=1, there are two distinct past trackable object candidates, objects 804 and 805. At time t=1, since object 809 is being occluded, only object 808 is acquired as a new trackable object candidate. At this point, when comparing the similarity between past candidate 804 and object 808 with the similarity between past candidate 805 and object 806, the similarity between objects 805 and 808 is higher than the similarity between objects 804 and 808. This is because CNN features associated with each candidate have been learned to distinguish between objects, and the position and size of the block boundaries (BBs) change moderately over time. Therefore, the current candidate 808 is not associated with past candidate 804, but with past candidate 805. The latest feature values ​​of object 805 are updated to the feature values ​​of object 808. For object 804, which was not detected at time t=1, the feature values ​​acquired at time t=0 are retained. Then, at time t=2, the similarity between candidate objects is calculated. At time t=2, objects 804 and 808 are past candidates, while objects 811 and 812 are acquired as two new candidates. Since these two candidate objects are not occluded, the desired feature values ​​can be obtained for them. After similarity calculation, high similarity is obtained between objects 808 and 811, and between objects 804 and 812, while low similarity is obtained between objects 808 and 812, and between objects 804 and 811. Therefore, object 804, as the tracking target, is associated with object 812, enabling correct tracking of the target.

[0105] <Variation Example 1-1> Online Measurement Learning

[0106] In Variation 1-1, in formula (1-1) according to the first exemplary embodiment, the weights W2 of the feature quantities are updated sequentially using the feature quantities of the tracked object and similar objects obtained in time series.

[0107] An example is described below.

[0108] [Equation 4]

[0109] W2=λ|∑f target -∑f distractor | Formula (1-6)

[0110] f target This represents the feature quantity of the tracked object acquired at each time point, and f. distractor This represents the number of features of similar objects acquired at various times.

[0111] Based on formula (1-2), the weights are updated using features of the tracked object and similar objects. Therefore, similarity can be calculated by applying larger weights to feature dimensions that make it easier to distinguish between the tracked object and similar objects. This makes it easier to distinguish between the tracked object and similar objects even if their features are similar in the feature space.

[0112] <Variation Example 1-2> Offline Metric Learning

[0113] In Variation 1-2, in formula (1-1) according to the first exemplary embodiment, a transformation for obtaining the similarity between feature quantities is pre-calculated through metric learning. When F represents a function of the transformed feature quantities, formula (1-1) is expressed as formula (1-7).

[0114] [Equation 5]

[0115] L(c1, c2) = -W1||BB1-BB2||-W2||F(f1)-F(f2)|| Formula (1-7)

[0116] Transform F represents the structure of a neural network connecting one or more layers, and can be pre-learned using methods such as triplet loss. Learning transform F using triplet loss allows transform F to learn transformations where the distance is short if the past and current objects are the same, or long if they are different. The learning method using triplet loss is described in detail in "Wang, Learning Fine-grained Image Similarity with Deep Ranking, In: CVPR 2014".

[0117] <Second Exemplary Embodiment> Occlusion Detection

[0118] The second exemplary embodiment also applies to the first exemplary embodiment. Figure 7 The tracking object recognition process in S306 includes an occlusion determination process. By performing an occlusion determination, even when the tracking object is occluded, it prevents the tracking object from being switched to other similar objects. The processing, which differs from the first exemplary embodiment, will be described in detail. The hardware configuration is similar to that according to the first exemplary embodiment. Figure 18 An example of the functional configuration of the information processing device 1' according to a second exemplary embodiment is shown. This configuration is substantially the same as... Figure 2The configuration is similar to that according to the first exemplary embodiment, and additionally includes an occlusion determination unit 207 for performing occlusion determination. Components assigned the same reference numerals as those according to the first exemplary embodiment perform the same processing. The occlusion determination unit 207 determines the occlusion relationship between objects based on partial images of candidate objects detected in the image. Furthermore, the tracking unit 205' tracks the tracking object based on the determination result of the occlusion determination unit 207.

[0119] The processing performed by the information processing device 1' according to the second exemplary embodiment will now be described. The flowcharts and diagrams according to this exemplary embodiment are shown below. Figure 3 , Figure 10A and Figure 10B Correspondingly, the basic processing is similar to that according to the first exemplary embodiment, except for the processing in S306. Therefore, the differences in S306 will be described in detail below, and descriptions of other processing will be omitted. In S305, candidate objects similar to the tracked object are detected based on the feature values ​​of the tracked object. In this case, when the tracked object is occluded by other objects, if the object occluding the tracked object is similar to the tracked object, the other object is detected as a candidate object. In this case, although the tracked object is related to the position of the similar object that is being occluded in the occlusion determination process, since the previous tracking features are maintained, the tracked object can be tracked again at the time when the occlusion is lifted. On the other hand, when the tracked object is occluded by an obstacle such as a wall, the occluded tracked object will not be detected as a candidate object in S305. In this case, in the subsequent occlusion determination process, it is determined that there is no candidate object that can be related to the tracked object that was detected immediately before the occlusion, and the feature values ​​of the tracked object are stored in S303. Subsequently, tracking can be restarted at the time when the occlusion is lifted, and the tracked object can be detected again.

[0120] Figure 10A This is a flowchart illustrating the tracking object recognition process, including occlusion determination processing, in S306. First, in S701, the tracking unit 205' obtains the similarity between candidates for past times pre-stored in the storage unit 206 and candidates for the current time obtained by the object detection unit 204. The processing in S701 is performed in a manner similar to that in S701 according to the first exemplary embodiment. Next, in S702, the tracking unit 205' performs correlation based on the similarity between the past and current candidates. The processing in S702 is also performed in a manner similar to that in S702 according to the first exemplary embodiment.

[0121] In S1002, the occlusion determination unit 207 determines the existence or non-existence of an occlusion region for an occluding candidate object based on the position of the candidate object in the current processing object image (second image). More specifically, the occlusion determination unit 207 performs occlusion determination on each candidate object in the current image. (Refer to...) Figure 10B The occlusion determination process in S1002 is described in more detail. In this case, the occlusion determination unit 207 performs occlusion determination on candidates (referred to as objects of interest) for which no relevant candidates were found in S702. First, in S10021, the occlusion determination unit 207 determines whether a correlation has been established for all previously detected candidate objects in S702. When a correlation has been established between all candidate objects detected in the past image (first image) and the candidate objects detected in the current image, the process proceeds to S10025. If, among the candidate objects detected in the past image, there exists a past candidate object (object of interest) with a similarity less than or equal to a threshold with the candidate object detected in the current image, the process proceeds to S10022. More specifically, when the process proceeds to S10022, there may be occluded candidate objects. In S10022, for the current candidate object (object of interest), the occlusion determination unit 207 obtains information indicating the degree of overlap between the BB of the candidate and the BB of other candidates. As an indicator of the degree of overlap between objects, the intersection of unions (hereinafter referred to as IoU) is calculated. More specifically, when the partial image (BB) of a candidate object detected in the current image is region A of object A and region B of object B, the IoU of object A and object B is calculated as region (A ∩ B) / (A ∪ B). A higher IoU indicates a higher degree of overlap between objects. Other candidate objects with an IoU exceeding a threshold are set as occluders related to the candidate. In this case, the state of candidate object A is judged as "occluded". In S10024, the position of the candidate judged as "occluded" by the occlusion judgment unit 207 is updated based on the position of the occluder. For example, it can be updated according to formula (2-1).

[0122] p s = (1 - α)p o + αp s Formula (2-1)

[0123] p s Indicate the position of the candidate, and p o This indicates the location of the obstruction. α is a value set based on experience.

[0124] In S703, the tracking unit 205 identifies the correlation between candidate objects in the first image and candidate objects in the second image based on the occlusion judgment result. More specifically, the tracking unit 205 identifies the position of the tracked object in the second image. In S702, when a candidate object identified as the previously tracked object is identified in the current image, the tracking unit 205 identifies the position of the tracked object in the current image. If the previously tracked object is not identified from the candidate objects in the current image in S702, an occlusion judgment is then performed in S1002. When it is determined that the tracked object is occluded in the current image, the occluder is identified, and the position of the tracked object is updated based on formula (2-1). On the other hand, the feature values ​​of the tracked object are not updated. In S704, the storage unit 206 stores the position and feature values ​​of the tracked object identified by the tracking unit 205. If occlusion occurs, the above processing allows tracking to resume after the occlusion is removed, because in some cases, the position of the tracked object is updated while maintaining the feature values ​​of the tracked object.

[0125] <Variation Example 2-1>

[0126] Variation 2-1 uses a neural network for occlusion detection. An example of occlusion detection using a neural network is "Zhou, Bi-box Regression for Pedestrian Detection and Occlusion, In: ECCV2018". In this example, in S1002, the tracking unit 205 estimates the object's bi-box (BB) and simultaneously estimates the unoccluded area (visible area) of the object region. Then, when the ratio of the occluded area to the object region exceeds a predetermined threshold, the occlusion detection unit 207 can determine it as occlusion.

[0127] <Effects of the Second Exemplary Embodiment>

[0128] Figure 11 This demonstrates the effect of updating candidate positions by using occlusion judgment and candidate position updates based on the position of occluders.

[0129] Figure 11Images 1211, 1212, 1213, and 1214, acquired at times t = 0, 1, 2, and 3, and the tracked object 1216 are shown. At time t = 0, the tracked object 1216 and a similar object 1215 exist, and both objects are being tracked. At time t = 1, the tracked object 1216 is hidden by the similar object 1217, and at time t = 1, only the similar object 1216 exists as a candidate object. At this time, when the IoU of 1216 and 1217 exceeds the threshold and is determined to be occluded, the position of 1216 is updated based on formula (2-1) to match the position of 1217. At time t = 2, since the occlusion has not been removed, the position of 1216 is updated to match the position of 1218, which is the occluder. At time t = 3, the occlusion is removed, and three different candidates 1219, 1220, and 1221 exist. At this point, the correct correlation results are between 1218 and 1219, and between 1216 and 1220. However, if the position of 1216 is not updated to match the positions of 1217 and 1218 without occlusion detection, then at time t=3, 1216 will exist around candidate 1221. Therefore, 1216 is likely to be correlated with the newly acquired candidate 1221 rather than with 1220, which may lead to false tracking. On the other hand, when the position of candidate 1216 is updated based on formula (2-1), the position of 1216 becomes close to the position of 1220, making it possible to correlate 1216 with 1220. Therefore, the possibility of false tracking can be reduced.

[0130] <Third Exemplary Embodiment> Single Object Tracking (Combined with Tracking Methods Learned Online)

[0131] In a third exemplary embodiment, for an online learning-based tracking method, multiple candidate objects are tracked simultaneously to stably track each similar object among multiple similar objects. The hardware configuration is similar to that according to the first exemplary embodiment. Figure 19 An example of the functional configuration of the information processing device 3 according to a third exemplary embodiment is shown. This configuration is substantially the same as... Figure 2The configuration is similar to that of the first exemplary embodiment, and additionally includes a learning unit 1902 for online learning. The tracking unit 1901 identifies the location of the object to be tracked by inputting the current image into the learned model. The learning unit 1902 updates the connection weighting parameters of the learned model used to estimate the object's location based on the estimated location of the object in the current image. The learned model used in this case is a Multi-Domain Convolutional Neural Network (MDNet) (“Nam, Learning Multi-Domain Convolutional Neural networks for Visual Tracking”, In: CVPR 2016). MDNet inputs the image into a CNN (learned model) to obtain feature quantities indicating the object. Furthermore, the obtained feature quantities are input into fully connected layers (hereinafter referred to as FC layers), and it is determined whether the input feature quantity is a feature quantity of the object to be tracked. Based on online learning, the FC layers output larger values ​​for objects that are more likely to be tracked. In online learning, the FC layers are learned in the initial frame, and then at intervals of several frames. Descriptions of processes similar to those in the first exemplary embodiment will be omitted, and processes different from those in the first exemplary embodiment will be described in detail.

[0132] Reference Figure 20 The processing performed by the information processing device 3 according to this exemplary embodiment is described. The processing in S301 to S304 is similar to the processing in S301 to S304 according to the first exemplary embodiment. In S304, a search range is set based on the acquired image. The search range image is determined based on the position and size of past candidate objects. In S305, the search range image acquired in S304 is input into the learned model to input the various feature quantities acquired from the search range image into the FC layer, and then objects whose likelihood (similarity) of the acquired tracked object exceeds a threshold are acquired as candidate objects. The MDNet described above is used as the learned model. In S2001, the tracking unit 1901 identifies the position of the tracked object from the candidate objects. In S2002, the learning unit 1902 updates the parameters of the learned model based on the determination result of the tracked object.

[0133] This online learning-based tracking method can reduce false tracking by tracking multiple candidates simultaneously in a manner similar to the first exemplary embodiment.

[0134] <Fourth Exemplary Embodiment> Multiple Object Tracking

[0135] In the fourth exemplary embodiment, a case is described where multiple tracking objects are set instead of a single tracking object. Even when tracking multiple similar objects, stable tracking can be achieved even if a tracking object is lost at one point by simultaneously tracking previously detected candidate objects. The hardware configuration is similar to that according to the first exemplary embodiment. Except for the processing differences between the tracking object determination unit 202 and the tracking unit 205, the information processing device according to this exemplary embodiment has a functional configuration similar to that of the information processing device 1 according to the first exemplary embodiment. The tracking object determination unit 202 determines multiple objects as tracking objects. The tracking object determination unit 202 determines the tracking objects using a method similar to that according to the first exemplary embodiment. All objects included in a specific image can be acquired as tracking objects. The tracking unit 205 tracks the detected objects for multiple tracking objects. More specifically, the tracking unit 205 maintains the CNN features of multiple candidate objects and correlates them using the similarity between candidate objects at times t and t+1.

[0136] The processing performed by the information processing device 1 according to this exemplary embodiment will be described. A flowchart and... Figure 3Correspondingly, in S301, the image acquisition unit 201 acquires an image (initial image) for capturing a predetermined object. In S302, the tracking object determination unit 202 determines multiple tracking objects in the image acquired in S301. In S303, the holding unit 203 holds the feature values ​​of the multiple tracking objects in the image containing the determined tracking objects based on a learned model. In this case, the detection-tracking method is used as a learned model ("Feichtenhofer, Detect to Track and Track to Detect, In: ICCV2017"). The detection-tracking method performs object detection using a CNN for each frame in a continuous time series. Then, in S304, the image acquisition unit 201 acquires images captured at multiple times for tracking processing. In S305, the object detection unit 204 detects the positions of candidate objects in the temporally continuous images acquired by the image acquisition unit 201 based on a learned model. First, the object detection unit 204 detects candidate objects for each consecutive frame in the time series using a CNN (learned model). More specifically, the object detection unit 204 acquires CNN features at time t and time t+1. Next, the object detection unit 204 detects the position of candidate objects by calculating the cross-correlation between the CNN features acquired at time t and the CNN features acquired at time t+1. In S306, the tracking unit 205 identifies multiple tracking objects in the current image (t+1). In this case, the tracking unit 205 first estimates the change in BB (ΔBB) of each object (change in BB position and change in BB size). More specifically, the tracking unit 205 estimates the change in BB by comparing BB(t+1) with BB(t) + ΔBB(t). In this case, objects with similar changes in BB position and similar changes in BB size are identified as the same object, and thus correlation between the objects can be performed. Next, the tracking unit 205 calculates the distance between the CNN features of the relevant candidate object at time t and the CNN features of the relevant candidate object at time t+1 based on formula (1-1) to calculate similarity. When a correlation with a similarity higher than a predetermined value exists, tracking is performed in relation to the previous detection results. Relevance can be established in descending order of relative similarity. When no relevance exists with a similarity higher than a predetermined value, the current detection result (feature quantity and location) is maintained without being correlated with previous detection results.

[0137] If two objects are detected at time t and one object is detected at time t+1, it is assumed that the object identical to the object at time t+1 is a highly similar object among the two objects detected at time t. Associating highly similar objects reduces the possibility of erroneous tracking. However, it is possible that due to occlusion, the object detected at time t is detected at time t+1. In this case, if at least one candidate object exists at time t+1 besides the target object, erroneous tracking of candidate objects at close range may begin. Therefore, in S306, the CNN features of multiple objects to be considered as candidates can be preserved, and the similarity with the feature values ​​of the candidate objects preserved during similarity calculation can be calculated. When the target object is occluded, correlation cannot be identified, but tracking can resume when the occlusion is removed.

[0138] The present invention is further implemented by the following process. More specifically, software (programs) for implementing the functions of the exemplary embodiments described above are supplied to the system or device via a data communication network or various types of storage media, and the computer (CPU or microprocessor unit (MPU) of the system or device reads and executes the program. The program may be provided by recording it on a computer-readable recording medium.

[0139] This invention is not limited to the exemplary embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the appended claims are attached to inform the public of the scope of the invention.

[0140] This application claims the benefit of Japanese Patent Application 2020-123796, filed on July 20, 2020, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device configured to track a specific object in images captured at multiple times, the information processing device comprising: A holding unit is configured to hold feature quantities of a tracked object based on a learned model, the learned model being configured to detect the position of a predetermined object in an input image; An acquisition unit is configured to acquire feature quantities of objects in multiple images based on the learned model; A detection unit is configured to detect candidate objects similar to the tracked object based on feature quantities of the tracked object and feature quantities of objects obtained from the plurality of images; as well as The recognition unit is configured to recognize the correlation between candidate objects detected in a first image of the plurality of images and candidate objects in a second image taken at a different time than the first image. The recognition unit is configured to, in addition to the tracked object, also identify the correlation of candidate objects in the plurality of images other than the tracked object. The holding unit is configured to hold the feature value of the object specified by the user as the feature value of the tracked object. The identification unit identifies the tracked object in the second image based on the correlation between the candidate objects detected in the first image and the candidate objects detected in the second image.

2. The information processing device according to claim 1, wherein, The holding unit holds the feature values ​​of the tracked object in the second image and the feature values ​​of the candidate object in the second image.

3. The information processing apparatus according to claim 1 or 2 further includes an updating unit configured to update the feature quantity of the tracked object held by the holding unit based on the identified correlation.

4. The information processing device according to claim 3, in, If, in the second image, a candidate object is detected whose similarity between the feature value of the tracked object held by the holding unit and the feature value of the tracked object in the first image is higher than a predetermined threshold, the updating unit updates the feature value of the tracked object using the feature value obtained from the second image. If no candidate object with a similarity to the feature quantity of the tracked object higher than the predetermined threshold is detected in the second image, the updating unit uses the feature quantity obtained from the first image to update the feature quantity of the tracked object.

5. The information processing device according to claim 4, in, The detection unit detects the positions of candidate objects in the plurality of images. Specifically, when a candidate object is detected in the second image whose feature similarity to the tracked object in the first image is higher than a predetermined threshold, the updating unit updates the position of the tracked object in the second image. If no candidate object with a feature similarity to the tracked object higher than the predetermined threshold is detected in the second image, the update unit does not update the position of the tracked object.

6. The information processing device according to claim 1 or 2, in, The first image was taken before the second image. The information processing device further includes an extraction unit configured to extract a portion of the second image to detect candidate objects based on the positions of candidate objects detected in the first image. The acquisition unit obtains the feature quantity of the object from the second image based on a portion of the second image extracted by the extraction unit.

7. The information processing device according to claim 6, wherein, The extraction unit extracts a portion of the image of a predetermined size from the region corresponding to the vicinity of the candidate object detected in the first image in the second image.

8. The information processing apparatus of claim 1 or 2 further includes a learning unit configured to update the parameters of the learned model based on teacher data used to indicate the location of the tracked object in an image.

9. The information processing apparatus of claim 8, further comprising a computing unit configured to acquire a loss for estimating the location of objects in the image similar to the tracked object, based on the teacher data used to indicate the location of the tracked object in the image. in, The learning unit updates the parameters of the learned model based on the loss obtained by the computing unit.

10. The information processing device according to claim 1 or 2, in, The first image was taken before the second image. The information processing device further includes a judgment unit configured to determine, based on the position of the candidate object in the second image, whether an occlusion region of the candidate object exists or does not exist. The identification unit identifies the correlation between candidate objects in the first image and candidate objects in the second image based on the judgment result of the judgment unit.

11. The information processing device according to claim 10, wherein, If the judgment unit determines that the tracked object in the second image is occluded, the recognition unit will identify the object occluding the tracked object in the second image as the tracked object.

12. The information processing device according to claim 10, wherein, For an object in the first image whose similarity to a candidate object in the second image is less than a predetermined threshold, the determination unit determines that the occlusion region exists.

13. The information processing device according to claim 12, wherein, For an object in the first image whose similarity to a candidate object in the second image is less than a predetermined threshold, the judging unit determines the degree of overlap between the region of the candidate object in the second image and the region of the object, and determines that the object is occluded when the degree of overlap is higher than the predetermined threshold.

14. The information processing device according to claim 13, in, When the object is occluded, the determining unit identifies the candidate object occluding the object as the occluder. The identification unit identifies the position of the obstruction as the position of the object.

15. An information processing method for tracking a specific object in images captured at multiple times, the information processing method comprising: Based on a learned model to maintain the feature quantity of the tracked object, the learned model is used to detect the position of a predetermined object in the input image; Based on the learned model, feature quantities of objects in multiple images are obtained; Candidate objects similar to the tracked object are detected based on the feature values ​​of the tracked object and the feature values ​​of objects obtained from the plurality of images; as well as Identify the correlation between candidate objects detected in a first image and candidate objects in a second image taken at a different time than the first image. In the identification process, in addition to the tracked object, the correlation of candidate objects other than the tracked object in the plurality of images is also identified, and In the "holding" process, the feature values ​​of the object specified by the user are held as the feature values ​​of the tracked object. In the identification process, the tracking object in the second image is identified based on the correlation between the candidate objects detected in the first image and the candidate objects detected in the second image.

Images