Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method and a program for appropriately selecting a discriminator to be used to tracking in accordance with the completion degree of online learning in the information processing device having a plurality of discriminators including a discriminator for performing the online learning to track a subject.SOLUTION: An information processing device for tracking a subject by using either or both a first discriminator (a tracking part) for tracking a subject in an input image and a second discriminator (a tracking part) different from the first discriminator to track the subject includes an image acquisition part for acquiring learning data to be used to learn tracking of the first discriminator, the first discriminator for performing online learning during tracking the subject by using the learning data, and a completion degree determination part for evaluating the completion degree of the online learning to determine whether to use the first discriminator in tracking the subject in accordance with the evaluation of the completion degree.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, technology using deep neural networks (DNNs) has been attracting attention as a highly accurate tracking technology. Online learning, in which classification results are accumulated for learning and a classifier for tracked / non-tracked objects is sequentially updated while tracking, is useful for improving tracking accuracy if there are sufficient learning times or sufficient training data. Non-Patent Document 1 discloses a technology for achieving highly accurate tracking by combining a pre-trained tracking method such as that described in Non-Patent Document 2 with a tracking method that performs online learning. Here, the offline learning method described in Non-Patent Document 2 is referred to as Fully-Convolutional Siamese Networks for Object Tracking (Siam Tracking Method). Patent Document 1 also discloses a technology for combining a tracking method that performs online learning with a tracking method that performs only offline learning in order to improve tracking accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-262331 [Non-patent literature]

[0004] [Non-Patent Document 1] Jinghao Zhou, “Discriminative and Robust Online Learning for Siamese Visual Tracking”, [online], April 3, 2020, AAAI, [Searched on April 1, 2021], Internet<URL:https: / / ojs.aaai.org / / index.php / AAAI / article / view / 7002> [Non-patent document 2] Luca Bertinetto, “Fully-Convolutional Siamese Networks for Object Tracking”, [online], June 30, 2016, AAAI, [searched on April 1, 2021], Internet<URL:https: / / arxiv.org / abs / 1606.09549> Summary of the Invention [Problem to be solved by the invention]

[0005] However, when online learning is used for tracking, if the number of learning sessions or the number of learning data is insufficient, the tracking accuracy may actually be lower than when online learning is not used. As a result, at a stage when the number of learning data or the number of learning sessions is insufficient, such as in the early stages of tracking, the tracking accuracy may actually be lower when online learning is used in combination.

[0006] The present invention aims to appropriately select a classifier to be used for tracking in accordance with the degree of completion of online learning in an information processing device having multiple classifiers, including a classifier that performs online learning, in order to track a subject. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an information processing apparatus according to one embodiment includes the following configuration: tracking means for tracking a subject using either or both of a first classifier that tracks a subject in an input image and a second classifier that tracks the subject and is different from the first classifier, acquiring means for acquiring training data used to train the first classifier to track the subject, learning means for performing online learning using the training data to train the first classifier while tracking the subject, evaluation means for evaluating the completion of the online learning, and determination means for determining whether the tracking means should use the first classifier in tracking the subject based on the evaluation of the completion. [Effects of the Invention]

[0008] In an information processing device having a plurality of classifiers including a classifier that performs online learning to track a subject, a classifier to be used for tracking is appropriately selected depending on the degree of completion of the online learning. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the hardware configuration of an information processing apparatus according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. [Figure 3] 3 is a flowchart showing an example of a processing method according to the first embodiment. [Figure 4] 10 is a flowchart showing an example of likelihood map integration processing according to the first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of a likelihood map generated by the information processing device according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a likelihood map according to the passage of time according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a change in loss over time in online learning according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a processing method according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] In this embodiment, a classifier that distinguishes between a tracked target and a non-tracked target in an input image is sequentially updated while tracking a subject (tracking target) in the input image through online learning. Hereinafter, in tracking of a subject, candidate areas for the tracking target are extracted from an image that serves as a search range for the subject, and the likelihood of each extracted candidate area being a tracking target is calculated, thereby determining the area in which the tracking target exists. However, the method of tracking a subject using a classifier that performs online learning is not particularly limited to this, and tracking may be performed using any known method, such as tracking using template matching.

[0012] On the other hand, in online learning, when the number of training sessions is extremely small, it is difficult for the classifier to adapt to the training data. Furthermore, in online learning, learning tends to become unstable when there is an extremely small amount of training data. Furthermore, there are cases where an appropriate classification surface cannot be obtained even when the classifier is adapted to the training data, such as when there is a dissociation between the distribution of possible appearances of the tracked and non-tracked targets and the distribution of the training data. Hereinafter, such a state will be referred to as a state in which online learning is not complete.

[0013] The information processing device 1 according to this embodiment includes a first tracking unit (classifier) ​​that performs tracking through online learning and a second classifier different from the first tracking unit, and tracks a subject using either or both of these classifiers. When the online learning of the first tracking unit is not complete, the information processing device 1 does not use the first tracking unit for tracking, and instead tracks the subject using the second tracking unit, thereby suppressing the occurrence of erroneous tracking. In view of this, the information processing device according to this embodiment evaluates the degree of completion of online learning in the first tracking unit that performs tracking through online learning. Next, the information processing device determines whether to use the first tracking unit to track the subject, depending on the evaluation of the degree of completion. The configuration and processing of such an information processing device will be described below with reference to FIGS. 1 to 7.

[0014] In this embodiment, the second tracking unit is described as having been trained in advance, rather than undergoing online learning. The training method of the second tracking unit is not particularly limited, and the training format of the second tracking unit (whether online learning is performed or not) is also not particularly limited. In this embodiment, the second tracking unit is trained in advance to perform tracking using the Siam tracking method described in Non-Patent Document 2, which detects the cross-correlation between features extracted by CNN from a template image of the tracking target and each candidate region as the likelihood of the tracking target. Cases where the training format of the second tracking unit is different or where there are two or more second tracking units will be described in Embodiment 3.

[0015] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device according to this embodiment. The information processing device 1 according to this embodiment includes a CPU 101, a ROM 102, a RAM 103, a storage unit 104, an input unit 105, a display unit 106, and a communication unit 107. The SPU 101 executes a control program stored in the ROM 102 and controls the overall processing performed by the information processing device 1. The RAM 103 functions as a storage area or work area for temporarily storing various data generated by the processing performed by each functional unit. The storage unit 104 is a storage area for storing various data, and media such as a hard disk drive (HDD), flash memory, and various optical media are used. The input unit 105 is configured, for example, with a keyboard, touch panel, or dial, and accepts input from a user. The user can enter user input, such as settings for a tracking target, via the input unit 105. The display unit 106 is, for example, an LCD display, and presents various processing results, such as the subject and tracking results, to the user. The information processing device 1 can also communicate with external devices, such as an imaging device, via the communication unit 107. This communication may be wireless communication via a network or may be priority communication, and the information processing device 1 may have an imaging function.

[0016] Next, functions of the information processing device 1 according to this embodiment will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a block diagram showing an example of the functional configuration of the information processing device 1 according to this embodiment. Fig. 3 is a flowchart showing an example of processing performed by the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment includes a first tracking unit that performs tracking by online learning as described above, and a second tracking unit that is different from the first tracking unit, and tracks the tracking target using these tracking units. Note that in this embodiment, The second tracking unit does not perform online learning, but is pre-trained to detect the subject.

[0017] In step S301, the image acquisition unit 201 acquires an image for setting a tracking target. The image acquisition unit 201 may acquire an image from an imaging device connected to the information processing device 1, may acquire an image stored in the storage unit 104, or may capture an image using an imaging unit (not shown).

[0018] In step S302, the target designation unit 202 determines the tracking target in the image acquired in step S301. Here, the target designation unit 202 can determine the tracking target in accordance with an instruction specified by the input unit 105 (for example, by touching the subject displayed on the display unit 106). The target designation unit 202 also generates a template that represents the characteristics of the determined tracking target. In the subsequent tracking process, the tracking target determined in step S302 is tracked.

[0019] In step S303, image acquisition unit 201 acquires an image to be tracked (here, an image including the tracking target determined in step S302 in its imaging range). In step S304, area acquisition unit 203 determines an area to search for the tracking target (search area), and cuts out the determined area from the image acquired in step S303. Here, area acquisition unit 203 may determine the search area as the entire image acquired in step S303, or may determine the area around the position of the tracking target in the immediately preceding process (for example, a predetermined range centered on the tracking target).

[0020] In step S305, the result acquisition unit 204 tracks the subject using each of the first and second tracking units. Here, the first tracking unit creates a likelihood map using a target determiner from the image of the search area acquired in step S303. The second tracking unit calculates the correlation between the template obtained in step S302 and the image of the search area acquired in step S303, and obtains a likelihood map indicating the probability that the subject exists at each position. Details of the processing performed by the first tracking unit will be described later in the description of FIG. 3, but the tracking method used by these tracking units is not particularly limited, and any known tracking method may be adopted. Here, as described above, the second tracking unit performs tracking using the Siam tracking method described in Non-Patent Document 2, in which at least one template of the tracked subject is used to distinguish between the tracked subject and the non-tracked subject.

[0021] In step S306, the completion determination unit 205 calculates the completion level of the first tracking unit. In step S307, the combination unit 206 determines whether to use the first tracking unit to track the tracking target based on the completion level of the tracking unit calculated in step S305, and calculates a final likelihood map to be used for tracking based on the determination result. The process of calculating the completion level and the process of determining whether to use the first tracking unit based on the completion level will be described later. Here, the combination unit 206 uses the first tracking unit for tracking, and creates a single likelihood map by integrating the likelihood maps acquired from each tracking unit in step S305 based on the completion level of each tracking unit. The combination unit 206 then detects the area with the highest likelihood as the tracking target. In step S308, the combination unit 206 acquires learning data to be used for training the target determiner of the first tracking unit from the image of the search area and stores the data in the storage unit 207. Next, in step S309, combination unit 206 performs learning using the learning data stored in storage unit 207. A detailed description of the learning process of the first tracking unit will be given later. In step S310, combination unit 206 decides whether to end the tracking process. If not, the process returns to S303 and the tracking process continues. The combination unit 206 may determine whether to end the tracking process by, for example, acquiring a user input, or may determine that the tracking process has ended after tracking has been performed for a predetermined period of time, and the criteria may be set arbitrarily by the user.

[0022] The learning performed by the first tracking unit in steps S306, S307, S308, and S309 will be described in detail below. The first tracking unit inputs a feature map calculated from the image of the search area using a CNN or the like to a target determiner to obtain a likelihood map. Here, the feature map is a three-dimensional tensor of width Wf × height Hf × number of channels Cf, and is an arrangement of C-dimensional vectors representing the features of each small area obtained by dividing the image of the search area into a grid of width Wf and height Hf. The likelihood map is a map that responds strongly to areas within the search area where the tracking target is highly likely to exist. Each cell (small area) of the feature map and likelihood map corresponds to the feature and likelihood of the small area obtained by dividing the image of the search area into a grid.

[0023] In step S306, the completion determination unit 205 calculates the completion level of the online learning of the first tracking unit according to the learning status of the online learning. In this embodiment, the change amount of loss calculated in the learning of the first tracking unit before and after new learning data is input is calculated, and if this change amount is equal to or less than a predetermined threshold, the completion level is set to 1; otherwise, the completion level is set to 0. Note that here, evaluation is performed by binary discrimination between whether the completion level is 1 (online learning is complete) or 0 (online learning is not complete). For an image at the start of tracking, the completion determination unit 205 sets the completion level of the first tracking unit to 0. For example, if the change amount ΔL in the total loss value before and after parameter update in the learning of the first tracking unit is smaller than a predetermined value, it is determined to be complete, and the completion level is set to 1. ΔL is calculated, for example, as follows: ΔL=|L t-1 -L t | Formula (1)

[0024] where L t is the loss value at the t-th iteration. An iteration refers to the number of parameter updates since the start of tracking. Here, when ΔL is less than a predetermined threshold, the degree of completion is determined to be 1, and it is determined that the learning of the first tracking unit has been sufficiently completed by the t-th learning, but the criteria for determination are not particularly limited to this. For example, the degree of completion determination unit 205 may determine that the degree of completion is 1 when the average of ΔL over several iterations is smaller than a predetermined threshold. Alternatively, the degree of completion determination unit 205 may determine that the degree of completion is 1 when the loss value is smaller than a predetermined value.

[0025] Here, it is assumed that the degree of completion needs to be calculated for the first tracking unit, but the degree of completion may also be acquired for the second tracking unit in consideration of use in the weighted sum of a likelihood map, which will be described later. Also, as described above, completion determination unit 205 according to this embodiment will be described as making a binary determination as to whether or not the first tracking unit is complete, but the degree of completion may also be output as a continuous value between 0 and 1 (the closer to 1, the higher the degree of completion) based on the magnitude of the loss value, etc. Also, for the second tracking unit that does not perform online learning, the degree of completion may be set to 1 from the start of tracking, while for the first tracking unit, the degree of completion may be set to 0 at the start of tracking.

[0026] An example of the process of combining the tracking results in step S307 is shown in the flowchart of FIG. 4. First, in step S401, the combination unit 206 aligns the resolution of the likelihood maps output by each tracker in step S305 using any method such as bilinear interpolation. Next, in step S402, the combination unit 206 acquires the value of each cell of the likelihood map to be finally used (final likelihood map) based on the completion rate of each tracker calculated in step S306. Here, the combination unit 206 acquires the final likelihood map by integrating the likelihood map calculated by the first tracker and the likelihood map calculated by the second tracker. Furthermore, if the completion rate of the first tracker is 0, the combination unit 206 adopts the likelihood map calculated by the second tracker as the final likelihood map.

[0027] Here, the degree of completion is output as a continuous value, and combining unit 206 calculates a weighted sum of the values ​​of each cell of the likelihood map using the degree of completion value calculated by each tracking unit as a weighting coefficient to obtain the final likelihood map. This processing makes it possible to limit the rate at which the tracking results of the first tracking unit are reflected in the final likelihood map while the degree of completion of the online learning of the first tracking unit is low, and to increase it as the degree of completion improves.

[0028] Thereafter, in step S403, combination unit 206 estimates the region with the highest likelihood as the tracking target and the other regions as non-tracking targets, and proceeds to step S308. An example of the integrated likelihood map is shown in FIG. 5. In likelihood map 501, tracking target 502 is displayed, and cell 503 near the center of tracking target 502 is displayed in black as an indication of a high likelihood value. In this case, it can be estimated that tracking target 502 is located in cell 503 with the highest correlation value. Here, a method of calculating a weighted sum of likelihood maps has been given as an example of a method of integrating likelihood maps into a single final likelihood map, but the method is not limited to this as long as the integration method reflects the respective likelihood values, such as taking the product of likelihood maps.

[0029] In step S308, the combination unit 206 first assigns, to each cell of the image in the search area, a label representing the result of the classification of tracked / non-tracked targets performed in step S403. As described above, the combination unit 206 may selectively use, as the classification result, the result of only the first tracking unit or the result of only the second tracking unit, depending on the degree of completion.

[0030] Next, the details of the learning process by the first tracking unit in step S309 will be explained. First, the combination unit 206 acquires multiple pairs of feature amounts and labels, which are learning data, from the storage unit 207. Next, the combination unit 206 inputs the feature amounts to a target classifier, obtains the likelihood of the feature amount being a tracking target, and then calculates a loss based on the likelihood and label. Next, the combination unit 206 updates the parameters of the target classifier using a gradient method based on the calculated loss. This process is similar to general learning processes, so a detailed description will be omitted.

[0031] The loss function is designed so that the loss is small when the tracking target is correctly estimated, and the loss value is large when a non-tracked target is estimated as a tracked target or when a tracked target is estimated as a non-tracked target. The loss value L can be written, for example, as in the following equation (2).

number

[0032] Here, L t is the loss value at the t-th iteration, and N t is the number of training data used to calculate the loss value at the t iteration, and loss i t is the loss for the i-th training data. Also, C in is the likelihood of the tracking target of the i-th training data calculated in step S308, and C gt is the label of the i-th training data. The loss value L t is used to determine the degree of completion. Equation (2) is an example, and the loss calculation formula is not limited to this.

[0033] Note that, in this description, it is assumed that the timing when online learning is completed and learning ends is when the completion rate of the first tracking unit reaches 1, but the condition for completing online learning is not particularly limited to this. For example, the combination unit 206 may determine the number of parameter updates in advance, and terminate the learning process by determining that online learning is completed when the number of iterations t reaches a predetermined value.

[0034] Furthermore, the information processing device 1 may continue learning in parallel with the processing of step S303, step S304, etc., and is not limited to these. The updated parameters are stored as learned parameters in the storage unit 207. Furthermore, although the first tracking unit has been described as sequentially updating the parameters of the target determiner by a gradient method, a different method may be used as long as it is a method of sequentially updating the discrimination plane of the tracked target / non-tracked target using data of the tracked target / non-tracked target obtained during tracking.

[0035] With this configuration, the degree of completion of the first tracking unit that performs online learning is calculated, and the first tracking unit and the second tracking unit can be combined or either one can be used for tracking depending on the calculated degree of completion. Therefore, if the first tracking unit is not complete, the first tracking unit is not used for tracking, thereby making it possible to suppress a decrease in tracking accuracy due to the use of a tracking means with a low degree of completion. Furthermore, even when the first tracking unit is used for tracking, by performing weighted integration of the likelihood map depending on the calculated degree of completion, it is possible to adjust the rate at which the tracking result by the first tracking unit is reflected in the final tracking result depending on the degree of completion.

[0036] As described above, the information processing device 1 according to this embodiment combines the tracking units used for tracking depending on the degree of completion of the first tracking unit that performs online learning. The principle of suppressing a decrease in tracking accuracy through such processing will be described below with reference to FIGS. 6 and 7. FIG. 6 shows likelihood maps obtained as the output of the first tracking unit and the output of the second tracking unit at times t=0, 1, 2, and 3, respectively. In the likelihood maps, areas with high likelihood of the tracking target are indicated in dark colors. FIG. 7 shows the time change in loss by the first tracking unit corresponding to time t.

[0037] In the example in Figure 6, at the beginning of learning (t = 0), the peak position of the likelihood map in the second tracking unit coincides with the position of the tracked target. On the other hand, in the first tracking unit, as shown in Figure 7, the learning of the classifier that distinguishes between tracked and non-tracked targets has not progressed, and the likelihood of areas other than the tracked target in the likelihood map is also likely to be high. Therefore, if the likelihood map of the first tracking unit is used for tracking, there is a high possibility of mistracking.

[0038] However, after online learning progresses after the initial stage of tracking (for example, the period from t = 0 to 2), the likelihood map of the first tracking unit is able to distinguish between the tracked target and non-tracked targets. In particular, at time t = 3, when the tracked target and its imitation are close to each other, the likelihood map of the second tracking unit reacts to both the tracked target and its imitation, which could result in mistracking. On the other hand, the likelihood map of the first tracking unit is able to distinguish between the tracked target and its imitation through sufficient online learning. Therefore, in this case, it is possible to suppress mistracking by using the likelihood map of the first tracking unit (or by integrating the likelihood maps of both tracking units).

[0039] [Variation 1] The combination unit 206 may also determine that online learning is complete when the number N of training data stored in the storage unit 104 exceeds a predetermined number, or when the number of feature variations σ of the training data exceeds a predetermined number. The variation σ is calculated by, for example, the eigenvalue λ of the covariance matrix of the feature, using the following equation (3): i It can be calculated as the sum of (i=1, 2, ..., d).

number

[0040] Here, d is the number of dimensions of the feature. As the variation α, all eigenvalues ​​may be used as in Equation (3), or only a predetermined number of eigenvalues ​​arranged in descending order may be used, allowing for the desired extraction process. In this example, using a threshold thN or thσ, if either N>thN or σ>thσ, or both, are satisfied, the completion rate of the first tracking unit is set to 1, and otherwise the completion rate is set to 0.

[0041] Furthermore, in this embodiment, the number of learning data, measurement of feature variation, setting of thresholds, and comparison with thresholds have been described as being performed for each tracking target, but this processing may be performed for both tracking targets and non-tracking targets. Furthermore, the targets classified into categories by the information processing device 1 according to this embodiment are not limited to tracking targets / non-tracking targets, and any three or more classification categories may be provided, such as tracking targets / objects similar to the tracking targets / background, for example.

[0042] In addition, in the present embodiment, the determination of the degree of completion in the first tracking unit and the update of parameters are described as being performed from the start of tracking, but the start timing of these processes is not particularly limited to this and may be linked to, for example, the convergence of learning. That is, the update of the parameters of the first tracking unit may be started when it is determined that learning data has been collected, and tracking by the first tracking unit may be started when the learning has converged.

[0043] When there is a small amount of training data at the beginning of tracking or when the appearance of the tracked target changes drastically, online learning of the target determiner is likely to become unstable. For example, if training is performed using training data with a special appearance at the beginning of tracking, such as when motion blur occurs only in images at the beginning of tracking, it may become impossible to identify any tracking target other than the special-looking tracking target. On the other hand, according to the above-mentioned process, by using the tracking results from the first tracking unit after the number of training data or the variation in feature values ​​has increased to a certain extent, the distribution of the training data approaches the possible distribution of tracked / non-tracked targets, making it easier to suppress erroneous tracking.

[0044] [Variation 2] Furthermore, in the process of step S306, the completion determination unit 205 may calculate the completion level according to the degree of matching described below. Here, the degree of matching is assumed to be the degree of closeness of the distributions of the feature quantities of the newly added training data and the feature quantities of the stored training data. In this example, the completion determination unit 205 quantifies the difference in distribution between the newly added training data and the stored training data, and determines that online learning is complete when this difference is smaller than a predetermined value. As a method of quantification, a separate CNN that distinguishes between the newly added training data and the stored training data may be used, or an index of the distance between distributions such as Kernel Mean Matching may be used.

[0045] According to this processing, by using the tracking results obtained by online learning for tracking, the distribution of the learning data approaches the possible distribution of the tracked / non-tracked targets, making it easier to suppress erroneous tracking.

[0046] [Variation 3] Furthermore, in the completion determination in step S306, the completion determination unit 205 may determine that the online learning is complete when the change in the distribution of the feature quantities of the learning data becomes small to a certain extent when new learning data is added to the learning data stored in the storage unit 207.

[0047] For example, the completion degree determination unit 205 defines a change in the distribution of the feature quantity as a change Δc in the center of the distribution and a change Δσ in the variation. If either or both of these changes become smaller than predetermined thresholds (thc, thσ), the completion degree determination unit 205 sets the completion degree to 1, and otherwise sets the completion degree to 0. Δc and Δσ are defined as follows:

number

[0048] where time t is the current time, N is the number of training samples obtained at time t, and fi (i=1, 2, ..., N) is the training sample obtained at time t, c t-1 and c t is the variance of the training samples obtained at time t-1 and time t. Also, σ t-1 and σ t may be calculated by Equation (1). Note that the change in distribution of the feature is not limited to this, and the change in distribution may be defined using other values ​​that vary depending on the distribution, such as the mean or variance of the feature.

[0049] According to this process, it is possible to determine that online learning is complete when the change in the distribution of the feature quantities of the learning data becomes smaller than a predetermined value. Therefore, when the distribution of the learning data of the first tracking unit matches the possible distribution of the tracking target / non-tracking target, tracking can be started using the tracking results of the first tracking unit, thereby suppressing erroneous tracking.

[0050] [Variation 4] Furthermore, even after the learning of the first tracker is completed, if the degree of completion of the first tracker changes, completion determination unit 205 may adjust the weighting of the tracking result combination in the likelihood map according to the degree of completion. For example, if the degree of completion of the first tracker decreases, completion determination unit 205 may reduce the weight in the likelihood map for the first tracker, or may set the weight to 0. Furthermore, if the degree of completion of the first tracker decreases, completion determination unit 205 may initialize the target determiner and perform learning again. That is, completion determination unit 205 may delete some or all of the learning data that has already been saved from storage unit 207, and acquire learning data again by performing the processes of steps S308 to S309, and perform learning again.

[0051] According to this process, if the tracking result by the first tracking unit is used for tracking once and the completion rate of the first tracking unit changes, a determination may be made as to whether to combine the first tracking unit and the second tracking unit again to track the subject, depending on the changed completion rate. Therefore, even in cases other than the initial stage of tracking, by reducing the reflection rate of the output of a tracking unit with a low completion rate in the likelihood map (or not using it for tracking), a decrease in tracking accuracy can be suppressed. This assumes a case where, for example, the appearance of the tracking target changes significantly during tracking, even with a tracking unit that was once considered complete, it may be difficult to distinguish between the tracking target and the non-tracking target. Here, the completion rate determination unit 205 can calculate the completion rate using, for example, a calculation method similar to that of Modification 3.

[0052] [Variation 5] Alternatively, the first tracking unit may detect the tracking target by performing target determination after extracting multiple tracking target candidates using the second tracking unit. In this example, the result acquisition unit 204 does not first output the likelihood map of the first tracking unit, but outputs the likelihood map only for the second tracking unit. If the first tracking unit is not well-established, the combination unit 206 detects the area with the highest likelihood in the likelihood map of the second tracking unit as the tracking target. On the other hand, if the first tracking unit is well-established, the combination unit 206 first extracts areas with likelihoods equal to or greater than a threshold from the likelihood map of the second tracking unit as tracking target candidates. Next, the combination unit 206 uses the first tracking unit to perform feature extraction and target determination for each extracted area to calculate the likelihood, and detects the area with the highest likelihood as the tracking target. Furthermore, when acquiring learning data, the combination unit 206 may assign labels of tracking targets not only to the area detected as the tracking target but also to the surrounding area.

[0053] [Variation 6] In this embodiment, the parameters of the first tracking unit are sequentially updated using a gradient method, and the degree of completion is evaluated based on the detection results using the updated parameters. However, the online learning of the first tracking unit is not limited to this format. For example, the first tracking unit may perform online learning on a tracking method using multiple templates using the method described in "Sauer, Tracking Holistic Object Representations. In: BMVC2019." In this case, the information processing device 1 learns the templates used for tracking and calculates the degree of completion based on the learned templates. For example, the combination unit 206 may determine that the online learning is complete when the number of templates exceeds a predetermined value or when the variation in the feature values ​​of the templates exceeds a predetermined value. Furthermore, for example, the completion determination unit 205 may output the degree of completion of the online learning as a continuous value between 0 and 1 depending on the number of templates or the variation in the feature values ​​of the templates.

[0054] Furthermore, for example, the first tracking unit may perform online learning on a tracking method for tracking a tracking target using a k-nearest neighbor method. In this case, the completion determination unit 205 can calculate the completion of the online learning by the same process as in the first or second modification.

[0055] [Embodiment 2] In the first embodiment, at the start of online learning, the completion rate of the first tracker performing online learning is set to 0, and the completion rate of the second tracker that has been trained in advance is set to 1. Also, in the first embodiment, an example has been described in which the completion rate of the first tracker is set to 1 when online learning is completed. In this embodiment, the information processing device 1 also performs processing similar to the above example, but when online learning is completed, the completion rate of the second tracker is set to 0. That is, the result acquisition unit 204 tracks the subject using only the likelihood map of the first tracker from the frame next after online learning is completed. Note that in this embodiment, tracking processing and online learning processing are basically performed by processing similar to that in FIG. 3 of the first embodiment, and therefore redundant description will be omitted.

[0056] According to this processing, when online learning is completed, it is possible to track the subject using only the first tracking unit that performs online learning. Therefore, when the amount of calculation or the amount of parameter data of the first tracking unit is small compared to the second tracking unit that does not perform online learning, it is possible to achieve tracking with lighter processing by switching to tracking using only the first tracking unit. This makes it possible to improve speed and save memory.

[0057] [Embodiment 3] The information processing device 1 according to the first embodiment includes two tracking units for tracking a subject: a first tracking unit that performs online learning and a second tracking unit that has been trained in advance. On the other hand, the information processing device according to the present embodiment includes multiple tracking units for tracking a subject. However, as long as the information processing device includes a first tracking unit that performs online learning, the number of other tracking units and the type of these tracking units (whether or not they perform online learning) are not particularly limited. When three or more tracking units are used, the information processing device tracks the subject using each tracking unit through the same processing as in the first embodiment in steps S305 to S307. That is, a likelihood map is acquired from each tracking unit, and whether or not to combine the first tracking unit with another tracking unit is determined depending on the degree of completion of the first tracking unit. In this embodiment, the tracking processing and the online learning processing are basically performed through the same processing as in FIG. 3 of the first embodiment, and therefore, redundant description will be omitted.

[0058] Here, the degree of completion and likelihood map are output for each tracker, and the combining unit 206 may output a final likelihood map to be used for tracking by performing a weighted sum integration of the likelihood maps, with the degree of completion used as a weighting coefficient. The combining unit 206 may also determine whether online learning of the first tracker is complete (whether the degree of completion is 1 or 0), as in the first embodiment, and perform tracking using the first tracker if it is complete, or perform tracking using another tracker if it is not. When tracking using another tracker is performed, a tracker to be preferentially used among those trackers may be determined in advance, or the tracker to be used for tracking may be selected based on the tracking results. For example, the combining unit 206 may refer to the degree of completion of each tracker and select the tracker with the highest degree of completion as the one to be used for tracking.

[0059] Furthermore, the information processing device 1 may also have a tracking unit (hereinafter referred to as a third tracking unit) that performs online learning in addition to the first tracking unit. In this case, the third tracking unit performs learning that is different from the first tracking unit in terms of the learning data, the resolution of the input data, or the frequency of parameter updates. Even in this case, the combination unit 206 outputs a final likelihood map based on the likelihood maps of the first tracking unit and the third tracking unit (not necessarily one). Here, the degree of completion is calculated for each tracking unit, and the combination unit 206 may output the final likelihood map by taking a weighted sum of the likelihood maps, with each degree of completion used as a weighting coefficient, or the likelihood map of the tracking unit with the highest degree of completion may be used as the final likelihood map.

[0060] Furthermore, it is possible that the sizes of the search ranges of the first and third trackers are different. In this case, when the tracker with the smaller search range loses sight of the tracking target, switching to tracking by the tracker with the larger search range (using the likelihood map of the tracker with the larger search range as the final likelihood map) makes it easier to continue tracking. Note that while there are two trackers here, if there are three or more trackers with different search ranges, the combination unit 206 may switch between trackers so that the search range increases in order, starting with the tracker with the smallest search range. Also, while tracking is being performed by the tracker with the larger search range, if the tracking target can also be detected by the tracker with the smaller search range, the combination unit 206 may switch to tracking by the tracker with the smaller search range. By reducing the search range for tracking, the possibility of an object similar to the tracking target being present within the search range is reduced, making it possible to suppress erroneous tracking.

[0061] Here, at the beginning of tracking, completion determination unit 205 sets the completion level of the tracker with a smaller search range (tracker A) to 1, and sets the completion level of the tracker with a larger search range (tracker B) to 0. Next, completion determination unit 205 calculates the reliability of the tracking result based on the final likelihood map acquired by result acquisition unit 204, and if the calculated reliability is less than a predetermined value, sets the completion level of tracker A to 0 and the completion level of tracker B to 1. If the reliability thereafter becomes equal to or greater than the predetermined value, completion determination unit 205 again sets the completion level of tracker A to 1 and the completion level of tracker B to 0. For example, completion determination unit 205 may acquire the largest value in the cells of the final likelihood map as the reliability.

[0062] According to this process, it is possible to select a tracker to be used from among a plurality of trackers according to the degree of completion and output a likelihood map, thereby improving tracking performance.

[0063] [Embodiment 4] The information processing device 1 according to this embodiment includes a tracker L and a tracker S that both perform online learning to track a subject. Tracker L is a tracker that updates parameters less frequently than tracker S, and learns long-term changes in the characteristics (appearance, etc.) of the tracked target and non-tracked targets. On the other hand, tracker S is a tracker that updates parameters more frequently than tracker L, and learns sudden changes in the characteristics of the tracked target and non-tracked targets. Note that tracker L and tracker S each basically perform online learning processing in the same way as the first tracker of embodiment 1, and therefore redundant explanations will be omitted.

[0064] In this embodiment, the tracking units L and S will be described as determining the degree of completion using the same processing as in Modification 1 or Modification 2 of Embodiment 1. That is, these tracking units determine whether learning is complete or not based on the number of times learning is performed, the number of pieces of training data, the variation in feature quantities of the training data, or changes in the distribution of feature quantities. Here, it is assumed that the threshold used for this determination in tracking unit L is set higher than the threshold used in tracking unit S. By setting the required level of completion in tracking unit S lower than that of tracking unit L, it is possible to make it easier for the tracking unit to complete, enabling more responsive learning.

[0065] In this example, the following description will be given assuming that a tracking unit S is used instead of the first tracking unit of the first embodiment. That is, depending on the degree of completion of the tracking unit S, it is determined whether or not to perform tracking by combining the tracking unit S with the tracking unit L. However, this determination may also be made depending on the degree of completion of the tracking unit L, and a desired setting may be made.

[0066] An example of the target determiner learning process (steps S308 to S309) performed by the combination unit 206 of the information processing device 1 according to this embodiment will be described below with reference to Fig. 8. This process is performed following the process of step S307. In step S801, the combination unit 206 controls the timing of parameter update for each tracker. Here, the combination unit 206 determines whether or not to update parameters for each of tracker L and tracker S. For trackers for which it is determined that parameters should be updated, the process proceeds to step S802, and for trackers for which it is not determined that parameters should be updated, the process ends.

[0067] As described above, in this embodiment, the parameters of tracker S are updated more frequently than those of tracker L. In the process of FIG. 3, the learning loop process is performed until the tracking process is completed. The update frequency of the parameters of each tracker is not particularly limited. However, here, the combination unit 206 updates the parameters of tracker S for each loop process, and updates the parameters of tracker L only for part of the loop process. For example, the combination unit 206 may update the parameters of tracker L in a loop in which the occurrence of an event indicating that updating the tracking accuracy is preferable is detected. The combination unit 206 may update the parameters of tracker L only when, for example, the reliability of the tracking result is low (determined in the same manner as in the third embodiment). Furthermore, the combination unit 206 may update the parameters of tracker L when, in step S305, a similar object with a high likelihood of being the tracking target is detected separately from the tracking target. The combination unit 206 may update the parameters of tracker L at a predetermined interval, for example, every two loop processes.

[0068] In step S802, combination unit 206 acquires learning data to be used for learning the target determiners of the trackers from the data stored in storage unit 207. This process is similar to the process performed in step S309 in Fig. 3, but the range of data referenced in storage unit 207 differs between tracker L and tracker S. For tracker L, combination unit 206 sequentially acquires learning data acquired up to N times ago from the current time, and for tracker S, it sequentially acquires learning data acquired up to M times ago from the current time (N>M). The subsequent parameter update process is performed similarly to the process in step S309 in Fig. 3, and therefore will not be described again.

[0069] According to this processing, online learning is performed with different parameter update frequencies for each tracker, and it is possible to determine whether to combine these trackers for tracking depending on the degree of completion of the learning. By performing tracking using tracker L, it is possible to track the tracked / non-tracked objects based on features that are maintained (hard to change) over the long term. Furthermore, by performing tracking using tracker S, which learns using only data collected over a shorter period than tracker L, it becomes possible to quickly respond to drastic changes in the features of the tracked / non-tracked objects, improving tracking performance.

[0070] Furthermore, when the reliability of the tracking result of the tracking unit currently used for tracking is low, the combination unit 206 may switch to use a tracking unit that updates parameters more frequently than the current tracking unit for tracking. By switching the tracking unit to be used for tracking to a tracking unit that updates parameters more frequently, it is possible to expect early improvement even when the reliability of the tracking result is low.

[0071] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0072] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0073] 1: information processing device, 201: image acquisition unit, 202: target designation unit, 203: area acquisition unit, 204: result acquisition unit, 205: completion determination unit, 206: combination unit, 207: storage unit

Claims

1. a tracking unit that tracks the subject using either or both of a first classifier that tracks the subject in an input image and a second classifier that tracks the subject and is different from the first classifier; an acquisition means for acquiring learning data used for learning tracking of the first classifier; a learning means for performing online learning to train the first classifier using the learning data while tracking the subject; evaluation means for evaluating the degree of completion of the online learning; a determining means for determining whether or not the tracking means should use the first classifier in tracking the subject, depending on the evaluation of the degree of completion; The information processing device further comprises:

2. 2. The information processing apparatus according to claim 1, wherein said evaluation means evaluates the degree of completion of said online learning in accordance with the learning status of said online learning.

3. 3. The information processing device according to claim 2, wherein the evaluation means evaluates the degree of completion of the online learning based on tracking results by the first classifier before and after new learning data is input in the online learning.

4. further comprising a calculation means for calculating a loss in the tracking result based on the training data and a tracking result using the training data as an input; 4. The information processing device according to claim 3, wherein the evaluation means evaluates a degree of completion of the online learning based on an amount of change, when new training data is input in the online learning, between the loss calculated in the first classifier before the online learning and the loss calculated in the first classifier after the online learning.

5. 3. The information processing apparatus according to claim 2, wherein the evaluation means evaluates the degree of completion of the online learning based on a distribution of the feature values ​​of each of the learning data.

6. 6. The information processing device according to claim 5, wherein the evaluation means evaluates the degree of completion of the online learning based on a change in variance of the feature quantities of the learning data when new learning data is input in the online learning.

7. using the reliability of the tracking result by the tracking means to evaluate the degree of completion of the online learning; 3. The information processing apparatus according to claim 2, wherein a degree of completion indicating completion is given to said tracking means when said reliability is equal to or greater than a predetermined value.

8. a first generating means for generating a first likelihood map indicating the likelihood of the object for each region in the input image; The information processing apparatus according to claim 7 , wherein the reliability is set based on the likelihood of the subject indicated in the first likelihood map.

9. a second generating means for generating a second likelihood map indicating the likelihood of the subject for each region in the input image by the first classifier; a third generating means for generating a third likelihood map indicating the likelihood of the subject for each region in the input image by the second classifier; 9. The information processing apparatus according to claim 8, wherein said first creating means creates said first likelihood map using said second likelihood map and said third likelihood map.

10. 10. The information processing device according to claim 9, wherein the first creation means, when the evaluation means determines that the first classifier should be used in tracking the subject, creates the first likelihood map by integrating the second likelihood map and the third likelihood map using a weight based on the degree of completion.

11. The first creating means setting the third likelihood map as the first likelihood map when the evaluation means determines that the first classifier will not be used in tracking the subject; 10. The information processing apparatus according to claim 9, wherein the evaluation means sets the second likelihood map as the first likelihood map when it determines that the first classifier is to be used in tracking the subject.

12. the evaluation means evaluates the degree of completion by determining whether the online learning has been completed or not; 12. The information processing device according to claim 2, wherein the determination means determines not to use the first classifier in tracking the subject when it is determined that the online learning has not been completed, and determines to use the first classifier in tracking the subject when it is determined that the online learning has been completed.

13. 13. The information processing device according to claim 12, wherein the tracking means tracks the subject using only the second classifier when it is determined not to use the first classifier, and tracks the subject using only the first classifier when it is determined to use the first classifier.

14. 14. The information processing device according to claim 12, wherein the evaluation means determines that the online learning is not complete if the reliability of the tracking result by the tracking means is less than a predetermined threshold, and determines that the online learning is complete if the reliability of the tracking result by the tracking means is equal to or greater than a predetermined threshold.

15. the second classifier is a classifier that is trained by online learning while the second classifier is tracking the subject, the first classifier and the second classifier have different search ranges for detecting the subject; when the subject is no longer detected while the tracking means is using one of the first classifier and the second classifier, which has the smaller search range, to track the subject, the tracking means switches the classifier used by the tracking means to the one of the first classifier and the second classifier, which has the larger search range; 15. The information processing device according to claim 1, further comprising: a switching unit that switches the classifier used by the tracking unit to the classifier having the smaller search range when the subject can also be detected by the classifier having the smaller search range while the tracking unit is using the classifier having the larger search range to track the subject.

16. The information processing device according to claim 1 , wherein the second classifier is a classifier that has been trained in advance to track the subject.

17. The information processing apparatus according to claim 1 , wherein the second classifier is a classifier having a parameter update frequency by online learning that is different from that of the first classifier.

18. 18. The information processing device according to claim 17, wherein, when it is determined that the reliability of a tracking result by the tracking means is low while both the first classifier and the second classifier are being used to track the subject, the tracking means switches to performing tracking using only the classifier that is updated more frequently out of the first classifier and the second classifier.

19. 19. The information processing apparatus according to claim 17, wherein the first classifier and the second classifier have different lengths of periods for collecting learning data used in online learning.

20. a step of tracking the subject using either or both of a first classifier that tracks the subject in the input image and a second classifier that tracks the subject and is different from the first classifier; acquiring training data used for training the tracking of the first classifier; performing online learning using the learning data to train the first classifier while tracking the subject; evaluating the degree of completion of the online learning; determining whether to use the first classifier in tracking the subject according to the evaluation of the degree of completion; An information processing method, further comprising:

21. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 19.

Images